Evidence record · captured 2026-10-11

Datadog LLM Eval Bootstrap

Bootstraps LLM evaluators from production traces in Datadog LLM Observability: it samples traces, proposes online LLM-judge evaluators, and after explicit confirmation creates them in Datadog as disabled drafts (never auto-enabled); on request it emits Python SDK code or a framework-agnostic JSON spec instead. It auto-selects between the Datadog MCP server and the `pup` CLI as the backend.

Official publisherUnverifiedSource reviewedBehavior not tested

Source review

Editorial assessment

This is a source-only assessment. It does not claim runtime or security testing.

What it does

Samples an app's production LLM traces, maps existing evaluators, and proposes 8 to 15 LLM judges each tied to a cited trace, then writes the confirmed set to Datadog only as disabled drafts.

Best for

Building a first evaluator suite for an LLM app already traced in Datadog LLM Observability
Turning a root-cause report or failure hypothesis into targeted evaluators
Exporting a confirmed suite as offline Python SDK code or a framework-agnostic JSON spec

Strengths in the source

Stops at a mandatory checkpoint where the user keeps, drops or renames evaluators, and publishes every one with enabled set to false.
Before overwriting an existing evaluator it fetches the full config and merges, because the create call is full-replace and would reset omitted fields.
Classifies each evaluator as span, trace or session scope, and only proposes session scope after a probe confirms spans carry a session_id.
Requires a clickable trace link as evidence for each proposal and flags any evaluator backed by fewer than five traces as tentative.

Limitations

Needs a Datadog account with LLM Observability plus either the Datadog MCP server, served from an /api/unstable/ endpoint, or the pup CLI.
Writes evaluator configs and optional notebooks into the Datadog org; once enabled, judges call an LLM (default openai gpt-5.4-mini) on sampled production traffic, which costs money.
Pulls production prompts and outputs into the agent's context; PII is scrubbed only in the files it writes, and every MCP call carries a skill telemetry intent tag.

When not to use it

Your traces are not in Datadog LLM Observability.
You need deterministic code checks running online; the publish path is LLM-judge only, so those come out as offline SDK code.

Reviewed by Retenta editorial on 2026-10-11; scope: source only.

Files included in this review

agent-observability/agent-observability-eval-bootstrap/SKILL.md

Evidence and boundaries

Execution · not tested

Retenta has not executed this skill.

Security · not tested

The skill folder bundles no executable scripts. Retenta has not run a security scan.

Pinned publisher source: agent-observability/agent-observability-eval-bootstrap/SKILL.md
Revision: d2411cc85d340c63ef955edff949179adf0e5ac8
License evidence: MITRetenta preserves the license text captured with this immutable upstream snapshot; no independent legal or security review is claimed.
Last source change: 2026-08-21
Independent verification: Published by Datadog, but not independently verified by a third-party reviewer.
Upstream repository: Open publisher repository