RetentaRetenta

Evidence record · captured 2026-10-11

Skill Creator

Guides the full loop of authoring an Agent Skill: draft it, write test prompts, run Claude with and without the skill, grade outputs with grader/comparator/analyzer sub-agent prompts, show results in a generated HTML review viewer, and iterate. It also runs trigger evals that test and rewrite the skill's description for triggering accuracy, then packages the skill.

Official publisherUnverifiedSource reviewedBehavior not tested
Source review

Editorial assessment

This is a source-only assessment. It does not claim runtime or security testing.

What it does

Treats skill authoring as an experiment: each test prompt runs with and without the skill in parallel subagents, outputs go to a review viewer before revision, and the description is tuned on a held-out set.

Best for

  • Turning a workflow from the current conversation into a reusable skill
  • Checking whether a revised skill actually beats the previous version on the same prompts
  • Fixing a skill that triggers too rarely or on the wrong requests

Strengths in the source

  • Spawns a with-skill run and a baseline run for every test case in the same turn, and snapshots the old skill first when improving one, so comparisons are like for like.
  • Requires grading.json to use the exact fields text, passed and evidence, then aggregates pass rate, time and tokens as mean and standard deviation per configuration.
  • Description tuning splits 20 trigger queries 60/40 into train and held-out test, runs each query three times, and picks the winner by test score to avoid overfitting.
  • Tells the agent to bundle a script into the skill when every test run independently wrote the same helper, so later invocations stop reinventing it.

Limitations

  • The trigger eval and description loop shell out to claude -p subprocesses that consume the user's Claude Code usage; on Claude.ai the description optimizer is skipped.
  • The full loop assumes subagents, Python and a browser or static HTML; on Claude.ai the skill drops baseline runs and quantitative benchmarking entirely.
  • Creates a skill-name-workspace directory of iteration folders beside the skill and opens local HTML files, which needs cleaning up afterward.

When not to use it

  • A small edit to an existing skill where baselines and a review viewer cost more than the change is worth.
  • Skills with purely subjective output, where the skill itself says assertions add little over human review.

Reviewed by Retenta editorial on 2026-10-11; scope: source only.

Files included in this review
  • skills/skill-creator/SKILL.md

Evidence and boundaries

Execution · not tested

Retenta has not executed this skill.

Security · not tested

Bundles 4 scripts (scripts/run_eval.py, scripts/improve_description.py, scripts/run_loop.py, aggregate_benchmark.py, generate_report.py, quick_validate.py, package_skill.py, utils.py, eval-viewer/generate_review.py) that the agent may run. Retenta has not run them or scanned them.

Pinned publisher source
skills/skill-creator/SKILL.md
Revision
dbd4588f9e1033efb41dad4bef2f7947c8993d44
License evidence
Apache-2.0Retenta preserves the license text captured with this immutable upstream snapshot; no independent legal or security review is claimed.
Last source change
2026-03-06
Independent verification
Published by Anthropic, but not independently verified by a third-party reviewer.
Upstream repository
Open publisher repository

Compatibility

Claude Code (plugin marketplace)

documented

Named in the upstream skill or repository documentation; not tested by Retenta.

Claude.ai

documented

Named in the upstream skill or repository documentation; not tested by Retenta.

Claude API (Skills API)

documented

Named in the upstream skill or repository documentation; not tested by Retenta.

Requirements and access

  • Python 3
  • Claude Code CLI (`claude -p`) for the eval and description-improvement scripts

Referenced tools

No named tool is required by the reviewed SKILL.md.

Install from the source

Clones the current default branch, which may be newer than the reviewed revision dbd4588. The destination shown is Claude Code's personal skills folder; other agents read the same folder from their own skills directory.

View the SKILL.md at its source Verify the publisher's install documentation

More AI development skills →

Install the full skill folder

git clone --depth 1 https://github.com/anthropics/skills.git
cp -R skills/skills/skill-creator ~/.claude/skills/skill-creator

Thyme RAG Capsules

Make this skill’s state durable, governed, and portable.

A Retenta capsule lets agents recall prior work, append provenance-stamped progress, and carry the memory to the next run.

Create durable agent memoryUnderstand the evidence labels