CLAUDE AGENTS TEAM
Claude Agents Team
ページ · Claude Agents Team

Evaluator — score an agent against a citation-backed rubric

The evaluator is the architectural anchor of the plugin. It scores any agent or skill file across six dimensions, every rule cites a source (Anthropic spec or published prior art), and it produces the same v1-schema JSON whether run in static, judge, or behavioral mode.

Three phases:

PhaseSkillWhat it scoresWhen
1. Static linter/evaluate-agentDefinition (frontmatter, description, tools, model fit, body, anti-patterns)Always. Free. CI-safe.
2. LLM-as-judge/evaluate-agent --deep (same skill)Description clarity, contradictions, role/tool mismatch nuanceWhen you want a human-grade read; costs tokens.
3. Behavioral/evaluate-agent-behaviorTool-whitelist adherence + step efficiency from a transcript (v0.1); domain adherence + instruction-following gap (v0.2)After a release; periodic team-quality coverage.

Quick start

bash

Output shape

JSON conforming to plugins/agents-team/lib/eval/schema/v1.json:

json

Markdown render:

text

Verdict thresholds

ScoreGradeDefault verdict
90–100Aship
80–89Bship
65–79Crevise
50–64Drevise
<50Freject

Override: any critical finding forces verdict = reject regardless of score. The score reflects overall quality; the verdict is the gate.

Rubric

The canonical rubric lives at plugins/agents-team/lib/eval/rubric.md. Every rule has a stable ID (tool_hygiene.write_on_review_role), severity, phase (static or judge), and a citation. Rules without citations go in experimental/ and are off by default.

Calibration

The LLM-as-judge is measured against a hand-graded calibration set under plugins/agents-team/lib/eval/calibration/. 15 fixture pairs span A–F grades. The Galileo-2026 target is ≥0.80 Spearman correlation per dimension; the CI floor is 0.75.

bash

The expected.json files are seeded from the static linter as the v0.1 baseline and must be hand-tuned over time to reflect a real human rater. See lib/eval/calibration/README.md for the full procedure.

Any PR that touches lib/eval/rubric.md or lib/eval/judge.py runs calibration in CI when ANTHROPIC_API_KEY is available; a drop below the threshold blocks the merge.

CI integration

Add a step to your repo's CI:

yaml

For the deep judge in CI, set ANTHROPIC_API_KEY in repo secrets and pass --deep. Cache hits keep cost bounded.

Anti-goals

  • Don't auto-rewrite agent files. Suggest fixes; the human applies.
  • Don't score the agent's generated artifacts (code, copy). That's a different evaluator.
  • Don't replace the LLM's own judgment on subjective trade-offs — the evaluator flags risks; humans pick.
  • Don't add rubric items without a citation. Unsourced rules go in experimental/ and are off by default.
  • Don't couple to a single judge model. The judge model is swappable via ANTHROPIC_JUDGE_MODEL.

Sources