- name
SKILL.md
Tệp skill vẫn nằm trong kho mã của nhà phát hành.
Dán lệnh đã sao chép vào terminal. Lệnh sẽ cài đặt tệp SKILL.md này vào agent đó.
npx skills add ericrisco/rsc-harness --skill agent-eval -a cursor -yDán lệnh đã sao chép vào terminal. Lệnh sẽ cài đặt tệp SKILL.md này vào agent đó.
npx skills add ericrisco/rsc-harness --skill agent-eval -a claude-code -yDán lệnh đã sao chép vào terminal. Lệnh sẽ cài đặt tệp SKILL.md này vào agent đó.
npx skills add ericrisco/rsc-harness --skill agent-eval -a codex -yDán lệnh đã sao chép vào terminal. Lệnh sẽ cài đặt tệp SKILL.md này vào agent đó.
npx skills add ericrisco/rsc-harness --skill agent-eval -a grok -yGửi tin nhắn đã sao chép trong cuộc trò chuyện Grok Bot. Bot sẽ lấy tệp SKILL.md này từ GitHub và thêm vào bộ skill mà mọi Bot của bạn dùng chung.
Install this skill: https://github.com/ericrisco/rsc-harness/tree/HEAD/skills/agent-evalTừ nhà phát hành
Turn "the agent feels better" into a number you can put in a PR check. You own the eval dataset, the scorer mix, the LLM-as-judge calibration, and the block-on-regression CI gate — framework-neutral, provider-neutral.
Every framework instantiates the same five-stage pipeline. Learn it once; the tool is a detail.
dataset ──▶ runner ──▶ scorers ──▶ metrics ──▶ gate
(JSONL (calls the (det / judge (aggregate + (pass/fail
golden system per / human) bootstrap CI) exit code)
set) case)
DeepEval, Inspect AI, and promptfoo are all just opinionated wrappers around this. If you understand the stages you can switch tools without relearning the craft.
Build your own golden set — a public leaderboard number is not your number, because identical model weights swing SWE-Bench Verified by 10–20 points just by changing the harness. Measure your task on your data. The dataset is the asset; everything else is replaceable. Rules:
Case schema — one JSON object per line:
{"id":"refund-001","input":"Where is my refund for order 4821?","expected":"States refunds take 5-7 business days and asks for nothing already on file","context":["policy: refunds 5-7 business days"],"meta":{"failure_mode":"hallucinated_policy","source":"ticket#4821"}}
{"id":"refund-002","input":"Cancel my subscription and refund this month","expected":"Cancels, refunds prorated amount, confirms no future charge","context":["policy: prorated refund on cancel"],"meta":{"failure_mode":"missed_tool_call","source":"ticket#5190"}}
failure_mode in meta is what lets you slice metrics by mode and find which kind of bug regressed — not just that the aggregate dropped.
Bad: "generate 1000 test questions with GPT and use those." Good: "80 real failure-mode cases pulled from support tickets, hand-labeled, tagged by failure mode."
Reach for the cheapest scorer that correlates with human judgment. Default mix:
One Scorer protocol, two implementations behind it — deterministic and judge are interchangeable to the runner:
from typing import Protocol
class Scorer(Protocol):
name: str
def score(self, case: dict, output: str) -> float: ... # 0.0–1.0
class JsonSchemaScorer:
name = "schema_valid"
def score(self, case, output): # deterministic, free, no drift
import json
try:
json.loads(output)
return 1.0
except ValueError:
return 0.0
class FaithfulnessJudge:
name = "faithfulness"
def __init__(self, judge_model): self.judge = judge_model
def score(self, case, output): # judge only where meaning matters
return self.judge.rate(case["context"], output) # see judge-design.md
A score you do not trust is worse than no score: an uncalibrated judge gives false confidence, which is more dangerous than admitted ignorance. Each rule, with its why:
Bad judge prompt: "Rate this answer 1–10." → everything lands 8–9, useless. Good: "Compare answer A and answer B against the reference. First write one sentence on each per the rubric, then output the better label." → forces reasoning, gives a stable signal.
Full rubric templates (pointwise + pairwise), the position-swap harness, the calibration script (agreement / Cohen's kappa vs human gold), G-Eval vs DAG, and the judge bias catalog (length, position, self-preference) with mitigations live in references/judge-design.md.
Score the path, not only the destination. Beyond exact/judge:
RAG (DeepEval / RAGAS names):
Agent:
The system side of these (how the loop and tools are built) is ../building-agents/SKILL.md; a common system-under-test is ../chatbot/SKILL.md.
Gate policy: block on regression vs a committed baseline, not on an absolute threshold. An absolute threshold flaps CI on judge noise and gives no signal on drift; "did this PR make a tracked metric worse than
main?" is the question that matters.
eval-report.json (metrics, per-failure-mode slices, baseline, pass/fail) and exits non-zero on a real regression so the merge is blocked.import json, sys
def gate(current: dict, baseline: dict, margin: float = 0.0) -> int:
regressed = []
for metric, score in current.items():
if metric in baseline and score < baseline[metric] - margin:
regressed.append((metric, baseline[metric], score))
report = {"metrics": current, "baseline": baseline, "regressed": regressed,
"passed": not regressed}
with open("eval-report.json", "w") as f:
json.dump(report, f, indent=2)
if regressed:
for m, b, c in regressed:
print(f"REGRESSION {m}: {b:.3f} -> {c:.3f}", file=sys.stderr)
return 1
return 0
sys.exit(gate(run_eval(), json.load(open("eval-baseline.json"))))
The complete provider-neutral runner (JSONL loader, scorer registry, bootstrap-CI metrics), the GitHub Actions workflow, and side-by-side DeepEval-pytest + Inspect-AI Task/Solver/Scorer versions of the same eval live in references/runner-and-gate.md.
Pick by where the eval runs and what it must do. Versions as of 2026-06 — re-verify, they rot.
The two-tool pattern is normal, not over-engineering: a light CI gate (DeepEval / RAGAS / promptfoo) plus a platform (Braintrust / LangSmith / Arize) for annotation and history. They share data; different jobs.
If the workspace has a 02-DOCS/ harness, record the eval policy in 02-DOCS/wiki/stack/evals.md: dataset location, scorer mix, gate baseline file, judge model, and the failure modes covered. Follow the harness wiki-article-template.md (type: stack) and index it in 02-DOCS/wiki/index.md. This is recorded, not gated — skip silently if there is no harness.
scripts/verify.sh is read-only and tool-detecting. It validates that every *.jsonl golden set in the project parses and that each line carries the required id, input, expected keys; checks the shape of any eval-report.json; and runs ruff / mypy on example Python and markdownlint on docs when those tools are installed. Every missing tool prints a yellow WARN and is skipped — never a failure. An empty or clean target exits 0.
Use when measuring whether an LLM or agent system actually got better and gating merges on it: golden sets, fixing an inflated LLM-as-judge, scoring RAG (faithfulness, contextual recall) or agent traj
---name: agent-evaldescription: Use when measuring whether an LLM or agent system actually got better and gating merges on it: golden sets…---# skills/agent-eval/SKILL.mdTổng quan
Use when measuring whether an LLM or agent system actually got better and gating merges on it: golden sets, fixing an inflated LLM-as-judge, scoring RAG (faithfulness, contextual recall) or agent trajectories (tool correctness, completion), or picking an eval framework. NOT building the agent loop, tools or RAG plumbing (that is `building-agents`).
Câu hỏi thường gặp
Tải skills/agent-eval/SKILL.md từ nhà phát hành, hoặc sao chép đường dẫn và mở kho mã. PlugMCP chỉ liệt kê skill và không lưu trữ tệp này.
Nhà phát hành duy trì nó tại https://github.com/ericrisco/rsc-harness.
Claude, Cursor, Codex và Grok Build có thể đọc tệp SKILL.md mà bạn thêm từ kho mã của nhà phát hành. Grok Bot cài đặt skill khi bạn gửi liên kết thư mục GitHub của skill trong cuộc trò chuyện.