Loading
Diagnoses and fixes skills in the dotnet/skills repository that lose to their own baseline, fail to activate, time out, or return "no credible improvement". Use when an evaluation verdict is a regress
The skill file stays in the publisher's repository.
Paste the copied command in a terminal. It installs this SKILL.md into that agent.
npx skills add dotnet/skills --skill improve-skill-quality -a cursor -y
Diagnoses and fixes skills in the dotnet/skills repository that lose to their own baseline, fail to activate, time out, or return "no credible improvement". Use when an evaluation verdict is a regression or underpowered, when a skill regressed after a change, when /evaluate reports no results, or when deciding whether a weak skill should be strengthened or retired. Do not use for scaffolding a brand-new skill (use create-skill) or a brand-new eval (use create-skill-test).
Diagnoses and fixes skills in the dotnet/skills repository that lose to their own baseline, fail to activate, time out, or return "no credible improvement". Use when an evaluation verdict is a regression or underpowered, when a skill regressed after a change, when /evaluate reports no results, or when deciding whether a weak skill should be strengthened or retired. Do not use for scaffolding a brand-new skill (use create-skill) or a brand-new eval (use create-skill-test).
Turn a failing or unconvincing evaluation into a targeted fix. The single most common mistake in this repo is rewriting skill prose in response to a verdict whose real cause was the eval, the fixtures, or the harness. Classify first, then fix.
/evaluate reports "Evaluation ran but produced no results".create-skill.eval.yaml from scratch — use create-skill-test.eng/skill-validator, eng/vally-adapter, evaluation*.yml).| Input | Required | Description |
|---|---|---|
| Verdict evidence | Yes | The /evaluate PR comment, or results.json from the run artifacts |
| Losing trial transcripts | Yes for content fixes | Baseline vs. skilled output plus the judge's stated reason |
| Stimulus-vote W/T/L and repeated-run W/T/L | Yes | Separates cross-task evidence from reliability |
| Activation status per arm | Yes | Isolated and plugin activation are different failures |
Read InvestigatingResults.md for how to
download artifacts and read results.json. Extract, per failing stimulus:
Do not change skill content until you can quote a losing trial and the judge's reason for it. For the other cause classes the evidence is different: harness failures are diagnosed from the job log and the spec, and power problems from the trial record — neither has a losing trial to quote, and demanding one is what sends people rewriting prose instead.
Work down this table and stop at the first row that matches. Rows are ordered by how often the symptom has been misdiagnosed as a skill-content problem — the fixture row is first because a fixture failure also presents as a setup or reliability failure and gets misfiled as one.
| Symptom | Real cause class | Go to |
|---|---|---|
| A fixture does not build, is untracked by git, breaks for the wrong reason, or contradicts itself | Fixture | Step 4 |
No results.json, "produced no results", or the spec never loaded | Harness / spec-load | Step 3 |
| Trials errored, timed out, or returned empty output | Reliability | Step 3 |
| Trajectories unmatched, a trial errored, or the summary disagrees — verdict reported inconclusive | Reliability (not power) | Step 3 |
| Positive record (e.g. 16W/8T/1L), comparison conclusive, verdict still not a pass | Statistical power | Step 5 |
| Skilled arm equals baseline arm by construction | Eval design | Step 6 |
| Activated and lost on quality, judge names a concrete defect | Skill content | Step 7 |
| Activated in isolation, not in plugin | Activation / routing | Step 8 |
| Not activated in either arm | Frontmatter description | Step 8 |
| Wins but costs far more than baseline | Scope and cost | Step 7 |
A verdict is only a measured result when the comparison was conclusive: adapt.mjs requires zero
errored trials, zero unmatched trajectories, and an agreeing summary before it will report a pass or
a regression. Confirm that before reading a record as a power problem.
See references/eval-triage.md for the full catalogue. The recurring ones:
config: alias, and Vally rejects a spec
that declares both config: and defaults:. Replace the alias with one defaults: block.session.idle
failures look identical from the verdict and need harness fixes, not SDK pins.expect_tools: [bash] on an advisory question forces a restore or build and turns an answer into
a timeout with no quality gain.check_eval_quality.py and deterministic golden-workspace replay.Run python eng/eval-quality/check_eval_quality.py — it blocks 22 defect classes that can
cost a real result here. Then confirm by hand:
git ls-files), not merely on disk — .gitignore
has silently swallowed committed coverage fixtures;line-rate, summary totals and <line> elements differ is the canonical case — or the
two arms legitimately read different truths.The gate has two independent bars, and confusing them is the usual misdiagnosis:
underpowered — never a pass, never a regression.| discordant stimulus votes | records that pass | p |
|---|---|---|
| ≤ 4 | none, however good the skill | ≥ 0.0625 |
| 5–7 | zero losses only (5W/0L) | 0.031 |
| 8 | one loss survivable (7W/1L) | 0.035 |
So at exactly 5 stimuli a single tie is fatal — it leaves 4 discordant. At 6 stimuli one tie is survivable (5W/1T/0L); at 7, up to two are (5W/2T/0L). A loss is not.
So a positive record with a failing verdict is a power problem, not a content problem. Fix it by
adding discriminating stimuli. Raising runs measures reliability for the same task and cannot
clear the floor.
An eval that compares the skill against itself measures judge noise:
expect_activation: false) must not also set constraints.reject_skills.
That makes the skilled arm skill-free, so the activation contract cannot observe a hijack.
Schema version 4 retains the identical-arm comparison for diagnostics but excludes it from
preference inference; unexpected isolated activation still blocks a pass.disable-model-invocation: true is absent from the model-facing skilled arm, so its
direct eval compares two identical arms regardless of whether graders inspect activation or answer
content. Cover it through consumer outcomes instead; for example, filter-syntax is covered by
run-tests and mtp-hot-reload.config is missing its required key enforces nothing, so the stimulus has one
fewer assertion than it appears to.Only now change the skill. Apply the patterns in references/writing-for-baseline-delta.md; the ones that most often flip a loss:
references/ reads and size any
orchestration to the user's scope.Activation failures are frontmatter and routing failures, not body failures. See references/eval-triage.md. Summary:
| Failure | Fix |
|---|---|
| Not activated in any arm | Put the user's own words in description: symptoms, error codes, artifact names, quoted requests |
| A sibling skill wins the prompt | Claim the exact ambiguous words in description, and add matching exclusions on both siblings |
| Model answers with no skill at all | Raise the stakes in the description, de-crowd the plugin menu, verify with the plugin arm |
| Boundary excludes real scenarios | Re-read every "do not use for" clause against every eval prompt and real workflow phase |
| Description at the 1,024-char ceiling | Cut restated body content, not trigger phrases; check the plugin menu budget too |
dotnet run --project eng/skill-validator/src/SkillValidator.csproj -- check --plugin ./plugins/<plugin>
python eng/eval-quality/check_eval_quality.py
./eng/run-skill-evals.sh <plugin> <skill>
Use the production path at normal worker concurrency and with the declared defaults.timeout:
Vally for skill evals, and skill-validator evaluate for agent evals. For an agent eval, separately
run check_eval_quality.py, apply each golden patch to its materialized fixture, and run the
applicable deterministic file, output, and command graders against the golden result. Do not use a
serial-only pass or a larger ad hoc budget as completion evidence. For broad routing or behavior
changes, collect separate GPT-family and Claude-family results. Do not pool model families into
extra stimulus votes.
Then request the official run by submitting a PR review containing /evaluate (Files changed →
Review changes), which binds the run to the reviewed commit. Before declaring a regression on the
result, confirm the skill payload actually changed — reruns on byte-identical content have shifted
7W/2T/2L to 4W/5T/2L.
check_eval_quality.py and skill-validator check both pass.| Pitfall | Solution |
|---|---|
| Rewriting skill prose in response to an underpowered verdict | Underpowered means too few distinct stimuli; add discriminating stimuli instead |
Using the deprecated top-level config: alias | Rename it to defaults: and preserve its settings; the repository gate rejects the alias |
Padding runs to clear the stimulus floor | Repeats measure reliability for one task; add stimuli |
| Treating an errored trial as fixture nondeterminism | Read the stderr first; judge-side auth failures need harness fixes |
| Fixing a "wrong" answer that the fixture actually made wrong | Check fixture self-consistency before blaming the response |
| Strengthening a skill nobody uses and nothing passes | Weak eval signal plus thin telemetry is a valid retirement case |
| Landing a fix without re-running | Verify the invoked payload contains the fix; judge noise is real |
results.json. This is the current guide; the similarly-named eng/skill-validator/src/docs/InvestigatingResults.md documents the retired skill-validator evaluate schema and does not describe today's results.