Loading
Autonomous iterative experimentation loop for any programming task. Guides the user through defining goals, measurable metrics, and scope constraints, then runs an autonomous loop of code changes, tes
The skill file stays in the publisher's repository.
Paste the copied command in a terminal. It installs this SKILL.md into that agent.
npx skills add github/awesome-copilot --skill autoresearch -a cursor -y
Autonomous iterative experimentation loop for any programming task. Guides the user through defining goals, measurable metrics, and scope constraints, then runs an autonomous loop of code changes, testing, measuring, and keeping/discarding results. Inspired by Karpathy''s autoresearch. USE FOR: autonomous improvement, iterative optimization, experiment loop, auto research, performance tuning, automated experimentation, hill climbing, try things automatically, optimize code, run experiments, autonomous coding loop. DO NOT USE FOR: one-shot tasks, simple bug fixes, code review, or tasks without a measurable metric.
Autonomous iterative experimentation loop for any programming task. Guides the user through defining goals, measurable metrics, and scope constraints, then runs an autonomous loop of code changes, testing, measuring, and keeping/discarding results. Inspired by Karpathy''s autoresearch. USE FOR: autonomous improvement, iterative optimization, experiment loop, auto research, performance tuning, automated experimentation, hill climbing, try things automatically, optimize code, run experiments, autonomous coding loop. DO NOT USE FOR: one-shot tasks, simple bug fixes, code review, or tasks without a measurable metric.
An autonomous experimentation loop for any programming task. You define the goal and how to measure it; the agent iterates autonomously -- modifying code, running experiments, measuring results, and keeping or discarding changes -- until interrupted.
This skill is inspired by Karpathy's autoresearch, generalized from ML training to any programming task with a measurable outcome.
Before any experimentation begins, work with the user to establish these parameters. Ask the user directly for each item. Do not assume or skip any.
Ask the user:
What are you trying to improve or optimize?
Examples: execution time, memory usage, binary size, test pass rate, code coverage, API response latency, throughput, error rate, benchmark score, build time, bundle size, lines of code, cyclomatic complexity, etc.
Record the user's answer as the goal.
Ask the user:
How do we measure success? What exact command produces the metric?
I need:
- The command to run (e.g.,
dotnet test,npm run benchmark,time ./build.sh,pytest --tb=short)- How to extract the metric from the output (e.g., a regex pattern, a specific line, a JSON field)
- Direction: Is lower better or higher better?
Example: "Run
dotnet test --logger trx, count passing tests. Higher is better." Example: "Runhyperfine './my-program', extract mean time. Lower is better."
Record:
METRIC_COMMAND: the command to runMETRIC_EXTRACTION: how to extract the numeric metric from outputMETRIC_DIRECTION: lower_is_better or higher_is_betterAsk the user:
Which files or directories am I allowed to modify?
And which files are OFF LIMITS (read-only)?
Record:
IN_SCOPE_FILES: files/dirs the agent may editOUT_OF_SCOPE_FILES: files/dirs that must not be modifiedAsk the user:
Are there any constraints I should respect?
Examples:
- Time budget per experiment (e.g., "each run should take < 2 minutes")
- No new dependencies
- Must keep all existing tests passing
- Must not change the public API
- Must maintain backward compatibility
- VRAM/memory limit
- Code complexity limits (prefer simpler solutions)
Record as CONSTRAINTS.
Ask the user:
How many experiments should I run, or should I just keep going until you stop me?
You can say a number (e.g., "try 20 experiments") or "unlimited" (I'll run until you interrupt).
Record as MAX_EXPERIMENTS (number or unlimited).
Inform the user of the default simplicity policy:
Simplicity policy (default): All else being equal, simpler is better. A small improvement that adds ugly complexity is not worth it. Removing code while maintaining or improving the metric is a great outcome. I'll weigh the complexity cost against the improvement magnitude. Does this policy work for you, or do you want to adjust it?
Record any adjustments as SIMPLICITY_POLICY.
Summarize all parameters back to the user in a clear table:
| Parameter | Value |
|---|---|
| Goal | ... |
| Metric command | ... |
| Metric extraction | ... |
| Direction | lower is better / higher ... |
| In-scope files | ... |
| Out-of-scope files | ... |
| Constraints | ... |
| Max experiments | ... |
| Simplicity policy | ... |
Ask the user to confirm. Do not proceed until confirmed.
Once the user confirms:
Create a branch: Propose a tag based on today's date (e.g., autoresearch/mar17).
Create the branch: git checkout -b autoresearch/<tag>.
Read in-scope files: Read all files that are in scope to build full context of the current state.
Initialize results.tsv: Create results.tsv in the repo root with the header row:
experiment commit metric status description
Add results.tsv and run.log to .git/info/exclude (append if not already present) so they stay untracked without modifying any tracked files.
Run the baseline: Execute the metric command on the current unmodified code.
Record the result as experiment 0 with status baseline in results.tsv.
Report baseline to the user:
Baseline established: [metric_name] = [value] Starting autonomous experimentation loop.
Run this loop continuously. Do not stop to ask the user. Run until:
MAX_EXPERIMENTS is reached, ORLOOP:
1. THINK - Analyze previous results and the current code.
Generate an experiment hypothesis.
Consider: what worked, what didn't, what hasn't been tried.
2. EDIT - Modify the in-scope file(s) to implement the idea.
Keep changes focused and minimal per experiment.
3. COMMIT - git add + git commit with a short descriptive message.
Format: "experiment: <short description of what changed>"
4. RUN - Execute the metric command.
Redirect output to run.log so it does not flood the context window.
Use shell-appropriate redirection:
- Bash/Zsh: `<command> > run.log 2>&1`
- PowerShell: `<command> *> run.log`
5. MEASURE - Extract the metric from run.log.
If extraction fails (crash/error), read the last 50 lines
of run.log for the error.
6. DECIDE - Compare metric to the current best:
- IMPROVED: Keep the commit. Update the "best" baseline.
Log status = "keep".
- SAME OR WORSE: Revert. `git reset --hard HEAD~1`.
Log status = "discard".
- CRASH: Attempt a quick fix (typo, import, simple error).
Amend the experiment commit (`git commit --amend`) with the fix
and rerun. The experiment keeps its original number.
If unfixable after 2 attempts, revert the entire experiment
(`git reset --hard HEAD~1`) and log status = "crash".
7. LOG - Append a row to results.tsv:
experiment_number commit_hash metric_value status description
8. CONTINUE - Go to step 1.
When generating experiment ideas, follow this priority order:
When the loop ends (budget reached or user interrupts):
git log --oneline <start_commit>..HEADTab-separated, 5 columns:
experiment commit metric status description
0 a1b2c3d 0.997900 baseline unmodified code
1 b2c3d4e 0.993200 keep increase learning rate to 0.04
2 c3d4e5f 1.005000 discard switch to GeLU activation
3 d4e5f6g 0.000000 crash double model width (OOM)
autoresearch/<tag> branchgit reset --hard HEAD~1results.tsv and run.log stay untracked (added to .git/info/exclude)