Why it matters
A copilot is a loop around a model that must produce changes to a real repository. The hard parts are context selection, patch application and verification, not prompt wording. Take-homes fail when they demo a chat window that prints code. The reviewer wants to see that a patch is applied deterministically, that tests decide whether it stays, and that the system behaves when the model is wrong, which it will be.
Mental model
It is a junior engineer with a keyboard: useful when the task is scoped and the tests are fast, unreliable when the task is ambiguous. The metaphor breaks because a junior can ask a question, while your loop must decide when to stop and ask the user.
How it works
- 1Choose one task shape, such as fixing a failing test or adding a function with a docstring, and refuse the rest.
- 2Build context selection from the failing test, the file under test and its direct imports, with a hard token cap.
- 3Generate a patch in a structured format and apply it deterministically, rejecting anything that does not apply cleanly.
- 4Run the test suite after applying, and revert on failure so the repository is never left broken.
- 5Record the attempt, the patch and the test result for every run, so the loop is inspectable.
- 6Write a README with the success rate on a small task set, the failure cases and the next step you would take.
Step 1 of 6
Choose one task shape, such as fixing a failing test or adding a function with a docstring, and refuse the rest.
Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.
What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.
Text version
The page's steps: 1. Choose one task shape, such as fixing a failing test or adding a function with a docstrin… 2. Build context selection from the failing test, the file under test and its direct imports… 3. Generate a patch in a structured format and apply it deterministically, rejecting anythin… 4. Run the test suite after applying, and revert on failure so the repository is never left … 5. Record the attempt, the patch and the test result for every run, so the loop is inspectab… 6. Write a README with the success rate on a small task set, the failure cases and the next …
- Choose one task shape, such as fixing a failing test or adding a function with a docstring, and refuse the rest.
- Build context selection from the failing test, the file under test and its direct imports, with a hard token cap.
- Generate a patch in a structured format and apply it deterministically, rejecting anything that does not apply cleanly.
- Run the test suite after applying, and revert on failure so the repository is never left broken.
- Record the attempt, the patch and the test result for every run, so the loop is inspectable.
- Write a README with the success rate on a small task set, the failure cases and the next step you would take.
Trade-offs
| Choice | Good for | Cost | Fails when |
|---|---|---|---|
| Whole-file rewrites | Simple to apply | Destroys unrelated changes | Files are large or edited concurrently |
| Diff patches | Minimal and reviewable | Brittle when context shifts | The model has stale line context |
| Test-gated apply | Only verified changes persist | Slow and dependent on test quality | Tests are flaky or incomplete |
| Human confirmation | Safe for risky edits | Breaks automation | The task is high volume and low risk |
Good for
Simple to apply
Cost
Destroys unrelated changes
Fails when
Files are large or edited concurrently
- Files are large or edited concurrently
- The model has stale line context
- Tests are flaky or incomplete
- The task is high volume and low risk
Figure. Each choice with what it buys, what it costs, and the case where it loses.
What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.
Text version
Four trade-offs: Whole-file rewrites is good for Simple to apply, costs Destroys unrelated changes, and fails when Files are large or edited concurrently. Diff patches is good for Minimal and reviewable, costs Brittle when context shifts, and fails when The model has stale line context. Test-gated apply is good for Only verified changes persist, costs Slow and dependent on test quality, and fails when Tests are flaky or incomplete. Human confirmation is good for Safe for risky edits, costs Breaks automation, and fails when The task is high volume and low risk
Failure modes
- Patch applies to the wrong place. The diff matches similar text and edits an unrelated function. Debug: assert the changed line range against the intended symbol before applying.
- Broken repository after failure. A failed attempt leaves partial edits and the next run starts from a dirty tree. Debug: check git status after every attempt in the log.
- Demo-only success rate. The README claims it works and reports no task count. Debug: run the stated task set and compare pass counts with the claim.
In production
Cap attempts per task, keep every run on a branch, and log the test command and exit code. Rehearse the case where the model returns an unappliable patch, since it is common.
Common misconceptions
People think: A better model makes the copilot reliable.
Actually: Context selection, patch application and test gating decide reliability more than the model does.
People think: More agent autonomy is more impressive.
Actually: A bounded loop with a verification gate is easier to trust and easier to review.
Interview angle
On a copilot take-home, the candidate should be able to describe the loop, the patch format, the test gate and the measured failure rate. Weak answers show a chat interface and cannot say what happens when the patch is wrong.
Sources
- Jimenez et al., SWE-bench — repository-level patch generation and test-based evaluation.
- Aider documentation — an open implementation of repo-aware diff editing.
- Anthropic, Claude Code documentation — tool permissions and edit approval in a production copilot.