Skip to content
AI Engineer Guide.

drill · drills · beginner · concept 21 of 33

Take-home: copilot

0 of 33 complete

Why it matters

A copilot is a loop around a model that must produce changes to a real repository. The hard parts are context selection, patch application and verification, not prompt wording. Take-homes fail when they demo a chat window that prints code. The reviewer wants to see that a patch is applied deterministically, that tests decide whether it stays, and that the system behaves when the model is wrong, which it will be.

Mental model

It is a junior engineer with a keyboard: useful when the task is scoped and the tests are fast, unreliable when the task is ambiguous. The metaphor breaks because a junior can ask a question, while your loop must decide when to stop and ask the user.

How it works

Mechanism, step by stepInteractive
  1. 1Choose one task shape, such as fixing a failing test or adding a function with a docstring, and refuse the rest.
  2. 2Build context selection from the failing test, the file under test and its direct imports, with a hard token cap.
  3. 3Generate a patch in a structured format and apply it deterministically, rejecting anything that does not apply cleanly.
  4. 4Run the test suite after applying, and revert on failure so the repository is never left broken.
  5. 5Record the attempt, the patch and the test result for every run, so the loop is inspectable.
  6. 6Write a README with the success rate on a small task set, the failure cases and the next step you would take.

Step 1 of 6

Choose one task shape, such as fixing a failing test or adding a function with a docstring, and refuse the rest.

Step 1/6 · step 1

Figure. The mechanism in this page's own steps. The figure adds sequence, not new claims.

What to notice. A step can be read alone, but the order is the mechanism. The control lets you stop on any one.

Text version

The page's steps: 1. Choose one task shape, such as fixing a failing test or adding a function with a docstrin… 2. Build context selection from the failing test, the file under test and its direct imports… 3. Generate a patch in a structured format and apply it deterministically, rejecting anythin… 4. Run the test suite after applying, and revert on failure so the repository is never left … 5. Record the attempt, the patch and the test result for every run, so the loop is inspectab… 6. Write a README with the success rate on a small task set, the failure cases and the next …

  1. Choose one task shape, such as fixing a failing test or adding a function with a docstring, and refuse the rest.
  2. Build context selection from the failing test, the file under test and its direct imports, with a hard token cap.
  3. Generate a patch in a structured format and apply it deterministically, rejecting anything that does not apply cleanly.
  4. Run the test suite after applying, and revert on failure so the repository is never left broken.
  5. Record the attempt, the patch and the test result for every run, so the loop is inspectable.
  6. Write a README with the success rate on a small task set, the failure cases and the next step you would take.

Trade-offs

ChoiceGood forCostFails when
Whole-file rewritesSimple to applyDestroys unrelated changesFiles are large or edited concurrently
Diff patchesMinimal and reviewableBrittle when context shiftsThe model has stale line context
Test-gated applyOnly verified changes persistSlow and dependent on test qualityTests are flaky or incomplete
Human confirmationSafe for risky editsBreaks automationThe task is high volume and low risk
Trade-off mapInteractive

Good for

Simple to apply

Cost

Destroys unrelated changes

Fails when

Files are large or edited concurrently

  1. Files are large or edited concurrently
  2. The model has stale line context
  3. Tests are flaky or incomplete
  4. The task is high volume and low risk

Figure. Each choice with what it buys, what it costs, and the case where it loses.

What to notice. The fourth column is the decision: if the losing case describes your situation, the choice is wrong.

Text version

Four trade-offs: Whole-file rewrites is good for Simple to apply, costs Destroys unrelated changes, and fails when Files are large or edited concurrently. Diff patches is good for Minimal and reviewable, costs Brittle when context shifts, and fails when The model has stale line context. Test-gated apply is good for Only verified changes persist, costs Slow and dependent on test quality, and fails when Tests are flaky or incomplete. Human confirmation is good for Safe for risky edits, costs Breaks automation, and fails when The task is high volume and low risk

Failure modes

  • Patch applies to the wrong place. The diff matches similar text and edits an unrelated function. Debug: assert the changed line range against the intended symbol before applying.
  • Broken repository after failure. A failed attempt leaves partial edits and the next run starts from a dirty tree. Debug: check git status after every attempt in the log.
  • Demo-only success rate. The README claims it works and reports no task count. Debug: run the stated task set and compare pass counts with the claim.

In production

Cap attempts per task, keep every run on a branch, and log the test command and exit code. Rehearse the case where the model returns an unappliable patch, since it is common.

Common misconceptions

People think: A better model makes the copilot reliable.

Actually: Context selection, patch application and test gating decide reliability more than the model does.

People think: More agent autonomy is more impressive.

Actually: A bounded loop with a verification gate is easier to trust and easier to review.

Interview angle

On a copilot take-home, the candidate should be able to describe the loop, the patch format, the test gate and the measured failure rate. Weak answers show a chat interface and cannot say what happens when the patch is wrong.

Sources