Many agent failures look like model weakness. Look closer, and the real problem is often terrible on-site material.

The user says “fix CI.” The system stuffs the entire conversation history, README, the last ten tool outputs, the repository tree, old task summaries, and several retrieval results into the prompt. The model can read it, but it does not know which part is the current real failure, which part was last week’s guess, which log came from the latest commit, and which constraint must never be violated. It behaves like a new employee thrown into an archive room: whatever it reads first becomes believable.

I now prefer to call the main battlefield of agent development Context Engineering. Prompt still matters, but prompt mainly controls tone, role, and a few principles. Production quality is decided by what context the model receives at each step, how it is prioritized, how sources are recorded, how it is compressed and replayed, and when the system explicitly tells the model: “there is a conflict here; do not guess.”

This article uses a reproducible case: a code-fixing agent that used to guess from CI failure logs, then was redesigned into a context-package pipeline that can locate, patch, validate, and create a PR.

The Incident: CI Is Red, and the Agent Changes the Wrong Place

Take an ordinary project. The repository is called demo-billing-service; it is not a real company project. It has a Node backend, and CI runs npm test plus npm run lint. After one commit, a GitHub Actions test fails:

FAIL tests/invoice-discount.test.ts
  applyDiscount
    expected 9900 to equal 9000

Received invoice:
  subtotal: 10000
  coupon: "WELCOME10"
  customerType: "new"

The user’s input is short:

{
  "task": "CI failed. Please fix it and open a PR.",
  "repo": "demo-billing-service",
  "run_id": "ci-20260615-1842",
  "branch": "feature/coupon-refactor"
}

The old agent built context crudely. It concatenated the last ten conversation turns, the full CI log, the README, all filenames under src/, git diff, the test file, and tool instructions into one long prompt. The system prompt said: “You are a senior engineer. Make careful changes.” This looks reasonable in a demo, but it fails in stable ways.

It is misled by old conversation. If the old summary says “we discussed that member-price logic may need adjustment,” the model goes to memberPrice.ts. It is misled by log noise. The full CI log has 300 lines of dependency warnings, and the model may decide the dependency version is the root cause. It is also misled by the README. The README says the local test command is npm run test:unit, but the project now uses npm test.

In the worst case, the agent changed the assertion from 9000 to 9900. CI turned green, but the business logic was wrong. The test was protecting “a new-user welcome coupon gives 10% off”; the model changed the test to match a broken implementation. It was not lazy on purpose. It simply did not receive a clear task boundary: this task is to fix implementation, not relax tests; the failing test is evidence, not the target to modify; old conversation is background, not stronger than current CI.

Context Engineering exists to fix exactly this kind of “lots of material, messy workbench” problem.

First Change: From Prompt to Context Package

I would not start by writing a better system prompt. I would first define the context package that the agent receives at each step. A context package is not a human-facing report. It is the task site shared by the model and the audit system.

For a code-fixing task, the first version can be simple:

{
  "package_id": "ctx-ci-fix-01HZX",
  "package_version": "code_fix_context_v1",
  "objective": {
    "user_goal": "Fix CI failure and create PR",
    "success_criteria": [
      "Failing test passes again",
      "Do not delete or skip tests",
      "Do not broaden discount behavior",
      "PR description includes reason and test result"
    ],
    "latest_user_message_id": "msg-1842"
  },
  "environment": {
    "repo": "demo-billing-service",
    "branch": "feature/coupon-refactor",
    "base_branch": "main",
    "ci_run_id": "ci-20260615-1842",
    "workspace_status": "clean"
  },
  "evidence": [],
  "constraints": [],
  "tool_budget": {
    "max_shell_commands": 20,
    "max_file_reads": 30,
    "requires_human_approval_for_push": true
  }
}

This structure may look less “intelligent” than a long prompt, but it has one crucial benefit: every field has a responsibility. objective owns the goal, environment owns current state, evidence owns facts, constraints owns boundaries, and tool_budget owns action cost. When the model later does something wrong, we can ask specifically: was the goal wrong, the environment stale, the evidence missing, or the constraint too weak?

The biggest problem with the old prompt is that everything is mixed together. Latest user input, model guesses, README, and tool errors appear as plain natural language side by side. The model can read them, but it cannot reliably distinguish fact, guess, historical preference, and current constraint. The first job of a context package is to separate them.

Do Not Dump the Whole CI Log

The CI failure log is the most important evidence in this case, but it can easily pollute context. A full log usually includes dependency installation, cache restore, lint output, test progress, warnings, and failure stack. The old agent pasted the entire thing; the model treated warnings as the root cause.

After redesign, the CI tool should not return a block of raw text. It should return a structured summary and raw-reference pointers:

{
  "type": "ci_failure",
  "source": "github_actions",
  "run_id": "ci-20260615-1842",
  "collected_at": "2026-06-15T10:42:31Z",
  "command": "npm test",
  "status": "failed",
  "primary_failure": {
    "test_file": "tests/invoice-discount.test.ts",
    "test_name": "applyDiscount applies WELCOME10 for new customers",
    "assertion": "expected 9900 to equal 9000",
    "stack_top": "tests/invoice-discount.test.ts:28:22"
  },
  "noise": {
    "warnings_count": 37,
    "warnings_sample": ["deprecated package warning"],
    "classified_as": "non_blocking"
  },
  "raw_log_ref": "artifact://ci-20260615-1842/log.txt#L810-L846",
  "confidence": 0.94
}

Several details matter.

primary_failure keeps the closest facts to the failure, but does not draw a business conclusion. It says “9900 is not equal to 9000”; it does not say “the discount algorithm is wrong.” noise does not delete warnings; it classifies them as non-blocking. raw_log_ref keeps the replay path, so if the summarizer is wrong, we can return to the original log.

Context engineering is not compressing logs into one sentence. It is compressing logs into verifiable evidence. Compression must preserve source, scope, and lookup path. Without those, summaries become second-hand facts with no accountability.

File Selection: Do Not Let the Model Wander Blindly

After finding the failing test, the agent needs to read files. The old way is to let the model run ls, grep, and open many files. That can work, but it is expensive and often misled by similar names.

A steadier approach is to use a lightweight retriever first, generate candidate files, and put the candidate relationship into the context package. From the test stack, imports, recent diff, and code search:

{
  "type": "file_candidates",
  "generated_by": "repo_context_indexer_v2",
  "items": [
    {
      "path": "tests/invoice-discount.test.ts",
      "reason": "failing_test",
      "priority": 1
    },
    {
      "path": "src/billing/applyDiscount.ts",
      "reason": "imported_by_failing_test",
      "priority": 1
    },
    {
      "path": "src/billing/couponRules.ts",
      "reason": "called_by_applyDiscount",
      "priority": 2
    },
    {
      "path": "src/billing/memberPrice.ts",
      "reason": "mentioned_in_old_summary_only",
      "priority": 5
    }
  ],
  "excluded": [
    {
      "path": "README.md",
      "reason": "contains stale test command; not relevant to failing assertion"
    }
  ]
}

This does not completely forbid reading other files. Context should not become a rigid cage. It should provide a default path while allowing the model to request expansion when evidence is insufficient. The key is that expansion must have a reason and be recorded.

A reasonable file-read log looks like:

{
  "event": "tool_call",
  "trace_id": "fix-1842",
  "step": "read_related_files",
  "tool": "repo.read_files",
  "input": {
    "paths": [
      "tests/invoice-discount.test.ts",
      "src/billing/applyDiscount.ts",
      "src/billing/couponRules.ts"
    ]
  },
  "policy": {
    "allowed": true,
    "reason": "paths selected from failing test imports"
  },
  "output_summary": {
    "files_read": 3,
    "total_chars": 12380,
    "truncated": false
  }
}

This log is not decoration. When the agent modifies the wrong file, you can immediately see why it read that file: failing-test import, old-summary contamination, or self-directed expansion.

Hard Constraints Should Be Checkable

The most common bad behavior of code-fixing agents is not failing to fix. It is making CI green the wrong way: deleting tests, skipping assertions, broadening conditions, swallowing exceptions, updating snapshots blindly, or weakening type checks.

These constraints should not be hidden in one sentence inside the system prompt. They should enter the context package and be verifiable by static checks or diff gates.

{
  "constraints": [
    {
      "id": "no_test_weakening",
      "severity": "blocker",
      "rule": "Do not delete, skip, or weaken the failing test unless the user explicitly asks to update business expectations.",
      "check": "diff_scan:test_assertion_changed"
    },
    {
      "id": "minimal_behavior_change",
      "severity": "blocker",
      "rule": "Limit the fix to WELCOME10 new-user discount; do not change member pricing, threshold coupons, or manual discounts.",
      "check": "targeted_test_matrix"
    },
    {
      "id": "no_secret_output",
      "severity": "blocker",
      "rule": "Logs, PR descriptions, and comments must not contain environment variables, tokens, or private URLs.",
      "check": "secret_redaction_scan"
    }
  ]
}

With constraints written this way, the model is not merely reminded. The system can inspect the generated diff: did the test assertion change, did other coupon behavior change, did the PR description contain secrets? If a check fails, the agent cannot create the PR; it must return to the repair step.

Many teams put safety and quality entirely into prompts. That is not enough. Prompts express intent; constraint systems block bad actions.

What Should the Model See During One Repair Round?

When the agent reaches “analyze and propose patch,” I want the model to see a clean workbench, not a document warehouse. The final input can be rendered as natural language, but it must come from structured context.

The model should see:

Goal:
Fix CI run ci-20260615-1842 failure in tests/invoice-discount.test.ts and create a PR.

Acceptance:
- npm test passes
- Do not delete, skip, or weaken the failing test
- Do not change discount semantics outside WELCOME10

Primary evidence:
- npm test failed: applyDiscount applies WELCOME10 for new customers
- Assertion: expected 9900 to equal 9000
- Test location: tests/invoice-discount.test.ts:28
- Raw log: artifact://ci-20260615-1842/log.txt#L810-L846

Files read:
- tests/invoice-discount.test.ts
- src/billing/applyDiscount.ts
- src/billing/couponRules.ts

Current observations:
- Test expects WELCOME10 to give new customers a 10% discount
- couponRules.ts sets WELCOME10 rate to 0.1
- applyDiscount.ts subtracts rate directly from subtotal instead of subtotal * (1 - rate)

Forbidden:
- Do not change test expectation to match 9900
- Do not skip the test
- Do not modify member pricing logic

The most valuable thing here is boundary. Every observation comes from a read file or CI evidence. It does not mix in model guesses. The old conversation’s “member price may need adjustment” does not enter primary evidence because it came only from a historical summary and has no direct link to the current failure.

If the model needs more information, such as confirming rate units, it can request more files:

{
  "request": "read_more_files",
  "reason": "Confirm whether coupon rate is consistently represented as a decimal ratio in other tests.",
  "paths": ["tests/coupon-rules.test.ts"]
}

This is much better than letting the model open arbitrary files. We are not limiting thought; we are asking the model to pay a small explanation cost when expanding context.

After Patch Generation, Context Comes From the Diff

After the agent generates a patch, the context should not remain in “ready to fix” mode. The next workbench should be rebuilt around the diff and validation results.

Suppose the patch is:

- const discount = rule.rate
- return subtotal - discount
+ const discount = subtotal * rule.rate
+ return subtotal - discount

The context package adds proposed_change:

{
  "type": "proposed_change",
  "files": ["src/billing/applyDiscount.ts"],
  "diff_summary": [
    "Apply percentage coupon rate as subtotal * rate instead of subtracting rate as a fixed amount"
  ],
  "risk": {
    "touched_test_files": false,
    "touched_public_api": false,
    "possible_behavior_change": "All coupons using percentage rate may be affected"
  },
  "required_validation": [
    "npm test -- tests/invoice-discount.test.ts",
    "npm test -- tests/coupon-rules.test.ts",
    "npm test"
  ]
}

This reveals a real engineering issue: the patch may affect all percentage coupons, not only WELCOME10. If the model only runs the failing test, risk is not covered. The context system should trigger a validation matrix based on the diff instead of letting the model decide from intuition.

Validation logs should also be structured:

{
  "event": "validation_result",
  "trace_id": "fix-1842",
  "commands": [
    {
      "cmd": "npm test -- tests/invoice-discount.test.ts",
      "exit_code": 0,
      "duration_ms": 1840
    },
    {
      "cmd": "npm test -- tests/coupon-rules.test.ts",
      "exit_code": 0,
      "duration_ms": 1260
    },
    {
      "cmd": "npm test",
      "exit_code": 0,
      "duration_ms": 9340
    }
  ],
  "coverage_of_constraints": {
    "no_test_weakening": "passed",
    "minimal_behavior_change": "passed",
    "no_secret_output": "passed"
  }
}

The final PR description should not be improvised freely. It should be generated from goal, diff summary, and validation result:

## Summary
- Fix percentage coupon calculation so rate is applied as a proportion of subtotal
- Keep failing test expectation unchanged; no test files modified

## Test Plan
- npm test -- tests/invoice-discount.test.ts
- npm test -- tests/coupon-rules.test.ts
- npm test

This PR description is plain, but reliable. It does not leak raw CI logs, private URLs, or model guesses.

Failure Mode One: Historical Summary Pollutes the Task

Context engineering must handle failure cases. The first common one is historical-summary pollution.

Suppose an old task summary says:

{
  "type": "conversation_summary",
  "created_at": "2026-06-10T09:00:00Z",
  "content": "The user previously suspected member pricing logic may need changes in memberPrice.ts.",
  "confidence": 0.42
}

The old agent would put that sentence next to the CI failure. The model sees “member pricing” and starts reading memberPrice.ts. In the redesigned flow, this summary can still be preserved, but it must be downgraded:

{
  "admission": "background_only",
  "reason": "not linked to current failing test stack or diff",
  "conflicts_with": [],
  "may_enter_model_context": false
}

Not all history deserves to enter model context. History can enter audit logs and candidate pools, but should not automatically enter the current workbench. Low-confidence guesses especially must be prevented from turning into facts through summarization.

Failure Mode Two: Compression Drops the Key Line

The second failure comes from compression. If the CI summarizer only keeps the last failure, it may miss an earlier initialization error. The solution is not “never compress.” It is to give compression quality checks.

After each tool-output compression, record:

{
  "compression": {
    "raw_chars": 184220,
    "summary_chars": 2410,
    "strategy": "failure-focused",
    "dropped_sections": [
      {"name": "dependency_install", "reason": "exit_code_0"},
      {"name": "lint_warnings", "reason": "non_blocking"}
    ],
    "must_keep_patterns": [
      "FAIL ",
      "Error:",
      "AssertionError",
      "exit code"
    ],
    "quality_checks": {
      "contains_exit_code": true,
      "contains_stack_top": true,
      "contains_failed_test_name": true
    }
  }
}

If quality_checks.contains_failed_test_name is false, the summary should not enter primary context. It should trigger re-extraction. This is tedious, but better than asking the model to reason from a broken summary.

Failure Mode Three: CI Cannot Be Reproduced Locally

Real projects often fail in CI but pass locally. The old agent keeps guessing and eventually produces a “maybe helpful” patch. The context package should expose this state directly:

{
  "type": "reproduction_status",
  "ci_failed": true,
  "local_reproduction": {
    "command": "npm test -- tests/invoice-discount.test.ts",
    "exit_code": 0,
    "environment_diff": [
      "CI uses Node 22.2.0, local uses Node 20.11.1",
      "CI has TZ=UTC, local TZ=Asia/Shanghai"
    ]
  },
  "next_allowed_actions": [
    "align_node_version",
    "run_with_ci_env",
    "inspect_test_time_dependency"
  ],
  "blocked_actions": [
    "create_pr_without_reproduction_or_explanation"
  ]
}

This stops the agent from pretending it understands the problem. The context system should allow “unknown” and turn unknown into concrete next steps. One of the biggest risks in production agents is confident output when evidence is insufficient.

Failure Mode Four: The Fix Passes, but Validation Is Incomplete

Another subtle case: the failing test passes, full tests were not run, and the agent prepares a PR. Models tend to “finish the task,” especially when the user says “fix it quickly.”

Validation should not be left to model preference. It should be driven by required_validation. Before PR creation, the system checks:

{
  "pr_gate": {
    "required": [
      "targeted_failing_test_passed",
      "related_test_matrix_passed",
      "full_test_command_passed",
      "diff_policy_passed",
      "secret_scan_passed"
    ],
    "actual": {
      "targeted_failing_test_passed": true,
      "related_test_matrix_passed": true,
      "full_test_command_passed": false,
      "diff_policy_passed": true,
      "secret_scan_passed": true
    },
    "decision": "blocked",
    "reason": "full_test_command_missing"
  }
}

After being blocked, the model can run the full test, or explain that the environment cannot run it. It cannot quietly skip it.

Context Must Be Replayable

If an agent error leaves only the final answer, the engineering team cannot fix the system. A maintainable agent must replay the context the model saw at each step.

I would save five record types for every task:

1. context_package: the structured input package for each step.
2. model_rendered_context: the actual text rendered from the structured package into model messages.
3. tool_trace: tool calls, argument summaries, output summaries, duration, and policy decisions.
4. diff_trace: model-proposed patch, applied patch, and automatic checks.
5. decision_trace: why the model chose the next action.

A simplified trace:

{
  "trace_id": "fix-1842",
  "context_version": "code_fix_context_v1",
  "model": "example-code-model",
  "steps": [
    {
      "name": "collect_ci_failure",
      "input_refs": ["ci-20260615-1842"],
      "output_refs": ["evidence:ci_failure:01"]
    },
    {
      "name": "select_files",
      "input_refs": ["evidence:ci_failure:01"],
      "output_refs": ["evidence:file_candidates:01"]
    },
    {
      "name": "propose_patch",
      "model_context_hash": "sha256:7b1...",
      "output_refs": ["diff:patch:01"]
    },
    {
      "name": "validate",
      "output_refs": ["validation:01"]
    },
    {
      "name": "create_pr",
      "gate": "passed"
    }
  ]
}

With replay, “the model hallucinated again” becomes concrete engineering work: CI summarizer missed a line, file retrieval over-weighted an old summary, diff gate failed to inspect test assertions, or PR template wrote an unverified command. Once you can locate it, you can fix it.

Context Budget Is Not Just Token Limit

Long-context windows tempt people to be lazy. Being able to stuff hundreds of thousands of tokens into a request does not mean you should.

For code-fix tasks, I would allocate budgets by material type:

Area	Initial Budget	Notes
Current goal and hard constraints	10%	Must be preserved fully; not subject to normal trimming
CI failure evidence	15%	Keep failure summary, key stack, and raw reference
Related test and implementation code	40%	Prioritize current failure path
Recent diff	15%	Only include diff related to failing files
Project conventions	10%	Test command, style, PR requirements
Historical summary	5%	Only verified and still-applicable preferences
Tool instructions	5%	Only schema summary for tools available this round

These are not fixed numbers. If the CI log is short, give more space to code. If related code is long, preserve function signatures, call sites, and nearby failing lines first, then let the model request more. Historical summaries should not displace evidence unless they directly affect current constraints.

Trimming should also be recorded:

{
  "trim_log": [
    {
      "source": "README.md",
      "chars": 6200,
      "decision": "excluded",
      "reason": "stale command and no relation to failing assertion"
    },
    {
      "source": "src/billing/memberPrice.ts",
      "chars": 3100,
      "decision": "excluded_from_model_context",
      "reason": "only linked by low-confidence old summary"
    }
  ]
}

Without a trim log, context debugging is painful. You do not know whether the model missed information because retrieval failed, budget was insufficient, relevance ranking excluded it, or the code forgot it.

Evaluate Context Packages, Not Only Final Answers

Many teams evaluate agents by asking whether the task completed. That is too late. Context quality itself needs metrics.

For this code-fixing agent, I would track:

Metric	Meaning	Bad Signal
primary_failure_capture_rate	Whether the main failure is captured correctly	Agent wastes time on unrelated warnings
relevant_file_hit_rate	Whether candidate files cover the actual fix file	Model frequently searches blindly
stale_context_ratio	Ratio of stale material entering model context	Old README or summaries affect judgment
constraint_gate_block_rate	How often gates block bad attempts	Model often tries to edit tests or skip validation
reproduction_success_rate	Whether local reproduction succeeds	Environment context is insufficient
pr_reopen_rate	PRs reopened for incomplete fixes	Validation matrix is weak

These do not need full automation on day one. Start with a replay set of ten or twenty real failures. Every time you change CI summarization, file retrieval, context templates, or model versions, regenerate context packages and compare whether they are cleaner.

The same final answer does not mean the same system quality. One agent may get lucky after reading the wrong files and guessing. Another may produce the same patch with a clear evidence chain and complete validation. In production, I trust the second one.

Implementation Checklist: Start With One Failing Test

If you already have a working code agent, you do not need to rebuild everything. Change it in this order.

First, save the final content sent to the model. Do not optimize yet. Add source labels to each segment: user input, CI, file, history, tool instructions, model summary. Many problems will become visible immediately, such as old summaries taking one-third of context while the real failure log has only two lines.

Then change CI output into structured evidence. Keep failing test name, assertion, stack top, command, exit code, raw reference, and non-blocking noise. Do not let the full log directly enter model context.

Next, build a file candidate selector. The simplest version can use failing-test imports, stack paths, git diff --name-only, and code search. It does not need to be perfect, but it must record why each file was selected.

Then add hard constraints and gates. Do not modify tests just to pass CI. Do not skip tests. Do not output secrets. Do not create PRs when validation is incomplete. These rules must be checkable, not only visible to the model.

Finally, build a replay set. Pick twenty historical CI failures: wrong fixes, non-reproducible failures, environment problems, and cases where the test itself was wrong. Fix the input and save context packages. Every time you change the context template, run those samples first.

After one chain works, extend to more task types. Context engineering suffers when it starts as a broad platform abstraction. First make one real scenario explainable, verifiable, and reversible.

When the Agent Should Stop

There is one unpopular but important design: the agent must know when to stop.

I would stop automatic modification in these cases: CI fails but cannot be reproduced locally and the environment difference is unclear; the failing test conflicts with product requirements; the fix requires changing public APIs; data migrations are involved; validation commands are unstable; the patch deletes a lot of code; high-confidence evidence conflicts inside the context.

Stopping is not failure. For production agents, continuing wrongly is worse than stopping honestly. The context package should give the model a legitimate “stop” path instead of framing every task as something that must be completed.

For example, the PR gate can return:

{
  "decision": "needs_user_confirmation",
  "reason": "test expectation conflicts with current product spec document",
  "safe_next_message": "I found a conflict: the test expects WELCOME10 to give new users 10% off, but the current product spec says the first order gets a fixed 100 discount. Please confirm which source is authoritative."
}

This is far better than letting the model choose one direction and modify code.

Context Engineering Is a Team Interface

Many people treat context engineering as prompt concatenation before a model call. That is too narrow. It is actually an interface between teams.

Platform teams own context packages, trace, gates, versions, and replay. Business engineers provide real failure samples, domain constraints, and validation matrices. Security teams own secret scanning, permission boundaries, and log retention. Product or engineering leads define which actions may be automatic and which require confirmation.

If these responsibilities are unclear, agent mistakes become blame games. The model team says business context was missing; the business team says the model is unreliable; the platform team says the prompt already mentioned it; the security team says logs are insufficient. The value of a context package is that arguments land on concrete fields.

I judge whether a code agent has entered a maintainable phase by four questions: what highest-priority goal did it see? What evidence did it rely on? What material was trimmed, and why? Which gates passed before it created a PR?

If these questions cannot be answered, the system is still in prompt-concatenation mode. If they can, it is ready to talk about model choice, cost optimization, and more complex automation.

Prompt can make an agent sound like a senior engineer. Context Engineering helps it avoid treating old notes as requirements, warnings as root causes, and test weakening as bug fixing. In production, the deciding factor is not one magic prompt. It is the cleaned, replayable, challengeable workbench placed in front of the model before every action.

References

• Model Context Protocol documentation
• OpenTelemetry GenAI semantic conventions
• Ragas documentation: from intuition checks to systematic evaluation

WeChat

Follow ZiCode on WeChat

If this post was useful, you can follow later updates on WeChat as well.

X / Twitter

Follow @ax2_zicode

Faster technical notes, short thoughts, and new-post alerts are posted on X.

X Open X Profile @ax2_zicode