This article starts Volume Three of Programmer Craft in the AI Era: bringing AI-generated work into an engineering system. The previous articles covered programs, data structures, APIs, databases, and high concurrency. Those topics are about design. But even good design eventually has to become code. AI writes code fast, so fast that people can easily lower their guard.

In AI coding, testing is not decorative. It is a seatbelt. Without tests, AI-generated code is like an unfamiliar module that appears to run: you do not know which parts are assumptions, which parts merely worked by accident, and which parts will collapse under boundary conditions.

People who know how to use AI to write code should care more about tests, not less.

AI Is Very Good at Writing Code That Looks Real

The danger of AI-generated code is not that it is always wrong. Quite the opposite: it often looks real.

Function names look real. Layering looks real. Exception handling looks real. Comments look real too. It may pass the simplest manual check, while quietly failing under several boundary conditions.

For example, a price calculation function:

def calculate_total(items, discount_rate):
    total = sum(item.price * item.quantity for item in items)
    return total * (1 - discount_rate)

It looks fine, but many questions remain:

• Can discount_rate be negative?
• Can a discount exceed 1?
• Are amounts floating-point numbers?
• Can quantity be 0?
• How is the result rounded?
• Is there a discount cap?
• Should an empty list return 0 or raise an error?

AI can write this function quickly. The human job is to turn these rules into tests.

Without tests, you can only read the code and guess. The value of tests is turning “I think” into “this input must produce this output”.

Write Examples Before Writing the Implementation

In AI coding, I prefer to understand “write tests first” as “write examples first”. Do not ask AI to implement the feature immediately. Ask it to list inputs, outputs, boundaries, and failure scenarios first.

For example:

Do not write the implementation yet.

Please list test cases for "order total calculation":
- Normal product quantity and price.
- Empty item list.
- Discount is 0.
- Discount is 1.
- Discount is less than 0.
- Discount is greater than 1.
- Amounts must be represented in cents, not floating-point numbers.
- Discount amount has a maximum cap.

This step exposes whether the requirement is clear. Very often, after AI lists the examples, you realize the product rules are not decided yet.

After the examples are confirmed, ask AI to write the implementation and tests. Then the code grows around rules, rather than around one vague requirement.

Tests can look like this:

import pytest


@pytest.mark.parametrize(
    ("items", "discount_bps", "expected"),
    [
        ([Item(price_cents=1000, quantity=2)], 0, 2000),
        ([Item(price_cents=1000, quantity=2)], 1000, 1800),
        ([], 0, 0),
    ],
)
def test_calculate_total(items, discount_bps, expected):
    assert calculate_total(items, discount_bps) == expected


def test_reject_invalid_discount():
    with pytest.raises(ValueError):
        calculate_total([Item(price_cents=1000, quantity=1)], -1)

Here discount_bps means basis points, avoiding floating-point errors. The tests force the implementation to become steadier.

Unit Tests Check Rules, Not Frameworks

Unit tests are best for checking pure rules.

Price calculation, state transitions, permission decisions, pagination cursor encoding and decoding, retry backoff, rate-limit token buckets, and field masking should all be written as much as possible as functions or small objects that do not depend on a database, network, or framework.

For example, order-state transitions:

ALLOWED_TRANSITIONS = {
    "pending": {"paid", "cancelled"},
    "paid": {"shipped", "refunding"},
    "shipped": {"completed"},
    "cancelled": set(),
}


def can_transition(current: str, target: str) -> bool:
    return target in ALLOWED_TRANSITIONS.get(current, set())

Tests:

def test_pending_can_be_paid():
    assert can_transition("pending", "paid")


def test_cancelled_cannot_be_paid():
    assert not can_transition("cancelled", "paid")


def test_unknown_status_cannot_transition():
    assert not can_transition("unknown", "paid")

These tests run fast, failures are easy to locate, and AI can repeatedly modify the code against them.

A common mistake is making unit tests start the whole web framework, connect to a real database, and call real external services. That is not a unit test anymore. Once tests become heavy, people stop running them, and AI cannot quickly verify changes either.

When asking AI to write unit tests, be explicit:

First extract the core rules into pure functions.
Unit tests must not access the database, network, filesystem, or real time.
External dependencies must be passed in as parameters or mocked.

Light tests make iteration fast.

Integration Tests Check Boundaries

Unit tests check rules. Integration tests check module boundaries.

For example, in an order-creation API, unit tests can cover price calculation, inventory-deduction rules, and state transitions. Integration tests should check:

• HTTP request schema.
• Permission.
• Database transaction.
• Unique constraint.
• Idempotency key.
• Error response structure.
• Whether outbox records are written.

Integration tests can use a test database, but they must be repeatable and cleanable.

async def test_create_order_writes_outbox(client, db):
    response = await client.post(
        "/api/orders",
        json={
            "items": [{"sku": "book_001", "quantity": 1}],
            "client_order_id": "c_001",
        },
        headers={"Idempotency-Key": "create-order:u1:c001"},
    )

    assert response.status_code == 200
    body = response.json()

    outbox = await db.fetch_one(
        "select * from outbox where aggregate_id = :order_id",
        {"order_id": body["id"]},
    )
    assert outbox["event_type"] == "order.created"

This test is not testing one function. It is testing the promise between the API and the database.

Integration tests should be controlled in number. If every path is covered through integration tests, the suite becomes slow. Critical paths, transaction boundaries, and cross-module contracts are the places worth guarding with integration tests.

Fixtures Are the Language of Tests

The easiest part of a test suite to ruin is data preparation.

If every test manually writes a large block of user, order, product, and inventory data, the tests become noisy. The reader cannot see the point, only boilerplate.

Fixtures turn “give me a valid user”, “give me a paid order”, and “give me a product with insufficient inventory” into test language.

@pytest.fixture
def active_user():
    return User(id="u_1", status="active", role="user")


@pytest.fixture
def paid_order(active_user):
    return Order(id="o_1", user_id=active_user.id, status="paid")

Then tests can read like rules:

def test_paid_order_can_request_refund(paid_order):
    assert can_request_refund(paid_order)

But fixtures can also go bad. Overly shared large fixtures make tests depend on hidden state. Changing one fixture can affect many tests.

Several practical principles:

• Fixture names should express business state.
• Do not use one universal fixture.
• Each test should take only the data it needs.
• Fixtures should generate valid objects by default.
• Boundary and invalid data should be explicitly overridden inside the test.

When asking AI to write tests, require:

Please design fixtures first.
Fixture names should express business meaning, such as active_user, paid_order, expired_token.
Do not create one universal fixture containing every field.
Each test should use only the fixtures it needs.

Good fixtures make tests read like documentation.

Contract Tests Prevent Interfaces from Quietly Breaking

Once an API contract has consumers, it cannot change casually.

Contract tests do not care about internal implementation. They care only whether input and output keep the promise. The earlier API article discussed schemas, error codes, idempotency, and pagination. These should all have contract tests.

For example, error responses:

def test_validation_error_contract(client):
    response = client.post("/api/orders", json={"items": []})

    assert response.status_code == 400
    body = response.json()
    assert set(body.keys()) == {"error", "request_id"}
    assert body["error"]["code"] == "INVALID_ARGUMENT"
    assert body["error"]["retryable"] is False

Pagination contract:

def test_cursor_pagination_contract(client, seed_orders):
    first = client.get("/api/orders?limit=20").json()

    assert "items" in first
    assert "page_info" in first
    assert "next_cursor" in first["page_info"]
    assert "has_more" in first["page_info"]

During refactoring, AI may easily rename page_info to pagination, forget retryable, or change an empty list into null. Contract tests prevent these “small-looking” breaks.

When asking AI to change an API, say first:

Add contract tests before changing the implementation.
Existing response fields must not be removed or renamed.
New fields must remain backward compatible.
The error response structure must not change.

Contract tests protect callers, not implementations.

Regression Tests Should Grow From Bugs

After a production bug is fixed, the most important question is not “was the code changed?” It is “can this bug come back later?”

Regression tests should grow from bugs.

For example, suppose a real issue happened: after a task worker restarted, some jobs stayed in running forever. The fix may add lease recovery:

update jobs
set status = 'pending',
    locked_by = null,
    locked_at = null
where status = 'running'
  and locked_at < now() - interval '30 minutes';

The regression test:

async def test_recover_stale_running_jobs(db, freezer):
    job = await create_job(status="running", locked_at=minutes_ago(40))

    await recover_stale_jobs(now=freezer.now())

    refreshed = await get_job(job.id)
    assert refreshed.status == "pending"
    assert refreshed.locked_by is None

This test records a real incident. Later, if AI refactors the worker state machine and accidentally removes lease recovery, the test stops it.

Every important bug deserves these questions:

• What is the smallest reproducing input?
• Which state combination triggers the problem?
• Which external dependency needs to be mocked?
• Is this bug best covered by a unit test, integration test, or end-to-end test?
• Can the test name clearly describe the historical problem?

Do not only ask AI to “fix it”. Ask it to turn the bug into a test.

Failure Cases Are More Valuable Than the Happy Path

AI is good at writing the happy path. Engineering quality is mostly about failure paths.

Async jobs should test:

• Duplicate messages.
• Downstream timeout.
• Temporary-error retry.
• Permanent-error dead letter.
• Recovery after worker crash.

APIs should test:

• Missing parameters.
• Insufficient permission.
• Duplicate idempotency key.
• Reusing an idempotency key with a different request body.
• Downstream degradation.

Databases should test:

• Unique-constraint conflict.
• Concurrent deduction.
• Transaction rollback.
• Optimistic-lock conflict.

High-concurrency code should test:

• Full queue.
• Connection-pool wait timeout.
• Rate-limit hit.
• Open circuit breaker.

For example, testing idempotency-key reuse:

def test_reject_same_idempotency_key_with_different_payload(client):
    headers = {"Idempotency-Key": "create-order:u1:c001"}

    first = client.post("/api/orders", json={"items": [{"sku": "a", "quantity": 1}]}, headers=headers)
    second = client.post("/api/orders", json={"items": [{"sku": "b", "quantity": 1}]}, headers=headers)

    assert first.status_code == 200
    assert second.status_code == 409
    assert second.json()["error"]["code"] == "IDEMPOTENCY_KEY_REUSED"

This kind of test constrains the system more strongly than “order creation succeeds”.

When asking AI to write a test plan, require:

Happy paths should be at most 30%.
Please focus on failure cases, boundary cases, concurrency cases, and historical regression cases.
For each case, explain which business rule it protects.

This moves AI from “write several tests” to “guard the system behavior”.

Use Mocks With Restraint

Mocks are necessary, but overusing them makes tests fake.

If a test mocks everything and only verifies that a function was called once, it may prove almost nothing about business correctness.

Bad smell:

payment_client.charge.assert_called_once()
email_sender.send.assert_called_once()

This only proves that calls happened. It does not prove state correctness, idempotency correctness, or failure handling.

A better test verifies results and observable side effects:

assert order.status == "pending_payment"
assert outbox.event_type == "payment.requested"
assert outbox.payload["order_id"] == order.id

Mocks are suitable for isolating uncontrollable external systems, such as payment, SMS, third-party APIs, and real time. Do not mock the business rule you are testing.

Several principles:

• Mock external boundaries, not core rules.
• Prefer verifying state and output, not only call counts.
• Mock returns should cover success, timeout, temporary error, and permanent error.
• Do not let mocks hide real serialization, schema, or transaction problems.

When asking AI to write tests, require:

Only mock external systems. Do not mock core business functions in the current module.
Tests should verify output, state changes, database records, or outbox records, not only call counts.

This reduces the problem of “many tests, but none testing the point”.

End-to-End Tests Should Be Few and Hard

End-to-end tests are closest to real user paths, but they cost the most, run the slowest, and are easiest to make unstable.

So they should be few and hard.

Good end-to-end scenarios include:

• Login to completing a core business action.
• Creating an order until payment status is written back.
• Uploading a file until the job succeeds.
• Searching and then executing a tool.
• Critical admin-console operations.

Do not put every boundary condition into end-to-end tests. Boundary conditions should mainly be covered by unit tests, integration tests, and contract tests.

A healthy test structure roughly looks like:

Level	Quantity	Purpose
Unit tests	Many	Quickly cover rules and boundaries
Integration tests	Medium	Cover database, API, queue boundaries
Contract tests	Medium	Freeze external promises
End-to-end tests	Few	Cover critical user paths

The test pyramid is not dogma, but the direction is right: more tests at lower levels, fewer tests at upper levels.

The faster AI writes code, the more important it is to avoid making end-to-end tests the only defense. Otherwise every change becomes slow, and eventually nobody runs the tests.

Tests Should Be Part of the Prompt

When asking AI to write a feature, include testing in the working protocol:

Write the test plan before implementation.
The test plan must include:
1. Unit tests for core business rules.
2. Integration tests for database transactions and unique constraints.
3. Contract tests for API success and error responses.
4. At least 3 failure cases.
5. At least 1 historical regression case.

Give me the test list first. Do not write the implementation yet.
After I confirm, write the test code and implementation code.

For a bug fix:

First write a failing test based on the bug description.
After confirming the test fails, modify the implementation.
After the fix, the test must pass. Explain why this test prevents the issue from regressing.

This significantly improves the quality of AI coding. AI is no longer responsible only for “writing the code”. It is also responsible for “proving the code meets expectations”.

Test Code Needs Review Too

Tests are not automatically correct. AI-generated tests need review as well.

Look especially at:

• Whether the test name describes business behavior.
• Whether assertions truly verify the result.
• Whether there are only happy paths.
• Whether mocks are excessive.
• Whether fixtures hide too much state.
• Whether tests depend on execution order.
• Whether tests access real external services.
• Whether time, randomness, or concurrency can make them unstable.
• Whether tests exist only for coverage.

A weak test:

def test_create_order(client):
    response = client.post("/api/orders", json={...})
    assert response.status_code == 200

It only proves that no error was thrown. A better test states the business promise:

def test_create_order_persists_pending_order_and_outbox_event(client, db):
    response = client.post("/api/orders", json={...}, headers={...})

    assert response.status_code == 200
    order = db.get_order(response.json()["id"])
    assert order.status == "pending_payment"
    assert db.find_outbox("order.created", order.id) is not None

The test name, input, and assertions should let a reader understand what the system promises.

A Testing Prompt for AI

You can use this as a fixed prompt:

Do not write the implementation yet.

Please design a test plan for this requirement:
1. List core business rules and corresponding unit tests.
2. List database, queue, transaction, and API boundaries that need integration tests.
3. List API contract tests, including success response, error response, pagination, idempotency, and permission.
4. Design fixtures whose names express business state.
5. List failure cases: timeout, duplicate request, insufficient permission, concurrency conflict, downstream failure.
6. If this is a bugfix, first write a failing test that reproduces the bug.
7. Explain which dependencies can be mocked and which must not be mocked.
8. Explain the business promise protected by each test.
9. Only then provide an implementation plan.

The core of this prompt is: pin down behavior before generating code.

Make the Result Reproducible

In the AI coding era, code generation becomes faster and faster. The scarce thing is reproducible judgment.

Tests are reproducible judgment.

They tell you: this input must produce this output; this error must return this code; this job must not create side effects twice when executed repeatedly; this bug must not come back after it has been fixed.

Without tests, an AI-generated result is only an answer that looks good once. With tests, it has a chance to enter an engineering system, be handed over, refactored, released, and maintained for a long time.

A seatbelt is not for driving slowly. A seatbelt is what lets you drive the car onto the road with confidence.

WeChat

Follow ZiCode on WeChat

If this post was useful, you can follow later updates on WeChat as well.

X / Twitter

Follow @ax2_zicode

Faster technical notes, short thoughts, and new-post alerts are posted on X.

X Open X Profile @ax2_zicode