The previous article in Programmer Craft in the AI Era covered testing. Tests answer one question: under known inputs, does the system behave as expected?

Production systems are harder. Real user input may not appear in tests. Third-party APIs may be unstable. Queues may back up. Databases may become slow. A new version may fail only on a small subset of requests. At that point, beautiful code is not enough. You must be able to see what is happening inside the system.

This tenth article is about observability. Logs, metrics, traces, audit records, and product events are not a few print statements added after launch. They are part of system design, and they are engineering boundaries that AI-generated code must satisfy.

AI Easily Forgets to Add Eyes

Ask AI to write an API, and it will usually write the main flow:

async def create_report(request: ReportRequest) -> ReportResponse:
    report = await save_report_request(request)
    await enqueue_report_job(report.id)
    return ReportResponse(report_id=report.id, status="queued")

This code looks runnable. But if a production user says, “I clicked generate report and never got a result”, a series of questions immediately appears:

• Did the request arrive?
• Was the database write successful?
• Was the queue message delivered?
• Did the worker consume it?
• Did a third-party API time out?
• Was there a retry after failure?
• Did retry duplicate billing or duplicate result writes?
• Did this user hit a special parameter combination?
• Is only this user slow, or is everyone slow?

Without observability, these questions can only be guessed.

The most common problem in AI-generated code is treating “the feature can run” as “the system is deliverable”. A truly deliverable service must leave enough evidence when something goes wrong.

Logs, Metrics, and Traces Each See One Layer

Many people understand observability as logs. Logs are important, but logs alone are not enough.

A service needs at least three views:

Logs      What happened
Metrics   How the overall system state changed
Trace     Which steps one request went through

Logs answer “what happened this time”. For example, a report_id was created successfully, a third-party API returned 429, or one retry failed.

Metrics answer “how the system is doing overall”. For example, requests per minute, error rate, P95 latency, queue backlog, and database connection-pool usage.

Trace answers “where one request became slow”. For example, the API took 80ms, the database write took 20ms, queue delivery took 10ms, the worker waited 4 minutes, and a third-party call took 15 seconds.

If any one of these views is missing, troubleshooting becomes half-blind.

Logs Are Events, Not Strings

AI easily writes logs like this:

logger.info(f"create report success: {report.id}")

This is better than no log, but it is not enough. It is a string for humans, and machines cannot analyze it reliably.

Production systems should prefer structured logs:

logger.info(
    "report_request_created",
    extra={
        "event": "report_request_created",
        "report_id": report.id,
        "user_id": request.user_id,
        "tool_id": request.tool_id,
        "request_id": context.request_id,
        "trace_id": context.trace_id,
        "status": "queued",
    },
)

The key difference is fields.

Fields make logs searchable, aggregatable, and connectable. You can query the full process of one report_id, check the recent failure rate of one tool_id, or connect API logs and worker logs by trace_id.

Important fields in structured logs usually include:

timestamp       Event time
level           Log level
event           Stable event name
service         Service name
env             Environment
request_id      Single entry request ID
trace_id        Distributed trace ID
span_id         Current step ID
user_id         User ID, with masking and compliance in mind
resource_id     Business object ID, such as report_id / order_id
status          success, failed, skipped, retrying
error_code      Stable error code
duration_ms     Current step duration

A practical standard is this: logs are not for the person writing the code today. They are for the person on call three months later.

Log Levels Need Discipline

AI-generated code often uses log levels carelessly. A normal branch logs error, a large loop logs info, and an exception is swallowed with only a vague “failed”.

Log levels should have relatively stable meanings:

DEBUG  Development and temporary diagnosis; off by default in production or sampled
INFO   Normal lifecycle events, such as request creation, job completion, state change
WARN   Recoverable abnormal conditions, such as retry, degradation, rate limiting, temporary third-party failure
ERROR  Current operation failed and should be counted and possibly alerted
FATAL  Process cannot continue; usually rare

The easiest level to misuse is INFO. If a loop processes hundreds of thousands of objects and writes one INFO log for each object, the logging system slows down, and the database may be overloaded by log writes. A normal skipped branch should usually be a counter or sampled log, not a flood of lines.

Requirements for AI should be clear:

Do not write one INFO log per item inside high-frequency loops.
Use metrics counters for normal skip branches.
DEBUG logs may be sampled, and summaries can be dropped when the queue is full.
ERROR logs must include error_code, trace_id, resource_id, and exception type.

This is not nitpicking. Logs themselves can become system load.

Metrics Show Trends, Not Stories

Logs tell you what happened in one request. Metrics tell you whether the system is becoming worse.

The most common service-level metrics are RED:

Rate      Request rate
Errors    Error rate
Duration  Latency

For resources, people often look at USE:

Utilization  Usage, such as CPU, connection pool, disk
Saturation   Saturation, such as queue backlog or thread waiting
Errors       Error count

For a report-generation service, a minimal metric set may include:

http_requests_total{route, method, status}
http_request_duration_seconds_bucket{route, method}
report_requests_total{tool_id, status}
report_jobs_enqueued_total{queue}
report_jobs_processed_total{queue, status}
report_job_duration_seconds_bucket{queue}
report_queue_depth{queue}
third_party_calls_total{provider, status}
third_party_call_duration_seconds_bucket{provider}
db_pool_in_use{database}

Metric labels must be controlled.

Fields with low to medium cardinality, such as route, status, provider, and tool_id, are usually suitable labels. Fields like user_id, request_id, and report_id are usually not suitable metric labels, because their cardinality is too high and can overwhelm a time-series database.

This is another place where AI-generated code needs review. It may think “more fields are better”, but metric systems fear unbounded cardinality.

Histograms Are More Honest Than Averages

For latency, do not look only at averages.

An API with an average latency of 200ms may have most requests at 50ms and a few requests at 10 seconds. Users feel tail latency, not averages.

Engineering teams more often look at P50, P95, and P99:

P50   Half of requests are faster than this
P95   95% of requests are faster than this
P99   99% of requests are faster than this

If a report API has P50 at 120ms, P95 at 2s, and P99 at 15s, a small number of requests are very slow. At that point, follow traces for slow requests instead of staring at the average and saying “it looks fine”.

When asking AI to add metrics, be explicit:

Add duration histograms for external APIs, database queries, queue jobs, and overall requests.
Do not record only average duration.
Alerts should focus on error rate, P95/P99, queue backlog, and saturation.

Trace Is the Timeline of One Request

The core of tracing is splitting one request into multiple spans.

A report-generation flow might look like this:

trace_id=abc

API /reports
  ├─ validate_request                 3ms
  ├─ insert report_request           18ms
  ├─ enqueue report_job               9ms
  └─ return queued                    1ms

worker report_job
  ├─ load report_request             12ms
  ├─ call provider A               2400ms
  ├─ normalize result                35ms
  ├─ save report_result              22ms
  └─ publish notification             8ms

Without trace, you have to search API logs, queue logs, and worker logs back and forth. Trace puts these steps on one timeline.

The key is context propagation. The entry request creates a trace_id. When the API writes to the database, sends messages, or calls downstream services, it must pass the ID along. When a queue job is consumed by a worker, the worker should continue the same trace instead of creating an isolated new trace.

HTTP usually carries it through a header:

traceparent: 00-<trace_id>-<span_id>-01

Queue messages can carry it in metadata:

{
  "job_id": "job_123",
  "report_id": "rpt_456",
  "trace_id": "abc",
  "parent_span_id": "def"
}

If you ask AI to write async jobs, this must be part of the requirement. Otherwise the API and worker will produce separate sets of logs, and troubleshooting will rely on stitching by business IDs.

Audit Records Are Not a Replacement for Logs

Another kind of data is often mixed into logs: audit records.

Logs are for engineering troubleshooting. Audit records are for business facts. They cannot replace each other.

For example, when a user changes permissions, the log can record:

event=permission_update_saved trace_id=...

The audit record should store a more stable business fact:

{
  "actor_id": "user_1",
  "action": "permission.grant",
  "target_type": "project",
  "target_id": "project_9",
  "before": {"role": "viewer"},
  "after": {"role": "editor"},
  "reason": "owner_action",
  "created_at": "2026-07-03T08:00:00Z"
}

Audit records should be traceable, displayable, and retained for the long term. Logs may be cleaned by retention policy. Audit records often cannot be discarded casually.

When AI generates admin, permission, billing, or state-change code, always ask: does this action need an audit record?

Product Events Answer “How Users Use It”

Logs and traces mainly help engineers understand the system. Product events help product and operations teams understand user behavior.

For the same report-generation service, events may include:

report_create_clicked
report_create_submitted
report_create_succeeded
report_create_failed
report_result_viewed
report_result_exported

Each event needs a stable schema:

{
  "event": "report_create_submitted",
  "user_id": "user_1",
  "session_id": "sess_1",
  "tool_id": "tool_x",
  "source": "search_result",
  "input_size_bucket": "small",
  "created_at": "2026-07-03T08:00:00Z"
}

There are two common ways product events go wrong.

First, event names are invented casually. Today it is click_report, tomorrow it is report_click, and the day after tomorrow it is submit_report_request. Eventually nobody trusts the data.

Second, everything is stuffed into the event. Sensitive information, long text, raw parameters, and user input create serious problems later if they enter the analytics system without processing.

When asking AI to write product events, give it an event dictionary first. Do not let it improvise.

Observability Has a Cost Too

Observability is not “the more, the better”.

Too many logs increase storage cost. Too many metric labels can overwhelm the time-series database. Full trace sampling can be expensive. Overly detailed product events make analysis chaotic.

Real systems often use these strategies:

Logs: keep all ERROR logs; control INFO volume; DEBUG off by default or sampled.
Metrics: aggregate all core paths; avoid high-cardinality fields.
Trace: full sampling for low-traffic services; proportional sampling for high-traffic services; always keep error requests.
Audit: persist all critical business actions with stable fields.
Events: design around funnels and critical paths; do not instrument casually.

For a service with only a few thousand requests per day, simple logs and a small set of metrics may be enough. For a service with tens of millions of requests per day, if sampling and labels are not controlled, the observability system may fail before the business system does.

AI will not naturally make this judgment for you. It can generate integration code, but humans must specify the cost boundary.

An Observability Acceptance Checklist for One API

If I ask AI to write a production API, I put observability into the acceptance checklist:

1. Every entry request generates or inherits request_id and trace_id.
2. Success, failure, retry, and skip paths all have stable event names.
3. ERROR logs include error_code, exception type, trace_id, and business object ID.
4. High-frequency loops do not write one INFO log per item.
5. Request count, error count, and latency histogram are exposed as metrics.
6. External calls, database queries, and queue delivery each have separate spans.
7. Async jobs inherit upstream trace context.
8. Permission, billing, and state changes write audit records.
9. Key user actions write product events, with names and fields from the event dictionary.
10. Metric labels do not include high-cardinality fields such as user_id or request_id.

This checklist is far more effective than “add logs”.

A Prompt for AI

Observability should be written into the prompt before implementation:

Please implement the report-generation API and asynchronous worker.

Besides business logic, the implementation must satisfy these observability requirements:
- Use structured logs. Do not concatenate long strings that cannot be queried.
- Every entry request has request_id and trace_id.
- API, database write, queue delivery, worker consumption, and third-party calls each create trace spans.
- Queue messages must carry trace_id and parent_span_id.
- Expose RED metrics: request count, error count, latency histogram.
- Expose queue metrics: enqueued count, consumed count, failed count, queue backlog, job duration.
- ERROR logs must include error_code, trace_id, report_id, and exception type.
- High-frequency loops must not write one INFO log per item; normal skips use counters, DEBUG can be sampled.
- User actions for generating, viewing, and exporting reports need product events.
- Creation, cancellation, billing, and permission changes need audit records.

After implementation, list which queries can locate:
1. Where a report_id is stuck.
2. Whether error rate increased in the last 15 minutes.
3. Whether one provider became slower.
4. Whether the queue is backing up.
5. The key operation path of one user.

The purpose of this prompt is not to make AI write more boilerplate. It is to make AI understand that a system must remain understandable after it goes online.

Real Engineering Quality Means Not Panicking When Things Fail

When a system is calm, people tend to look at features, UI, and performance.

When things fail, what saves you is the evidence chain.

Where did the request come from? Which modules did it pass through? Which step slowed down? Which dependency failed? How many users were affected? Was anything executed twice? Did it create business loss? Is compensation needed?

These questions are answered by logs, metrics, traces, audit records, and product events together.

AI makes code generation faster, and it also creates more services that “look runnable”. The programmer’s responsibility is not to stop AI from writing code. It is to set engineering boundaries for it: testable, observable, debuggable, and handoff-ready.

Systems will fail sooner or later. The difference is whether you open the console and start locating the problem, or sit in front of a black box and guess.

WeChat

Follow ZiCode on WeChat

If this post was useful, you can follow later updates on WeChat as well.

X / Twitter

Follow @ax2_zicode

Faster technical notes, short thoughts, and new-post alerts are posted on X.

X Open X Profile @ax2_zicode