Quant Trading for Programmers 08: Clean Raw K-Lines Into Trustworthy Market Bars

Article 7 produced a clean group of stocks. Next comes market data.

Market data is easy to underestimate. It looks like only date, open, high, low, close, volume, and amount. But vendor formats, units, missing values, and abnormal rows can quietly bias a backtest if they are not handled.

Raw K-Lines Should Not Go Directly Into The Database

The same daily bar may look like this from one source:

日期, 开盘, 最高, 最低, 收盘, 成交量
2026-06-15, 10.0, 10.8, 9.9, 10.5, 120

Or like this from another:

{"date": "20260615", "open": "10.0", "high": "10.8", "low": "9.9", "close": "10.5", "volume": "12000"}

Volume may be measured in lots or shares. If that is not normalized, participation rate, liquidity filters, and backtest execution constraints will all be wrong.

Define The Clean Object First

Chapter 8 adds app/market_data.py. The cleaned bar object is called CleanMarketBar:

@dataclass(frozen=True)
class CleanMarketBar:
    symbol: str
    trade_date: date
    open: float
    high: float
    low: float
    close: float
    volume: float
    amount: float
    source: str
    payload: dict[str, Any] = field(default_factory=dict)

It is not an ORM object. It sits between raw vendor data and database writes. Its responsibility is clear: convert raw rows into one consistent internal format.

Date And Number Parsing Should Be Tolerant

Vendor fields are rarely identical. Dates may be 2026-06-16, 20260616, or 2026/06/16. Numbers may contain commas, percent signs, empty values, or --.

So we first write two small parsing functions:

def parse_trade_date(value: Any) -> date | None:
    if isinstance(value, date):
        return value
    text = str(value or "").strip().replace("/", "-")
    for fmt in ("%Y-%m-%d", "%Y%m%d"):
        try:
            return datetime.strptime(text[:10] if fmt == "%Y-%m-%d" else text, fmt).date()
        except ValueError:
            continue
    return None

def parse_number(value: Any) -> float | None:
    text = str(value).replace(",", "").replace("%", "").strip()
    if not text or text in {"-", "--", "nan", "None"}:
        return None
    return float(text)

In engineering terms: parsing may be tolerant, but validation must be strict.

OHLC Must Be Self-Consistent

Chapter 8’s clean_market_bars() checks several issues:

• Missing trade date: missing_trade_date
• Duplicate date: duplicate_trade_date
• Missing or non-positive close price: invalid_close
• Invalid high, low, open, and close relationship: invalid_ohlc_range

The key point is not to silently discard bad rows. The function returns two results: cleaned bars and rejected rows.

bars, rejected = clean_market_bars("600519.SH", rows, source="eastmoney", volume_unit="lot")

Rejected rows include reasons and original rows:

{"reason": "invalid_ohlc_range", "row": row}

This is useful when vendor fields change. In a real system, rejected can later enter audit logs or data-task results.

Normalize Volume Into Shares

In A-share markets, one lot commonly means 100 shares. If a vendor returns volume in lots, the internal system should convert it to shares:

volume_multiplier = 100 if volume_unit == "lot" else 1
normalized_volume = round(volume * volume_multiplier, 4)

This is written into payload:

payload={"raw": row, "volume_unit": "shares"}

The internal object therefore states that volume is already in shares. The raw row is still preserved in payload so later issues can be traced.

Coverage Report

After cleaning, we also need a small coverage summary:

def coverage_report(bars: Iterable[CleanMarketBar]) -> dict[str, object]:
    rows = list(bars)
    dates = [row.trade_date for row in rows]
    symbols = sorted({row.symbol for row in rows})
    return {
        "rows": len(rows),
        "symbols": len(symbols),
        "first_date": min(dates).isoformat() if dates else None,
        "latest_date": max(dates).isoformat() if dates else None,
        "sources": sorted({row.source for row in rows}),
    }

This is a small version of a future /api/data/quality endpoint. Before writing strategy code, we should know how many rows exist, how many symbols are covered, and what the earliest and latest dates are.

Runnable Foundation Check

Market-data cleaning should show both coverage and rejection reasons. The current project uses this command to reproduce the foundation capabilities from articles 01-08:

uv run python -m scripts.chapter_examples foundation-check

The output for this chapter is:

The screenshot shows that 2 cleaned rows remain, covering 1 stock, from 2026-01-02 to 2026-01-05. The rejected row keeps invalid_ohlc_range, proving that a row with high price below open price is not silently passed into later backtests.

Hands-On Task

Clone the chapter 8 code:

git clone https://github.com/ax2/zi-quant-platform.git
cd zi-quant-platform
git checkout chapter-08
uv sync --extra dev
uv run pytest

Run only the market-data tests:

uv run pytest tests/test_market_data.py

The full chapter 8 test suite passes with 159 passed, still with only the existing FastAPI deprecation warning.

Chapter Update And Repository

This chapter adds:

• app/market_data.py.
• Trade-date parsing, numeric parsing, OHLC validation, duplicate-row rejection, volume-unit normalization, and coverage summaries.
• tests/test_market_data.py, verifying cleaned results and rejection reasons.

Repository:

https://github.com/ax2/zi-quant-platform

Code for this chapter:

git clone https://github.com/ax2/zi-quant-platform.git
cd zi-quant-platform
git checkout chapter-08
uv sync --extra dev
uv run pytest tests/test_market_data.py

Summary

The point of market data is not “we downloaded it”. It is “after cleaning, can we explain it?”

This article turns date parsing, numeric parsing, OHLC validation, duplicate handling, volume units, and coverage summaries into pure functions. The next article starts calculating factors on top of these clean K-lines.

WeChat

Follow ZiCode on WeChat

If this post was useful, you can follow later updates on WeChat as well.

X / Twitter

Follow @ax2_zicode

Faster technical notes, short thoughts, and new-post alerts are posted on X.

X Open X Profile @ax2_zicode