By 2015, internet companies could hardly live without data platforms.

When I wrote about Weibo Toutiao the previous year, I said a recommendation system was not one algorithm point, but a whole set of capabilities. By 2015 I felt that even more deeply. The more a business wants to iterate quickly, the more a data platform is both engine and burden.

It is heavy.

Collection, cleaning, storage, scheduling, queries, permissions, definitions, quality, lineage, and cost: every word hides pits. If one event field is buried wrong, a whole chain of analysis becomes wrong. If one job runs late, upstream product teams cannot see that day’s effect. If one metric definition is not unified, meetings turn into different teams talking past one another.

It also has to be light.

Business cannot wait. Recommendation strategies need validation, operations campaigns need review, product redesigns need measurement, and commercialization needs revenue calculation. If the platform only talks about perfect architecture, no one will wait half a year.

The First Principle Of A Data Platform: Trust

The scariest thing for a data platform is not being slow. It is being untrusted.

Slow can be explained and optimized. Untrusted data damages organizational judgment. Once people think data is inaccurate, they return to guessing, experience, and arguments by rank.

Trust comes from many details.

Event definitions must be clear. Client and server collection must align. Log formats cannot be changed casually. Job failures must alert. Reruns need rules. Deduplication and filtering must be explainable. Metric definitions must be traceable.

These jobs do not sound cool, but they matter a great deal.

When many companies build data platforms, they first chase rich tools: reports, dashboards, self-service analysis, real-time big screens, and visual drag-and-drop. Tools are useful, but if the underlying data cannot be trusted, the more tools you have, the faster you go wrong.

Heavy Platform, Light Decisions

A mature data platform must be heavy underneath.

It has to process huge logs, support offline and real-time jobs, manage resources, guarantee stability, control cost, and let different teams share the same infrastructure.

But the upper-layer experience must be light.

What business teams really want is simple: if a strategy changed today, can we see results tomorrow? Did retention change after a feature launch? Did one recommendation rule improve clicks but hurt long-term experience? Why did a certain user segment decline?

If every question requires waiting for an engineer to write a script, the platform’s value is discounted.

So the contradiction of a data platform is this: the more complex the bottom is, the simpler the top should feel.

This is true of much infrastructure. Cloud computing is the same: scheduling, networking, storage, and isolation are complicated below, but users should preferably see only APIs. If a data platform cannot hide complexity, it becomes a system only a few specialists can use.

Engineering Changes Behind Hadoop And Spark

Around 2015, the Hadoop ecosystem was already mature, and Spark was drawing more attention from more teams.

Behind the stack change, data-processing needs had changed.

Earlier work was more offline batch processing: collect logs, run jobs at night, read results the next day. Later, businesses wanted faster loops. Recommendation, advertising, risk control, and operations all wanted shorter feedback cycles. Batch processing remained important, but real-time and near-real-time capability became more valuable.

This was not simply swapping frameworks.

When the framework changes, task models, resource management, error handling, data consistency, monitoring, and development habits all change with it. Engineers should not chase new terms; they should judge how fast, accurate, and cheap the business actually needs the system to be.

Some metrics are fine once a day. Some must be minute-level. Some need second-level. Making everything real time is wasteful; leaving everything offline is sluggish.

Data Must Eventually Return To The Business

After working on data platforms for a long time, it is easy to get trapped inside the platform itself.

Cluster size, job count, throughput, latency, and availability are of course important. But a data platform ultimately has to answer business questions: why users come, why they leave, why content spreads, why a strategy works, and why growth stops.

If a data platform only serves engineers, it loses half its value.

In 2015 I increasingly felt that platform engineers must understand business. This does not mean everyone has to become a product manager. It means at least knowing who uses the data, what decision it supports, and what damage an error will cause.

The next year, AlphaGo would push AI into public attention. For many people, it was a Go match. For engineers, it also reminded us that data, computation, and models would push software systems into a new stage.

IT Events Of 2015

• Mobile internet entered mature competition. User growth continued, but competition moved from grabbing entry points toward retention, recommendation, commercialization, and operational efficiency. Data-driven work became everyday capability for internet companies.
• Kubernetes 1.0 was released. Container orchestration entered a production-usable stage. Microservices, elastic scheduling, automated deployment, and platform engineering gradually formed a clearer technical route.
• TensorFlow was open sourced. Google open sourced TensorFlow, accelerating the spread of deep-learning engineering and model-training toolchains. AI was no longer only in papers and labs; it began entering more engineering teams.
• Hadoop and Spark evolved in parallel. Hadoop continued supporting offline computation, while Spark and newer frameworks expanded their influence. Data platforms had to face batch processing, near-real-time processing, machine learning, and interactive analysis at the same time.
• Pinduoduo was founded. Pinduoduo entered through social commerce, low-price supply, and lower-tier markets, reopening space in an e-commerce landscape where Alibaba and JD were already strong. Mobile internet could still grow new platforms from specific scenes and populations.
• Data quality became core platform value. Recommendation, advertising, risk control, and operations analysis all relied on data platforms. Metric definitions, job stability, feedback speed, and data trust began deciding whether a platform was truly usable.

References

• Apache Spark: Spark Release 1.0.0
• Apache Hadoop: Apache Hadoop project
• Google Cloud Blog: Kubernetes V1 Released
• Google Research Blog: TensorFlow open sourced
• PDD Holdings: Investor Relations

WeChat

Follow ZiCode on WeChat

If this post was useful, you can follow later updates on WeChat as well.

X / Twitter

Follow @ax2_zicode

Faster technical notes, short thoughts, and new-post alerts are posted on X.

X Open X Profile @ax2_zicode