1999: The First Socket Into Production
· 44 min read · Views --

1999: The First Socket Into Production

Author: Alex Xiang

Essays

In 1998, I wrote MIS systems: forms, fields, databases, and reports.

In 1999, those forms began connecting to harder things.

That year, I worked on bank systems, used SCO UNIX as a front-end machine, and truly used socket programming in production. This sentence sounds ordinary now, even a little old-fashioned. But for me at the time, it was the year when my sense of software responsibility changed.

Before that, I mostly wrote programs on my own machine, in internal departmental systems, or in environments that were relatively controllable. If something was wrong, it could be fixed. If it was slow, it could be tuned. Restarting was not necessarily a big deal. Banking systems were different. They connected to transactions, counters, users’ money, and an order of production that could not be stopped casually, could not be wrong casually, and could not go online merely because “it should be fine.”

A socket was no longer only an API in a book.

It became a real line.

On one end of the line was the program you wrote. On the other end was a larger system.

A Period Word: Front-end Machine

The Chinese term “front-end machine” now sounds dated.

But in the late 1990s and early 2000s, it was a crucial layer in many industry systems. Banks, securities firms, telecom systems, tax systems, and social-security systems did not allow peripheral business systems to connect directly to core systems. There was often a front-end layer in the middle responsible for protocol conversion, connection management, message sending and receiving, logging, permission isolation, and exception handling.

It was like a gatekeeper and a translator.

A counter system or business system sent a request. The front-end machine packed the message in the agreed format and sent it through a socket to the backend host or core system. The response was parsed, converted, recorded, and returned to the front end. Today we use terms such as API gateway, message queue, service mesh, load balancer, and observability platform for similar concerns. Back then, many systems were more direct: UNIX processes, TCP connections, fixed-format messages, and log files.

The value of a front-end machine is that it stands on an unstable boundary.

External systems may be slow. Networks may break. Core systems may return errors. Messages may be invalid. Counter staff may repeat an operation. The front-end machine cannot merely “forward.” It needs to know when to wait, when to time out, when to retry, when to reject, when to write a problem into logs, and when human intervention is required.

These choices are easy to understand in a textbook.

In production, every choice becomes concrete.

If the timeout is too short, normal transactions may fail. If it is too long, front-end users keep waiting. If logs are too sparse, problems cannot be investigated. If logs are too verbose, disk and performance are affected. If error codes are unclear, business staff cannot understand them. If error handling is casual, problems get swallowed.

This is the first lesson production systems give programmers: code must not only run. It must also be explainable, traceable, and able to withstand reality.

The Reality of SCO UNIX

The system I used then was SCO UNIX.

Today, when many people hear UNIX, they think of a macOS terminal, Linux servers, cloud hosts, or containers. SCO UNIX belonged to another era. It often appeared in industry systems, small servers, dedicated devices, and more traditional business environments. It was not flashy or friendly, but it was stable, direct, and carried the smell of “this is production.”

That environment makes people careful.

You do not casually move things on such a machine. Directory layout, startup scripts, log locations, configuration files, runtime users, and occupied ports all need to be checked clearly. Debugging was not comfortable either. There was no mature observability platform, no modern IDE remote debugging, no disposable container environment. Very often it was terminals, logs, commands, experience, and judgment about system behavior.

This is good training for programmers.

It forces you to respect the environment.

On your own development machine, it is easy to feel that the world revolves around code. On a production UNIX machine, code is only one process in a system. It shares resources with other processes, follows operational rules, leaves logs, handles signals, can be started, stopped, and restarted, and tries not to create a larger mess when exceptions occur.

This was very different from writing MIS systems in 1998.

MIS taught me business data and user flows. Bank front-end systems taught me production links and system boundaries.

A Socket Is Not Done When It Connects

When people first learn sockets, the exciting part is usually a successful connection.

Create a socket. Connect. Send. Receive. On the server side, accept, read, write. Once both sides can communicate, it feels as if the work is done.

After writing it for production, you learn that a successful connection is only the beginning.

Connections break. The other side becomes slow. Messages arrive partially. A single recv does not mean you have received a complete message. On the network, there is no such thing as “I thought it should arrive all at once.” You need to define message boundaries, handle length, handle encoding, handle timeouts, handle the peer closing unexpectedly, and handle how state recovers after your own process restarts.

A bank front-end machine and socket link around 1999

A socket request-response link among bank counter terminals, a front-end machine, and a core system. This is a period-inspired illustration.

Then there is the harder part: business semantics.

Sending a message technically does not mean the business succeeded. Receiving a response does not mean you can relax. If the connection times out, did the transaction reach the other side? Did the other side process it? If the user retries, will money be deducted twice? If the log contains only half of the information, how do you reconcile later?

These questions turn socket programming from “network API” into “business protocol.”

Many years later, internet systems would repeatedly talk about idempotency, retries, timeouts, distributed transactions, eventual consistency, and tracing. Financial systems had forced programmers to face these problems much earlier, only without today’s popular terminology or mature tooling.

The biggest change from writing front-end systems was that you began treating failure as normal.

Not “what if it fails,” but “it will fail; how should the system behave when it fails?”

That is the difference between engineering and writing small programs.

Logs Are the Memory of Production Systems

In production, logs are not a few temporary print statements for debugging.

Logs are the system’s memory.

When a request came in, where it came from, what the message summary was, where it was sent, when the other side responded, what result code came back, how long it took, why it failed, whether it was retried, and what final state it reached. In normal times, this information feels verbose. When something goes wrong, it is the lifeline.

But logs cannot be written carelessly.

If logs are too coarse, problems cannot be found. If logs are too detailed, performance suffers and key information is buried in noise. If the format is inconsistent, later analysis is difficult. If timestamps are inaccurate, different machines cannot be correlated. If key fields are missing, the final investigation becomes guesswork.

In 1999, logs were usually plain.

Text files, daily rotation, and command-line searching when necessary. Many programmers truly learned grep, tail, awk, and sed not to show off, but because they had to find what happened to one transaction quickly inside a pile of files.

Later, when I worked on internet systems, log platforms became much stronger, and tracing, metrics, alerting, and observability all emerged. But the essence did not change: a system must leave enough evidence to prove what it did.

A system without evidence is not suitable for production.

An Internal Forum Written in Perl

Around 1999, I also built an internal forum for the company using Perl.

Compared with the bank front-end system, this was a much lighter thing.

One was a production transaction system. The other was an internal communication tool. One had to face connections, protocols, and responsibility seriously. The other felt more like a small tool programmers built for themselves. But together, they represented two directions of software in those years.

On one side was industry informatization: banks, telecom, enterprises, governments, schools, and various businesses entering systems as data and processes.

On the other side were the Web and communities: forums, message boards, personal homepages, mailing lists, and chat rooms. For the first time, they made people feel that software was not only for processing business. It could also organize communication among people.

Perl was representative at the time.

It was not as close to the system layer as C, nor as suited for Windows window applications as Delphi. Perl was more like glue. It handled text, generated pages, operated files, wrote CGI, and stitched small tools together. It was flexible and carried the flavor of scripting languages.

The internal forum may not have been a large project, but it made me realize something else: the value of software does not always come from complexity. Sometimes it comes from being placed exactly where people need to talk.

If people are willing to use it, it lives.

The Air of 1999

There was another special atmosphere in 1999: Y2K.

Today the Year 2000 problem can sound like a historical joke. It was not. Many systems used two digits for years, and 00 could be interpreted as 1900. Finance, electricity, transportation, government, and enterprise systems all needed checks. It made the world realize, perhaps for the first time at such scale, that a field design inside software could become a social problem decades later.

For programmers, this was a very educational lesson.

Technical debt is not abstract. A design made to save space, save effort, or fit the habits of the time may come back at extremely high cost many years later.

The software industry in 1999 had both this tension and new excitement.

Linux 2.2 was released. GNOME 1.0 was released. Open-source desktop and server worlds kept growing. Napster appeared and pushed peer-to-peer sharing, copyright, and digital distribution into public debate. Salesforce was founded, and the seed of SaaS began to sprout. Alibaba was founded, and China’s internet commerce entered a new stage. OICQ, the predecessor of QQ, launched, and instant messaging would gradually become part of everyday Chinese internet life.

These big events and the front-end machine I was writing seemed to belong to different worlds.

But together they showed that software was entering more real scenarios.

It entered banks, enterprises, personal communication, music distribution, commerce, servers, desktops, and every place that needed connection.

In 1998, I learned that behind forms were organizations.

In 1999, I learned that behind connections was responsibility.

Both lessons affected me deeply. Forms show business rules. Connections show system boundaries. The former tells you how data enters a system. The latter tells you how one system relates to another.

Many years later, internet backend engineering looked completely different in its technology stack: HTTP, RPC, message queues, caches, databases, microservices, containers, Kubernetes, and observability. But the core questions did not change.

Where does the request come from?

Where does it go?

What happens on timeout?

Can failure be retried?

Will duplicate requests break things?

Can logs explain what happened?

How are responsibilities divided among upstream and downstream systems?

Sockets in 1999 had already asked these questions once.

There were not as many polished concepts then, but production systems have never lacked real problems.

Next: Beijing

After 1999 came 2000.

I came to Beijing. CSDN could not offer the salary I expected, and I eventually joined a foreign company. The internet bubble had inflated high and would soon burst. An editor from Sina said the compensation he received could cover an entire bed. That was another smell of the era: portal websites, startups, foreign-company offices, NASDAQ, stock options, layoffs, and the argument over whether the internet was a bubble.

In 1999, I was still writing sockets in production systems.

In 2000, the network kept rushing forward, and people moved along with cities and industries.

The next essay is about the bubble bursting while the net stayed.

IT Milestones of 1999

  • Y2K entered its final countdown. In 1999, systems around the world were doing final checks and fixes for the Year 2000 problem. It made the software industry collectively realize that dates, field lengths, and historical compatibility can become systemic risks.
  • Linux 2.2 was released. Linux kernel 2.2 improved networking, SMP, and other capabilities. Linux was moving from hobbyist systems and server choices into more serious production scenarios, laying groundwork for later internet infrastructure and cloud computing.
  • GNOME 1.0 was released. GNOME 1.0 appeared in 1999. Open source was not only competing on servers; it was also trying to provide a complete desktop experience. Together with KDE and other projects, it became an important part of the Linux desktop ecosystem.
  • Napster appeared. Napster launched in 1999 and pushed P2P file sharing into the mainstream. It changed music distribution and triggered debates over copyright, platforms, and digital-content business models.
  • Salesforce was founded. Salesforce was founded in 1999 and later became a representative SaaS company. It helped move enterprise software from local installation and project delivery toward subscription cloud services.
  • Alibaba was founded. Alibaba was founded in 1999 and later deeply influenced Chinese e-commerce, payments, cloud computing, and small-business digitization.
  • OICQ launched and later became QQ. Tencent launched OICQ in 1999, later renamed QQ. Instant messaging became one of the most important entrances to the Chinese internet.
  • Early VMware Workstation products appeared. VMware released early virtualization products in 1999. Virtualization later influenced development and testing, server consolidation, cloud computing, and local experimental environments.

References

More in this column

View all posts in "Thirty Years in IT and Me"