T18 · Realtime System

A Real-World Problem

Your indexer is now reliable (the result of T17). Next you need to push changes to users. You add a persistent connection and send events whenever the indexer finishes a block. Locally, delivery takes seconds. It feels great.

Two weeks after launch, support tickets start describing the same hard-to-reproduce problem:

• “Several entries are missing from my activity list, but they reappear when I refresh.”
• “One transaction still says pending even though it succeeded long ago.”
• “The balance in the interface does not match what the block explorer shows.”

You check the server logs. They are clean: every message was sent. The send function succeeded, with no exceptions or queue alerts.

Then you notice what these users have in common: they were on mobile networks and briefly disconnected.

The problem becomes clear. Your push system is “send and forget”: whoever is connected at that moment receives the message; whoever is not connected loses it forever. The message was lost in your design, not in the network, because you never planned for a user being absent.

The real question is: when a user returns after thirty seconds offline, where do you get the missed events to fill the gap?

Thought Experiment

Consider three ways to send messages.

First: a radio broadcast. You speak in the studio; everyone whose radio is on hears you. If you are away, that segment is gone, and nobody knows you missed it.

Here, the server remembers nothing about its listeners, so the cost is very low. The price is that delivery depends entirely on luck.

Second: a numbered notice board. Each message goes on a wall with a rising number. When you return, you tell the caretaker, “I saw number 812,” and they give you everything after 812.

A disconnection changes from an irrecoverable accident into a gap you can catch up on. Someone must maintain the wall, however, and its capacity is finite: old messages will eventually be erased.

Third: announce only that something changed and let people check for themselves. You send a signal, not the content. Anyone who hears it fetches the latest state.

This has an underappreciated advantage: losing a signal barely matters, because the next one refreshes the entire state. It is naturally idempotent and resilient to loss. It cannot answer “what happened in between”; it can only show “what is true now.”

All three point to the same criterion:

A real-time push system must either support replay or converge on the current state. Without either, it is gambling on the network.

The first method can neither replay nor converge, so it will inevitably lose data. The three support tickets are three symptoms of that one flaw.

Your Decision

Choose an approach for “update the interface whenever something changes onchain.”

What Happens

Rank the four approaches on different criteria and a distinction many people miss becomes visible:

The first and second columns are independent. This is the main point to take away:

“Real time” measures latency. “No loss” measures correctness. Entirely different mechanisms provide those guarantees.

Most “real-time push” systems deliver only the first. They look good at launch because developers and testers use stable networks. Users encounter the failures as occasional missing items, the hardest sort of complaint to investigate.

There is another conclusion that connects this chapter to the previous one:

Events near the chain head can be revoked. Events already pushed to a user do not retract themselves.

A push stream therefore needs a “revoke” message, or it should send only events deep enough in the chain. That is the delivery condition for the outbox in T17. “Do not lose it” and “do not report a false event” are separate requirements. You need both.

Build a Model

Three ingredients

A push system that does not lose events needs just three things:

Where should the cursor come from? Three candidates, two wrong

Timestamp: wrong. It is not strictly monotonic (several events can share a millisecond), clocks can be adjusted, and different instances drift. A timestamp cursor causes duplicates or omissions during catch-up.

Database auto-increment sequence: usually wrong too. The reason is subtle: numbers are allocated at insertion, but transactions do not commit in that order. Transaction A receives 102 and transaction B receives 103; B commits first. A reader sees 103 and moves its cursor to 103. A then commits, and 102 is never read by anyone.

The gap causes no error; some events simply vanish. To use such a sequence, you need either to read only sufficiently old records or to allocate serially through a single writer.

Onchain coordinates: right. The tuple (chain_id, block_number, log_index) is a strictly increasing, globally unique cursor consistent across instances and still verifiable after a restart. Clients can also compare it directly against the chain.

Encode the three parts as a comparable string or a large integer:

cursor = chainId : blockNumber : logIndex
    e.g.  1 : 00000000020134 : 0007

Message protocol: five messages, all necessary

Define the protocol before choosing a library:

client → server    subscribe  { topics, cursor }   Subscribe from the position I have seen
server → client    snapshot   { cursor, state }    Optional current-state snapshot
server → client    event      { cursor, type, payload }
server → client    revoke     { fromCursor }       Events after this position are invalid
server → client    reset      { reason }           Your cursor is too old; subscribe again
both directions   ping / pong                      Heartbeat

The last three are the ones beginners most often omit:

reset is essential. The replay window is finite. A client returning after an hour may have a cursor that has already been purged. Without reset, the server silently replays nothing and the client thinks it is up to date. That is worse than an obvious disconnection because everything appears normal.

revoke is the exit for reorgs. If you push unconfirmed events, you need a way to tell clients that earlier events were invalidated. If you do not want to implement it, push only deep enough events and let confirmation depth handle the risk.

Heartbeats must go both ways. A heartbeat only from the client lets the server detect a dead peer; one only from the server lets the client do so. Both sides need timeouts, or a half-open connection can leave one side believing the other is still present.

The seam between snapshots and deltas: the easiest place to make a mistake

“Fetch the current state, then subscribe to later changes” sounds obvious, but it is wrong:

Wrong order:
  t0  Query snapshot (state through cursor=100)
  t1  ← Event cursor=101 happens here, while nobody is listening
  t2  Establish subscription; receive 102 and later

  Result: 101 is lost forever, with no visible warning

The right order is subscribe, buffer, fetch the snapshot, then release the buffer:

1. Connect and subscribe; the server starts enqueuing events for this connection
2. Do not deliver them yet; buffer them on the client or server
3. Fetch a snapshot and its cursor
4. Release buffered events whose cursor is greater than the snapshot cursor, in order; discard older ones
5. Switch to normal delivery

Reversing this order is the most common cause of “a few items disappeared after reconnecting.” It is almost impossible to reproduce locally: an event must arrive at exactly that instant.

Backpressure: degrade, never wait

A client reads more slowly than you send (weak network, background tab, or poor client code). There are three responses:

The third is the only sound answer. Remember it this way:

The way out of backpressure is degradation, not waiting.

Give each connection a bounded queue, perhaps a thousand events. When full, discard all queued events for that connection and send a reset, sending it through the subscribe, snapshot, catch-up sequence again.

Do not merely discard a few oldest events. That creates a gap the client cannot detect. Discard the whole queue and explicitly say, “Start over.”

Reconnecting: jitter the backoff

A client that reconnects immediately after each failure can overwhelm a recovering server. Use exponential backoff:

Wait for retry n = min(base interval × 2^n, cap) × random factor
Random factor is between 0.5 and 1.5

The random factor is not optional. Without it, a server restart brings every client back in the same second. Another restart does the same, creating a periodic attack on your own service.

Delivery semantics: do not chase exactly once on the server

Exactly-once delivery is not achievable at the network layer. A workable combination is:

Server: at-least-once delivery (catch-up may resend events)
Client: deduplicate by cursor (discard anything at or below the last seen cursor)
Together: effectively exactly once

This is the idempotency of T15, the primary-key idempotency of T16, and the checkpoint idempotency of T17, appearing for the fourth time as the same pattern: do not require an operation to happen only once; make repeated execution have the same result.

Fanout across instances

Give every connection a bounded outgoing queue and maintain an index from topic to connections. Look up subscribers when an event arrives. That is straightforward on one instance.

With multiple instances, it is tempting to put events in a message broker and have each instance subscribe. That can work, but preserve the T16 rule: one source of truth. The broker transports events; it is not the source of facts. Catch-up must query the database by cursor, not ask the broker for history.

The reason is simple: a broker's retention policy, partition order, and consumer positions form another set of state that can fail. Let the broker provide speed and the database provide correctness.

What It Is Called

Hands-on

After a 30-second disconnection and reconnection, the event sequence received
must exactly match the sequence seen by a continuously connected control client
(count, order, and content).

select * from outbox
 where chain_id = $1
   and (block_number, log_index) > ($2, $3)   -- Cursor supplied by the client
   and block_number <= $safeHeight
 order by block_number, log_index
 limit 1000;

- Same count?
- Same order?
- Same content, event by event?
- Did the reconnecting client receive duplicates? If so, did deduplication work?

- 500 ms latency
- 10% packet loss
- Very low bandwidth, forcing backpressure

AI Lab

Step one:
Design a subscription protocol for pushing onchain events without loss after a disconnect.
List every message type and field, plus the client and server state machines.

Step two:
For each case, identify which part of your protocol handles it:
1. A client reconnects after 30 seconds; 12 events occurred meanwhile.
2. A client reconnects after 3 hours; the server's replay window is only 1 hour.
3. One client reads at one-tenth the push rate for 10 minutes.
4. The server restarts, disconnecting 50,000 clients at once.
5. An event happens exactly between "fetch snapshot" and "establish subscription."
6. Three events already pushed to a client are invalidated by a chain reorg.

Step three:
Which of those six cases can your protocol not yet handle?
For each, describe the smallest fix and its cost.

Real-World Case

Change One Variable

Takeaway Questions

Chapter Check