T15 · Backend for Crypto
If the data is already onchain, why do you still need a backend?
- Capabilities practiced
- Builder
- Hands on
- Add a cache and queue to an on-chain query and load-test the latency with and without them.
- AI Lab
- Have AI design an idempotency-key scheme. Construct duplicate requests to verify that it really is idempotent.
A Real-World Problem
A “My Assets” page has a simple requirement: list the user’s tokens and show each balance, current price, and 24-hour change.
Suppose a user holds 20 tokens. You count:
- 20 balance queries
- 20 decimals queries, because token precisions differ
- 20 name and symbol queries
- 20 prices from another data source
More than 80 network requests from the user’s browser.
The initial screen takes six seconds. Users think it is broken and refresh, sending 80 more requests. Three simultaneous refreshes trigger 429 errors on your RPC quota. Only three people are online.
Look closer: the results of many requests are the same for everyone. Names, symbols, and decimals rarely change; prices are shared across the site; and even a given address’s balance at a given block is deterministic.
You made each user recalculate answers that are the same for the world.
“Why have a backend?” is now a concrete question. The chain leaves work that the frontend does poorly and a backend needs to handle.
Thought Experiment
Think of RPC as an archive with one service window.
It has unusual properties:
- It is precise and answers only what you ask. It does not aggregate, sort, or tell you an address’s net inflows for the past month.
- It has no memory of you. Every query begins without a session or context.
- There is a shared queue. If one person asks too often, everyone waits.
- You pay by query. More questions cost more.
Now bring product requests to this archive:
“List balances of these 20 tokens, sorted by dollar value.” The archive does not sort; you retrieve all 20 and sort them. “Notify me as soon as this address receives money.” The archive never contacts you; you keep asking. “The user clicked withdraw three times; execute once.” The archive has no concept of your business operation. Send three transactions and it executes three. “Limit one person to ten queries a minute.” The archive does not know who a person is.
These are four backend jobs: aggregation, polling and notification, idempotency, and rate limiting. Add caching: when the same question has the same answer for a while, you need not ask again.
A backend handles what the chain does not do. Equally important, do not let it take over what the chain does do, particularly defining what is true.
Your Decision
You want to add one layer to that asset page. Which first?
What Happens
These layers build on one another. They solve two sides of the same problem:
| Backend capability | Why the chain does not provide it | Why the frontend struggles |
|---|---|---|
| Aggregation | The chain answers atomic questions | Dozens of requests and local calculations |
| Caching | The chain does not remember earlier questions | Browser caches are per user |
| Retries | The chain does not know your job failed | Closing the page ends client retries |
| Notifications | The chain does not contact you | Offline frontends cannot receive updates |
| Rate limits and quotas | The chain does not identify your users | Client restrictions can be bypassed |
| Key custody | Not a chain responsibility | A secret in frontend code is not secret |
| Idempotency | The chain knows nonces, not your business IDs | You cannot prevent every repeat click |
Draw a boundary immediately:
The chain is the source of truth; the backend is a projection and traffic-control layer.
The backend may cache, aggregate, process, and precompute. But every number it returns should be traceable to a chain state at some height. If your backend starts creating independent “facts,” such as a balance it updates itself, you have two sources of truth that eventually diverge. T16 turns this boundary into a database design.
Build a Model
The backend has a read path and a write path with very different failure modes:
- Read path: RPC and index database
- Cache
- Aggregation API
- Frontend
- Write path: request
- Idempotency check
- Queue
- Onchain worker
- Receipt confirmation
- Notification
Read Path: Three Cache Layers
Cache-key design matters more than the cache technology. Divide data into three groups:
| Category | Examples | Cache key | Strategy |
|---|---|---|---|
| Immutable | Token name, symbol, decimals; historical blocks; finalized receipts | No height | Cache long-term |
| Fixed at a height | Balance of an address at a block; contract state at a block | Include height | A new block naturally uses a new key |
| Constantly changing | Mempool and latest quotes | Short time window | Seconds-long TTL or no caching |
The first group is a nearly free win. Of the asset page’s 80 requests, about 40 ask for name, symbol, or decimals. Those values normally do not change; cache once and remove roughly half the requests.
The second group offers a useful technique: put block height in the key. A response for a fixed height remains correct forever, while a different height cannot accidentally use it. Hit rate falls as height advances, so this fits historical queries better than “latest.”
One thing many first versions omit is single flight. A hundred simultaneous requests arrive after a cache miss, so all hundred go upstream, defeating the cache exactly when it matters most. Allow only one upstream request per key while the others await its result.
Write Path: Three Layers of Idempotency
Repeat clicks, network retries, queue redelivery, and client retries make duplicate requests normal, not exceptional. Repeating an onchain operation can lose real money.
Idempotency must hold at three layers:
Layer one Request idempotency: one key creates only one job
Implementation: unique index; return existing result on insert conflict
Layer two Job idempotency: a redelivered job performs its side effect only once
Implementation: job state machine and conditional state transition, not read-then-write
Layer three Onchain idempotency: one business operation produces at most one onchain transaction
Implementation: bind the idempotency key to a nonce; each key always uses that nonceLayer three is often omitted and is the most expensive to miss. The backend sending once does not guarantee the chain receives one transaction. A worker may crash after broadcast, restart, build a new transaction with a new nonce, and land both.
Bind one idempotency key to one nonce and repeated attempts with that nonce compete to replace each other; at most one lands. This uses the chain’s own rule as a safety layer.
A table for request idempotency (SQL is stable enough to specify here):
create table submissions (
idempotency_key text primary key,
request_digest text not null, -- Hash of request body; detect same key with different content
status text not null, -- queued | broadcast | confirmed | failed
nonce bigint, -- Fixed once assigned
tx_hash text,
result jsonb,
created_at timestamptz not null default now(),
updated_at timestamptz not null default now()
);Use one statement for insertion, not “query and then insert”:
insert into submissions (idempotency_key, request_digest, status)
values ($1, $2, 'queued')
on conflict (idempotency_key) do nothing
returning idempotency_key;- One returned row: new request; enqueue it.
- Zero returned rows: duplicate request; fetch and return the existing result.
- Existing
request_digestdiffers: the same key was used for different content; return a conflict error. The client key generator may be broken; do not accept it silently.
“SELECT first; INSERT if missing” is wrong. Two concurrent requests can both observe “missing” and both insert. The unique index is the reliable lock.
Two Kinds of Rate Limit
Many systems implement only half:
- Downstream, for users: limit by user, address, and IP to prevent abuse. Return 429 with a retry time when exceeded.
- Upstream, for RPC quota: limit your own outgoing calls too. A traffic spike can consume a whole day’s allowance and leave the site unavailable.
Do not simply drop upstream requests. Queue them with a timeout; if they cannot get a token in time, fail quickly and return a cached older result where appropriate. A slightly stale answer is often better than an error page.
What It Is Called
An interface shaped for your frontend. It should not merely forward RPC calls; it should return all data needed by a page in the shape that page uses.
A practical test: how many requests does the frontend need to render one screen? More than three suggests this layer needs work.
Stores an answer already calculated. The special feature in Crypto is that an answer’s validity may depend on block height, not elapsed time.
A rule: cache settled facts, not unstable guesses. Historical blocks, finalized receipts, and token metadata are easy to cache. Latest balances need either a height or short TTL.
Separates accepting a request from performing it, enabling retries, smoothing spikes, priorities, and failure isolation.
Know two facts: most queues deliver at least once, and messages can arrive out of order. Consumers must be idempotent and cannot assume ordering.
A client-generated string identifying one business intent. Repeating it makes the server return the same result instead of executing again.
It must be generated by the client (a server-generated key changes on retry), unique within the business scope, and given an explicit lifetime.
For one key, permits only one simultaneous upstream fetch while other requests share its result.
It prevents a burst of misses from overwhelming the origin. Without it, a cache can fail worst at its moment of greatest need.
Holds messages that still fail after several retries for human investigation instead of retrying forever or silently discarding them.
Any queue dealing with funds needs one. Silently losing a transfer job can be worse than visibly reporting an error.
Hands-on
Use testnet for all transactions on the write path. If a private key is needed, use a temporary wallet with test tokens only.
The goal is your own measured comparison table, not another person’s benchmark.
Build a bare endpoint and measure the baseline.
Create GET /portfolio?address=... that makes dozens of RPC calls and aggregates the results.
Load-test with 50 concurrent users for 30 seconds. Record three numbers: p50 latency, p95 latency, and the total upstream RPC calls made.
The third maps directly to your bill.
Stop asking questions whose answers do not change.
Cache token names, symbols, and decimals as long-lived entries. Once obtained, reuse them.
Repeat the load test. Upstream calls should fall roughly in half, without changing business logic.
Add height-aware keys for mutable data.
Use a key shaped like chainId:method:params:blockNumber. Get the latest height once per request batch, then use it when querying the cache.
Repeat the test and compare p95. Watch how the hit rate changes when height advances.
Add single flight and prove it works.
Wrap upstream reads so only one concurrent call for a key goes through.
Clear the cache and send 100 identical requests at once. Count actual upstream calls. It must be 1. If it is 100, single flight is not working.
Make a write asynchronous.
Create POST /withdraw: validate arguments, insert a row in submissions, enqueue a job, then return 202 with a job ID immediately.
Add GET /withdraw/:id for status. The frontend can poll it, connecting to T14’s state machine.
Add an idempotency key, then attack with repeats.
Require the client to send the key in a header. Use the on conflict do nothing insertion above.
Attack one: submit the same key serially ten times. Exactly one job should exist; the next nine requests return its ID. Attack two: submit the same key concurrently ten times. Expect the same result. This breaks “SELECT then INSERT.” Attack three: reuse the key with a different body. Return a conflict rather than silently executing the first body.
After all three, count onchain transactions: there should be only one.
Kill the worker once.
After broadcast but before the receipt arrives, kill the worker process and restart it.
It should recover from submissions that “this job was broadcast with hash X” and resume waiting, not construct a new transaction.
If a second transaction lands, layer-three idempotency—the nonce binding—is missing. This is the lab’s most valuable test.
Produce the comparison table.
Three rows: no cache, cache, cache plus queue. Three columns: p50, p95, and upstream calls.
This table belongs in your stage project and gives you stronger evidence for architecture discussions because you measured it yourself.
AI Lab
Ask two sets of questions. The second is onchain, where models often stumble:
Step one:
Design idempotency for a backend endpoint that sends onchain transactions. Cover:
- Who generates the key, its format and lifetime
- Storage and correctness under concurrency
- What repeated requests return
- What happens when one key accompanies different request bodies
- Retry after failure and terminal status
Provide the exact schema and SQL.
Step two:
The worker crashes after broadcasting a transaction but before receiving a receipt.
What should it do after restart when it reads that record?
If it builds and sends a new transaction, what can happen onchain?
How do we ensure one business request produces at most one onchain transaction?The first answer often looks good, but watch for “SELECT to check first.” It works in a single-thread demo and fails under concurrent load.
The second separates ordinary web idempotency from onchain idempotency. Many explanations stop before nonce handling. If the model omits nonce binding, you have found a boundary in its answer: it knows the general concept but missed this domain’s extra layer.
Run the concurrent test yourself. Reading the proposed code is not enough to verify idempotency under load, which is why the first three checks above require execution.
Verify these points yourself after using AI
- Does it use “SELECT then INSERT” or a unique index and conflict handling? The former fails under concurrency
- You wrote a concurrent test proving only one of two simultaneous requests creates a job
- Does a repeated request return the first result, or execute business logic again?
- Does it reject the same key paired with a different request body? Many designs omit this
- Does it bind the key to a transaction nonce? One backend execution does not guarantee one onchain transaction
- Does retry logic distinguish retryable failures from terminal failures, including whether the key may be reused?
- You checked SQL syntax, locking behavior, and queue delivery guarantees against documentation for your actual versions
Real-World Case
An airdrop job enters a queue. A worker finishes it and the transaction lands, but the process restarts before acknowledging consumption.
The queue redelivers the message. A second worker sends another transaction, and one person receives two allocations.
The cause is treating “at least once” delivery as “exactly once.” Fix consumers through idempotency, not by switching queue products.
An API caches transaction status for ten minutes. A pending transaction is queried once, and “pending” goes into the cache.
It confirms twenty seconds later, but for the next ten minutes every query still reports pending. The user assumes it is stuck and sends another transaction.
Match TTL to how quickly the data can change. Do not cache an intermediate state, or give it a TTL shorter than its expected transition time.
A service without upstream rate limiting receives heavy crawler traffic for half an hour and consumes its daily RPC quota.
All real user queries fail afterward without warning. Quotas behave like cliffs, not gentle slopes.
Upstream limits preserve availability in bad conditions, not merely money. Keep a safety margin and alert near the threshold.
For performance, a team maintains user balances in its backend, adding and subtracting after operations without reconciling to the chain.
Two months later, hundreds of addresses disagree with onchain balances. Causes vary: a failed transaction treated as successful, an internal transfer missed, and a reorg not rolled back.
A backend may cache onchain facts but cannot invent its own version of them. Derived data must be recalculable from the chain and reconciled regularly. T16 makes this a database design rule.
Change One Variable
Read performance and cost improve. A display-focused product may stop here.
But writes still happen inside synchronous requests. Users watch a spinner while the server waits for receipts. Failure allows little beyond an error, and users retry by clicking again, creating duplicate submissions.
Conclusion: a read-only product may not need a queue, but once it sends transactions for users, queue and idempotency must come together.
Writes become more reliable, but read costs do not fall.
The queue itself may add many reads as workers fetch status, nonce, and receipts, increasing upstream pressure.
The two layers complement each other: cache reduces cost; queue improves reliability. Choose what to build first according to whether the current pain is the bill or incidents.
Everything held only in process memory stops being globally correct: you now have N divergent caches, an ineffective in-memory idempotency table, and single flight only within each instance.
This transition fails quietly, showing up as occasional duplicates that are hard to reproduce.
Keep idempotency state in shared storage; a database unique index is a strong anchor. Cache layers can differ: immutable data local, mutable shared.
Provider rate limits and quotas change, and latency may fall. Caching becomes less valuable but does not become useless, because aggregation still costs work.
Now you must monitor node synchronization. A node dozens of blocks behind may return stale data without any error. Check its height, switch providers or alert when it lags.
T4 covered the other side: running a node trades a quota problem for an operations problem.
Takeaway Questions
Who pays? You pay for chain queries, not your users. This explains much of the backend’s role: every duplicate upstream call removed lowers your marginal cost.
Who bears the risk? Without idempotency, users lose money to duplicate transactions. These incidents are rare, intermittent, and difficult to reproduce, often appearing only after a complaint. Verify idempotency with concurrent tests, not review alone.
What problem does it solve? The backend does not settle “what is true”; the chain does that. It handles experience, cost, and reliability. Confusing the two leads to a backend that claims authority over onchain facts.
Chapter Check
Seven things the chain does not do and the frontend handles poorly: aggregation, caching, retries, notifications, rate limiting, key custody, and idempotency.
They do not decide what is true. They deliver facts efficiently and reliably to users.
Keep the boundary clear: chain is the source of truth, backend the projection. Every number it gives should trace to a chain state at a block height.
Two concurrent requests can both find “missing” and both insert. There is a race between check and write.
Combine them into an atomic operation: a unique index plus conflict handling. Successful insertion means first request; conflict means duplicate.
Ask of any proposed idempotency design: does it still work concurrently? Test that under concurrency rather than trusting a reading of the code.
Backend idempotency governs how often your business logic runs. The chain uses the nonce to distinguish transactions.
A worker can broadcast and crash before receiving the receipt. On restart, it builds another transaction with a new nonce. Both are valid and can land, charging twice.
Bind one idempotency key to one nonce. Reconstructed attempts then share a nonce, so at most one can land.
Match it to the data:
- Immutable token decimals, historical blocks, and finalized receipts: no height; cache long-term.
- Fixed at a height balances and contract state: include height so a cached answer refers to the exact state requested.
- Continuously changing mempool and latest quotes: short TTL or no caching.
Height-aware keys turn over with each block and may have low hit rates for “latest.” Short TTL plus single flight can be more useful there.
There is no standard diagram. Check:
- Does every cache key on the read path include height when needed? When can it return stale data?
- Where can duplicates arise between request and chain, and how is each point handled?
- What are the queue’s delivery guarantees? Are consumers idempotent? Is there a dead-letter queue?
- Where is upstream rate limiting? If quota runs out, does the service return older data or an error?
- Has the backend calculated anything that cannot be derived again from the chain? That is where reconciliation will eventually fail.
Point 5 leads into T16.
Key idea
The backend handles what chains are bad at: aggregation, caching, retries, notifications, and rate limiting.