Effectively once: placing an order when every system answers twice

An investor taps Invest on a weak network. The tap becomes one purchase, and that purchase crosses four systems that share no transaction: the app, the payment gateway, the exchange order platform, and the registrar that allots the units. A message between two of them can arrive late, out of order, or more than once. Some arrive only because the sender tried again.

Two failures follow, and investors notice both: two orders for one tap, and money taken with no units allotted. This essay shows how a backend prevents both. It uses allot, an open-source order engine in Go that I built to test each idea (source on GitHub). Its chaos harness injects every fault named here on every run.

Why "exactly once" is the wrong promise

A network cannot tell a sender whether its message arrived. When a call times out, the sender does not know if the other side did the work. The only safe response is to try again, so every system in the chain retries, and every message can arrive more than once.

That makes duplicates normal, not rare. No network gives exactly-once delivery, so the goal is a duplicate with no extra effect. Delivery stays at least once; the effect happens once. This essay calls that effectively once.

Each technique below answers one source of duplicates: the app, the gateway, the exchange, and the registrar.

The order is a state machine, and the machine is pure

An order is a small state machine. It waits for payment, goes to the exchange, and ends allotted or refunded. Every outside message is an event, and the machine decides what each event means for the order in its current state.

In allot, that decision is one pure function. It takes an order and an event, and returns the next order with its side effects as data: the ledger postings to write and the messages to send. It performs no I/O.

// Apply returns the order after event e, or a *TransitionError that wraps ErrStale or ErrConflict.
func Apply(o Order, e Event) (Result, error)

// Result is an order after one event, with the side effects of the transition.
type Result struct {
	Order    Order
	Postings []ledger.Posting
	Messages []Message
}

Two kinds of refusal matter. A stale event is one the order has already moved past, such as a second copy of a payment that was already applied; it is safe to ignore. A conflicting event contradicts what the order knows, such as a second, different payment for one order; it needs a person. A system that treats the two as one either loses money or calls a person for a harmless retry.

A pure function is easy to test thoroughly. A table test applies ten events, the duplicates and contradictions included, to each of the nine states. A fuzz test drives one order through any sequence of events and checks, after every step, that the order made only legal moves and that its ledger matches its state.

Retries from the app: idempotency keys

The app sends each order with an Idempotency-Key header, a value it chooses once per purchase and reuses on every retry. Keys belong to one user, so two users can never collide. allot follows the IETF Idempotency-Key draft, an Internet-Draft that has expired without becoming an RFC but that describes the behavior clients expect:

• the same key and the same body get the stored response, byte for byte;
• the same key with a different body gets 422, because it is a different request;
• the same key while the first request still runs gets 409 and a Retry-After.

The key is claimed in its own short transaction, with a lease. The order and the stored response are then written in one transaction, so a crash can never leave an order without its response, or a response without its order. If the process dies while it holds the key, the lease runs out and the next retry takes the key over.

Retries from the gateway: the inbox

A payment gateway sends a webhook until it gets a 200. It sends copies after timeouts, and the copies arrive in any order. allot records each event in an inbox table keyed by the event's own ID, in the same transaction as the change the event causes.

INSERT INTO inbox_events (source, event_id, order_id, kind, payload)
VALUES ($1, $2, $3, $4, $5)
ON CONFLICT (source, event_id) DO NOTHING;

If the insert writes no row, the event was already processed, and the handler answers 200 without touching the order. The gateway stops retrying either way.

Hover a part to see its name

The inbox stops copies. The state machine stops the rest. A gateway can also send a failure for a payment it already reported as successful. That event has a new ID, so the inbox lets it through, and the state machine rejects it as stale: the order already moved past payment.

Calling the exchange: the transactional outbox

The order must reach the exchange after the payment is recorded, and the two cannot share a transaction. Calling the exchange inside the database transaction holds row locks for as long as the exchange takes, and loses the answer if the process dies after the call but before the commit.

allot writes the call as a row in an outbox table, in the same transaction as the payment. A relay claims due rows with a lease, commits the claim, makes the call, and then marks the row sent in the same transaction as the event the answer produces.

UPDATE outbox
SET lease_until = now() + make_interval(secs => $2), attempts = attempts + 1
WHERE id IN (
  SELECT id FROM outbox
  WHERE sent_at IS NULL AND available_at <= now()
    AND (lease_until IS NULL OR lease_until < now())
  ORDER BY available_at, id
  LIMIT $1
  FOR UPDATE SKIP LOCKED)
RETURNING id, order_id::text, kind, attempts;

SKIP LOCKED lets several relays share the table without waiting on each other. If a relay dies between the claim and the call, its lease runs out and another relay sends the message. The cost of this design is that a message can be sent twice, which is safe only if the exchange can recognize the second send.

A timeout is a state, not a failure

Every call to the exchange names the order by one reference that never changes. When the exchange does not answer in time, allot does not know whether the order exists there. It records exactly that: the order moves to submit_uncertain, and the relay asks the exchange for the order's status under the same reference. If the exchange has never seen the order, the relay sends it again under that reference, and the exchange records it once.

Hover a part to see its name

There are two tempting shortcuts, and both are wrong:

• Treat the timeout as a rejection, and refund. The exchange may have accepted the order, so the investor gets the money back and the units too.
• Resend under a new reference. That is how one tap becomes two orders.

The chaos harness catches both. The repository plants each shortcut in the relay behind a build tag, so anyone can run it. In one run of 150 orders, a new reference on every attempt produced 132 exchange orders that allot never made, and 100 accepted orders whose reference the exchange did not know. A timeout treated as a rejection produced 17 refunds for orders the exchange had accepted. The counts move a little between runs, because the seed fixes the faults but not the timing; every run of either shortcut fails. With the code as written, none of these appears.

Evidence moves an order forward

The registrar sends a file of allotted units. A row in that file is evidence: units for an order prove the exchange accepted it, even if the acceptance message never arrived. allot moves the order to allotted from any state that could have led there, so the order of arrival never decides the outcome.

Evidence is checked before it is believed. For each row, allot recomputes the units as amount over the net asset value (NAV), the price of one unit, and the NAV date from the cut-off rules. A row that disagrees opens a finding and is not applied. After the file, any paid order with no row opens a finding too. A person resolves findings; the code does not guess.

When the money arrives decides the NAV

Which day's NAV a purchase gets depends on when both the order and the money reach the fund, compared with a cut-off in India Standard Time.

The money matters as much as the tap. In a scheme with a 3:00 PM cut-off, an order placed at 2:00 PM whose payment reaches the fund at 3:05 PM gets the next business day's NAV. allot keeps these rules as data, one per category, with the date each took effect and its source, so a rule change is a data change, not a constant in code. Two simplifications are stated plainly in the repository: the time the gateway reports the payment stands in for the time the money reaches the fund, and rules are not yet chosen by date.

What the chaos run shows

The harness places orders with racing, retried, and reused keys. It pays them through a gateway that duplicates, delays, reorders, and sends stale failures. It submits them to an exchange that rejects, drops, and goes silent after accepting. It stops relays mid-flight, abandoning the messages they hold, and expires unpaid orders while late payments are still on their way. Then it checks six guarantees against the database, the exchange, and the gateway.

$ go run ./cmd/chaos -runs 50 -orders 200 -seed 1
50 runs, 10000 orders, 5m34s
requests by status    201=20280  409=9662  422=504
final states          accepted=394  allotted=7609  expired=471  payment_failed=417  refunded=1109
findings              missing_allotment=394  unit_mismatch=159
webhooks delivered    28893 (duplicates included)
exchange calls        9410 for 8736 exchange orders
refunds               1109
guarantee violations  0 (runs failed: 0)

Every order ended in a final state or with a finding for a person. The 394 orders left in accepted are those the registrar file did not allot, and each has a finding. "Exchange calls" counts submissions, retries included: 674 more than the orders they created. The fake exchange keeps one order per reference, so the proof is the check that found no exchange order under a reference allot did not create. These numbers come from one laptop with PostgreSQL in Docker; they show correctness under faults, not throughput.

The same discipline in inference serving

Nothing here is specific to mutual funds. Any service that calls a system it does not control meets the same three questions: what a retry means, what a timeout means, and how two records of the same fact are reconciled. A client that retries a generation request after a timeout needs an idempotency key, or it pays for the tokens twice. A usage meter that bills from events needs an inbox, or it counts one request twice. A prefill worker that hands a KV cache to a decode worker over the network faces the timeout question when the handoff stalls. The answer in each case is the same: name the work once, record what is known, and treat "I do not know" as a state.

Limits

• The exchange, gateway, and registrar in allot are fakes, modeled on public descriptions of these systems, not on private APIs or file formats.
• allot covers purchases only. Redemptions follow different rules.
• Nothing here describes how any company builds its systems.

References

• allot, and its source code with the design decisions and the alternatives rejected
• The Idempotency-Key HTTP Header Field, IETF Internet-Draft 07, expired April 2026
• RFC 9457: Problem Details for HTTP APIs
• PostgreSQL: the locking clause, including SKIP LOCKED
• SEBI Master Circular for Mutual Funds, on uniform cut-off timings for the applicability of NAV