Engine Room

Reliability: failure, retry and scale

Reliability on Solana is mostly a question of what you do with uncertainty. A submission that returned an error may still have landed. A confirmation that never arrived may only mean your subscription dropped. An engine that treats silence as failure will happily send the same value twice.

The notes here cover the four behaviours that decide whether an engine is safe to leave running: resending without rebuilding, absorbing rate limits without a stampede, measuring a run in terms that survive scrutiny, and growing load in stages with an abort condition written down in advance.

Safe retry
Resend identical bytes, never rebuild
Control signal
Queue depth, not error count
Core metric
Cost per successful swap
Ramp rule
One variable per stage, abort criteria first

4 notes in this section

Retries that do not double-spend, backpressure that protects an endpoint instead of hammering it, metrics that expose the truth, and a growth path that does not break on the way up.

01

Retry, backoff and idempotency

Why a resend on Solana is safe and a rebuild is not, what the blockhash actually guarantees, and how to make the retry loop terminate on evidence rather than on hope.

Open the note
02

Rate limits and backpressure

Credit accounting, concurrency caps, queue depth as the control signal, and how to shed load deliberately instead of letting an endpoint decide for you.

Open the note
03

Observability for an engine

The metric set that describes a run honestly: landing rate, time to confirmation, cost per successful swap, and the traces that connect a number back to a signature.

Open the note
04

Scaling a run safely

Load testing against the constraint you will actually hit, staged ramps with defined abort criteria, and the four ceilings that stop an engine before compute does.

Open the note

The failure this section is written against

Almost every serious incident in an automated trading system is the same shape: an ambiguous result was resolved optimistically, the engine acted on that assumption, and the correction arrived after the value had moved. Idempotency, backpressure and honest metrics are three angles on the same defence, which is refusing to let the engine decide what happened without evidence from the chain.