Scaling a Run Safely: Ceilings, Ramps and Abort Criteria

Scaling an engine safely means identifying which ceiling you will meet first, raising one variable at a time toward it, and writing the abort criteria before the run rather than during it. The four ceilings are the endpoint request budget, the fleet, the capital tied up in accounts, and the operator's ability to resolve what the engine cannot.

Compute is almost never the constraint, and neither is send speed. Both are the things people tune, which is why so many engines are optimised in the wrong dimension and then fall over at the request layer during the first genuinely busy afternoon.

Four ceilings, and which one you meet first

The fourth ceiling is the one people never plan for and the one that ends most ambitious runs. Every unknown outcome requires a human decision, and the rate of unknowns grows with the size of the run and with congestion. A run large enough to produce a dozen unresolved items an hour is a run that requires a person, and if that person is asleep the engine is accumulating quarantined accounts and undetermined value.

Work out which ceiling binds before scaling. It is a paper exercise: request volume from the capacity arithmetic, fleet size from release rate and observed p90 settle time, capital from account count and floors, and operator load from your historical unknown rate. Whichever is closest is the one your ramp is really testing.

Load testing the constraint that binds

A load test is only informative if it stresses the thing that will actually fail. Testing an engine by sending a burst of transactions on a quiet afternoon tests almost nothing, because the quiet afternoon is not the scenario. Three tests are worth running and they target different ceilings.

The request budget test does not need to trade at all. Reproduce the request profile at your target size, including confirmation polling at the ageing intervals you intend, and run it against your endpoint until you either reach your target or start being refused. This is cheap, safe, and it answers the question that binds most designs.

The fleet test is about lease behaviour rather than throughput. Force artificially long settle times, either by testing against a slow path or by holding leases deliberately, and confirm that the scheduler holds rather than starving, that quarantine works, and that the run recovers when settle times return to normal. What you are testing is the shape of the degradation, not the peak.

The recovery test is the one people skip. Kill the process mid-run and confirm that the in-flight set can be reconstructed from persisted state, that stored bytes are resent rather than rebuilt, and that nothing is double-sent. If that test has never been run, crash recovery is a design intention rather than a property.

The staged ramp

A ramp is a sequence of stages, each raising exactly one variable, each held long enough to be informative, each ending in an explicit promote or revert decision. The discipline is what makes the result attributable.

1. Establish a baseline. Run at a size you already trust. Record landing rate per intent, attempts per landed transaction, median and p90 settle time, and cost per successful swap. Record the ranges, not single values.
2. Write the abort criteria. Before anything changes, define the values at which the run stops. Include the observation window for each, and name who may override them, which should be a short list.
3. Raise one variable. Fleet size, release rate, swap size or venue count. One. Everything else stays at baseline.
4. Hold and observe. Run long enough to include a congested period. A stage that only saw quiet conditions has been rehearsed rather than tested, and the ceiling you are looking for only appears under load.
5. Promote or revert. If every metric stayed inside its baseline range, record the new baseline and move to the next variable. If any abort criterion fired, revert and investigate before retrying. A stage that neither passed cleanly nor aborted is a failed stage.

Doubling is a reasonable step size for the first few stages and becomes reckless later, because the ceilings are not linear. Going from four accounts to eight changes nothing structural; going from forty to eighty changes reconciliation from something you can eyeball to something that has to be automated. Step sizes should shrink as you approach a ceiling you have identified on paper.

Abort criteria are written before the run

An abort criterion written during an incident is not a criterion, it is a negotiation. Write them in advance, make them measurable, and give each one an observation window so that a single bad sample does not stop a healthy run.

• Landing rate per intent below a stated value for a stated number of consecutive minutes.
• Cost per successful swap above a stated ceiling, sustained rather than momentary.
• More than a stated number of unknown outcomes unresolved at any one time.
• Any unexplained negative balance delta on an executor, with no observation window at all: this one aborts immediately.
• Circuit breaker open for longer than a stated period, or failover to the last remaining endpoint.
• p90 settle time above a multiple of baseline for a sustained window.

Notice that one criterion has no observation window. An unexplained outflow is either duplicate execution or a compromised key, and in both cases every additional transaction makes the situation worse. Everything else can tolerate a few minutes of observation; that one cannot, and the asymmetry should be encoded rather than left to judgement.

Abort means stop releasing, keep confirming, and report what is outstanding. It does not mean kill the process, which would convert every in-flight transaction into an unknown and create precisely the problem the abort was meant to prevent.

One variable per stage

The temptation to move two settings at once is strongest when a stage passes easily, and it is exactly then that the discipline is worth keeping. Attribution is the entire product of a staged ramp. Two variables moving together give you one data point about a combination, which tells you nothing about either.

The four variables interact in ways that make joint changes especially misleading. Raising the release rate increases both request volume and fleet pressure. Raising fleet size relieves fleet pressure and increases capital and reconciliation load. Raising swap size changes nothing about request volume but changes the value at risk per unknown. Adding a venue changes route shape, which changes transaction size and compute consumption, which changes cost.

Because of those interactions, the useful order is usually fleet first, then release rate, then venue count, then swap size. Fleet is the cheapest to reverse and it relieves pressure that would otherwise contaminate later stages. Swap size goes last because it changes the consequence of a failure rather than its likelihood, and you want the failure behaviour understood before you raise the stakes.

What changes qualitatively as you grow

Scaling is not smooth. Certain sizes change the character of the system, and knowing where they are prevents a stage from failing for reasons that have nothing to do with the setting you raised.

Reconciliation changes first. A handful of accounts can be checked by inspection; a few dozen cannot, and past that point the reconciliation job stops being a safety net and becomes load-bearing infrastructure that itself needs monitoring. The same applies to funding: refilling a few accounts is manual, refilling many has to be batched or the transfers become a meaningful share of your transactions.

Confirmation strategy changes next. Polling per signature is fine at low volume and becomes the dominant request consumer as concurrency grows, at which point batching or a subscription-based approach stops being an optimisation and becomes a requirement. That change is a redesign, not a setting, and it is better done between stages than during one.

Operator load changes last and most sharply. Unknowns scale with attempts and with congestion, and each one needs a decision. Somewhere between a few per day and a few per hour, resolving them stops being something a person does between other tasks and starts requiring either automation or a smaller run. Deciding which, in advance, is more useful than discovering it during a busy window.

The point where building stops being the cheaper option

Every ceiling above has a cost to raise, and those costs are not symmetric. Endpoint capacity is money. Fleet size is rent plus engineering. Capital is opportunity cost. Operator load is the one that cannot be bought incrementally, because it requires either automation you have not written or attention you do not have.

That asymmetry is what usually decides the build-or-run question at scale rather than at the start. A small run is genuinely easy to build for, and the arguments for owning the ledger and the keys are strong. A large run requires batched confirmation, automated reconciliation, resolution tooling for unknowns and an endpoint plan with real capacity, and each of those is a project rather than a feature. Comparing what that costs against what a hosted alternative costs is a legitimate exercise, and anyone doing it seriously should be reading a comparison of the best Solana volume bot options against their own ramp results rather than against a feature list.

The honest framing is that neither answer is safer. Building means the failure modes are yours and you can see them. Running means somebody else has already solved these problems at a size you have not reached, and you cannot inspect how. What makes either defensible is having done the ceiling arithmetic, because Solana volume automation at a size nobody has capacity-planned for fails the same way regardless of who wrote it.

The post-run review

A ramp that produced no written record produced a feeling. The review after each stage should be short and should answer four questions, recorded with the run id so the sequence can be read later as a history.

1. Which variable changed, from what to what, and for how long was the stage held?
2. Did any metric leave its baseline range, and by how much? A metric that moved without aborting is still a finding.
3. How many unknowns occurred, how long did each take to resolve, and did any require a decision the engine could have made itself?
4. What is the next ceiling on the paper arithmetic, and did this stage change the estimate?

The fourth question is what keeps the ramp pointed at something. Each stage should either confirm or move the estimate of where the binding ceiling is, and a stage that does neither has told you nothing except that the engine still works.

Scaling readiness checklist

• Has the binding ceiling been identified on paper before the first stage?
• Do you have a baseline metric set with ranges, not single values?
• Are the abort criteria written, measurable, and each given an observation window?
• Does abort mean stop releasing and keep confirming, rather than kill the process?
• Has the crash recovery test actually been run, rather than assumed to work?
• Is the request profile at target size known, including confirmation polling under congestion?
• Is reconciliation automated at the size you are ramping toward, rather than at the size you are at?
• Do you know your unknown-resolution rate per hour, and is a person available at that rate?

The last item is the one to be honest about. Every other ceiling can be raised with money or engineering, and that one cannot be raised at short notice at all. An engine that produces more unresolved items than a person can clear is not scaling, it is accumulating, and the accumulation is invisible until reconciliation. Read the observability note for the metrics this procedure depends on, and the rate limit note for the ceiling you are most likely to meet first.

Questions this note gets asked