Observability for an Engine: Metrics That Survive Scrutiny

Four numbers describe a run honestly: landing rate, attempts per landed transaction, time to confirmation as a distribution, and cost per successful swap in lamports. Everything else is either a component of one of those or a detail you look at after one of them moves. All four are computed from the ledger, and all four can be traced back to individual signatures.

The reason to be strict about this is that the alternative is very easy and very comfortable. An engine that counts submitted transactions and displays a rising total will look identical whether it is working well or duplicating work, and the operator will only find out during reconciliation. Metrics are a design artefact, not a reporting afterthought.

The four numbers that describe a run

Each of the four exists because it can deliver bad news. That is the test for whether a metric is worth keeping: if there is no plausible run in which the number would look alarming, it is a counter and not a measurement.

Note that three of the four have a denominator that includes work which did not succeed. That is deliberate. Metrics whose denominator only counts successes are structurally incapable of describing a bad run, which is exactly the run you need them for.

Definitions, and the trap in each one

Metric definitions are where honest and flattering quietly diverge, and the divergence is rarely dishonest on purpose. It happens because the easy definition is also the generous one. Write the definitions down before computing anything, and include the commitment level, because a landing rate measured at a loose commitment and one measured at a strict commitment are different numbers with the same name.

Four decisions cover most of it. Does an intent that was never released count in the denominator of landing rate? Does a transaction that executed and failed on chain count as landed? Does an unknown outcome count as a failure, or is it excluded pending resolution? And are fees from failed executions included in cost? There are defensible answers on both sides of each, but only one answer per engine, written down.

The convention this desk uses, for the sake of being explicit rather than because it is the only valid one: released intents form the denominator; an on-chain execution failure counts as not landed but its fee is included in cost; unknowns are reported separately and never silently bucketed; and cost includes every lamport the run spent, whatever the outcome that produced it.

Landing rate, defined precisely

Landing rate answers how much of what you released actually executed. Measured per intent it is the number that matters to an operator; measured per transaction it is a diagnostic for the pipeline. Reporting only the second when someone asked for the first is the most common way engine reporting misleads without containing a false statement.

The illustrative arithmetic makes the gap concrete. Suppose 400 intents were released. Suppose 380 produced a landed transaction on the first try, 15 expired and were rebuilt of which 12 landed, and 5 ended unknown. Per intent, 392 of 400 landed, which is 98 per cent. Per transaction, there were 415 distinct transactions of which 392 landed, which is 94.5 per cent. Both are true, they measure different things, and quoting one without saying which is how a number stops being useful.

The five unknowns are the interesting part of that example and they are also the part most reporting quietly drops. They represent accounts that are quarantined, value whose disposition is undetermined, and work that a human has to resolve. A run summary that reports 98 per cent and does not mention them has described the pleasant part of the outcome.

Cost per successful swap

This is the metric that ties the rest of the engine together, because almost every design decision moves it. A higher priority fee raises it directly. A tighter compute limit lowers it until overruns start, then raises it again. More retries raise it through wasted execution fees. A better route lowers the value lost to slippage, which is a different line but belongs in the same report.

Compute it in lamports rather than in a fiat figure, and keep the components separate so the number can be explained: base fees on landed transactions, base fees on failed executions, priority fees, and any account creation rent that was not recovered. Rolling those into one total makes the metric harder to act on, because a rise could mean four different things.

Cost per successful swap is also the right axis for comparing an engine you built against one you could use. Throughput comparisons are easy and nearly meaningless, since both can be paced. Cost comparisons require both sides to define a successful swap the same way, which is why any serious discussion of how volume campaigns are measured has to start with definitions rather than with figures.

Time to confirmation is a distribution

Report median and p90 at a minimum, and keep the raw values so other percentiles can be computed later. The median tells you what normal looks like. The p90 is what you size the fleet against, because it describes how long an account is actually unavailable during the periods that matter.

Measure from first submission to settled outcome, not from build time and not from release time. Those other intervals are worth having separately, since a large gap between release and first submission points at the sender being backed up rather than the network being slow, but the confirmation metric should measure the network.

Track the distribution's shape over the run, not just its summary. A p90 that is stable at forty seconds is a capacity planning input. A p90 that was eight seconds for three hours and then became ninety is an event, and an engine that only reports the aggregate will show a mildly elevated number for a run that contained a serious incident.

Building the metric set

The procedure below is deliberately ordered so that definitions precede instrumentation and instrumentation precedes dashboards. Doing it in the other order produces a dashboard whose numbers nobody can defend.

1. Write the definitions first. Attempt, landed, expired, unknown, and the commitment level for each. One page, in the repository, dated.
2. Instrument the ledger, not the code paths. Every metric is a query over durable rows. A metric incremented from inside a function is a metric that survives a refactor incorrectly.
3. Compute the four headline metrics. Nothing else goes on the first screen. Additional metrics are available on demand, but a first screen with twenty numbers is a first screen nobody reads.
4. Attach a trace to every number. Each metric expands into the signatures it was computed from. This is what turns a surprising figure into an investigation instead of an argument.
5. Set thresholds from a baseline. Run small first, record the normal range, then set alerts relative to it. Thresholds chosen before a baseline exists are round numbers with no relationship to your system.
6. Re-derive after every configuration change. A change to pacing, fees or fleet size invalidates the baseline. Record old and new together so the effect is visible rather than inferred.

The log schema underneath

Metrics are aggregates; logs are the evidence. Write one JSON object per line with a stable field set, because a schema that drifts makes filters silently miss records rather than return nothing.

{ ts, run_id, intent_id, signature, wallet_label, venue,
  event, attempt, result, error_class,
  cu_limit, cu_price, fee_lamports, slot, amount_in, amount_out }

event   : planned | released | built | submitted | settled | quarantined
result  : landed | expired | unknown | rejected | failed_onchain
never   : private keys, seed words, endpoint credentials, tokens

Two fields do most of the work. The run id groups everything from one execution, which is what makes a post-mortem tractable. The signature is the join key between your records and the chain, which is what makes a claim verifiable by someone who does not have your logs. Use the wallet label rather than the public key as the human-facing account field; labels survive rotation and stay readable, and the public key belongs in the record for verification rather than in the field an operator scans.

The never list is not advisory. A secret that reaches a log line should be treated as compromised and rotated, because logs get copied to backups, shipped to aggregators and pasted into chats far more casually than key material ever is. Deleting the line is not remediation, since you cannot prove nothing read it first.

From a number back to a signature

The property that separates a useful observability setup from a decorative one is drill-down. A landing rate of 94 per cent is a starting point; the list of the transactions that did not land is the actual information. If getting that list requires writing a query by hand during an incident, it will not happen during the incident.

Build the drill-down as a first-class path rather than an afterthought. Every headline metric expands to a list, every row in the list carries a signature, and every signature can be checked against a public explorer by someone who does not trust your dashboard. That last property is what makes an engine's reporting auditable, and it is the difference between a claim and a demonstration.

It also changes how you argue about performance. When two people disagree about whether a run went well, a metric with a trace resolves it in a minute, because both can look at the same signatures. A metric without one produces a discussion about methodology that neither side can win.

The same property is the cleanest thing to ask for when you are assessing somebody else's system rather than your own. A Solana DEX volume bot that reports totals without signatures is asking you to trust an aggregate; one that hands you the list is making a claim you can check against a public explorer without its cooperation. That difference is worth more than any feature comparison, because it is the only part of a reported run that does not depend on the reporter.

Alerts worth waking someone for

Most metrics belong on a dashboard, not in a pager. The test for an alert is whether a human can do something about it in the next ten minutes that materially changes the outcome. Everything else is a report.

• Any unexplained negative balance delta on an executor. This is either a duplicate or a compromise, and both get worse with time.
• An unknown outcome that has not resolved within its resolution window. It is holding a quarantined account and undetermined value.
• Circuit breaker open, or failover to the last remaining endpoint. The engine is one incident from stopping and nobody has been told.
• Landing rate below the baseline range for a sustained window, not for a single sample. Single samples are noise.
• Cost per successful swap above its ceiling. The run may still be working and no longer worth continuing.
• The scheduler holding releases for longer than a defined period. Backpressure is doing its job, but a sustained hold is an incident.

Six alerts is close to the maximum a single operator will keep reacting to. Every additional one dilutes the rest, and an alert channel that fires often becomes a channel nobody reads, which is functionally the same as having no alerts while feeling like having many.

Observability checklist

• Are the metric definitions written down, including the commitment level each uses?
• Is landing rate reported per intent as well as per transaction?
• Are unknown outcomes reported separately rather than bucketed into failure?
• Is cost per successful swap computed from the ledger, in lamports, with components separated?
• Are timings reported as median and p90, with raw values retained?
• Does every headline metric expand into the signatures it was computed from?
• Is the log schema stable, one object per line, with wallet labels rather than raw keys as the human field?
• Is the alert list short enough that every alert still gets a response?

A run that can be described with these four numbers and defended with their traces is a run you can hand to somebody else. Read the scaling note next, because every threshold in this note is also an input to the decision about whether the engine is ready to run larger.

Questions this note gets asked