Metrx
Sign in
Methodology

How Metrx counts

This page covers the numbers on an agent’s contract, scorecard and holdout verdict. Each is plain arithmetic on what your systems sent, carries a label saying how sure it is, and is computed by the same code this page uses for its examples.

  1. Model spend

    Measured

    The sum of the cost of every model call your SDK, OpenTelemetry exporter or the Metrx gateway reported for the agent in the window.

    Worked example: 1,840 calls in 30 days → $48 of model spend.

    Spend is what was reported. A call that never reached Metrx is not in it, and is not counted as $0.

  2. Cost per outcome, and per 1,000

    Measured

    Model spend ÷ outcomes reported for the agent in the same window. Per 1,000 is the same figure × 1,000, named in the unit of work the contract states. Like a rate, it needs at least 20 outcomes in the window to be graded.

    Worked example: $48 ÷ 200 outcomes → $0.24 each, or $240 per 1,000 refund decisions.

    With no outcomes in the window this figure is undefined and shown as such, never $0. The agent P&L divides the same spend by the outcomes the attribution engine linked to the agent, so it carries the Attributed label.

  3. Rates (error-flagged share, confirmed share)

    Measured

    Outcomes in a tier ÷ all outcomes reported in the window, as a percent. A rate needs at least 20 outcomes in the window; with fewer it is not graded.

    Worked example: 6 error-flagged ÷ 200 outcomes → 3%. With 19 outcomes the term reads “Not graded”: Fewer than 20 outcomes in the window (Metrx's minimum for a rate, not a contract term).

    A confirmed outcome is one a person confirmed in Metrx or your own SDK asserted. Metrx cannot tell whether a person reviewed an SDK-asserted outcome.

  4. Spend against the contract budget

    Measured

    Spend over the budget period (the last 24 hours, 7 days or 30 days) compared with the whole budget. In the contract’s first period, spend since it started is compared with the whole budget, with no proration, and the term reads “Tracking” until the period has run; it reads “Misses” as soon as spend is over.

    Worked example: A $400 monthly budget, graded two days after the contract started → “$3.20 of $400 budget, 2 days in”.

    The budget is declared, not enforced: Metrx grades spend against it and does not stop calls.

  5. Meets, Misses, Not graded

    Measured

    Each term in a job contract is compared with the measured value over the window the contract names. Meets or Misses only when the value exists; otherwise Not graded, with the reason.

    Worked example: 3% error-flagged against “at most 5%” → Meets. 19 outcomes against the same term → Not graded.

    An absent signal is never graded as a zero: no calls means spend terms are not graded, and no outcomes means outcome terms are not graded.

  6. Human baseline

    Estimate · declared by you

    The cost of one unit of work done by a person, declared by the contract owner. Shown beside the agent’s cost per outcome; never graded.

    Worked example: Declared $4 per refund decision, shown next to the agent’s $0.24 each.

  7. Jobs

    Measured

    One job is one task ID your SDK or OpenTelemetry exporter sends with its calls. A call sent without a task ID counts as its own job.

    Worked example: Three calls sharing one task ID and two calls with none count as three jobs.

  8. Spend with no contract

    Measured

    Spend of agents that no job contract governed at the end of the window. A contract version counts if it was created by then and was still active, or was replaced only after it. Spend not tied to any agent is reported on its own line.

    Worked example: An agent whose contract started the day after the window ended counts as having none for that window.

  9. The holdout verdict

    Verified

    A cheaper model or route is tested against the current one on randomly assigned live traffic. Quality must not be worse (a hard gate); then the cost difference must clear the threshold set before the test. Too few results in either arm gives “insufficient evidence”, not a guess. Negative and inconclusive verdicts are kept.

    Worked example: SDK only. Demo-proven on Northwind AI, an illustrative company with synthetic data; no customer verdict is claimed.

  10. Synthetic

    Synthetic

    Demo data. Every figure for Northwind AI is generated, carries the dashed Synthetic chip, and a CSV exported from a demo workspace starts with a line saying so.

    Worked example: The examples on this page are worked arithmetic, not customer data.

Measured, not estimated

ValueThe usual estimate: Hours saved × an hourly rateMetrx: Spend ÷ the outcomes your own systems counted, per unit of work
The barThe usual estimate: Set after the factMetrx: Written in the job contract before anything is graded, with a version history
Too little dataThe usual estimate: A number anywayMetrx: Not graded below 20 outcomes, with the reason shown
Did a change help?The usual estimate: Before and afterMetrx: A randomized holdout, for agents you run through the SDK
The outputThe usual estimate: A dashboardMetrx: A record per agent you can export and hand to finance

Metrx counts from what your systems send: call cost, token counts, outcomes and the fields listed on the trust page. The SDKs and OpenTelemetry exporter send no prompt or completion text. What Metrx stores · Tasks, outcomes, contracts and scorecards (docs)