How Peer Benchmarks protects your data
Peer Benchmarks compares your usage against other Costiva accounts. This page states, plainly, what is collected, how cohorts are formed, and the mechanism used to keep any published figure from exposing a single account — including what this mechanism does not guarantee.
1. Opt-in, off by default
Peer Benchmarks is off for every account until an org manager explicitly opts in. Opting in requires choosing a company size band, an industry sector, and an AI adoption maturity level — the three attributes used to match your account to a cohort of similar accounts — and checking an explicit consent box. There is no default-on state and no pre-checked box. Any org can opt out at any time; doing so immediately stops that org from contributing to or seeing cohort comparisons.
2. Cohort dimensions
Cohorts are formed from exactly three self-reported attributes — never inferred from your usage data, since usage volume and mix are exactly what this feature compares:
- 1-10 employees
- 11-50 employees
- 51-200 employees
- 201-1000 employees
- 1000+ employees
The coarse, top-level NAICS sector list — e.g. Agriculture, forestry, fishing and hunting, Mining, quarrying, and oil and gas extraction — never the granular business-type list, which would make most cohorts too narrow to ever reach the minimum size below.
- Exploring — Evaluating AI use cases, little or no production usage yet
- Piloting — One or two production use cases, still proving value
- Scaling — Multiple production use cases, actively growing usage
- Optimizing — AI is core to operations, focus has shifted to efficiency
3. Minimum cohort size
No figure is ever published for a cohort segment (a specific company-size × industry × AI-maturity combination) with fewer than 20 distinct contributing accounts. A segment below that threshold simply shows "not enough peers yet" — there is no fallback to a smaller, more identifiable group.
4. The noise and clipping mechanism
For each qualifying segment, every contributing account's metric value is first winsorized (clipped to within a bounded multiple of the segment's median absolute deviation), so a single extreme account cannot visibly skew the published figures. Five percentile boundaries (10th, 25th, 50th, 75th, 90th) are then computed and each one has Laplace-mechanism noise added, calibrated per metric (epsilon = 1, metric-specific sensitivity — see the table below). Only these five noised numbers per segment are ever stored or shown — never an individual account's raw value, and never a value in between two contributing accounts that could be reverse-engineered back to one of them.
Published figures are replaced in place on every recomputation, not kept as a history. An append-only log of past snapshots would let someone infer when a specific account joined, left, or changed a segment by diffing consecutive runs; keeping only the current snapshot avoids that.
5. The six benchmarked metrics
| Metric | Definition |
|---|---|
| Cost per token | Total cost ÷ total tokens, expressed as USD per million tokens. |
| Cache hit rate | Cached input tokens ÷ total input tokens. |
| Model mix | Share of spend on premium-tier models (Opus/Fable family), vs. total spend. |
| Batch API share | Tokens processed on the Batch API ÷ all tokens. |
| Cost per call (proxy) | There is no per-request cost signal in Anthropic's usage data, so this is approximated as output tokens ÷ total tokens — a proxy for larger, more expensive responses per call, not a literal per-call cost. |
| Error rate | Errors ÷ requests, from OpenTelemetry-instrumented traffic only. An account with no OTel instrumentation is excluded from this specific comparison — it is never assumed "healthy" by default. |
| Month-on-month spend growth | Month-to-date spend vs. the same number of days at the start of the previous calendar month. |
6. What this does not guarantee
This is a documented engineering approximation of k-anonymity and differential-privacy-style protection, not a certified formal guarantee. It has not yet been reviewed by an external privacy or legal specialist. If you have questions about what this means for your account's data, contact Costiva support before opting in.