Our AWS dashboard said $16.67 over 30 days. The bill said $229.35 for Cost Explorer alone in July.

That is roughly 22,935 calls to GetCostAndUsage, because AWS charges $0.01 per Cost Explorer API request. The tool we used to ask what our infrastructure cost had become the largest AWS line item it was supposed to observe.

It also could not see that line item.

The shareholder found the problem by reading the actual bill. We did not catch it. That is the honest version, and it points to a more useful failure than “we forgot to add a chart.” The dashboard's attribution model made its own consumption structurally invisible.

The query excluded its own answer

The AWS account contains resources belonging to more than one project, so an unfiltered total would be wrong. Our cost query filtered for resources carrying a project tag. That is a normal cost-allocation pattern:

cost belongs to project
  if resource.tags["Project"] == project_name

It works for a virtual machine, object bucket, or container registry entry. Those charges attach to resources that can carry the tag.

An API-request charge has no project resource to tag. AWS's own billing documentation describes service charges that cannot be tagged at the resource level. A filter over resource tags therefore cannot return the charge for running the filter. No missing dashboard widget can fix that. The row was excluded before the result existed.

The attribution rule had to become explicit set logic:

attributed cost = tagged resources
                OR audited untaggable services attributed to us

We kept the tag filter because removing it would include other projects' resources. We added a narrow allowlist for untaggable services only when we can state why the charge belongs to us. Every result now carries its attribution basis and the untaggable subtotal. A number with an unknown basis renders as incomplete instead of quietly presenting itself as a total.

That last part matters. Attribution is a claim, not a query implementation detail. If the claim does not travel with the number, every downstream sum inherits an exclusion nobody can see.

The cache was real in one environment

The second surprise was that caching already existed.

We had four cache keys with a six-hour TTL. Production used a database-backed cache, so the expected upper bound there was about 16 calls per day. Yet July averaged about 740 calls per day. From August 1 through August 5, the bill showed $5.25, or about 131 calls per day.

The same code did not have the same cache in every environment:

• Production used a shared persistent store.
• Test used a null store, where every read is a guaranteed miss.
• Development used an in-memory store, where every one-shot process starts cold.

A review of the production cache configuration proved only that production page loads were cached. It said nothing about test processes or local one-shot commands. CI environments commonly carry cloud credentials, so an unstubbed test can quietly become recurring real spend. Development scripts have the same problem when a new process means a new empty cache.

The environment analysis explains how calls escaped the production cache. It does not explain why the observed rate fell from roughly 740 per day in July to roughly 131 per day in early August before the full fix landed. We do not have a measured explanation for that discrepancy, so we are leaving it unexplained.

A read path should not own a purchase

We replaced “cache the dashboard call” with a harder boundary: dashboard reads never fetch.

One scheduled job owns the daily refresh. It writes the results to durable last-known-good snapshots. The dashboard and briefing read those snapshots and display when they were fetched. A cache eviction, deploy, or cold process can make a number stale, but it cannot turn a read into another billable request.

We also added an environment gate. Outside the production refresh path, the API is disabled unless someone deliberately opts in for a one-off diagnostic call. Tests have a network guard that rejects Cost Explorer access even if application code regresses around the first gate.

The cache still exists, but it is now an optimization behind the boundary, not the boundary itself.

When the harm is the call, fail closed

Our default for operational checks is fail open. If a monitoring probe breaks, it should not wedge the system it watches.

That instinct is backwards for a spend ceiling. Here, the guarded action is the harmful action. If the budget ledger is unreadable, continuing to call is not resilience; it is an unlimited budget disguised as error handling.

Every real request now claims a unit from a persistent daily ledger before opening the network connection. If the ledger cannot be written or the daily ceiling is spent, the request stops. The row records the environment, caller, time, and outcome. A crashed process may leave a pending row, but the counter will never understate how many calls it authorized.

This makes the failure direction deliberate:

budget healthy      -> claim, then call
budget spent        -> stop
budget unreadable   -> stop
provider unavailable -> keep last-known-good

The system can serve a stale cost snapshot without buying a fresh one. Freshness has its own alarm based on snapshot age, separate from the spend budget. “The job stopped” and “the job spent too much” are opposite failures; one counter cannot represent both.

Verify the instrument against the vendor

We did not close this incident because our new ledger said the call count was low. That would be trusting the replacement instrument on its first day.

On August 6, AWS billed $0.05, implying five requests. Our ledger also recorded five: four from the scheduled refresh and one deliberate verification query. We loaded both read surfaces between ledger checks; the count did not move.

The agreement matters more than the small number. It shows that the free local readout matches the vendor's bill, including calls that might otherwise bypass our application metrics. We paid once to validate the instrument against ground truth. Future checks can read the ledger without spending another cent to ask how many cents we spent asking.

This incident is adjacent to cost per completed task, but the unit is different. That post asks how to attribute model usage to useful work. This failure happened one layer earlier: the attribution filter could not represent the observer's own consumption at all.

The general rule is: for every cost or usage report you run, write down what its filter can never match. Then ask whether the report's own consumption lives in that gap.