Nothing but the cache: killing our token bill by 90%

We recently caught our own product doing something ridiculous: paying full price to re-read the same web page. Over and over. Within the same minute.

One log line summed it up. On a single request, our agent sent the model about 318,000 tokens — roughly two novels' worth of text — and only 32,000 of them were recognized as "seen before." The other 286,000? Full price. For a page that had not changed at all since the previous request, fifteen seconds earlier.

Fixing that turned into one of the most satisfying engineering weeks we've had. Here's the story, minus the scary parts.

The five moves, up front

If you only take away the checklist, take this one. Every fix we made was a version of the same idea — make the request look as much like the last request as possible, for as long as possible:

1. Squash the prompt variants. We had one "system prompt" that quietly rendered differently depending on which pass, which settings, and which tools were in play. Every variation is a different document to the cache. Now the instructions are byte-identical for the whole task, and the tool definitions are sorted into a fixed order so they can't shuffle.
2. Order everything by how often it changes. The single highest-value change in this whole post. Instructions and tool definitions first, then attached files and profile, then the page snapshot, then the append-only history, then the actual request — and the clock dead last.
3. Never re-send what you can point at. The model's own notes let us reference old page snapshots instead of shipping them again.
4. Keep related calls in the same cache "segment". Two requests with identical text can still miss if a single request setting differs. We lost months of discounts to one flag.
5. Measure it, or it silently rots. A broken cache throws no errors. It just costs fifty times more.

The rest of this post is how we learned each one the hard way.

The 50× coupon

First, the thing most people don't know about AI pricing.

Every time you send a message to a model, it doesn't just read your message — it re-reads the entire conversation so far. All the instructions, all the history, everything. And you pay for every word of that re-read, every single time. This is why AI agents that work in long sessions get expensive fast: the bill isn't driven by what the model says, it's driven by what it re-reads.

To soften this, DeepSeek offers a deal — and it is a genuinely startling one. Any part of your request they've already seen recently is billed at a fraction of the normal rate:

USD per 1M tokens, from DeepSeek's published API pricing.

Read those first two columns again. On Flash, text the model has seen before costs 1/50th of new text. On Pro, 1/120th. That's not a rounding-error discount — recognized tokens are effectively free, and unrecognized tokens are the entire bill.

But there's a catch, and it's brutal: the discount only applies from the very beginning of your request up to the first character that's different from last time. Think of it as a chain of dominoes. As long as every domino matches yesterday's chain, they fall for free. Flip one — one — and you pay full price for every domino behind it, no matter how identical they are.

There is no partial credit. No "this is 99% the same." A single changed character a third of the way in means the remaining two-thirds is billed as brand new.

So the entire game is: arrange your request so the stuff that never changes comes first, and the stuff that always changes comes last. Pack the suitcase with the things you never touch at the bottom.

We thought we were doing that. We were not.

Why we care so much about DeepSeek's version

A side note on why this post is about DeepSeek specifically.

Google's Gemini has a caching discount too, but the default flavour is implicit caching — meaning Google decides what to cache, when, and whether to bother, and you find out afterwards from a usage field. When we finally built proper measurement for it, we found long stretches where it cached exactly nothing: millions of eligible tokens, zero discount. The clearest signal came from retries — the same request, sent again seconds later with a byte-for-byte identical context, still reported zero recognized tokens. On a good day it recognizes maybe a quarter of what it's seen before.

Gemini does offer explicit caching, where you upload a chunk of context, get a handle, and reuse it. We looked hard at it and passed, for two practical reasons. First, maintenance: our prompts change weekly, and every edit means minting, versioning, and garbage-collecting a new cache object for every prompt variant in flight — a whole cache-invalidation problem bolted onto a prompt-engineering problem. Second, phantom bills: explicit caches bill for storage by the hour whether or not anything reads them, so a forgotten handle quietly meters money in the background. We didn't want a second inventory system whose failure mode is an invoice.

DeepSeek's cache, by contrast, is dumb in the best way: no handles, no TTLs, no storage line item — just one strict, published rule applied automatically. And you can engineer against a rule. You can test against it, measure against it, and build alarms that fire when you break it. Against "we may cache your prompt if conditions are favorable," all you can do is hope. Hope, it turns out, hits about a quarter of the time.

Some genuinely fun token facts

Because the discount is character-exact, we spent quality time with DeepSeek's actual tokenizer — the machine that chops text into the "tokens" you're billed for. They publish it as a downloadable package, which matters more than it sounds: it let us stop estimating token counts and start measuring them, then use it as a bench to test how we encode a web page.

That turned into a small research project of its own. The page snapshot is the biggest thing we send, so we tried encoding the same page a dozen ways — different indentation, different bracket styles, flattened vs nested, with and without element attributes, with and without link URLs — and measured each against the real tokenizer instead of guessing. Some results were worth real money (dropping link URLs out of the tree text and resolving them at runtime cut about a fifth of the tokens on link-heavy pages; a bug where table cells re-emitted their entire subtree as a label was costing us 43%). Others were beautifully pointless. Running a real 2.6-million-character snapshot through it:

The tokenizer bundles runs of spaces together on its own, so indentation is essentially free — and tabs actively cost you. We came close to shipping a whitespace-stripping "optimization" that would have saved 0.0002% of our bill and made everything harder for the model to read.

The rule we came away with: encoding choices that look expensive to a human are often free, and the expensive ones are invisible. Download the tokenizer and weigh your actual payload before you redesign anything.

Why browser agents break the coupon

Here's where our product makes life hard for itself.

A browser agent's request to the model is dominated by one thing: a fresh snapshot of the web page it's working on — a big structured "map" of everything on the page, where every button and link gets a numbered label the model can refer to. On real pages, that map is routinely 80% of everything we send.

And web pages are fidgety. A points counter ticks on Hacker News. A "2 hours ago" rolls over on LinkedIn. One new element appears at the top of the page — and because the labels are numbered in order, every label after it shifts by one. To a character-exact discount rule, a page that is 99% the same can look 95% different.

We measured exactly how much two snapshots of the "same" page actually have in common:

That third and fourth row are the whole ballgame. When the agent doesn't touch the page between two looks — because it was busy writing to a spreadsheet, or double-checking its work — the new snapshot comes back identical, down to the character. The shared-navigation-bar theory of caching is dead; the untouched-page reality is very much alive.

So the question was never "how do we make web pages hold still." It's: on the passes where the page happens to be identical, does our request layout let the discount kick in?

Ours didn't. And the reason is almost funny.

The suitcase was packed wrong

Our agent's request was arranged in the obvious storytelling order:

Perfectly natural. Terrible for the coupon.

"What you've done so far" grows on every step — that's the point of it. Which means the giant page map sitting behind it gets pushed to a different position every single time. Same map, same characters, different starting point — and to a strict start-to-first-difference rule, that's a brand-new document. That's the log line at the top of this post: the instructions matched, then the diary had one new entry, and the two-novels-long map behind it got re-billed in full. For a page that hadn't changed.

The fix is one line of surgery: put the map before the diary.

Now the map sits at a fixed position. When the page hasn't changed, it matches perfectly — and the biggest thing in the request costs a fiftieth of list price. The diary grows behind it, where growing is harmless. And when the page really did change, we only lose the discount on the small stuff, instead of losing a two-novel map to one new diary entry.

We'd actually designed something much fancier first — a scheme where the map would "float" to wherever it was first captured, with the server comparing snapshots to decide where it belongs. Then we realized the simple version captured the entire win with zero moving parts. The best thing we designed that week was the deletion of the clever thing.

A handful of smaller fixes rode along, all the same principle — sort everything by how often it changes:

The one setting that split our cache in two

This one cost us the most and was the hardest to see, because the text was never the problem.

Our agent does two kinds of call: one where it decides what to do next, and one right after where it pulls structured data out of what it found. The second call deliberately replays the first call's entire conversation, so it should be a perfect, free re-read — and we'd carefully verified, character by character, that it was.

It never got the discount. Not once.

The culprit turned out to be a single request setting, unrelated to the text: the extraction call asked for its answer in JSON mode. Flip that one flag and the provider files your request under a different cache "segment" — same words, different universe, no match. We only found it by sending the same prompt twice with one parameter changed and watching one hit and the other miss.

The rule we now hold everyone to: two calls that should share a cache must be identical in their settings, not just their text. We dropped JSON mode from the extraction call (the format is enforced by the instructions and the parser anyway) and it went from paying full freight every time to riding the previous call's prefix almost entirely — one of our biggest single wins, from deleting one line.

The model that takes notes on itself

One more trick, and it turns DeepSeek's most-mocked trait into a feature.

DeepSeek talks a lot. Its internal reasoning is long and chatty — as it works, it narrates everything it sees: "the upload button is number 9385," "the Post button is 5954, currently grayed out." Most products throw that narration away after every step.

We keep it. All of it, word for word, in the growing diary.

Why? Because those notes mean we never need to re-send old page snapshots at all. When a later step needs something from a page the agent saw three steps ago, it doesn't need the old map back in the conversation — it just cites its own note ("element 25, from step 0, tab 0") and our system looks up the real thing from storage behind the scenes.

The model's rambling is its filing system. It takes notes because it can't help itself; we made the notes load-bearing. And since the diary is exactly the append-only, never-edited text the coupon loves, every one of those notes is billed once — and re-read at a fiftieth of the price forever after.

The agent that stopped believing its own eyes

Honesty section: one of these changes bit us within hours, and it's too good not to share.

Right after we moved the page map to the front, one of our agents — halfway through filling out a job application — became convinced it couldn't see the page anymore. From its actual reasoning log: "the page snapshot was given at the very start, before any of my actions. Since then, I've only been shown web addresses."

The snapshot was right there. Fresh on every step, sitting at the top of its request. But the model reads like a person: top to bottom, as a story. And in story order, a page that appears before your list of actions reads like something from before those actions — old news. So the agent did what a confused person would do: it started taking screenshots. Scroll, screenshot, "describe this image to me," wait, screenshot again — six rounds of trying to re-observe a page it was already holding, tripling the cost of the task.

The fix wasn't moving the map back — the savings are real. The fix was adding one plain sentence at the end of every request, where the model's attention is sharpest: the page snapshot above was captured for this turn — it's live, it's current, trust it. The agent read it, calmed down, and finished the application.

Lesson learned: a prompt's layout is a contract with two parties — the cache and the model. Renegotiate with one, and you'd better inform the other.

The receipts

Everything below was measured on real tasks, against the real DeepSeek API, reading the discount straight off the bill.

The middle bar is the honest one. The LinkedIn task involves a composer modal that genuinely rebuilds the page on half its steps — those steps should miss, and they do. The job-application task, where the agent works a stable form, lands at 87% discounted across the whole task: 867,000 input tokens for about $0.019 instead of $0.12.

And because recognized tokens also skip a chunk of the model's reading work, responses start noticeably faster too. Same intelligence, same results — a fraction of the bill, at higher speed.

Curious what this is worth for your own agent? The shape that matters is snapshot size × rounds × hit rate:

What's next

The unglamorous truth: none of this was a trick. It was measurement. Cache regressions are silent — nothing crashes, nothing errors, you just quietly pay fifty times more. The only defense is instrumentation that tells you exactly which character broke the chain, and automated tests that fail loudly when a code change re-packs the suitcase wrong.

Coming next: labels for page elements that survive page changes (so a 99%-similar page finally gets 99% credit instead of 5%), skipping the re-photograph entirely when we can prove a page hasn't changed, and rolling all of this out across our cloud agent fleet, where the same math applies at bigger volume.

The takeaway if you're building — or just buying — anything on top of these models: a prompt isn't a document, it's a suitcase. Pack what never changes at the bottom, what always changes on top. Providers are handing out a 98%-off coupon for material they've already read. Most products leave it on the table.

We decided we wanted nothing but the cache.