How do retries affect AI costs?

Every retry is a second bill. A failure retried 3 times costs 4 times, and retries pile up at exactly the moments something is already going wrong. Cap the attempts on every item, sort failures by kind, and send the ones that keep failing to a pile you read.

340 times in an hour

My phone buzzed on the kitchen counter, and kept buzzing. (For once, it was on loud.) It was the document product. One of its jobs puts a document back in the queue when the model's attempt fails. It had fired 340 times in the past hour, on the same handful of documents, every one sent back to the model and failing again.

Each re-queue was correct on its own terms. The fault was that nothing cleared the thing that made the document fail, so the moment it failed, it went straight back. An hour of my money went round and round the same few pages. I stopped it by hand, which took longer than it should have. The switch that stops everything in one command was built that same week.

3 kinds of failure

• The provider had a moment. A timeout, or an error that says try later. Retry after a pause. 1 or 2 is usually enough.
• The input is the problem. Unreadable, too long, empty. Sending it again gets the same failure at the same price. No retry: it goes to the pile.
• The answer was wrong. The check rejected it. Asking the same way usually gets a different wrong answer at double the price. One retry at most, ideally with the stronger model, then the pile.

The pile

Every item that has used up its attempts goes into one list, with the reason. I read it on Sunday. It costs nothing to keep, and it tells me which inputs and which bugs are mine to fix.

Who Pays for the AI? has a worked example of how fast retries multiply the cost when failures rise.