When is a cheaper AI model good enough?

A cheaper model is good enough for the inputs where it agrees with the strong one on your own past work, and where your check catches the times it doesn't. Find out by running it over 50 real inputs and reading every disagreement. Then route: cheap first, the strong model only when the check fails.

2 evenings, 50 documents

I sat in the spare room with 50 real documents, customers' details stripped out, and 2 columns of answers: the strong model I'd used since launch, and a much cheaper one from the same provider, with the same instructions and the same check.

I'd chosen the strong one by reputation. (It's how most people choose a model. It's also how most people choose a wine.) I expected the cheap one to be hopeless.

On most of the 50, the columns agreed exactly. On a few more, they differed in ways that didn't matter, like a middle initial my code formatted away anyway. The rest the cheap model got wrong, and those had something in common. They were long, and badly scanned: a page photographed at an angle, a fax of a fax, a coffee ring across the middle. (Real customers. Real coffee.)

Route on what you learned

So now every document goes to the cheap model first. The check runs. If it passes, that's the answer. If it fails, or the document is longer than a length I set from those 2 evenings, it goes to the strong model. Most documents never meet it. The hard ones still do.

It took an afternoon to write. The 2 evenings of reading were the part that mattered.

Read the disagreements

A match rate alone gives you 2 bad options: accept the cheap model's failures, or pay for the strong one on everything. Only reading the disagreements tells you why the cheap one fails, and the why is what lets you write the routing rule.

Who Pays for the AI? has the test and a worked example of the saving.