Should you call the model live or queue the work?

Queue it. The work goes into a queue, a job takes it every few minutes, sends it to the model, checks the result, stores it, and then tells the customer. Call the model live only when a person is genuinely sitting there waiting for the answer.

The afternoon I did nothing

I found out about an outage at lunch, from the provider's status page: a red banner, elevated errors, engineers investigating. It had been going on for an hour, in the middle of my document product's working day. I finished my sandwich and did nothing at all.

That evening the provider came back, the queue moved, and every document from the afternoon went through, was checked, and was emailed a few hours late. Nobody wrote in. (That's the best review an outage can get.)

The afternoon I did everything

The year before, my price list comparer sent each upload to the model straight away while the customer watched a spinning wheel. For a fortnight that was lovely. Then the provider had a bad afternoon. Pages hung. Customers pressed the button again, which sent the work again. My code retried the failures, which sent it again too. Some lists were processed twice over; some not at all. The bill that month was several times normal, and because the product kept nothing but what it showed on the page, I couldn't tell customers which lists had gone through.

I rebuilt it as a loop that same week. That product died anyway, for other reasons. The loop outlived it.

Why the queue is cheaper

• Work is never done twice by accident. Pressing the button again adds nothing if the work is already queued.
• An outage costs nothing. The queue waits.
• Records are kept. Every result is stored before the customer sees it, so "which of mine went through?" takes a minute.
• You choose the pace. Some providers charge less for work that doesn't need an answer straight away. Check their current terms.

Customers barely notice minutes. None of mine has ever left because a result took minutes rather than seconds.

Who Pays for the AI? has the loop, with a diagram of both shapes.