How we decide when a small model is enough for document work

For narrow document tasks the largest model is often the expensive choice, not the safe one. How we measure whether a smaller model can do the job.

Anyone who has seen a demo with a large frontier model tends to send every document to that same model afterwards. For narrow, repetitive tasks that is rarely the cheapest choice, and not always the safest either. Classifying incoming post, pulling a fixed set of fields from invoices, checking whether a contract follows the template: smaller models now do that work well enough that the largest model mostly ends up being the most expensive option.

There is another factor. We can run a small model in a European data centre or on the client's own infrastructure, so the question of where the documents go has been answered before the legal department asks it. Whether a small model is good enough, though, we only know after measuring, and we measure the same way every time.

Start with an evaluation set built from real documents

We don't start with a model but with a pile of the client's own documents. Not tidied-up examples from a manual, but whatever actually arrives on an ordinary Tuesday. For each document we record the correct answer: the right category, the right value for each field, whether or not the document meets the template. The labelling is done together with the people who currently do the work by hand, because they know which exceptions matter in practice.

The first time we did this, our set was mostly clean PDFs, because they were quick to label. The small model looked ready, until the first week in production turned out to be full of skewed scans and bilingual forms. Since then we go looking for the ugly cases on purpose and include them in roughly the proportion in which they arrive.

• scans of photocopies, skewed or smudged

• documents that switch from Dutch to French halfway through

• handwritten notes in the margin or on a stamp

• forms based on an old version of the template

• attachments that are really a different document type

• fields that are empty and should stay empty

Looking at where it goes wrong

We run a small and a large model over that set, with the same instructions and the same output format. The average on its own says little. Two models with a similar overall score can fail in very different ways: one occasionally misses a field in random places, while the other fails consistently on scans or on French-language documents.

So we break the results down by document type, language, field and source. Often the small model turns out to be as good as the large one on the ordinary cases, with its errors concentrated in one recognisable group. A group like that we can deal with. Errors spread across every category are a much worse sign, even when the average looks reasonable.

Where the errors are recognisable, we route. The small model processes everything, and a simple rule or a separate classification step passes the difficult cases on to a larger model or to a person. Signals we use include low OCR quality, a detected switch of language, output that fails schema validation, and fields that contradict each other, such as a total that doesn't equal the sum of the lines. The large model then sees only a small share of the volume, and precisely the share where it earns its price.

The average tells you whether a model is usable. The distribution of the errors tells you whether you can trust it.

A fixed schema does more for a small model

Give a large model a loose instruction like 'return the relevant details' and it will usually still produce something useful. A small model has less margin: it renames fields, wraps explanation around the JSON or picks a different structure for each document. Each of those deviations is an error that has nothing to do with the content of the document, yet it counts just as heavily in the score.

So we impose a fixed schema, with fixed field names, types, a closed list of permitted categories and an explicit value for 'not found'. That last one is needed because otherwise a small model will happily fill a missing VAT number with something plausible-looking. Where the inference server allows it, we enforce the structure during generation with constrained decoding, so invalid output can no longer occur. The model then only has to decide what to answer, not how, and in our experience that makes a bigger difference for a small model than for a large one.

What it costs, and where the large model stays

For some tasks we don't even try a small model. Weighing a clause on page forty against a definition on page three, or working out what has changed across an entire file, takes reasoning over a long context. Small models lose the thread there sooner, even when their context window is large enough on paper. Chunking helps less than people hope, because the connections run straight across the boundaries between the chunks.

The savings aren't free either. Hosting a model yourself means renting or reserving GPU capacity, maintaining an inference server, keeping up with updates and having someone on hand when the service falls over at night. At low volumes that rarely beats an API bill. On top of that, the evaluation set has to be better than it would for a large model: more representative, and updated whenever a new document type or a new supplier comes along. That maintenance carries on well after handover.

If you are thinking of moving part of your document flow to a smaller model, start with the evaluation set and only then choose the model. If you would like us to look at your own documents with you, you can reach us through the contact page.