We’re having great results from the Chatbase chat engine on fintechbenchmark.com. But we have to look ahead, and smaller models have caught our attention.
For years the industry assumed bigger was always better, with companies building ever-larger data centres to house models with trillions of parameters. But a shift is under way towards smaller, more efficient models, and for good reason.
The power of specialisation
If you’re using AI to provide intelligent search across a directory of 8,000 fintech products, you don’t need a model that can discuss Shakespeare, quantum physics or the Treaty of Westphalia. The questions are predictable and the subject matter is bounded, so a smaller model is not just feasible. For many tasks it’s good enough, at a fraction of the cost.
Don’t underestimate the significance of this. Large general-purpose models remain essential for broad AI applications, but there are far more cases in limited domains where smaller models deliver what’s needed for much less.
What “size” means
The size of an AI model is measured in parameters. Think of them as the knobs and dials the model adjusts during training to recognise patterns. A 7-billion-parameter model has 7 billion of these settings. The largest frontier models are estimated to run to trillions, so 7 billion is small by comparison.
The remarkable part is this: a 7-billion-parameter model takes around 14GB of storage, and compressed (“quantised”) versions need only 4–5GB, small enough to run on a good laptop. No data centre required.
The model doesn’t need to know your products
What surprised us most was this. A small model doesn’t need to know anything about fintech to power a fintech search engine, and neither does a large one. The trick is retrieval. The system first finds the products relevant to a question, then hands them to the language model to read and explain. The model needs to be a good reader and writer, not an encyclopaedia.
And the “search” part is often done by a tiny model. An embedding model of 20–100 million parameters (a few hundred megabytes, small enough to run on a basic server) turns every product description into a mathematical fingerprint, so a buyer’s question can be matched by meaning rather than keywords. A search for “tools for predicting corporate liquidity” finds cash forecasting software even though the words barely overlap. The language model only steps in at the end, to explain the results. That’s specialisation in practice: small models where they’re sufficient, and a larger one only where it adds value.
Where to get them
Hugging Face is the main hub for open-weight models. You can download models such as Google’s Gemma, Microsoft’s Phi and Alibaba’s Qwen. Choose the “instruct” versions, which are tuned to follow instructions and hold a conversation.
Two separate decisions
It helps to separate two questions that often get blurred:
- Model size. Small models are available through APIs as well as for download, so you can choose a smaller, cheaper, faster model without running any hardware yourself.
- Hosting. Running models on your own servers usually only pays off at high volumes or where speed is critical (think fraud detection or trading). But it gives you complete control over your data, which can matter a great deal for fintech compliance.
The takeaway
For fintech companies evaluating AI infrastructure, the key point is this: match the model to the problem. Don’t pay for a trillion-parameter generalist when a lean model, given the right information, will do the job faster and at a fraction of the cost. This field is still in its infancy. Watch this space.
Leave a Reply