In the previous article, we covered evaluation before buying AI infrastructure. That is not a separate step before model selection. It is the basis for that choice. Without your own test set, it is easy to choose the model with the best-known name, the highest benchmark score or the most impressive demo.
That is risky. A model can look strong in general benchmarks and still be a poor fit for Dutch support questions, legal documents, tenant isolation, structured output or your latency requirements. With private inference, there is another layer: you choose not only a model, but also how that model runs.
Model selection is therefore about more than parameters. You choose a combination of model type, license, context length, quantization, runtime settings, performance and operations. The best model is usually not the largest model. It is the smallest model that passes your own evaluation, under conditions that fit your organisation.
Start with the task, not the model name
The question is not: which model is the best? The question is: what behaviour do you need?
A classification model for tickets has different requirements from a document assistant for contracts. An internal search function with RAG has different weak spots from a code assistant. A SaaS feature for customers has to be tested differently from an internal summariser for employees.
Start with a short description of the workload:
- What should the system do: classify, summarise, search, answer, extract, write code or process documents?
- What data does it touch: public, internal, confidential or regulated?
- Is the application real time, or can it run in batch?
- Should the output be creative, or predictable?
- What counts as a serious failure: a wrong source, a data leak, invalid JSON, too much latency or too much cost?
Only then does model selection become concrete. A smaller model can be perfectly fine for routing, extraction or a bounded RAG flow. For complex reasoning, code, multimodal input or difficult domain language, you may need a larger or more specialised model. You only know after testing.
Open weights, commercial model or hybrid
Private inference does not always mean that you own the model weights or run everything on your own hardware. There are several routes.
With open weights, you can run a model yourself on dedicated hardware, private cloud or a locked-down environment. That gives you a lot of control over runtime, logging, network access, updates and cost structure. It also adds operational work: patching, security, monitoring, incident response, license checks and evaluations.
A commercial model can also be used in a private deployment, for example through a private endpoint, VPC or VNet and clear contractual agreements about data. You do not manage the weights, but you do manage access, network boundaries and data processing within the chosen cloud environment.
For many organisations, a hybrid approach is the most practical. Sensitive or predictable workloads run privately. Heavier or less sensitive tasks can use a managed model, as long as data flows, logging and contracts are clear.
Use precise language here. Do not call a model open source automatically because the weights are available. Check whether it uses Apache 2.0, MIT, a community license, a custom model license or a commercial service. For SaaS companies, terms around hosted services, distribution and derived models are not a detail.
Choose the smallest model that passes your evaluation
Larger models can do more, but they also cost more. They need more memory, often have higher latency and limit how many users you can serve at the same time. With self-hosting, that quickly means more GPU capacity or stricter limits on context and batch size.
A practical approach is to avoid starting with the largest model. Test a small and a medium-sized model on the same evaluation set. Look at where the smaller model fails and whether the larger model actually fixes those failures. If the difference is mostly nicer wording, but not task score, latency or reliability, larger is not automatically better.
For a support flow, a smaller instruct model with good retrieval can work better than a large model with too much irrelevant context. For legal extraction, you may need higher precision or a larger model. For code, a specialised model can fit better than a general chat model.
The eval harness from the previous article should decide this. Not the model name.
Context window is not memory
A large context window sounds attractive. 32K, 128K or more gives room for long documents, detailed instructions and many examples. But context is not free memory.
More context means more tokens. That increases cost and latency. With self-hosted inference, the KV cache also grows with the number of tokens, batches and concurrent requests. A model that can technically handle 128K context can still be impractical for a real-time SaaS feature with many users.
Use long context deliberately. Do not put everything in the prompt just because it fits. For most business applications, better context selection is more important than more context. Think of good chunking, metadata filters, authorisation per document, retrieval and reranking.
A large context window is a buffer, not an architecture.
Test English, Dutch and domain language separately
Many model cards mention multilingual support. That is useful, but not enough for Dutch B2B use cases.
Test with real examples: support tickets, emails, contract passages, knowledge base articles, product specifications and sector-specific terms. Do not only look at grammar. Also assess tone, jargon, English loanwords, source use and failure behaviour.
For RAG, the same applies to embeddings and rerankers. A strong generation model does not help if the right passages are not retrieved. If retrieval fails, the answer still goes in the wrong direction.
Test three things separately:
- does the embedding model retrieve the right documents?
- does the reranker put the best passages at the top?
- does the generation model give a correct and useful answer?
Base, instruct, embeddings and rerankers
A base model is not simply a chatbot without a system prompt. Base models are mainly a starting point for further training, research or specialisation. For end-user applications, you usually want an instruct or chat model, because it is trained to follow instructions and handle conversational patterns.
For RAG, the generator is only one part of the chain. You also choose an embedding model and sometimes a reranker. Those three do not have to come from the same supplier.
A reranker can sometimes help more than a larger generation model. If the problem is that the wrong passages are retrieved, a larger LLM will not fix that cleanly. The search layer has to improve.
Quantization: run smaller, measure again
Quantization lowers the precision in which model weights or cache data are stored. That means a model can fit in less RAM or VRAM, and throughput can improve. But quantization is not free compression.
There are several families, each with its own use:
- BF16 or FP16 as a quality baseline, if your hardware supports it.
- GGUF with Q4, Q5, Q6 or Q8 for llama.cpp, Ollama and local or hybrid deployments.
- AWQ or GPTQ for 4-bit GPU serving when VRAM is tight.
- FP8 on modern data centre GPUs with a suitable runtime.
- Quantized KV cache for long context and many concurrent requests.
Q4 can be enough for a simple assistant, but too coarse for legal extraction, code, tool calling or Dutch domain text. Q5 or Q6 can be a better balance when quality matters more than maximum savings. FP8 can be interesting in data centre environments, but depends on hardware and runtime.
The practical approach: use BF16 or FP16 as the measuring stick if you can. Then test the quantized variants on the same evaluation set. Measure not only memory use and tokens per second, but also task score, hallucinations, structured output, source use, language and safety.
Also watch calibration. AWQ, GPTQ, FP8 and importance matrices work better when the calibration data resembles the real workload. Do not use random English examples if production runs on Dutch customer questions or contract texts.
Parameters shape behaviour
Even with the same model and the same quantization, the application can feel different because of parameters.
Temperature, top-p and top-k determine how much variation the model may use. For classification, extraction, JSON output and RAG answers, you usually want low randomness. For creative text, more variation can be useful, but then you also accept less reproducibility.
Max tokens determines how much the model may answer. It is also a cost and latency control. Long answers are not only more expensive, they also keep capacity occupied for longer.
Structured output needs separate attention. For integrations with databases, CRM, ticketing systems or workflows, "return only JSON" is not robust enough. Use JSON Schema or constrained decoding where possible. That reduces parse errors, retries and repair code.
With self-hosting, runtime choices are added to that list: context length, KV cache, prefix caching, continuous batching, batch size, tensor parallelism and runtime version. These choices determine how many users you can serve at the same time and how stable p95 and p99 latency remain.
Reproducibility is not absolute either. A seed helps, but hardware, backend version, scheduling and online batching can still cause differences. Pin model version, parameters, prompt template, runtime and hardware profile in your evaluation results.
A practical selection process
A good selection process does not have to be large. It mainly has to be repeatable.
- Describe the workload, data class and failure limits.
- Decide whether you want to test open weights, a commercial model or a hybrid route.
- Check license, hosted service terms and data processing.
- Test a small and a medium-sized instruct model on your own evaluation set.
- Test English, Dutch, domain language, RAG retrieval and structured output separately.
- Run a baseline and one or more quantized variants.
- Set parameters per use case, not as a global default.
- Measure latency, throughput, cost and failure behaviour under real load.
- Record model, quant, runtime, parameters and eval version.
This makes model selection less sensitive to hype. You can explain why a model was chosen, when it must be reviewed and which signals point to a regression.
Conclusion
Private inference does not start with the question of which model is most popular. It starts with your workload, data requirements and acceptance criteria.
Once those are clear, you can compare model size, license, context, quantization and parameters. Model selection then becomes a grounded infrastructure decision instead of a bet based on demos.
Do you want to use private inference seriously? cloud.nl can help bring workload, data requirements, model choice and infrastructure together, so you do not start by buying hardware but by making a choice that works technically and operationally.