Private by design: shipping production AI that never leaves your building

Almost every serious AI conversation with a bank, a hospital, or a law firm ends up in the same place: this is great, but our data cannot leave our network. For years that was a conversation-ender. You either accepted a cloud API and the compliance headache that came with it, or you walked away. That trade-off is mostly gone now, and a lot of teams have not noticed yet.

The myth: on-prem means worse

Cloud AI APIs are wonderfully convenient. They are also the reason a whole category of organisations could not use modern AI at all. The unspoken assumption was that self-hosting meant weaker models, more work, and a worse product. Two years ago that was fair. Today, open-weight models have closed most of the quality gap, GPU inference has gotten cheap and boring, and the tooling to run it is mature.

We know because we have shipped three very different private systems (a document-intelligence platform for a fintech, a real-time voice agent, and a document-extraction pipeline) all designed so the sensitive data never leaves the customer's environment. The patterns below are what made each one work.

Pattern 1: Governed RAG that stays in the VPC

For the fintech platform, the requirement was absolute: no document, no embedding, and no query could touch a third-party API. That constraint shapes every layer. Documents are parsed and embedded with an open model (BGE-m3) on the client's own GPU, and the vectors land in a per-organisation Qdrant collection. There is no embeddings API in the loop. The vector space is generated and stored entirely in-house.

The language model is the part people assume you cannot self-host. You can. We built the AI core to be provider-agnostic, with a forced on-prem mode that routes every generation to a locally served open-weight model (via vLLM or Ollama). Cloud providers are available when latency matters more than residency, but flip one flag and the whole thing runs air-gapped.

Privacy is not just about where the model runs, either. Retrieval itself is governed: role-based, need-to-know filtering means a user only ever sees context they are entitled to, and every answer is returned with de-duplicated, page-level citations so it can be verified.

• Local embeddings (BGE-m3 on your GPU): no embeddings API
• A self-hosted vector store, one collection per tenant
• A provider-agnostic LLM core with a hard on-prem switch
• Role-based, need-to-know retrieval (governed RAG)
• Reliability patterns (transactional outbox, idempotent IDs) so it holds up in production

Pattern 2: Voice AI that runs locally

Voice is the case where people are most sure you need the cloud, and they are mostly wrong. Telephony aside, the entire AI stack of our voice agent can run on your own box: a local Whisper model for speech-to-text, a self-hosted LLM for the dialogue, and a self-hosted vector store for the knowledge base. Transcripts, caller memory and knowledge never leave your network.

The hard part of local voice is not the models. It is the milliseconds between them. A caller will forgive an imperfect answer, but not a two-second silence. We stream the pipeline end to end and speak the first sentence of a reply while the rest is still generating, so the caller hears a response in roughly 0.4 to 0.8 seconds. A full conversational turn lands around 1.3 to 1.6 seconds on a GPU box.

The hard part of local voice isn't the models. It's the milliseconds between them.

Pattern 3: Document AI on open-source models

Our document-extraction technique reads messy, real-world documents (crumpled receipts, photographed invoices, handwritten cheques) and returns clean structured data. It fuses an open-source vision-language model, PaddleOCR and classical computer vision, and every model in that stack is open source. There is no per-page cloud cost and, crucially, no page ever gets uploaded to a third party. For anyone processing financial documents, that last point is the whole ballgame.

The honest trade-offs

On-prem is not free, and pretending otherwise does nobody any favours. You are taking on real responsibilities in exchange for control.

• A GPU box (or a small fleet) to own, size and keep fed
• Ops: monitoring, updates, capacity planning, the unglamorous parts
• A model-quality gap that is small and shrinking, but not always zero
• Slower iteration than calling an API and moving on

And sometimes the cloud is simply the right answer: bursty low-volume workloads, anything with no sensitive data, or cases where you want the absolute frontier model today. Plenty of good systems are hybrid: a private core for the sensitive path, a cloud escape hatch for the rest.

How to decide

The decision comes down to four questions: how sensitive is the data, how tight are your latency needs, what is your volume, and what does compliance actually require (not what it vaguely worries about). If the data is regulated and the volume is steady, on-prem usually wins on both risk and cost. If it is neither, the cloud is probably fine. Most real deployments sit somewhere in between, and the architecture should make that a config choice, not a rewrite.

The bottom line

Private AI is not a compromise anymore. It is a design choice you can make on day one without giving up much. The models are good enough, the tooling is ready, and the patterns are proven. If your data cannot leave the building, that is no longer a reason to sit out this wave. It is just an architecture.