01
The data can’t leave.
Patient records, contracts, source code, anything under NDA or a client’s security policy. If the model runs inside your network, nothing has to be sent out to be processed.
L2 Private LLM on-prem · your cloud · air-gapped
A private LLM on your own hardware.
On-prem, in your cloud account or air-gapped: the model runs where your data already is, and prompts never reach an outside AI provider. Here’s what fits, how fast it runs and what it takes, from the box I run every day.
§01 why local
Three reasons teams move a model in-house.
02
The bill grows with usage.
API pricing scales per token. A model on your own GPU costs the same at 3 a.m. as at peak, so sustained, high-volume workloads can come out cheaper.
03
You control the version.
No silent model updates, no deprecations on someone else’s schedule. You pin the weights, the quantisation and the prompt format, and your evals stay valid.
§02 what fits RTX 4090 · 24 GB
A 27B model with 262K context on one 24 GB card.
This is my own box: one RTX 4090 serving a 27B Qwen model, quantised to Q4_K_M, to my coding agents all day. The weights take about 17 GB. The KV cache, the memory that holds the conversation, is what usually decides whether long context fits.
- tokens of context
- 262K
- tokens of context
- VRAM measured in use
- ~20 GB
- VRAM measured in use
- tok/s peak decode
- 82
- tok/s peak decode
- FP16 KV ≈33 GB · doesn’t fit
- TBQ4 KV ≈21 GB · fits
§03 beyond the model n=6
What it takes besides weights on a GPU.
- API An OpenAI-compatible endpoint, so your existing tools, SDKs and agent frameworks keep working unchanged.
- quant A quantisation and KV-cache strategy matched to your context length and your GPU, measured rather than guessed.
- evals Evals on your own data before anything switches over, so you know what the local model gets right and where it doesn’t.
- access Authentication, per-team access and prompt logs, so you can show who asked what and when.
- network Network isolation: the inference server doesn’t need outbound internet, and in an air-gapped setup it never gets it.
- routing Optional routing: sensitive work stays local, everything else can still go to a frontier API.
§04 gdpr
Does a local model make GDPR go away?
No, but it removes the hardest part. When the model runs on your hardware, prompts and documents aren’t sent to an outside AI provider, so the AI provider drops out of the picture for that system: no extra processor to vet and no transfer of prompts outside your infrastructure.
What stays is yours either way: a legal basis, retention rules, access control and, where the use case calls for it, a data protection impact assessment. I build the system so those are easy to evidence, with logs, access control and isolation in place. Your DPO or lawyer signs off; I’m an engineer, not a lawyer.
when not to
When local is the wrong answer.
- Light or bursty usage. If the model sits idle most of the day, an API is usually cheaper and simpler.
- You need the strongest frontier model on the hardest tasks. An API gives you that without buying hardware.
- Nobody can own it. A local model is a service someone has to run. If that’s nobody, I’ll tell you on the call.
§05 answers
Short answers.
Can a 27B model run on one RTX 4090?
Yes. At Q4_K_M the weights take about 17 GB. With a 4.125-bit KV cache, 262K tokens of context fit in about 20 GB in total, and decode peaks at 82 tok/s on my box.
Local LLM or API: which is cheaper?
It depends on volume and on how sensitive the data is. Low volume usually favours an API; sustained volume on data that can’t leave usually favours local. I run the numbers on your workload during the teardown.
Which local model should we use?
The one that passes your evals on your tasks. For agentic coding on 24 GB, the dense 27B models led my benchmark of ten local models.
Can it run fully offline?
Yes. Inference, retrieval over your documents and the API all run without internet access; models and updates are brought in deliberately.
Data that can’t leave the building?
Describe your problem30-minute technical call with the engineer who builds it. No sales team, no slides.