How to run an LLM locally

A local LLM is one where the model weights live on hardware you control and inference happens on that hardware. No prompt leaves the machine and there is no per-token bill. This is different from a private API endpoint (still someone else's servers) and different from a fine-tuned model on a hosted platform (still hosted).

Local can mean a laptop, a workstation, a rack-mounted GPU server in your office, or an air-gapped appliance in a secure facility. The architectural property, weights on your hardware, is what matters.

• 1. Pick an open-weight model: The Llama, Qwen, Mistral, Gemma, Phi, DeepSeek and Nemotron families all publish open weights, and the leading model in each changes often, so check a current leaderboard rather than a blog post. Pick a parameter count that fits your hardware (see below).
• 2. Pick a runtime: Ollama is the easiest entry point. llama.cpp underlies most local setups and runs well on CPU and Apple Silicon. vLLM is the production-grade choice on Nvidia GPUs. LM Studio gives a chat UI without the command line.
• 3. Pick a quantisation: Quantisation (Q4_K_M, Q5_K_M, Q8, FP16) trades a small amount of quality for a large amount of memory. Q4_K_M is the typical sweet spot for local use.
• 4. Serve it: Most runtimes expose an OpenAI-compatible HTTP API. Point your existing tooling at the local endpoint and most code paths just work.
• 5. Wire up retrieval: A local LLM with no access to your documents is a toy. Add a retrieval layer (LlamaIndex, LangChain, or a hand-rolled pgvector setup) so it can answer over your data.

• 7B models: Run comfortably on an Apple Silicon MacBook with 16-32GB unified memory, or a single 12-16GB Nvidia GPU. Fine for one developer.
• 13B-14B models: Need a 24GB GPU (RTX 4090, A5000) or a 32-64GB Apple Silicon machine. Quality starts becoming useful for real work.
• 30B-32B models: Need 48GB+ of VRAM, typically a workstation with an RTX 6000 Ada or two consumer GPUs. The minimum for serious business use.
• 70B+ models: Need a multi-GPU server, an H100, or a dedicated appliance. Beyond a single workstation.
• Concurrency multiplies all of this: The numbers above are for a single user. Ten concurrent users need batched serving (vLLM, TensorRT-LLM) and significantly more VRAM, this is where DIY usually breaks.

• Concurrency and uptime: One developer is easy. Twenty users hitting one chat endpoint with retrieval is a distributed-systems problem, not an Ollama problem.
• Identity, audit, retention: Enterprise procurement requires SSO, RBAC, prompt logging with configurable retention, and an audit trail. None of this comes in the box.
• Retrieval at organisational scale: A useful enterprise LLM has to retrieve over thousands or millions of documents with permissions. That is an ingest, embedding, and access-control pipeline, not a vector store with 100 PDFs in it.
• Domain accuracy: Open-weight base models are general. For domain accuracy you need post-training (not just RAG), and serious post-training is its own discipline (catastrophic forgetting, evaluation, regression suites).
• Ongoing model lifecycle: Models improve. Re-evaluating, re-quantising, and re-deploying every few months is a small ML platform team, not a side project.

Local LLMs are the right answer when the data must not leave the building, when you want a fixed cost rather than a per-token bill, when latency must be predictable, or when you need to operate offline or air-gapped. Those properties do not change between a laptop with Ollama and a Locai One in the corner of the office, what changes is how many users, how much data, and how serious the governance.

For a single developer experimenting, Ollama plus a 13B model and a small RAG layer is the right call and costs nothing. For a business that wants the same architectural properties at organisational scale, with identity, audit, retrieval over real document estates, and a model trained on your domain, you want an appliance rather than a side project.

Locai One is the managed, owned version of "running an LLM locally". It bundles our open-weight Juno models with production serving, retrieval, identity, and an application layer in one on-prem appliance, and a model post-trained on your own data (using Forget-Me-Not) can be deployed on the same machine. The model still runs entirely inside your perimeter, your prompts still never leave, but you do not have to operate the stack. If you have prototyped with Ollama and are looking at scaling to a business, that is the natural next step.