Run models on your own computer
A model is a large file of numbers, and a program on your computer can load it and run it. This lesson shows how, how to tell whether a model will fit, and when a local model is the better choice.
What "running a model locally" means
In lesson 1 you met parameters, the billions of numbers that make up a model. To run a model, a program called an inference engine loads those numbers into memory and does the next-token loop. Nothing is sent to a company's servers: the prompt, the answer and the model all stay on your machine.
Ollama is a free inference engine that many tools build on, including LocalRAG and DL Prism's beginner setup. It downloads models, keeps them on disk, and serves them on your own computer at http://localhost:11434.
your app (LocalRAG, a script, a chat window)
│ HTTP on localhost:11434
▼
Ollama ── loads ──► model file on disk (e.g. llama3.2)
│
▼
CPU and memory, or a graphics card (GPU)Ollama in five commands
These are Ollama's standard commands. They need Ollama installed from its website.
| Command | What it does |
|---|---|
ollama pull llama3.2 | Downloads a model, once |
ollama run llama3.2 | Opens a chat with it in the terminal. Type /bye to leave |
ollama list | Lists the models on disk, with their sizes |
ollama ps | Lists the models loaded in memory right now, and whether they run on the CPU or the GPU |
ollama rm llama3.2 | Deletes a model from disk |
A typical first session:
ollama pull llama3.2
ollama run llama3.2>>> In one sentence, what is a token?
A token is a small piece of text, such as a word or part of a word, that a
language model reads and writes.
>>> /byeYour answer will be worded differently: the model samples, as you saw in lesson 1.
llama3.2 is the answer model LocalRAG uses by default; you pull it once during LocalRAG's install. DL Prism's beginner guide pulls two others, llama3.1:8b and qwen2.5-coder:7b, a general model and a coding model. The part after the colon is a tag: it picks a size or variant of the same model family.
The first answer after starting is slow, because Ollama has to load the model from disk into memory. Later answers are faster. LocalRAG and DL Prism both warn about this.
How much memory a model needs
The model's numbers must fit in memory while it runs. A rough way to estimate:
memory ≈ parameters × bytes per parameter (+ some extra for the context)How many bytes each number takes depends on how precisely it is stored:
| Stored as | 3B model | 8B model | 70B model |
|---|---|---|---|
| 16-bit, 2 bytes each (full quality) | 6 GB | 16 GB | 140 GB |
| 8-bit, 1 byte each | 3 GB | 8 GB | 70 GB |
| 4-bit, half a byte each | 1.5 GB | 4 GB | 35 GB |
On top of this the engine needs room for the context: the longer the conversation or document, the more memory. Leave a few gigabytes free for that and for your other programs.
Worked example. Your laptop has 16 GB of memory and no graphics card. An 8B model at 16-bit needs 16 GB before anything else: it will not fit. The same model at 4-bit needs about 4 GB, plus context: it fits, with room to spare. A 70B model does not fit at any precision.
Quantisation: smaller numbers, smaller model
Quantisation stores each parameter with fewer bits, for example 4 instead of 16. The file shrinks to about a quarter, it loads faster and it runs faster, because there is less to move through memory.
The cost is a small loss of quality. At 8 bits it is usually hard to notice; at 4 bits it is often acceptable for everyday use; below that, answers get noticeably worse. Model names and tags often carry hints such as q4 or q8 for 4-bit or 8-bit. Ollama's default downloads are usually 4-bit versions, which is why the two models in DL Prism's beginner guide need only about 10 GB of disk between them.
CPU or GPU
| Runs on the CPU | Runs on a GPU | |
|---|---|---|
| Memory used | Normal system memory (RAM) | The graphics card's own memory (VRAM) |
| Speed | Works, but slower, especially for mid-size and larger models | Much faster, often many times |
| What you need | Any recent computer | A supported graphics card with enough VRAM for the model |
A model that does not fit in VRAM is split between the GPU and the CPU, and slows down a lot. ollama ps shows how a loaded model was placed.
You do not always need a GPU. LocalRAG's embedding model runs on a normal processor, and Ollama runs on the CPU too and uses a graphics card if one is set up. Small models (1 to 4 billion parameters) are usable on a CPU; larger ones really want a GPU.
Not every on-device model is a chatbot
SignVox runs a different kind of model on an Android phone: it watches your hand in the camera and recognises fingerspelled letters, then speaks the words. Everything runs on the phone, and the app does not even ask for internet access. See SignVox.
It is also a good lesson in reading accuracy claims. The project publishes two numbers:
| Measured on | Score per letter |
|---|---|
| Held-out photos from the same collections the model trained on | 99.3% |
| A photo collection the model never saw in training (single models) | 69% to 76% |
The first figure flatters the model, as the project itself notes: photos from the same collections look like the training data. A person signing at a real phone, in real light, is different again, and that has not been measured yet. When you see a model's score, ask: measured on what? See How accurate is it?.
Local or cloud?
| Local model | Cloud model | |
|---|---|---|
| Privacy | Prompts and documents never leave your machine | Everything you send goes to the provider |
| Cost | Free per request; you pay for the hardware and electricity | Pay per token, or a subscription |
| Quality | Good for many tasks; the biggest models will not fit | The most capable models are usually here |
| Speed | Depends on your hardware; no network delay | Usually fast; depends on the network and the provider's load |
| Works offline | Yes | No |
| Limits | Your memory and patience | Rate limits, outages, account rules |
Many people mix the two: a local model for private or simple work, a cloud model for hard problems. DL Prism exists for exactly that, and lesson 4 shows how it routes between them.
Stay safe
- Keep the engine on your own computer. Ollama listens on
localhostby default, so other devices cannot reach it. Do not change that on a shared or public network: anyone who can reach the port can use your models. - Download models from sources you trust. A model file is large and you cannot inspect it by eye. Use the engine's official library or the publisher's own page.
- Read the model's licence. "Open" models come with very different licences: some allow almost anything, others restrict commercial use or certain uses. Check before you build a product on one. Lesson 6 has more.
- Watch your disk and memory. Models take gigabytes each. Remove ones you no longer use with
ollama rm, or with Delete in DL Prism's desktop companion.
Check yourself
Question 1 of 6
Your computer has 16 GB of memory and no graphics card. Which of these models is the sensible choice?
What you learned
- An inference engine such as Ollama loads a model and runs it on your machine, at
localhost:11434. - Memory is roughly parameters times bytes per parameter, plus room for the context.
- Quantisation stores numbers in fewer bits: much smaller, slightly worse.
- A GPU is much faster, but only if the model fits in its memory.
- Accuracy numbers depend on what they were measured on.
- Local means private, free per request and offline; cloud usually means more capable and pay per token.