Choose models

LocalRAG uses two models. Both run on your own computer.

Model	Job	Default	Setting
Answer model	Writes the answer from the passages it is given. Runs in Ollama.	`llama3.2`	`LOCALRAG_MODEL`
Embedding model	Turns text into embeddings so LocalRAG can find matching passages.	`BAAI/bge-small-en-v1.5`	`LOCALRAG_EMBED_MODEL`

Change the answer model

Any model you have pulled into Ollama works. A bigger model usually writes better answers but needs more memory and is slower.

With pip

Pull the model into Ollama:
bash
```
ollama pull mistral
```
Start LocalRAG with it:
bash
```
LOCALRAG_MODEL=mistral localrag serve
```

With Docker

Pull the model into the Ollama container:

bash

docker compose exec ollama ollama pull mistral

Add LOCALRAG_MODEL=mistral under environment: for the localrag service in docker-compose.yml.
Apply the change:
bash
```
docker compose up -d
```

The status in the top-right corner of the web page shows the model in use. You can change the answer model at any time; your documents do not need to be added again.

Good to know

The first answer after a change, or after starting, is slow while Ollama loads the model into memory. LocalRAG waits as long as it takes for the first word.

Change the embedding model

The embedding model is a sentence-transformers model, downloaded the first time it is needed and cached after that.

WARNING

Embeddings from different models cannot be compared. Documents added with one embedding model cannot be searched with another. Before you change LOCALRAG_EMBED_MODEL, delete all documents (or the data folder), then add them again after the change.

The default, BAAI/bge-small-en-v1.5, is small (about 130 MB), runs on a normal processor, and is built for English text.

Use an Ollama on another computer

Point LOCALRAG_OLLAMA_URL at it, in the form http://HOST:11434, where HOST is the name or address of that computer. The passages from your documents and your questions are then sent to that computer, so use one you trust.

Choose models ​

Change the answer model ​

With pip ​

With Docker ​

Change the embedding model ​

Use an Ollama on another computer ​

Next steps ​