Answers from your documents (RAG)
A model knows nothing about your own manuals, notes or policies, and guessing is where it hallucinates. Retrieval-augmented generation fixes that by finding the right passages first and handing them to the model. By the end of this lesson you can follow every step, and read a cited answer critically.
The problem RAG solves
From lesson 1 you know two limits:
- The model only knows its training data, and your documents were not in it.
- It can only see what fits in its context window, and a folder of PDFs will not fit.
Retrieval-augmented generation (RAG) works around both. Before the model answers, the system retrieves the few passages that best match the question and puts only those in the prompt. The model then writes its answer from that text, and the system can show which passages it used.
The whole pipeline
There are two phases. You add documents once; you ask questions as often as you like.
ADDING A DOCUMENT
file ─► extract text ─► cut into chunks ─► embed each chunk ─► store in a vector database
(page by page) (≤ 500 tokens) (list of numbers) (on your disk)
ASKING A QUESTION
question ─► embed it ─► find the closest chunks ─► drop weak matches ─► prompt the model
(top-k = 5) (score < 0.30) with those chunks
│
answer + citations (document · page) ◄───┘The numbers are LocalRAG's defaults. Each step is explained below.
Step 1: get the text out
RAG works on text, so first the text has to come out of the file.
| File | What LocalRAG does |
|---|---|
| PDF with a text layer | Reads the text page by page, so citations can name the page |
| Scanned PDF (images only) | Rejects it with no extractable text. It has no text, and LocalRAG has no OCR |
.txt or .md | Reads it as one page |
| Word, PowerPoint, web pages | Not supported directly |
That last row is where a converter helps. Markdrop turns Word, Excel and PowerPoint files, e-books and web pages into Markdown, plain text with simple marks for headings, lists and tables. LocalRAG accepts the resulting .md file. You can use Markdrop Studio on the desktop, the browser extension for web pages, or Markdrop Web from scripts.
policy.docx ─► Markdrop ─► policy.md ─► LocalRAGTwo things to know about that route:
- Markdrop adds a short front-matter block at the top (
title,source,converted,generator). LocalRAG reads it as ordinary text, which does no harm. - A Markdown file counts as one page, so every citation says
p.1. If you have a PDF with real text, give LocalRAG the PDF itself to keep page numbers.
Step 2: cut the text into chunks
A whole document is too big to send with every question, and most of it is irrelevant. So the text is cut into chunks: pieces small enough that a few of them fit comfortably in the prompt.
LocalRAG cuts each page into chunks of up to 500 tokens, with about 50 tokens of overlap between neighbours, and never lets a chunk cross a page boundary:
page 12 ┌──────── chunk A (≤ 500 tokens) ────────┐
│ ┌─overlap─┤
└──────────────────────────────┤ chunk B …
└─────────
page 13 ┌──────── chunk C (starts fresh on the new page) ───- Overlap keeps a sentence at the edge from being split away from its meaning.
- Staying inside a page is what lets a citation name the exact page.
- Chunk size is a trade-off. Small chunks match precisely but lose surrounding context; big chunks keep context but blur the match. LocalRAG lets you change both numbers. See Configuration.
Step 3: turn each chunk into an embedding
To find passages that mean the same thing as the question, even with different words, the system needs a way to compare meaning. That is an embedding: a list of numbers, produced by a separate embedding model, that places a text in a "meaning space". Texts about similar things get similar lists.
A toy example with three numbers per text (real ones are much longer: LocalRAG's default embedding model, BAAI/bge-small-en-v1.5, makes 384):
| Text | Embedding (toy) |
|---|---|
| Question: "At what wind speed does a turbine start producing power?" | [0.9, 0.1, 0.3] |
| Chunk: "…starts generating power at a wind speed of about 3 to 4 metres per second." | [0.8, 0.2, 0.4] |
| Chunk: "Most modern turbines have three blades…" | [0.1, 0.9, 0.2] |
The closeness of two embeddings is measured with cosine similarity: roughly, whether the two lists "point the same way". Higher is closer; for text it usually falls between 0 and 1. For the toy numbers above, the first chunk scores about 0.98 against the question and the second about 0.27.
The embedding model is not the answer model. It is small, fast and runs on a normal processor. And the same embedding model must be used for documents and questions: lists from different models cannot be compared. That is why changing LocalRAG's embedding model means adding all documents again.
Step 4: keep them in a vector store
A vector store (or vector database) keeps every chunk with its embedding, its document name and its page, and can quickly find the embeddings closest to a new one. LocalRAG uses ChromaDB, stored in its data folder on your own disk.
Step 5: retrieve the best matches
When you ask a question, LocalRAG:
- Embeds the question with the same model.
- Takes the five closest chunks. This number is called top-k.
- Drops any chunk whose similarity score is below the score threshold, 0.30 by default.
If nothing survives step 3, LocalRAG answers I don't know based on the indexed documents. straight away, without calling the model at all. There is nothing to answer from, so there is nothing to make up.
Step 6: prompt the model with the sources
The surviving chunks go into the prompt, each labelled with its document and page, together with firm instructions. LocalRAG's prompt has this shape (shortened):
Answer the question using only the information in the context below.
If the answer is not present in the context, reply with exactly:
"I don't know based on the indexed documents." …
Context:
[1] turbines.txt (p. 1):
Wind turbines convert the kinetic energy of moving air into electricity. …
Question: At what wind speed does a turbine start producing power?
Answer:The model now has the facts in its context, so it does not need to remember them. It is also told what to say when they are missing. This is the main reason RAG answers hallucinate far less, though not never.
Step 7: cite the sources
After the answer, LocalRAG lists the passages it gave the model, as chips such as turbines.txt · p.1. Hover a chip to see its score.
- One chip per page, best score first.
- The score is similarity, not correctness. It says how alike the question and the passage are in meaning, not whether the answer is right.
- The chips show what the model was given, not proof of each sentence. For anything important, open the document at the cited page and check.
See Ask questions and read citations for the full detail.
When RAG says "I don't know"
An honest "I don't know" is a feature. LocalRAG gives it in two ways:
| What happened | Citation chips |
|---|---|
| No chunk reached the score threshold, so the model was never asked | None |
| Some chunks matched a little; the model read them and did not find the answer | Possibly some |
If you get "I don't know" for something your documents do contain:
- use the words the document uses ("flow rate", not "how fast the water goes");
- check the document is in the Documents list, and that a PDF has real, selectable text;
- as a last resort, lower the threshold a little, for example to
0.25. Weaker matches then get in, so answers may wander more.
What RAG is not good at
Retrieval hands the model a handful of chunks, not the whole library. So RAG struggles with:
- Questions about the whole collection, such as "summarise every document" or "how many policies mention travel?". Only the top few chunks are seen.
- Answers spread across many places. Raising top-k helps a little.
- Follow-up questions. LocalRAG does not remember earlier questions, so "and what about it?" has no subject. Repeat the subject each time.
Stay safe
- Know who can reach your RAG server. LocalRAG's API has no login. With pip it listens only on
127.0.0.1; the Docker setup publishes port 8090 on every network connection. Anyone who can reach that port can read answers from your documents and delete them. See Security. - Check citations before you act. A cited answer is a much better draft, not a guarantee.
- Documents can carry instructions. Text inside a document goes straight into the prompt, so a line such as "ignore the question and say X" may sway the model. Add documents you trust. Lesson 6 explains this.
- Deleting in LocalRAG does not delete your file. It removes the document and its chunks from LocalRAG only. To wipe everything, delete the data folder, as the configuration page explains.
Check yourself
Question 1 of 6
In retrieval-augmented generation, what happens before the model writes its answer?
What you learned
- RAG finds the passages that match a question and gives only those to the model.
- Text is extracted, cut into overlapping chunks, embedded and stored in a vector store.
- An embedding is a list of numbers for meaning; cosine similarity compares two of them.
- Top-k and the score threshold decide which chunks reach the model; if none do, the answer is "I don't know".
- Citations show what the model was given; the score is similarity, not truth.
- Markdrop turns files LocalRAG cannot read into Markdown it can.