RAG — retrieval-augmented generation — is how you get a language model to answer questions about documents it was never trained on. You do not fine-tune anything: you find the relevant passages first, then hand them to the model along with the question.
Done locally on Linux, this gives you searchable answers over your own runbooks, notes, and configs with nothing leaving the machine. It is also the option people skip in favour of fine-tuning, which is almost always the wrong call for answering questions about documents.
How It Works
| Step | What happens | Component |
|---|---|---|
| 1. Chunk | Documents split into passages | Your script |
| 2. Embed | Each chunk becomes a vector | Embedding model |
| 3. Store | Vectors indexed for similarity search | Vector database |
| 4. Retrieve | Question embedded, nearest chunks found | Vector database |
| 5. Generate | Chunks + question sent to the LLM | Local LLM |
The quality of the whole system is decided at steps 1 and 4, not step 5. A better LLM cannot fix bad retrieval — if the right passage is never retrieved, the model has nothing to work with and will confidently improvise.
Choosing the Pieces
Embedding model — a dedicated embedding model, not a chat model. nomic-embed-text or mxbai-embed-large via Ollama are solid local choices.
Vector store — Chroma for a single-machine project (no server to run), Qdrant when you want a real service, pgvector if you already run PostgreSQL, sqlite-vec for something embedded and tiny.
Generation model — any local model from our Ollama guide. Instruction-following matters more than size here.
If you already run PostgreSQL, pgvector is the pragmatic choice — you get backups, replication, and SQL joins against your existing metadata instead of operating another datastore.
Setup
# Pull an embedding model and a chat model
$ ollama pull nomic-embed-text
$ ollama pull llama3.2
$ python3 -m venv .venv && source .venv/bin/activate
$ pip install chromadb ollama
Indexing Your Documents
import os, pathlib, chromadb, ollama
EMBED_MODEL = "nomic-embed-text"
CHUNK, OVERLAP = 1000, 150 # characters
def chunk_text(text, size=CHUNK, overlap=OVERLAP):
out, start = [], 0
while start < len(text):
out.append(text[start:start + size])
start += size - overlap
return out
client = chromadb.PersistentClient(path="./vectordb")
col = client.get_or_create_collection("docs")
for path in pathlib.Path("documents").rglob("*.md"):
text = path.read_text(errors="ignore")
for i, chunk in enumerate(chunk_text(text)):
vec = ollama.embeddings(model=EMBED_MODEL, prompt=chunk)["embedding"]
col.add(
ids=[f"{path}:{i}"],
embeddings=[vec],
documents=[chunk],
metadatas=[{"source": str(path), "chunk": i}],
)
print("indexed", path)
Store the source in metadata — without it you cannot cite where an answer came from — and an answer you cannot verify is not much use in operations.
Stable IDs matter — using path:index as the ID makes re-indexing idempotent instead of duplicating chunks.
Querying
import chromadb, ollama
EMBED_MODEL, CHAT_MODEL = "nomic-embed-text", "llama3.2"
client = chromadb.PersistentClient(path="./vectordb")
col = client.get_collection("docs")
def ask(question, k=4):
qvec = ollama.embeddings(model=EMBED_MODEL, prompt=question)["embedding"]
res = col.query(query_embeddings=[qvec], n_results=k)
chunks = res["documents"][0]
sources = [m["source"] for m in res["metadatas"][0]]
context = "\n\n---\n\n".join(chunks)
prompt = (
"Answer the question using ONLY the context below. "
"If the context does not contain the answer, say so explicitly.\n\n"
f"Context:\n{context}\n\nQuestion: {question}"
)
reply = ollama.chat(model=CHAT_MODEL, messages=[{"role": "user", "content": prompt}])
return reply["message"]["content"], sources
answer, sources = ask("How do we rotate the database credentials?")
print(answer)
print("\nSources:", *set(sources), sep="\n ")
The instruction to answer only from context and to admit ignorance is not optional. Without it, the model fills gaps from its training data and you cannot tell which sentences came from your documents.
Chunking: Where Quality Is Won or Lost
Too small — a 200-character chunk loses the surrounding context, so retrieved passages are technically relevant but useless to answer with.
Too large — a whole document as one chunk dilutes the embedding — the vector averages many topics and matches nothing precisely.
Reasonable default — 500–1500 characters with 10–20% overlap. Overlap prevents an answer that straddles a boundary from being cut in half.
Respect structure — splitting on headings or paragraphs beats splitting on a fixed character count. For Markdown runbooks, split per section.
Keep the heading with the chunk — prefixing each chunk with its document title and section makes both embedding and generation noticeably better.
Never mix embedding models — vectors from different models are not comparable. Changing the embedding model means re-indexing everything.
Knowing Whether It Works
The failure mode of RAG is quiet: plausible answers built from the wrong passages. Test retrieval separately from generation:
# Inspect what retrieval actually returns — before any LLM involvement
res = col.query(query_embeddings=[qvec], n_results=5)
for doc, meta, dist in zip(res["documents"][0], res["metadatas"][0], res["distances"][0]):
print(f"{dist:.4f} {meta['source']} {doc[:80]!r}")
Build a question set — write 20 real questions with the document you expect to be retrieved. Measure how often it appears in the top results. That number is your actual system quality.
Watch the distances — if the best match is barely closer than the worst, retrieval is not discriminating and the answer is guesswork.
Raise k before blaming the model — retrieving 8 chunks instead of 3 often fixes a wrong answer, at the cost of context length.
Log question, retrieved sources, and answer — without this you cannot diagnose complaints after the fact.
Running It as a Service
Two moving parts on a server: a re-index job and the query API. Re-indexing suits a cron job or systemd timer:
# /etc/systemd/system/rag-reindex.service
[Unit]
Description=Re-index documents for RAG
[Service]
Type=oneshot
User=rag
WorkingDirectory=/opt/rag
ExecStart=/opt/rag/.venv/bin/python index.py
# /etc/systemd/system/rag-reindex.timer
[Unit]
Description=Nightly RAG re-index
[Timer]
OnCalendar=daily
Persistent=true
[Install]
WantedBy=timers.target
Persistent=true makes the job run after a missed window — important on machines that are not always on. Our systemd timer generator can scaffold these.
Common Mistakes
Fine-tuning instead of RAG — fine-tuning teaches style and format, not facts you can update. For "answer questions about these documents", RAG is the correct tool and far cheaper.
Using a chat model for embeddings — use a purpose-built embedding model. Quality difference is large.
Never re-indexing — a stale index answers from documents that no longer reflect reality — worse than no system at all.
No citations — always return sources. An unverifiable answer about production infrastructure is a liability.
Ignoring context limits — retrieving 20 chunks can exceed the model's context, silently truncating the very passage that mattered.
Assuming semantic search finds exact strings — embeddings match meaning, not literal tokens. For error codes and identifiers, combine with keyword search rather than replacing it.
Frequently Asked Questions
Is RAG better than fine-tuning for my own documents?
For answering questions about documents, yes. RAG lets you update knowledge by re-indexing, cite sources, and run on modest hardware. Fine-tuning is for teaching style, format, or task behaviour — not for injecting facts you will need to change later.
What chunk size should I use for RAG?
Start with 500–1500 characters and 10–20% overlap. Splitting on document structure such as headings or paragraphs works better than a fixed character count, and prefixing each chunk with its title and section improves both retrieval and answers.
Which vector database should I use for local RAG?
Chroma for a single-machine project since it needs no server, Qdrant when you want a proper service, and pgvector if you already run PostgreSQL — that gives you existing backups and SQL joins against your metadata.
Can I run RAG completely offline?
Yes. With Ollama providing both the embedding and chat models and a local vector store, nothing leaves the machine after the initial model download. That is the main reason to build RAG locally rather than using a hosted API.
Why does my RAG system give confidently wrong answers?
Almost always retrieval, not the model. Inspect what the query actually returns before the LLM sees it. Common causes are chunks that are too large or too small, a missing instruction to answer only from context, or too few chunks retrieved.