Fine-tuning has a reputation as the answer to every LLM shortcoming. It is not. It changes how a model responds — format, tone, task behaviour — and is a poor and expensive way to teach it facts, because facts change and retraining does not scale.
When fine-tuning genuinely is the right tool, LoRA and QLoRA make it possible on a single consumer GPU instead of a cluster. This covers when to use it, how to run it, and how to get the result into Ollama.
Decide Before You Train
| Goal | Right tool |
|---|---|
| Answer questions about my documents | RAG — see our local RAG guide |
| Follow a strict output format every time | Fine-tuning (or a good prompt first) |
| Adopt a domain's tone and vocabulary | Fine-tuning |
| Learn an internal task or workflow | Fine-tuning |
| Know facts that change monthly | RAG — never fine-tuning |
| Be smarter in general | A larger base model |
Try prompting first, seriously — a careful system prompt with two or three examples solves a surprising share of what people reach for fine-tuning to fix — at zero training cost and with instant iteration.
What LoRA and QLoRA Actually Do
Full fine-tuning updates every weight, which needs memory for the weights, their gradients, and optimizer state — far beyond a consumer card for anything but tiny models.
LoRA — freezes the base model and trains small low-rank adapter matrices injected into specific layers. You update a tiny fraction of the parameters, so gradients and optimizer state shrink accordingly.
QLoRA — loads the frozen base model in 4-bit and trains LoRA adapters on top. This is what brings 7B–13B fine-tuning onto a single consumer GPU.
The adapter is small — a LoRA adapter is megabytes, not gigabytes. You can keep several for different tasks and swap them against one base model.
| Approach | Rough VRAM for a 7B model | Quality |
|---|---|---|
| Full fine-tune | Far beyond consumer cards | Best, rarely necessary |
| LoRA (16-bit base) | High — needs a large card | Very close to full |
| QLoRA (4-bit base) | Fits mainstream GPUs | Close to LoRA for most tasks |
Preparing the Dataset
Format matters less than consistency. A JSONL file of instruction/response pairs is the common shape:
{"instruction": "Summarize this alert for an on-call engineer.", "input": "DiskPressure on node-7, 92% used on /var", "output": "node-7 is at 92% disk on /var and under DiskPressure. Clear logs or expand the volume; pods may be evicted."}
{"instruction": "Summarize this alert for an on-call engineer.", "input": "OOMKilled: api-gateway restarted 4x in 10m", "output": "api-gateway is being OOM-killed and has restarted 4 times in 10 minutes. Raise its memory limit or investigate a leak."}
Consistency beats volume — a few hundred examples that all follow the same format teach that format. A few thousand inconsistent ones teach inconsistency.
Include the hard cases — if you only train on clean inputs, the model will not handle messy ones.
Hold back a validation split — 10% you never train on. Without it you cannot tell learning from memorizing.
Match the chat template — apply the base model's own template. A mismatch between training and inference formatting is the most common cause of a fine-tune that seems to have learned nothing.
Running a QLoRA Fine-Tune
$ python3 -m venv .venv && source .venv/bin/activate
$ pip install torch --index-url https://download.pytorch.org/whl/cu124
$ pip install transformers peft trl datasets bitsandbytes accelerate
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import LoraConfig
from trl import SFTTrainer, SFTConfig
import torch
BASE = "meta-llama/Llama-3.2-3B-Instruct"
quant = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(
BASE, quantization_config=quant, device_map="auto", torch_dtype=torch.bfloat16
)
peft_config = LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
bias="none",
task_type="CAUSAL_LM",
target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj"],
)
ds = load_dataset("json", data_files={"train": "train.jsonl", "test": "val.jsonl"})
trainer = SFTTrainer(
model=model,
train_dataset=ds["train"],
eval_dataset=ds["test"],
peft_config=peft_config,
processing_class=tok,
args=SFTConfig(
output_dir="./out",
num_train_epochs=3,
per_device_train_batch_size=1,
gradient_accumulation_steps=8, # effective batch size 8
gradient_checkpointing=True, # trades compute for memory
learning_rate=2e-4,
bf16=True,
logging_steps=10,
eval_strategy="epoch",
save_strategy="epoch",
max_length=1024,
),
)
trainer.train()
trainer.save_model("./out/adapter")
The Hyperparameters That Matter
r (rank) — adapter capacity. 8–16 handles most style and format tasks; higher only if the model is clearly failing to learn. Larger r means more parameters and more memory.
lora_alpha — scaling for the adapter. Setting it to roughly 2× r is a widely used convention.
target_modules — which projections get adapters. Attention-only is lighter; including the MLP projections generally learns more.
learning_rate — LoRA tolerates much higher rates than full fine-tuning — around 1e-4 to 2e-4 is a normal starting range.
gradient_accumulation_steps — the memory lever. Batch size 1 with accumulation 8 behaves like batch 8 without the memory cost.
gradient_checkpointing — recomputes activations instead of storing them. Meaningfully slower, and often the difference between fitting and OOM.
num_train_epochs — 2–3 is typical on a small dataset. Watch validation loss — when it rises while training loss falls, you are overfitting.
Using the Result
You can load the adapter alongside the base model, or merge it into standalone weights. Merging is what you want in order to convert to GGUF:
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained(
"meta-llama/Llama-3.2-3B-Instruct", torch_dtype="auto", device_map="cpu"
)
merged = PeftModel.from_pretrained(base, "./out/adapter").merge_and_unload()
merged.save_pretrained("./merged")
AutoTokenizer.from_pretrained("meta-llama/Llama-3.2-3B-Instruct").save_pretrained("./merged")
Then convert and quantize with llama.cpp, and register it with Ollama:
# In your llama.cpp checkout
$ python3 convert_hf_to_gguf.py ./merged --outfile custom-f16.gguf --outtype f16
$ ./build/bin/llama-quantize custom-f16.gguf custom-Q4_K_M.gguf Q4_K_M
# Modelfile
$ printf 'FROM ./custom-Q4_K_M.gguf\n' > Modelfile
$ ollama create my-model -f Modelfile
$ ollama run my-model
Merging must happen at 16-bit, not against the 4-bit quantized base — merge first, then quantize. Details on formats are in our quantization guide.
Troubleshooting
CUDA out of memory — in order: reduce max_length, set batch size 1 and raise accumulation, enable gradient checkpointing, lower r, then use a smaller base model.
Loss does not decrease — usually the chat template is not applied, so the model sees malformed examples. Print one fully formatted training sample and inspect it.
Model learned the style but forgot everything else — catastrophic forgetting from too many epochs or too high a learning rate. Fewer epochs, lower rate, more diverse data.
Validation loss rising — overfitting. Stop at the best checkpoint — this is exactly what the held-back split is for.
Output is fine in training but broken in Ollama — a template mismatch between training and serving. Ensure the Modelfile template matches how you formatted training data.
Merge produces garbage — you merged into a quantized base. Load the base in 16-bit for merging.
Training is extremely slow — confirm the GPU is actually being used — see our GPU monitoring guide; silent CPU fallback is common.
Frequently Asked Questions
Should I fine-tune or use RAG?
Use RAG to answer questions about documents or any facts that change — you update it by re-indexing. Use fine-tuning to change behaviour: enforcing an output format, adopting a domain's tone, or learning an internal task. Fine-tuning is a poor way to store facts.
What is the difference between LoRA and QLoRA?
LoRA freezes the base model and trains small low-rank adapters. QLoRA does the same but loads the frozen base in 4-bit, cutting memory further. QLoRA is what makes fine-tuning 7B-class models practical on a single consumer GPU.
How much VRAM do I need to fine-tune an LLM?
With QLoRA, mainstream consumer GPUs can fine-tune models in the 3B–8B range, and larger cards handle more. The levers that reduce memory are sequence length, batch size with gradient accumulation, gradient checkpointing, and the LoRA rank.
How many training examples do I need?
For format and style tasks, a few hundred consistent examples often suffice. Consistency matters far more than volume — inconsistent data teaches inconsistency. Always hold back around 10% as a validation split.
Why does my fine-tuned model behave differently in Ollama than in training?
The chat template differs between training and serving. Make sure the Modelfile template matches the format used to build training examples — template mismatch is the most common reason a fine-tune appears to have had no effect.