A Raspberry Pi can run real AI workloads — just not the ones most people ask about first. Chat-style LLM use is technically possible and practically frustrating. Wake-word detection, object detection, transcription, and classification are genuinely useful and run well.
This is a realistic guide to what works, what to set up, and where the ceilings are.
What Runs Usefully
| Workload | Viability on Pi 5 | Notes |
|---|---|---|
| Wake word / keyword spotting | Excellent | Tiny models, real-time, low power |
| Speech-to-text (Whisper tiny/base) | Good | Faster than real-time on short clips |
| Image classification | Good | Especially with an accelerator |
| Object detection (small models) | Workable | Accelerator strongly recommended for video |
| Text embeddings / semantic search | Good | Small embedding models are cheap to run |
| LLM chat (1–3B params) | Slow but real | A few tokens per second; fine for automation |
| LLM chat (7B+) | Impractical | Memory-bound; expect painful latency |
| Any training or fine-tuning | No | Use a real GPU machine |
The pattern: the Pi is good at small models doing narrow jobs, and poor at large models doing general ones. That maps well onto home automation, kiosks, and sensors — and badly onto replacing a chat assistant.
Hardware That Matters
Pi 5 over Pi 4 — the CPU and memory bandwidth improvement is large, and memory bandwidth is the limiting factor for LLM generation.
RAM is the hard limit — an 8 GB or larger board opens up meaningfully bigger models. On 4 GB you are restricted to the smallest ones.
NVMe over SD card — a Pi 5 with an NVMe HAT loads multi-gigabyte models in seconds instead of minutes, and avoids wearing out flash.
Active cooling is mandatory — sustained inference is a sustained 100% CPU load. Without a fan the Pi throttles and your throughput quietly halves.
Power supply — use the official supply. Brownouts under sustained load cause corruption that looks like software bugs.
Running an LLM with Ollama
Ollama installs on ARM64 the same way as on x86 and is the simplest path:
$ curl -fsSL https://ollama.com/install.sh | sh
# Stick to small models — 1B to 3B parameter class
$ ollama run llama3.2:1b
# Check what is loaded and how much memory it took
$ ollama ps
Set expectations — generation runs at a few tokens per second on 1–3B models. That is unusable for interactive chat and perfectly fine for a script that classifies a sensor reading or summarizes a short log.
Keep the model resident — set OLLAMA_KEEP_ALIVE high so you do not pay model load time from disk on every request.
Reduce context — a large context window costs memory the Pi does not have to spare. Keep it modest.
Speech-to-Text with whisper.cpp
This is where a Pi genuinely shines — a small always-on transcription box. whisper.cpp builds cleanly on ARM:
$ sudo apt install -y build-essential cmake git ffmpeg
$ git clone https://github.com/ggml-org/whisper.cpp && cd whisper.cpp
$ cmake -B build && cmake --build build -j 4
$ ./models/download-ggml-model.sh base.en
$ ffmpeg -i note.m4a -ar 16000 -ac 1 -c:a pcm_s16le note.wav
$ ./build/bin/whisper-cli -m models/ggml-base.en.bin -f note.wav -t 4
The tiny.en and base.en models are the sweet spot here. See our Whisper guide for output formats and batch scripting.
Adding an AI Accelerator
For vision work, an accelerator changes the picture completely — it moves inference off the CPU entirely, so the Pi stays responsive and power draw stays low:
| Option | Interface | Good for |
|---|---|---|
| Raspberry Pi AI HAT (Hailo) | PCIe (Pi 5) | Real-time object detection on video |
| Coral USB Accelerator | USB | TensorFlow Lite classification/detection |
| No accelerator | — | Audio, embeddings, small LLMs, still images |
Important limitation — these accelerators run compiled vision-style models, not general LLMs. An AI HAT will not speed up Ollama. Match the accelerator to the workload before buying.
Models must be converted — each accelerator needs models compiled for its runtime. Check that the model you want is supported before committing.
Tuning the Pi for Inference
# Watch throttling and temperature during a run
$ vcgencmd measure_temp
$ vcgencmd get_throttled # 0x0 means no throttling has occurred
# Confirm you are on 64-bit — required for most AI tooling
$ uname -m # should print aarch64
# Add zram so a brief memory spike does not kill the process
$ sudo apt install -y zram-tools
# Watch memory during inference
$ watch -n1 free -h
get_throttled matters — a non-zero value means the Pi has thermally or electrically throttled. Any benchmark taken while throttled is meaningless.
Use 64-bit Raspberry Pi OS — 32-bit builds cannot address enough memory and much AI tooling ships aarch64 binaries only.
Swap is a safety net, not a strategy — a model that needs to swap will be unusably slow. Size the model to fit RAM.
Match threads to cores — pass -t 4 on a 4-core Pi; oversubscribing threads makes things slower, not faster.
Projects That Make Sense
Offline voice notes — a systemd path unit that transcribes any audio file dropped into a watched directory.
Doorbell / camera detection — object detection with an accelerator, triggering a notification rather than recording everything.
Local log triage — a small model that classifies log lines and only escalates the interesting ones.
Semantic search over your notes — embeddings plus a small vector store — cheap to run and genuinely useful.
Sensor anomaly flagging — classification on time-series data, no LLM needed.
Private smart-speaker frontend — wake word plus local transcription on the Pi, handing heavier work to a GPU box on your LAN.
That last pattern is the most practical use of a Pi in an AI setup: the Pi handles capture and small models, and forwards anything heavy to a machine with a GPU running a real inference server.
Frequently Asked Questions
Can a Raspberry Pi run an LLM?
Yes, in the 1–3B parameter class using Ollama or llama.cpp, at a few tokens per second. That is too slow for interactive chat but workable for scripted tasks like classification or short summaries. 7B and larger models are impractical.
Which Raspberry Pi is best for AI work?
A Pi 5 with as much RAM as you can get, active cooling, and NVMe storage. Memory bandwidth and RAM capacity are the binding constraints, and sustained inference will thermally throttle a passively cooled board.
Does the Raspberry Pi AI HAT speed up LLMs?
No. Accelerators like the Hailo-based AI HAT and Coral USB run compiled vision-style models, not general language models. They dramatically help object detection and classification but do nothing for Ollama or llama.cpp.
Is Whisper usable on a Raspberry Pi?
Yes — this is one of the best fits. whisper.cpp with the tiny.en or base.en model transcribes short clips faster than real time on a Pi 5, making it practical for voice notes and always-on transcription.
Can I train or fine-tune models on a Raspberry Pi?
No. Training needs far more memory and compute than a Pi has. Train on a machine with a GPU and deploy the resulting small model to the Pi for inference.