Whisper is the practical local alternative to cloud transcription APIs: it runs entirely offline, handles many languages, and produces subtitle files directly. On Linux it fits naturally into pipelines — ffmpeg in, SRT out, no account required and nothing uploaded.
This guide covers the three implementations worth knowing, how to pick a model size, the audio format requirement that trips everyone up, and how to batch-transcribe a directory.
Which Implementation?
| Project | Runs on | Best for |
|---|---|---|
whisper.cpp | CPU, CUDA, Vulkan, Metal | Single binary, no Python, great on CPU |
faster-whisper | CPU, CUDA (CTranslate2) | Highest speed on GPU; Python library |
openai-whisper | CPU, CUDA (PyTorch) | Reference implementation; heaviest install |
For command-line and server use, whisper.cpp is usually the right default — no Python environment to maintain and it performs well on plain CPUs. Choose faster-whisper when you have an NVIDIA GPU and are transcribing at volume from Python.
Building whisper.cpp
$ sudo apt install -y build-essential cmake git ffmpeg
$ git clone https://github.com/ggml-org/whisper.cpp
$ cd whisper.cpp
# CPU build
$ cmake -B build && cmake --build build -j $(nproc)
# NVIDIA GPU build
$ cmake -B build -DGGML_CUDA=ON && cmake --build build -j $(nproc)
# Download a model (see sizes below)
$ ./models/download-ggml-model.sh base.en
Models land in models/ as ggml-*.bin files, and the binaries in build/bin/. The same GGML backends as llama.cpp apply here, so a Vulkan or ROCm build works the same way.
Choosing a Model Size
| Model | Relative speed | Use it when |
|---|---|---|
| tiny | Fastest | Real-time on weak hardware; accuracy is rough |
| base | Fast | Clear audio, quick drafts, Raspberry Pi |
| small | Moderate | Good general-purpose accuracy/speed balance |
| medium | Slow | Accented speech, noisy recordings |
| large | Slowest | Best accuracy; multilingual work |
The .en suffix — English-only variants (base.en, small.en) are more accurate than the multilingual model of the same size for English audio. Use them when you only need English.
Start at small — it is the size most people settle on. Move up only if the transcript quality is actually failing you.
The Audio Format Requirement
whisper.cpp expects 16 kHz mono 16-bit WAV. Feeding it an MP3 or a 48 kHz stereo file is the single most common failure. Convert first — ffmpeg handles anything:
$ ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le output.wav
# Straight from a video file
$ ffmpeg -i lecture.mp4 -ar 16000 -ac 1 -c:a pcm_s16le lecture.wav
# -ar 16000 resample to 16 kHz
# -ac 1 downmix to mono
# -c:a pcm_s16le 16-bit signed little-endian PCM
Transcribing
# Basic transcription to stdout
$ ./build/bin/whisper-cli -m models/ggml-base.en.bin -f audio.wav
# Write subtitle and text files next to the input
$ ./build/bin/whisper-cli -m models/ggml-small.en.bin -f talk.wav --output-srt --output-txt
# JSON with timestamps, for scripting
$ ./build/bin/whisper-cli -m models/ggml-small.en.bin -f talk.wav --output-json
# Non-English audio, and translation to English
$ ./build/bin/whisper-cli -m models/ggml-medium.bin -f interview.wav -l de
$ ./build/bin/whisper-cli -m models/ggml-medium.bin -f interview.wav -l de --translate
# Use all cores
$ ./build/bin/whisper-cli -m models/ggml-small.en.bin -f talk.wav -t $(nproc)
Batch Transcribing a Directory
This is where local transcription pays off — an afternoon of recordings processed overnight with no API bill:
#!/usr/bin/env bash
set -euo pipefail
MODEL="models/ggml-small.en.bin"
BIN="./build/bin/whisper-cli"
OUT="transcripts"
mkdir -p "$OUT"
shopt -s nullglob
for f in recordings/*.{mp3,m4a,mp4,wav}; do
base=$(basename "${f%.*}")
[[ -f "$OUT/$base.srt" ]] && { echo "skip $base"; continue; }
echo ">> $base"
ffmpeg -loglevel error -y -i "$f" -ar 16000 -ac 1 -c:a pcm_s16le "/tmp/$base.wav"
"$BIN" -m "$MODEL" -f "/tmp/$base.wav" --output-srt --output-txt -of "$OUT/$base"
rm -f "/tmp/$base.wav"
done
The skip-if-exists check makes the script resumable — important when transcribing hundreds of files, since a crash three hours in should not mean starting over.
faster-whisper for GPU Throughput
When you have an NVIDIA GPU and volume to process, faster-whisper (built on CTranslate2) is the fastest common option:
$ python3 -m venv .venv && source .venv/bin/activate
$ pip install faster-whisper
from faster_whisper import WhisperModel
model = WhisperModel("small.en", device="cuda", compute_type="float16")
segments, info = model.transcribe("audio.wav", beam_size=5)
print(f"language={info.language} duration={info.duration:.1f}s")
for s in segments:
print(f"[{s.start:.2f} -> {s.end:.2f}] {s.text}")
compute_type — use float16 on GPU and int8 on CPU for a substantial speedup at minor accuracy cost.
It accepts most formats — faster-whisper decodes audio itself, so the strict 16 kHz WAV requirement does not apply as it does with whisper.cpp.
Practical Uses on a Linux Box
Meeting notes — transcribe a recording, then pipe the text into a local LLM for a summary — fully offline.
Subtitle generation — the --output-srt flag produces files that video players and editors read directly.
Searchable archives — transcribe a podcast or lecture library once and grep it forever.
Voice notes on a server — a cron job or systemd path unit that transcribes anything dropped in a watched directory.
Accessibility — captioning internal recordings without sending audio to a third party.
Troubleshooting
"failed to read audio" / silence in output — the input is not 16 kHz mono WAV. Convert with ffmpeg first — this is the most common issue by far.
Repeated or looping phrases — typical on silence or music. Trim non-speech sections, or try a larger model.
Wrong language detected — set it explicitly with -l rather than relying on auto-detection on short clips.
Very slow on CPU — use an .en model one size down, pass -t $(nproc), and prefer small over medium.
GPU build still uses CPU — check the startup log line naming the backend; a stale build directory configured without the GPU flag is the usual cause.
No speaker labels — Whisper does not do diarization. You need a separate diarization step to attribute speakers.
Frequently Asked Questions
Does Whisper run fully offline on Linux?
Yes. After the model file is downloaded, transcription is entirely local — no network access and no data leaves the machine. That makes it suitable for confidential recordings that cannot go to a cloud API.
Why does whisper.cpp require 16 kHz mono WAV?
The model is trained on 16 kHz mono audio and whisper.cpp does not resample internally. Convert with ffmpeg using -ar 16000 -ac 1 -c:a pcm_s16le. Silent or garbled output is almost always a format problem.
Which Whisper model size should I use?
Start with small, or small.en for English-only audio. Move up to medium or large for accented speech, noisy recordings, or multilingual work, and down to base or tiny when you need speed on weak hardware.
Do I need a GPU for Whisper?
No. whisper.cpp performs well on modern CPUs, especially with the smaller and English-only models. A GPU mainly matters when transcribing at volume, where faster-whisper with float16 gives the largest speedup.
Can Whisper identify who is speaking?
No. Whisper transcribes speech but does not perform speaker diarization. Identifying speakers requires a separate diarization tool run alongside or after transcription.