Local models are the only realistic option for genuinely sensitive work — classified environments, isolated OT networks, regulated data that legally cannot leave a boundary. The models themselves run offline happily. The hard part is getting them in, because every tool in the ecosystem assumes it can reach the internet.
This covers transferring models, Python dependencies and container images across an air gap, verifying what you moved, and keeping it maintainable.
The Approach
You need a staging host with internet access running the same OS and architecture as the target, plus removable media or a one-way transfer mechanism. Everything is downloaded, verified, and packaged on the staging host, then moved.
| What you move | Method |
|---|---|
| LLM weights | GGUF file + Modelfile, or an Ollama blob directory |
| Python packages | pip download wheelhouse |
| Container images | docker save / docker load |
| System packages | Downloaded .deb/.rpm set or a local mirror |
| The runtime itself | Ollama or llama.cpp binary/tarball |
Match the architecture — a wheel or binary built on x86-64 will not run on ARM. Stage on hardware matching the target, or download explicitly for the target platform.
Model licences travel too — many open-weight models carry usage restrictions. Record the licence with the weights — in a controlled environment, someone will eventually ask.
Transferring an LLM
The cleanest method — and the one that survives version changes — is to move a plain .gguf file and register it with a Modelfile on the target. No internal directory layout to depend on:
# --- On the staging host ---
# Download the GGUF file directly from the model publisher
$ curl -L -o model-Q4_K_M.gguf "<url-from-publisher>"
# Record a checksum to verify after transfer
$ sha256sum model-Q4_K_M.gguf | tee model-Q4_K_M.gguf.sha256
# --- On the air-gapped host ---
$ sha256sum -c model-Q4_K_M.gguf.sha256 # MUST pass before proceeding
$ printf 'FROM ./model-Q4_K_M.gguf\n' > Modelfile
$ ollama create local-llm -f Modelfile
$ ollama run local-llm
The alternative is copying Ollama's model store wholesale, which is faster for many models but ties you to its on-disk layout:
# --- Staging host: pull, then archive the model store ---
$ ollama pull llama3.2
$ sudo tar -C /usr/share/ollama/.ollama -czf ollama-models.tar.gz models
$ sha256sum ollama-models.tar.gz > ollama-models.tar.gz.sha256
# --- Air-gapped host ---
$ sha256sum -c ollama-models.tar.gz.sha256
$ sudo systemctl stop ollama
$ sudo tar -C /usr/share/ollama/.ollama -xzf ollama-models.tar.gz
$ sudo chown -R ollama:ollama /usr/share/ollama/.ollama
$ sudo systemctl start ollama
$ ollama list
Fix ownership after extraction — tar preserves the source UIDs. If the ollama user has a different UID on the target, the service cannot read its own models — a confusing failure that looks like corruption.
Prefer the GGUF + Modelfile route — it is explicit, portable across Ollama versions, and works equally well with llama.cpp.
Python Dependencies Offline
pip download builds a wheelhouse you can move and install from with no index:
# --- Staging host ---
$ python3 -m venv .venv && source .venv/bin/activate
$ pip download -r requirements.txt -d wheelhouse
# For a different target platform, be explicit:
$ pip download -r requirements.txt -d wheelhouse \
--platform manylinux2014_x86_64 --python-version 3.11 \
--only-binary=:all:
$ tar -czf wheelhouse.tar.gz wheelhouse requirements.txt
# --- Air-gapped host ---
$ tar -xzf wheelhouse.tar.gz
$ pip install --no-index --find-links=./wheelhouse -r requirements.txt
Pin every version — use a fully pinned requirements.txt. Resolving on the staging host and installing something different on the target defeats the point of the exercise.
--only-binary avoids surprises — a source distribution will try to compile on the target, which needs a toolchain and sometimes network access for build dependencies.
GPU wheels are large and specific — PyTorch CUDA wheels are multi-gigabyte and tied to a CUDA version. Download from the correct index URL on the staging host.
Container Images
# --- Staging host ---
$ docker pull ollama/ollama:latest
$ docker save ollama/ollama:latest | gzip > ollama-image.tar.gz
$ sha256sum ollama-image.tar.gz > ollama-image.tar.gz.sha256
# --- Air-gapped host ---
$ sha256sum -c ollama-image.tar.gz.sha256
$ gunzip -c ollama-image.tar.gz | docker load
$ docker images | grep ollama
Pull by digest, not tag — use image@sha256:... so you know exactly what you transferred. Tags move; digests do not.
Consider a local registry — beyond a handful of images, run a registry inside the enclave and push to it once. Better than managing tarballs indefinitely.
GPU images still need host setup — the driver and container toolkit must be installed on the offline host — see our GPU containers guide.
Verification Is Not Optional
Multi-gigabyte transfers over removable media fail silently more often than people expect, and a truncated model file produces plausible-looking gibberish rather than an error:
# Generate a manifest for everything you are transferring
$ sha256sum *.gguf *.tar.gz > MANIFEST.sha256
# On the target — verify before installing anything
$ sha256sum -c MANIFEST.sha256
# Compare publisher-provided hashes where available
$ sha256sum model-Q4_K_M.gguf
Verify on the target, not the source — the point is detecting corruption introduced by the transfer itself.
Check publisher hashes on the staging host — that is where you confirm you downloaded the genuine artifact. The manifest then confirms it arrived intact.
Record provenance — note the source URL, date, and publisher hash for every model. In a controlled environment you will be asked to prove where it came from.
Making It Sustainable
Batch transfers on a schedule — monthly or quarterly rather than ad hoc. Each crossing is a controlled event with paperwork; fewer and larger beats many and small.
Keep a bill of materials — every model, wheel, and image with its version, hash, and licence. This is what makes an audit answerable.
Stage identically to production — same distro, same architecture, same Python version. Differences surface as failures you cannot debug offline.
Keep the previous version — you cannot re-download a rollback. Retain the working artifacts until the new ones are proven.
Test on a staging replica first — an offline environment is a terrible place to discover a dependency you forgot.
Document the runbook — the person doing the next transfer may not be you. Combine with our hardening checklist for the target build.
Troubleshooting
ollama list is empty after copying models — ownership or path. The model store must be owned by the ollama service user, under the directory that build uses.
Model produces gibberish — an incomplete transfer. Verify checksums — this is the classic symptom of a truncated GGUF.
pip install still tries to reach the network — a dependency is missing from the wheelhouse. Use --no-index so it fails loudly, then add what it names.
"No matching distribution found" offline — the wheel was downloaded for a different Python version or platform. Re-download with explicit --platform and --python-version.
docker load succeeds but the image will not run — an architecture mismatch. Confirm the staging host matches the target platform.
Ollama tries to phone home — it only reaches the network to pull models. Once weights are local, no request is made — but keep egress blocked and verify with ss if you must prove it.
Frequently Asked Questions
Can Ollama run completely offline?
Yes. Ollama only contacts the network to download models. Once the weights are present locally, inference is entirely offline. You can register a model from a local GGUF file with a Modelfile containing a FROM line, without any network access.
What is the best way to move an LLM to an air-gapped machine?
Copy the GGUF file and register it with a Modelfile using ollama create. That is explicit, portable across versions, and works with llama.cpp too. Copying Ollama's whole model store also works but depends on its on-disk layout and requires fixing file ownership afterwards.
How do I install Python packages without internet access?
Run pip download -r requirements.txt -d wheelhouse on a staging host with the same platform and Python version, transfer the directory, then install with pip install --no-index --find-links=./wheelhouse -r requirements.txt. Pin every version and prefer --only-binary to avoid compiling on the target.
Why does my transferred model produce nonsense output?
Almost always an incomplete or corrupted transfer. Large files copied via removable media can truncate without an error, and a partial GGUF generates gibberish rather than failing. Verify sha256 checksums on the target before using any model.
How do I move Docker images across an air gap?
Use docker save to write the image to a tarball, transfer it with a checksum, then docker load on the target. Pull by digest rather than tag so you know exactly what you moved, and consider running an internal registry once you have more than a few images.