The alert says the application cannot write. The log says No space left on device. You run df -h and the filesystem is 44% full, with more than fifty gigabytes free. Nothing is wrong with the disk, and deleting a big log file will not help, because the filesystem has not run out of space. It has run out of inodes.
The kernel returns the same error, ENOSPC, whether a filesystem is out of data blocks or out of inodes, so the message never tells you which. This guide explains what an inode is, why ext4 can exhaust them long before its blocks, how to find the directory responsible, and why XFS and Btrfs rarely have the problem at all.
What an Inode Actually Holds
Every file, directory, symlink, socket and device node on a Unix filesystem is represented by one inode: a fixed-size record of metadata, identified by a number that is unique within that filesystem. stat shows most of it:
$ stat /etc/hosts
File: /etc/hosts
Size: 287 Blocks: 8 IO Block: 4096 regular file
Device: 259,2 Inode: 1835023 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2026-10-01 08:12:44.512837201 +0000
Modify: 2026-09-14 10:22:03.118823104 +0000
Change: 2026-09-14 10:22:03.118823104 +0000
Birth: 2026-03-02 17:40:11.902114562 +0000
$ ls -i /etc/hosts
1835023 /etc/hosts
| Stored in the inode | Where it shows up in stat |
|---|---|
| File type and permission bits | regular file, Access: (0644/-rw-r--r--) |
| Owner UID and group GID | Uid: 0 root, Gid: 0 root |
| Size and allocated block count | Size: 287, Blocks: 8 |
| Timestamps: atime, mtime, ctime, and birth time on ext4 and XFS | Access, Modify, Change, Birth |
| Hard link count | Links: 1 |
| Extents or block pointers locating the data | Not shown |
| Small extended attributes, such as ACLs and SELinux labels | Not shown; use getfacl or ls -Z |
What is missing is the name. It lives in the directory. A directory is a special file whose contents are a list of entries, each one a name and an inode number. Opening /etc/hosts means reading the root directory to find the inode of etc, reading that directory to find the inode of hosts, then reading that inode to find the data. The permission bits explained in our file permissions guide are stored here too, which is why chmod changes ctime but not mtime.
Hard links are extra directory entries — ln a b adds a second name pointing at the same inode and raises its link count to 2. Neither name is the original. Because inode numbers are only unique per filesystem, hard links cannot cross filesystems; symlinks can, because they store a path.
rm removes a name, not a file — it deletes a directory entry and decrements the link count. The inode and its blocks are freed only when the count reaches zero and no process still has the file open.
mv within a filesystem is instant — it rewrites directory entries and leaves the inode, and the data, where they are. Across filesystems it has to copy the data and create a new inode.
Diagnosing Inode Exhaustion
The tell is a disk that reports free space and still refuses new files. Writing to an existing file can keep working, because that needs blocks, not inodes; creating anything new fails:
$ touch /var/tmp/probe
touch: cannot touch '/var/tmp/probe': No space left on device
$ df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/sda2 99G 41G 54G 44% /
$ df -i /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda2 6553600 6553600 0 100% /
That is the whole diagnosis: blocks at 44%, inodes at 100%. On ext4 you can confirm the fixed totals with tune2fs:
$ sudo tune2fs -l /dev/sda2 | grep -Ei '^(inode count|block count|free inodes|inode size)'
Inode count: 6553600
Block count: 26214400
Free inodes: 0
Inode size: 256
Monitor IUse% as well as Use% — most disk alerts only watch blocks. df -x tmpfs -x devtmpfs --output=ipcent,target | awk 'NR > 1 && $1+0 >= 80' lists every filesystem above 80% inode use and is easy to drop into an existing check.
Root still has reserved blocks, not reserved inodes — ext4 reserves 5% of blocks for root by default, but there is no equivalent reserve of inodes. When they run out, root cannot create files either, which can break logins and services that write PID or lock files.
Finding the Directories That Ate the Inodes
Inode exhaustion is almost always one directory holding millions of tiny files. du --inodes counts entries instead of bytes; -x keeps it on one filesystem:
$ sudo du --inodes -x / 2>/dev/null | sort -n | tail -5
412870 /usr
598133 /var/lib/docker/overlay2
4127906 /var/lib/php/sessions
4789512 /var/lib/php
6553589 /
# Alternative: count files per parent directory, deepest culprit first
$ sudo find / -xdev -printf '%h\n' | sort | uniq -c | sort -rn | head -3
4127904 /var/lib/php/sessions
98411 /var/spool/postfix/deferred/A
51322 /usr/src/linux-headers-6.8.0-45/include
| Usual suspect | Path | Why it grows |
|---|---|---|
| PHP sessions | /var/lib/php/sessions | One file per session; cleanup cron or gc disabled, broken, or outpaced. RHEL uses /var/lib/php/session |
| Mail queue | /var/spool/postfix | A broken relay leaves every outgoing message deferred |
| Application caches | /var/cache, app/cache | Template or thumbnail caches with no expiry |
| Old kernel headers | /usr/src | On Ubuntu, each linux-headers package adds tens of thousands of files |
| Container layers | /var/lib/docker/overlay2 | Unused images and build cache; see our Docker guide |
| Build trees | node_modules, .cache | CI runners that never clean their workspaces |
Huge directories are visible by size — a directory's own size grows with its entry count, and on ext4 it does not shrink when entries are removed. find / -xdev -type d -size +10M finds directories that have held millions of entries, even after cleanup.
Docker has its own cleanup — use docker system df and docker image prune rather than deleting files under overlay2 by hand. The Docker guide for admins covers storage housekeeping.
Deleting Millions of Files Without Making It Worse
rm /var/lib/php/sessions/* fails with Argument list too long, because the shell expands the glob into millions of arguments before rm starts. Let find do the walking and deleting itself:
# Delete session files older than one day, at low I/O priority
$ sudo ionice -c3 nice -n19 find /var/lib/php/sessions -type f -mmin +1440 -delete
# Remove everything below a directory but keep the directory itself
$ sudo find /var/cache/app -mindepth 1 -delete
# Watch inodes come back while it runs
$ watch -n5 df -i /
Fix the cause first — on Debian and Ubuntu, PHP sessions are cleaned by the phpsessionclean timer, not by PHP itself. If the timer is disabled or failing, the directory refills right after you empty it. Check systemctl list-timers and the application's own cleanup job.
Free one inode to get a shell working — if logins or sudo fail because nothing can be created, deleting a handful of files in the culprit directory first restores enough room for normal tools to run.
Expect it to take a while — unlinking millions of files is metadata-bound. On ext4 with a busy disk it can run for tens of minutes; ionice keeps it from starving the application.
Why ext4 Runs Out and XFS Rarely Does
ext4 allocates its whole inode table when the filesystem is created, based on a bytes-per-inode ratio. The default in /etc/mke2fs.conf is one inode per 16 KiB of space, which assumes an average file size around that. The filesystem above is 100 GiB (df shows 99G after overhead), and 100 GiB divided by 16 KiB is 6,553,600 inodes, and that number never changes unless the filesystem is resized or recreated.
# /etc/mke2fs.conf (excerpt)
[defaults]
inode_size = 256
inode_ratio = 16384
[fs_types]
news = {
inode_ratio = 4096
}
largefile = {
inode_ratio = 1048576
blocksize = -1
}
# Plan for many small files at mkfs time: one inode per 4 KiB
$ sudo mkfs.ext4 -i 4096 /dev/sdc1
$ sudo mkfs.ext4 -T news /dev/sdc1 # same ratio via the usage type
$ sudo mkfs.ext4 -N 20000000 /dev/sdc1 # or an explicit inode count
| Filesystem | Inode allocation | Can you run out? |
|---|---|---|
| ext4 | Fixed at mkfs from the inode ratio; resize2fs adds more in proportion when growing | Yes. More inodes means growing or recreating the filesystem |
| XFS | Dynamic, up to imaxpct of the space (25% under 1 TB by default) | Rarely. Raise the cap with xfs_growfs -m on the mount point |
| Btrfs | Dynamic; inodes are items in metadata trees | Not as a separate limit. df -i shows 0 totals |
| tmpfs | nr_inodes, by default half the RAM pages | Yes, on busy /tmp. Remount with a higher nr_inodes |
RHEL defaults to XFS, Debian and Ubuntu to ext4 — which is one reason inode exhaustion tickets are far more common on Debian-family servers holding session stores or mail queues.
Each small file still costs a block — on a 4 KiB-block ext4 filesystem a 100-byte file uses one inode and one full block. Millions of tiny files waste both, which is a sign the data belongs in a database or key-value store.
The Other Phantom Full Disk: Deleted but Open
A related puzzle has the opposite symptom: df says the disk is full but du cannot find the space. Here the inode, not the name, is the answer. A process still holds a deleted file open, so its link count is zero but its inode and blocks stay allocated:
$ sudo lsof +L1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
java 2214 app 1w REG 259,2 21474836480 0 1966114 /var/log/app/server.log (deleted)
# Truncate it through the process's file descriptor, or restart the process
$ sudo truncate -s 0 /proc/2214/fd/1
+L1 asks lsof for open files with a link count below one. Fix the log rotation that deleted a file still being written, usually by switching to copytruncate or signalling the process to reopen its logs. For more disk and filesystem commands, see the Linux tools cheat sheet.
Frequently Asked Questions
Why do I get No space left on device when df shows free space?
The filesystem has probably run out of inodes rather than data blocks. The kernel returns the same ENOSPC error for both. Run df -i: if IUse% is at or near 100%, some directory holds a very large number of small files. Find it with du --inodes -x and delete or clean up those files.
Can I increase the number of inodes on an existing ext4 filesystem?
Not in place. ext4 creates its inode table at mkfs time. Growing the filesystem with resize2fs adds inodes in proportion to the added space, but otherwise you need to back up the data, recreate the filesystem with a lower bytes-per-inode ratio such as mkfs.ext4 -i 4096, and restore.
What information does an inode store?
The file type, permission bits, owner and group, size, timestamps, hard link count, the locations of the data blocks, and small extended attributes such as ACLs and SELinux labels. It does not store the file name, which lives in the directory entry that points to the inode.
Does XFS run out of inodes?
Rarely. XFS allocates inodes dynamically as files are created, up to a percentage of the filesystem set by imaxpct, which defaults to 25% on filesystems under 1 TB. If you do reach it, xfs_growfs -m with a higher percentage on the mounted filesystem raises the limit without reformatting.
What is the difference between a hard link and a symlink in terms of inodes?
A hard link is another directory entry pointing at the same inode, so both names share permissions, contents and timestamps, and the data is freed only when the last link is removed. A symlink is its own inode whose content is a path. It can cross filesystems and can dangle if the target is removed.