Troubleshooting a Full Filesystem or Inode Errors

One unfortunate day encounter error messages may appear indicating that the hard drive in the firewall is full, such as these:

/: write failed, filesystem is full
/: create/symlink failed, no inodes free
Warning: session_start(): open(/tmp/sess_XXXXXX, O_RDWR) failed: No space left on device

The typical cause of such errors is rarely that the drive is full, but that the operating system is unable to contact the drive. In short, the disk (HDD, SSD, CF, etc) is dead or dying.

In cases when the drive is dying, the OS tries to write to the drive, and receives back an error code that there aren’t any inodes left. Typically other messages about the underlying problem are logged to the console but not passed back to PHP so they can be seen by the GUI. Usually those refer to things like g_vfs_done or Troubleshooting DMA and LBA Errors.

In order to eliminate the possibility that the drive is actually full (perhaps a package went crazy eating up space), try to get to a shell from the console or ssh, and run the following command:

: df -hi

The output shows disk space usage for both capacity and inodes, using human-readable numbers. The System Information widget on the Dashboard on pfSense® software version 2.2 shows the usage for all mounted partitions. Earlier versions only showed usage for the root (/) slice. The following output is from an ALIX system running NanoBSD:

Filesystem           Size    Used   Avail Capacity iused ifree %iused  Mounted on
/dev/ufs/pfsense1    442M    167M    239M    41%    6.3k   52k   11%   /
devfs                1.0k    1.0k      0B   100%       0     0  100%   /dev
/dev/ufs/cf           48M    1.7M     43M     4%      26  6.4k    0%   /cf
/dev/md0              48M     84k     44M     0%      56  6.6k    1%   /tmp
/dev/md1              77M     16M     54M    24%     133   10k    1%   /var
devfs                1.0k    1.0k      0B   100%       0     0  100%   /var/dhcpd/dev

Note

The devfs lines do NOT indicate an actual problem; The devfs filesystem is virtual and used for housing device nodes not for files.

Of special concern on NanoBSD is the space on /var and /tmp, since they are RAM disks. Some things, such as an abnormally large DHCP leases file, can in fact fill up the /var memory disk and that is one way to encounter the problem.

If the root (/) slice has space and inodes remaining, and so do /var and /tmp and so on, then the problem is most likely a failing disk.

If disk space has been exhausted, find a way to free it up. This usually involves uninstalling or removing packages such as squid, or changing the settings so they use less space.

If the disk is failing, swap it out as soon as possible.

Often when the drive is failing or full the system will continue routing packets indefinitely until it would need to access the hard drive, at which point it would quit. We have seen them run for months with a dead drive unnoticed before, though of course that is not advisable.

Another possible cause is mild filesystem corruption, which could be helped by Forcing a Filesystem Check.