Ubuntu UniFi server running out of space on /run

Had an issue that /run was randomly running out of space which in turn would interfere with the unifi-video service causing it to run, but not record.

/run looks like a tmpfs or ramdisk that Ubuntu sets up.  So you can do a “temporary” fix by remounting the tmpfs with a larger size.  Example below.  If /run is a 2GB directory, you can remount changing the size from 2GB to 2.5GB.

sudo mount -t tmpsfs tmpfs /run -o remount,size=2500M

Note that it is a temporary fix and goes away after a reboot.

The issue ended up being that the WiFi UniFi controller was setup to auto backup everything once a week.  So as it was backing stuff up, it would eat up the available space in the tmpfs, think there may be an issue with the size of the UniFi data and maybe not being able to fit it all in RAM?

Running the following command

df -h --max=1 /var | sort

shows the following

1.1M /run/udev
2.5G /run/
2.5G /run/unifi   <-- UniFi controller
4.0K /run/initramfs
8.0K /run/network
12K /run/user
288K /run/samba
404K /run/systemd

Looking inside the unifi directory shows the following folders.  Looks like the they are temp files.

200M /run/unifi/ExpTmp351719567129045774
696M /run/unifi/ExpTmp3406220793759111216
1.6G /run/unifi/ExpTmp3368400690321364109
0 /run/unifi/work
2.5G /run/unifi

Running an ls inside the folder shows

-rw-r----- 1 unifi unifi 13971807 Jul 2 02:30 db.gz
-rw-r----- 1 unifi unifi 1665223462 Jul 2 02:56 db_stat.gz

Looking inside the UniFi controller it is set to auto backup on Monday at 2:30AM

Looks like “/run/unifi” is used as a temporary folder to create the backups and when it is completed converts it to a .unf file and moves it to “/usr/lib/unifi/data/backup/autobackup/”  So the left over temp files were never completed or something caused them to stop working.  Maybe the backup was big enough to take up all the tmpfs space and cause the backup to fail.

Resolution.  To fix the problem I turned the data retention down to a week and deleted the temp files in /var/unifi.  Had to sudo su to root.  sudo wasn’t cutting it for some reason.

sudo su
rm -rf /var/unifi/Exp*
exit

Should be all good now.

Leave a Reply