/run/udev/data folder fills up over time, eventually crashing dashboard

[ethanxrosen]

Hi All,

I've seen an issue twice in the last few months where /run/udev/data will fill up the /run partition overtime, eventually crashing the dashboard. Restarting the server seems to clear this folder and the server comes back online, but obviously it would be best not to have the server go down every few weeks.

Below is output of some disk usage commands. Please let me know if there's any other info I can provide!

Best,
Ethan

root@localhost:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            2.0G     0  2.0G   0% /dev
tmpfs           393M  393M     0 100% /run
/dev/sda2        20G   11G  8.0G  57% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
/dev/sda1       487M   36M  422M   8% /boot
/dev/sda3       1.8T  795G  927G  47% /home
overlay         1.8T  795G  927G  47% /home/docker/overlay2/9a5e282fdcd9ec4681c0d1c0fb15f7a7ff7eda0ece5e4b9e0a68762692eae35e/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/0bdf5e13f8a83ca3ebebc87212922638eb4d9ded7d7accaa3e20ddeda5dced92/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/45613b260840f06be967ef5fdf248007199891fa2f7f9d4847e6ba6398804db4/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/476f326bbc83b1900d923072d7ba3e263b27e9467d4cf4a6e1920bf33ce03e58/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/a237010796d9a5da3b27631eca065d2e103da31dcaba2e1877d7c4fade19d218/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/7169515c216da9e1913b8c39ea96296790571f7c022c988496b76ae49ba73436/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/193a0fd036a1b7deaba4fbcf59b7cb64e37178df637b7cd65c041d55151bdf3e/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/a29651b2ee737f945aadbf52e025e5a1bb0fad78b4a2eed0f1e50f51eec1b268/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/8d142cc736c9e0512dc809a99e45158f9a9ba2552fb741935a2de26ea2a80165/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/942491a3968de0663f8116d94afe22f0e0e8755676e7c4f2d82e8649fb587857/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/0ca4d556581ee79190f7948caa097be42fc9cf7c8e928f759a7af8022c682fb9/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/242df83b4d74aa35ca500b2eda722b96210fcbffa1cce39a6332d4d2d23fc8ac/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/a196773d462838d7b38a8f44d9b44557022ed24784bcc2f700331fa471fc5272/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/dd4da96ca820fbe61e77a27cb02346742d4c1afc710097b74d8f52d31e8b0517/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/bc6b0bea1be9dae8eb85b73b38f54c7504216b1941cf9f1e9a528c442696c6d5/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/37fbe6e6b9b452c17f1f8de2a5267018f85c6a3983f46998052f80278dbe50b1/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/a2b82c8008d0f0e210866b2c77864d26b2b1fcb1d8ea195ecf81f50f7bdbb4ed/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/9ad83b3b2ab5055882503162f952c9c442a10fbc17c80d09f62084c42f23bfe6/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/cfd2bd3867197e60d4eebf99e8df3423ab5dde864b3191fdf7b9393afe22b3ef/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/d7e51bc7bae2d8cdc16799e7b9579e8a1f85c443d35acd6b028c07e2f8576b74/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/02dcfe4cdb631a7121cb8a37efccefe70ef5adbef9aa1f2ccaff2bd09fa48002/merged
overlay         1.8T  795G  927G  47% /home/docker/overlay2/aba28e860dab3232388fee0d24a7e36dbbbe9bffa8738742fe2461b74f42569e/merged
tmpfs           393M     0  393M   0% /run/user/0

root@localhost:/run# du -a | sort -n -r | head -n 5
 
402000	.
400320	./udev/data
400320	./udev
708	./docker/runtime-runc/moby
708	./docker/runtime-runc

[girish]

@ethanxrosen What ubuntu version are you using?

[ethanxrosen]

@girish Ubuntu 16.04.4 LTS, ah I just spotted this article: https://docs.cloudron.io/guides/upgrade-ubuntu-20/ will try upgrading and see if the issue persists

[girish]

@ethanxrosen right, first migrate to 18 - https://docs.cloudron.io/guides/upgrade-ubuntu-18/ and then 20.