Cloudron update exited with code 1 and no space left in /boot

[CarbonBee]

Hi,

Cloudron update are failing for some day, and the message is update exited with code 1 signal null

I went to journalctl -u cloudron-updater and found:

Nov 15 21:23:43 my.domain.com installer.sh[354]: dpkg: dependency problems prevent configuration of linux-image-generic:
Nov 15 21:23:43 my.domain.com installer.sh[354]:  linux-image-generic depends on linux-image-4.4.0-190-generic | linux-image-unsigned-4.4.0-190-generic; however:
Nov 15 21:23:43 my.domain.com installer.sh[354]:   Package linux-image-4.4.0-190-generic is not installed.
Nov 15 21:23:43 my.domain.com installer.sh[354]:   Package linux-image-unsigned-4.4.0-190-generic is not installed.
Nov 15 21:23:43 my.domain.com installer.sh[354]:  linux-image-generic depends on linux-modules-extra-4.4.0-190-generic; however:
Nov 15 21:23:43 my.domain.com installer.sh[354]:   Package linux-modules-extra-4.4.0-190-generic is not configured yet.
Nov 15 21:23:43 my.domain.com installer.sh[354]: dpkg: error processing package linux-image-generic (--configure):
Nov 15 21:23:43 my.domain.com installer.sh[354]:  dependency problems - leaving unconfigured
Nov 15 21:23:43 my.domain.com installer.sh[354]: Processing triggers for linux-image-4.4.0-189-generic (4.4.0-189.219) ...
Nov 15 21:23:44 my.domain.com installer.sh[354]: /etc/kernel/postinst.d/initramfs-tools:
Nov 15 21:23:44 my.domain.com installer.sh[354]: update-initramfs: Generating /boot/initrd.img-4.4.0-189-generic
Nov 15 21:23:49 my.domain.com installer.sh[354]: cat: write error: No space left on device
Nov 15 21:23:49 my.domain.com installer.sh[354]: update-initramfs: failed for /boot/initrd.img-4.4.0-189-generic with 1.
Nov 15 21:23:49 my.domain.com installer.sh[354]: run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
Nov 15 21:23:49 my.domain.com installer.sh[354]: dpkg: error processing package linux-image-4.4.0-189-generic (--configure):
Nov 15 21:23:49 my.domain.com installer.sh[354]:  subprocess installed post-installation script returned error exit status 1
Nov 15 21:23:49 my.domain.com installer.sh[354]: Errors were encountered while processing:
Nov 15 21:23:49 my.domain.com installer.sh[354]:  linux-modules-extra-4.4.0-190-generic
Nov 15 21:23:49 my.domain.com installer.sh[354]:  linux-image-generic
Nov 15 21:23:49 my.domain.com installer.sh[354]:  linux-image-4.4.0-189-generic
Nov 15 21:23:49 my.domain.com installer.sh[354]: ==> installer: Failed to fix packages. Retry

I saw the line No space left on device and indeed df -h said /dev/md2 487M 482M 0 100% /boot
What is weird is that uname -nar says 4.4.0-154-generic but in /boot I have

initrd.img-4.4.0-104-generic
initrd.img-4.4.0-112-generic
initrd.img-4.4.0-154-generic
initrd.img-4.4.0-161-generic
initrd.img-4.4.0-164-generic
initrd.img-4.4.0-178-generic
initrd.img-4.4.0-179-generic
initrd.img-4.4.0-184-generic
initrd.img-4.4.0-185-generic
initrd.img-4.4.0-186-generic
initrd.img-4.4.0-187-generic
initrd.img-4.4.0-189-generic.new

Finally, I tired dpkg --confiure -a and my dpkg is locked : dpkg: error: dpkg frontend is locked by another process . Using lsof and ps, I found it is used by root 1213 0.0 0.0 65512 3000 ? Ss 2019 37:23 /usr/sbin/sshd -D

So, what should I do? Remove every files that are not 4.4.0-154-generic? Can I kill this sshd process that is locking dpkg?

[msbt]

hey, you can try following this guide here: https://docs.cloudron.io/troubleshooting/#recovery-after-disk-full

[CarbonBee]

@msbt Well, any call to apt-get tells me that

You might want to run 'apt-get -f install' to correct these.
The following packages have unmet dependencies:
 linux-image-generic

Which of course I can't do because of the lack of space in /boot.
I followed parts of this link :
dpkg --list 'linux-image*'|awk '{ if($1=="ii") print $2}'|grep -vuname -r``
rm -rf /boot/*-4.4.0-{161,164,178,179,184,185,186,187}-*
apt-get -f install
apt-get autoremove

And it worked fine!

[d19dotca]

@carbonbee That’s strange because the command from the documentation actually has to uninstall unused packages rather than install anything. Are you sure you ran the correct command?

[mehdi]

@d19dotca it's just a trick to remove old kernels

[d19dotca]

@mehdi Yeah I know, but I don't understand why that couldn't run "because of the lack of space" when it's removing files that aren't used. Shouldn't that succeed still? Or is that output just implying that there weren't any to remove?

[msbt]

@d19dotca it's a bit tricky, sometimes there isn't enough space to remove things, because that requires some bytes of free space to run and fails eventually because there aren't any left

[mehdi]

@d19dotca I think the -f flag on install just "cleans up" the installation : it notices that some files required for some packages were removed (in the previous line), so it removes said packages. What actually frees up the space is the rm. The apt-get -f install just makes the system notice that the packages in question are not installed anymore