Solved Upgrade still not progress ...

[saglagla]

Hello,
Since yesterday, my Cloudron is in upgrading phase :

The issue is, that after 24h, it's still in update. I've tried to stop DB, and reboot after 2h of waiting but that change nothing.

Now, my Cloudron is blocked on this page (App still reachable by URL). What should I do ?

[girish]

@saglagla Can you check what it says in journalctl -u box -u cloudron-updater output? I suspect it has got some dpkg related error.

[girish]

Also, please check if / and boot have enough space (df -h)

[saglagla]

@girish said in Upgrade still not progress ...:

Also, please check if / and boot have enough space (df -h)

I've enough free space on '/' but probably not on /boot (96% used, 7.8Mo free)

@girish said in Upgrade still not progress ...:

@saglagla Can you check what it says in journalctl -u box -u cloudron-updater output? I suspect it has got some dpkg related error.

As far as I've seen in the log :

Jun 19 00:48:49 my.XXXXXXinstaller.sh[18503]: Running depmod.
Jun 19 00:48:49 my.XXXXXXinstallerinstaller.sh[18503]: update-initramfs: deferring update (hook will be called later)
Jun 19 00:48:49 my.XXXXXXinstaller.sh[18503]: The link /initrd.img is a dangling linkto /boot/initrd.img-4.4.0-127-generic
Jun 19 00:48:49 my.XXXXXXinstaller.sh[18503]: Examining /etc/kernel/postinst.d.
Jun 19 00:48:49 my.XXXXXXinstaller.sh[18503]: run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.4.0-128-generic /boot/vmlinuz-4.4.0-128-generic
Jun 19 00:48:49 my.XXXXXXinstaller.sh[18503]: run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.4.0-128-generic /boot/vmlinuz-4.4.0-128-generic
Jun 19 00:48:49 my.XXXXXXinstaller.sh[18503]: update-initramfs: Generating /boot/initrd.img-4.4.0-128-generic
Jun 19 00:48:54 my.XXXXXXinstaller.sh[18503]: gzip: stdout: No space left on device
Jun 19 00:48:54 my.XXXXXXinstaller.sh[18503]: cpio: write error: Broken pipe
Jun 19 00:48:54 my.XXXXXXinstaller.sh[18503]: E: mkinitramfs failure cpio 1 gzip 1
Jun 19 00:48:54 my.XXXXXXinstaller.sh[18503]: update-initramfs: failed for /boot/initrd.img-4.4.0-128-generic with 1.
Jun 19 00:48:54 my.XXXXXXinstaller.sh[18503]: run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
Jun 19 00:48:54 my.XXXXXXinstaller.sh[18503]: Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/linux-image-4.4.0-128-generic.postinst line 1052.
Jun 19 00:48:54 my.XXXXXXinstaller.sh[18503]: dpkg: error processing package linux-image-4.4.0-128-generic (--configure):
Jun 19 00:48:54 my.XXXXXXinstaller.sh[18503]:  subprocess installed post-installation script returned error exit status 2
Jun 19 00:48:54 my.XXXXXXinstaller.sh[18503]: Setting up linux-image-4.4.0-127-generic (4.4.0-127.153) ...
Jun 19 00:48:54 my.XXXXXX installer.sh[18503]: Running depmod.
Jun 19 00:48:55 my.XXXXXX installer.sh[18503]: update-initramfs: deferring update (hook will be called later)
Jun 19 00:48:55 my.XXXXXX installer.sh[18503]: The link /initrd.img is a dangling linkto /boot/initrd.img-4.4.0-128-generic
Jun 19 00:48:55 my.XXXXXX installer.sh[18503]: Examining /etc/kernel/postinst.d.
Jun 19 00:48:55 my.XXXXXX installer.sh[18503]: run-parts: executing /etc/kernel/postinst.d/apt-auto-removal 4.4.0-127-generic /boot/vmlinuz-4.4.0-127-generic
Jun 19 00:48:55 my.XXXXXX installer.sh[18503]: run-parts: executing /etc/kernel/postinst.d/initramfs-tools 4.4.0-127-generic /boot/vmlinuz-4.4.0-127-generic
Jun 19 00:48:55 my.XXXXXX installer.sh[18503]: update-initramfs: Generating /boot/initrd.img-4.4.0-127-generic
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]: gzip: stdout: No space left on device
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]: cpio: write error: Broken pipe
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]: E: mkinitramfs failure cpio 1 gzip 1
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]: update-initramfs: failed for /boot/initrd.img-4.4.0-127-generic with 1.
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]: run-parts: /etc/kernel/postinst.d/initramfs-tools exited with return code 1
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]: Failed to process /etc/kernel/postinst.d at /var/lib/dpkg/info/linux-image-4.4.0-127-generic.postinst line 1052.
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]: dpkg: error processing package linux-image-4.4.0-127-generic (--configure):
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]:  subprocess installed post-installation script returned error exit status 2
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]: Setting up initramfs-tools (0.122ubuntu8.11) ...
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]: update-initramfs: deferring update (trigger activated)
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]: dpkg: dependency problems prevent configuration of linux-image-extra-4.4.0-128-generic:
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]:  linux-image-extra-4.4.0-128-generic depends on linux-image-4.4.0-128-generic; however:
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]:   Package linux-image-4.4.0-128-generic is not configured yet.
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]: dpkg: error processing package linux-image-extra-4.4.0-128-generic (--configure):
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]:  dependency problems - leaving unconfigured
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]: dpkg: dependency problems prevent configuration of linux-image-extra-4.4.0-127-generic:
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]:  linux-image-extra-4.4.0-127-generic depends on linux-image-4.4.0-127-generic; however:
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]:   Package linux-image-4.4.0-127-generic is not configured yet.
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]: dpkg: error processing package linux-image-extra-4.4.0-127-generic (--configure):
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]:  dependency problems - leaving unconfigured
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]: dpkg: dependency problems prevent configuration of linux-image-generic:
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]:  linux-image-generic depends on linux-image-4.4.0-128-generic; however:
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]:   Package linux-image-4.4.0-128-generic is not configured yet.
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]:  linux-image-generic depends on linux-image-extra-4.4.0-128-generic; however:
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]:   Package linux-image-extra-4.4.0-128-generic is not configured yet.
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]: dpkg: error processing package linux-image-generic (--configure):
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]:  dependency problems - leaving unconfigured
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]: dpkg: dependency problems prevent configuration of linux-generic:
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]:  linux-generic depends on linux-image-generic (= 4.4.0.128.134); however:
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]:   Package linux-image-generic is not configured yet.
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]: dpkg: error processing package linux-generic (--configure):
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]:  dependency problems - leaving unconfigured
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]: Processing triggers for initramfs-tools (0.122ubuntu8.11) ...
Jun 19 00:49:00 my.XXXXXX installer.sh[18503]: update-initramfs: Generating /boot/initrd.img-4.4.0-124-generic
Jun 19 00:49:05 my.XXXXXX installer.sh[18503]: gzip: stdout: No space left on device
Jun 19 00:49:05 my.XXXXXX installer.sh[18503]: cpio: write error: Broken pipe
Jun 19 00:49:05 my.XXXXXX installer.sh[18503]: E: mkinitramfs failure cpio 1 gzip 1
Jun 19 00:49:05 my.XXXXXX installer.sh[18503]: update-initramfs: failed for /boot/initrd.img-4.4.0-124-generic with 1.
Jun 19 00:49:05 my.XXXXXX installer.sh[18503]: dpkg: error processing package initramfs-tools (--configure):
Jun 19 00:49:05 my.XXXXXX installer.sh[18503]:  subprocess installed post-installation script returned error exit status 1
Jun 19 00:49:05 my.XXXXXX installer.sh[18503]: Errors were encountered while processing:
Jun 19 00:49:05 my.XXXXXX installer.sh[18503]:  linux-image-4.4.0-128-generic
Jun 19 00:49:05 my.XXXXXX installer.sh[18503]:  linux-image-4.4.0-127-generic
Jun 19 00:49:05 my.XXXXXX installer.sh[18503]:  linux-image-extra-4.4.0-128-generic
Jun 19 00:49:05 my.XXXXXX installer.sh[18503]:  linux-image-extra-4.4.0-127-generic
Jun 19 00:49:05 my.XXXXXX installer.sh[18503]:  linux-image-generic
Jun 19 00:49:05 my.XXXXXX installer.sh[18503]:  linux-generic
Jun 19 00:49:05 my.XXXXXX installer.sh[18503]:  initramfs-tools
Jun 19 00:49:05 my.XXXXXX installer.sh[18503]: ==> installer: Failed to fix packages. Retry

[saglagla]

@girish said in Upgrade still not progress ...:

injournalctl -u box -u cloudron-updater

I've follow up your solution: https://forum.cloudron.io/topic/1310/disk-filled-up-with-kernel-update-artifacts/2

It's work !

[girish]

Sorry about that. In our updater logic, we repeatedly try to install a package if it fails. I think this ends up making temporary initramfs images and fills up the disk. I will fix our retry logic atleast to workaround this.

[mehdi]

For the record, I hit that issue too. /boot got full. However, I could not fix it with the solution at https://forum.cloudron.io/topic/1310/disk-filled-up-with-kernel-update-artifacts/2 . I had to follow https://askubuntu.com/a/430944 (slightly adapted)