Issue after 8.0.3 upgrade with one app because of manually configured domain name - a post mortem (kind of)

[fbartels]

Hi, I just had a small odyssey (partly self inflicted) to upgrade one server from 7.7.x to 8.0.3 and wanted to put my steps in here in case anyone else encounters the same.

We noticed that dns resolutin in the installed apps stopped working and since I already read that there are some dns changes in 8.x I decided to upgrade to 8.0.3 before further troubleshooting.

The automatic app upgrade (Hetzner Cloud Server) failed because of a dpkg error in regards to grub.

Apr 01 02:00:00 http://archive.ubuntu.com/ubuntu focal-backports InRelease
[no timestamp]  package lists...
[no timestamp]  dependency tree...
[no timestamp]  state information...
Jan 01 01:00:00 packages can be upgraded. Run 'apt list --upgradable' to see them.
[no timestamp]  up grub-efi-amd64-signed (1.187.6~20.04.1+2.06-2ubuntu14.4) ...
[no timestamp]  /var/lib/grub/esp: special device /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi0-0-0-0-part15 does not exist.
[no timestamp]  error processing package grub-efi-amd64-signed (--configure):
[no timestamp]  installed grub-efi-amd64-signed package post-installation script subprocess returned error exit status 32
[no timestamp]  dependency problems prevent processing triggers for shim-signed:
[no timestamp]  shim-signed depends on grub-efi-amd64-signed (>= 1.187.2~) | grub-efi-arm64-signed (>= 1.187.2~); however:
[no timestamp]  Package grub-efi-amd64-signed is not configured yet.
[no timestamp]  Package grub-efi-arm64-signed is not installed.
[no timestamp] 
[no timestamp]  error processing package shim-signed (--configure):
[no timestamp]  dependency problems - leaving triggers unprocessed
[no timestamp]  were encountered while processing:
[no timestamp]  grub-efi-amd64-signed
[no timestamp]  shim-signed
Aug 15 16:30:26 ==> installer: dpkg reconfigure failed (try 3)

(for whatever reason the timestamps on the web log viewer are all over the place)

So I first installed package upgrades manually and could then in the interactive package configuration select the partition that should hold the grub record. Afterwards the Cloudron update was able to proceed.

Once the update was complete one of our custom apps was hanging in the "configuring reverse proxy" step. This was because one domain that was added as an alias to the app, and which was set to be configured manually was no longer pointing towards the app (in fact according to our documentation the domain has not been pointing towards the cloudron since at least may 2023), which made the lets encrypt http verification fail.

Unfortunately while the app is configuring the reverse proxy, the admin cannot make any changes to the aliases of the app. so i had to cancel the task, in the domain advanced settings set the certificate to "custom wildcard certificate" and was then able to retry the last task and had the app back online again.

One the app was online I removed the domain alias and set the certificate settings for the domain back to lets encrypt.

[girish]

Thanks for sharing . Right... our current domain/location configuration follows an all or nothing approach. If one or more fail, it will refuse to bring the app up and will get "stuck" forever . We have a task somewhere to make this more smart and also make the app repair feature smarter but it always gets pushed out as low priority.