SOLVED "Error" for multiple apps

[uiharu]

Multiple apps on my Cloudron (version 2.3.2) are in "Error" status.
Accessing the apps throw an nginx 404 page.

Before I discovered this situation, I did

apt update 
apt autoremove
dpkg --configure --a
reboot

because the admin panel was stuck at an automatic box update, and /boot was nearly full due to kernel compilations.
However, I'm not sure if the problem already existed beforehand (or not).

Attempting to solve the problem, I tried to

• repair the app
• restore from the latest backup
• restore from earlier backups
• uninstall the app
  but none of the above worked and ended with "Error".

I also attempted

• installing an new instance of one of the faulty apps -> "Error"
• installing a new instance of an app that's not one of the faulty ones -> "Error"
• Clicking "Backup now" in settings -> "external error"

Edit: now I'm pretty sure I accessed the admin panel yesterday and things seemed fine. so the problem's prob due to

• fiddling with apt and dpkg manually
• the update got aborted midway by reboot and now I end up with a broken installation

[msbt]

Hey, what worked for me after /boot was 100% full and the update-installer was stuck:

//show all kernels not in use
sudo dpkg --list 'linux-image*'|awk '{ if ($1=="ii") print $2}'|grep -v `uname -r`
sudo systemctl stop box
sudo systemctl stop docker
//do ps -ax to find the id of the installer
sudo kill -9 ID_of_installer.sh
//remove old kernels, adjust this to the results from above
sudo rm -rf /boot/*-4.4.0-{98,97,96,93,62}-*
//fix packages
sudo apt-get -f install
//automatically remove unneeded kernels
sudo purge-old-kernels
//bring it back on
sudo systemctl restart box
sudo systemctl restart docker
sudo systemctl restart cloudron.target

edit: hope i didn't forget to escape any ' or *

[uiharu]

Thanks @msbt for the your answer.
sudo apt-get -f install doesn't (re)install anything...

service box status gives some errors (partly redacted)

  node[2046]: box:apphealthmonitor [domain] org.ghost.cloudronapp2 skipped. istate:error rstate:stopped - 
  node[2046]: box:apphealthmonitor apps alive: [naked_domain|com.nextcloud.cloudronapp, note|de.nebulon.guacamoly, git|com.gitlab.cloudronapp, ...
  node[2046]: box:apphealthmonitor [domain] org.etherpad.cloudronapp skipped. istate:error rstate:stopped - 
  node[2046]: box:apphealthmonitor [domain] com.monicahq.cloudronapp skipped. istate:error rstate:stopped - 
  node[2046]: box:apphealthmonitor [domain] com.invoiceninja.cloudronapp skipped. istate:error rstate:stopped - 
  node[2046]: box:apphealthmonitor [domain] org.wordpress.cloudronapp skipped. istate:error rstate:stopped - 
  node[2046]: box:apphealthmonitor [domain] org.tt_rss.cloudronapp skipped. istate:error rstate:stopped - 
  node[2046]: box:apphealthmonitor [domain] org.ghost.cloudronapp2 skipped. istate:error rstate:stopped - 
  node[2046]: box:apphealthmonitor [domain] org.ghost.cloudronapp2 skipped. istate:error rstate:stopped - 
  node[2046]: box:apphealthmonitor apps alive: [naked_domain|com.nextcloud.cloudronapp, note|de.nebulon.guacamoly, git|com.gitlab.cloudronapp, ...

Please kindly note that I'm not having problems with /boot anymore nor am I trying to upgrade - /boot now has some free space and apt upgrade has cleared...

[uiharu]

I'm wondering if the system is missing an necessary kernal...
I now have 4.4.0-127 and 4.4.0-128

$ apt list linux-image* --installed
Listing... Done
linux-image-4.4.0-127-generic/xenial-updates,xenial-security,now 4.4.0-127.153 amd64 [installed,automatic]
linux-image-4.4.0-128-generic/xenial-updates,xenial-security,now 4.4.0-128.154 amd64 [installed,automatic]
linux-image-extra-4.4.0-127-generic/xenial-updates,xenial-security,now 4.4.0-127.153 amd64 [installed,automatic]
linux-image-extra-4.4.0-128-generic/xenial-updates,xenial-security,now 4.4.0-128.154 amd64 [installed,automatic]

[girish]

@uiharu if you cleared the boot issue, just press configure button of an app and press save. The app should come back up.

If not, can you paste the output of 'docker ps'?

[uiharu]

Thanks @girish

I tried again but still get "Error"

• repair the app
• restore from the latest backup
• restore from earlier backups
• uninstall the app
  but none of the above worked and ended with "Error".

I also attempted

• installing an new instance of one of the faulty apps -> "Error"
• installing a new instance of an app that's not one of the faulty ones -> "Error"
• Clicking "Backup now" in settings -> "external error"

[deleted]

[girish]

@uiharu thanks, if you click on the app icon usually it will display the full error. Does it not say anything?

[girish]

Actually, please try 'sudo systemctl restart docker' and wait for a minute and then reconfigure an app to see if it changes anything.

[uiharu]

Unfortunately, the "Error for doamin" overlays are blank (except the buttons)...
restarting docker...

[deleted]

[uiharu]

@girish

The apps still throw "Error" when i attempt to repair/reconfigure them
Edit: Uninstalling an app also ends with "Error"

[girish]

@uiharu I sent you a mail from support. I think I will need ssh access to debug what is happening. It's most likely something docker related since the cloudron code itself seems to run fine.

[girish]

Looks like this Cloudron hit https://forum.cloudron.io/topic/1316/apps-using-mysql-are-not-working/2. MySQL apps are not working. The workaround is also not working.

[uiharu]

I forgot to post a follow up on this thread. I'd like to thank @girish for solving my problem. He responded quickly and arranged to ssh into my server to diagnose. He's been very patient and professional and I really appreciate it.

Basically the problem is due to MySQL addons not properly starting/building, so all apps using MySQL error out and cannot start or install. Hope the problem has been resolved in the final 2.4 releases