Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task

[humptydumpty]

I have 3 apps that are stuck applying updates. The status of all 3 apps are identical in the stage of " Updating - Cleaning up old install ". What should I do now?

Cloudron v7.3.2 (Ubuntu 20.04.3 LTS)

The three sites are:

• ghost blog
• wordpress dev site
• listmonk

Here are the most recent logs under System Info > Logs

Nov 19 09:29:50 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:30:00 box:disks checkDiskSpace: checking disk space
Nov 19 09:30:00 box:janitor Cleaning up expired tokens
Nov 19 09:30:00 box:janitor Cleaned up 0 expired tokens
Nov 19 09:30:00 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:30:00 box:disks checkDiskSpace: disk space checked. out of space: no
Nov 19 09:30:10 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:30:20 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:30:30 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:30:40 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:30:50 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:31:00 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:31:10 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:31:20 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:31:30 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:31:40 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:31:50 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:32:00 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:32:10 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:32:20 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:32:30 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:32:36 box:locker Lock unreleased apptask
Nov 19 09:32:40 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:32:50 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:33:00 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:33:10 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:33:20 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:33:30 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:33:40 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:33:50 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:34:00 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:34:10 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:34:20 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:34:30 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:34:40 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:34:50 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:35:00 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:35:10 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:35:20 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:35:30 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:35:40 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:35:50 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:36:00 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:36:08 box:updatechecker checkBoxUpdates: checking for updates
Nov 19 09:36:08 box:updatechecker checkBoxUpdates: no updates
Nov 19 09:36:08 box:updatechecker checkAppUpdates: checking for updates
Nov 19 09:36:10 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:36:14 box:updatechecker checkBoxUpdates: checking for updates
Nov 19 09:36:14 box:updatechecker checkBoxUpdates: no updates
Nov 19 09:36:14 box:updatechecker checkAppUpdates: checking for updates
Nov 19 09:36:19 box:updatechecker checkBoxUpdates: checking for updates
Nov 19 09:36:19 box:updatechecker checkBoxUpdates: no updates
Nov 19 09:36:19 box:updatechecker checkAppUpdates: checking for updates
Nov 19 09:36:20 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:36:30 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:36:40 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:36:50 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:36:52 box:updatechecker checkBoxUpdates: checking for updates
Nov 19 09:36:52 box:updatechecker checkBoxUpdates: no updates
Nov 19 09:36:52 box:updatechecker checkAppUpdates: checking for updates
Nov 19 09:36:57 box:updatechecker checkBoxUpdates: checking for updates
Nov 19 09:36:58 box:updatechecker checkBoxUpdates: no updates
Nov 19 09:36:58 box:updatechecker checkAppUpdates: checking for updates
Nov 19 09:37:00 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:37:01 box:updatechecker checkBoxUpdates: checking for updates
Nov 19 09:37:01 box:updatechecker checkBoxUpdates: no updates
Nov 19 09:37:01 box:updatechecker checkAppUpdates: checking for updates
Nov 19 09:37:06 box:updatechecker checkBoxUpdates: checking for updates
Nov 19 09:37:06 box:updatechecker checkBoxUpdates: no updates
Nov 19 09:37:06 box:updatechecker checkAppUpdates: checking for updates
Nov 19 09:37:10 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:37:20 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:37:30 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:37:36 box:locker Lock unreleased apptask
Nov 19 09:37:40 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:37:50 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:38:00 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:38:10 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:38:20 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:38:30 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:38:40 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:38:50 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:39:00 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:39:10 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:39:20 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:39:30 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:39:40 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:39:50 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:40:00 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:40:00 box:cloudron Getting logs for box as json
Nov 19 09:40:10 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:40:20 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:40:30 box:apphealthmonitor app health: 21 alive / 4 dead.
Nov 19 09:40:40 box:apphealthmonitor app health: 21 alive / 4 dead.

Note: It states 4 dead sites. One of them is a purposefully stopped app so please disregard that.

Services just keep on spinning for some:

Disk is not full

Boot is fine

[humptydumpty]

I'm able to stop the task and "retry update task" just to get the site back up. However, if I try to manually update any of the apps, they get stuck at "cleaning up old install". Suggestions?

Edit: after stopping all update tasks and the apps are up again. I rechecked the services page and it takes a full minute for the service check to populate. All are green.

All redis apps are green.

[girish]

@humptydumpty Can you try docker image prune -a ? Is it getting stuck?

[humptydumpty]

@girish can i try that in any of the app terminals or do i need to use ssh? nvm i figured out how to log in to ssh.

[humptydumpty]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

docker image prune -a

nothing happening
edit: still no change to the above

[girish]

@humptydumpty Yeah, this happens quite happen in contabo servers. Docker simply basically becomes unresponsive, I am not sure why. My guess is that disk access becomes super slow and it commands that use disk a lot (like pruning disk images), just get stuck.

In the past, just rebooting the server seemed to do the trick. Maybe you can try that?

[humptydumpty]

@girish Thank you. Restarting did the trick. I was able to update the apps manually and ran the command you asked and got a result back "Total reclaimed space: 70.23MB". This is the first time I've ran into this issue with Contabo. I'll keep what you said in mind. Thank you!

[avatar1024]

I've running into this issue as well over the past couple of days. I'm using Netcup as a provider.

[imc67]

@avatar1024 same here on one of my 4 NetCup Cloudrons 2 apps hung in this task, after rebooting the server from the Cloudron GUI (2 days ago via the NetCup GUI) it was solved.

[jdaviescoates]

@avatar1024 I've just moved to netcup and hit it too, so was wondering if you had too.

Now I'm beginning to wonder if I should've moved to them at all

[girish]

I am wondering there is a sudden flurry of docker related issue suddenly. Did anything change? Did you all upgrade to 7.3?

[avatar1024]

@girish on that instance I'm still on 7.2.5

[imc67]

@girish nope still on 7.2.5

[avatar1024]

Also seems to have issues in my other instances which is on 7.3.2. Also it feels like it's only a problem with manual updates...but that's unsure. I can report tomorrow after the next auto update cycle.

[avatar1024]

@avatar1024 sorry I withdraw this. On that instance (7.3.2) updates work just fine. For some reasons the backup storage wasn't mounted which made the update fails (no error message though, the update process just stopped and the app restarted, I only noticed cause I went to see my notifications which indicates a backup had failed)

[jdaviescoates]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

Did you all upgrade to 7.3?

In my case, yes, but that was a while ago and I only started seeing this issue after a server move to netcup.

I think @avatar1024 might be onto something about only effecting manual updates, but I'm not sure.

[jdaviescoates]

@jdaviescoates said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

I think @avatar1024 might be onto something about only effecting manual updates, but I'm not sure.

Actually overnight something has happened to one of my instances of n8n. It was not responding, restarting the app brought the app back up but it shows an update that it wont apply, it flashes Updating (Queued) - Queued for a second and then just goes to starting

[jdaviescoates]

@jdaviescoates said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

it shows an update that it wont apply, it flashes Updating (Queued) - Queued for a second and then just goes to starting

Indeed I'm currently getting this behaviour on ALL apps that are showing an update, none of them will actually update!

Very concerning.

I'm on my phone on a train with patchy signal so can't easily properly investigate further at this stage...

[jdaviescoates]

@avatar1024 said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

For some reasons the backup storage wasn't mounted which made the update fails

ah, this was actually my issue too!

for some reason my backup destination had become unmounted which was stopping updates working because they couldn't backup first, but why did that happen?!?

[jdaviescoates]

@jdaviescoates said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

my backup destination had become unmounted which was stopping updates working because they couldn't backup first,

But now the backup destination is reconnected they are still getting stuck at "Updating - Cleaning up old install".

This topic is NOT solved.

Either Cloudron 7.3.2 borked or Netcup is no longer compatible with Cloudron, or ?