Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task
-
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
Did you all upgrade to 7.3?
In my case, yes, but that was a while ago and I only started seeing this issue after a server move to netcup.
I think @avatar1024 might be onto something about only effecting manual updates, but I'm not sure.
@jdaviescoates said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
I think @avatar1024 might be onto something about only effecting manual updates, but I'm not sure.
Actually overnight something has happened to one of my instances of n8n. It was not responding, restarting the app brought the app back up but it shows an update that it wont apply, it flashes Updating (Queued) - Queued for a second and then just goes to starting
-
@jdaviescoates said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
I think @avatar1024 might be onto something about only effecting manual updates, but I'm not sure.
Actually overnight something has happened to one of my instances of n8n. It was not responding, restarting the app brought the app back up but it shows an update that it wont apply, it flashes Updating (Queued) - Queued for a second and then just goes to starting
@jdaviescoates said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
it shows an update that it wont apply, it flashes Updating (Queued) - Queued for a second and then just goes to starting
Indeed I'm currently getting this behaviour on ALL apps that are showing an update, none of them will actually update!
Very concerning.
I'm on my phone on a train with patchy signal so can't easily properly investigate further at this stage...
-
@avatar1024 sorry I withdraw this. On that instance (7.3.2) updates work just fine. For some reasons the backup storage wasn't mounted which made the update fails (no error message though, the update process just stopped and the app restarted, I only noticed cause I went to see my notifications which indicates a backup had failed)
@avatar1024 said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
For some reasons the backup storage wasn't mounted which made the update fails
ah, this was actually my issue too!
for some reason my backup destination had become unmounted which was stopping updates working because they couldn't backup first, but why did that happen?!?
-
@avatar1024 said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
For some reasons the backup storage wasn't mounted which made the update fails
ah, this was actually my issue too!
for some reason my backup destination had become unmounted which was stopping updates working because they couldn't backup first, but why did that happen?!?
@jdaviescoates said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
my backup destination had become unmounted which was stopping updates working because they couldn't backup first,
But now the backup destination is reconnected they are still getting stuck at "Updating - Cleaning up old install".
This topic is NOT solved.
Either Cloudron 7.3.2 borked or Netcup is no longer compatible with Cloudron, or
? -
@jdaviescoates said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
my backup destination had become unmounted which was stopping updates working because they couldn't backup first,
But now the backup destination is reconnected they are still getting stuck at "Updating - Cleaning up old install".
This topic is NOT solved.
Either Cloudron 7.3.2 borked or Netcup is no longer compatible with Cloudron, or
?@jdaviescoates I feel like this topic mixes different issues. If app tasks are queued, then this is mostly the case if the system backup or too many tasks are currently running.
-
@jdaviescoates I feel like this topic mixes different issues. If app tasks are queued, then this is mostly the case if the system backup or too many tasks are currently running.
@nebulon true. It just happened to me again. Automatic updates don’t seem to run. Unless you pushed these updates after my scheduled auto updater. So i tried to update them manually and the app got stuck at cleaning up old install again. I restarted the server and it manually updated just fine. I’ll refrain from manually updating apps until i confirm the auto updates are working and will report back.
@jdaviescoates I’m with Contabo on an NVME plan. What’s yours?
-
@jdaviescoates I feel like this topic mixes different issues. If app tasks are queued, then this is mostly the case if the system backup or too many tasks are currently running.
@nebulon no, that wasn't the reason for the queueing thing flashing up, it was because the backup destination was unmounted (as mentioned above)
This issue needs a proper investigation, I've emailed support@ and enabled SSH access for you on my server.
There are lots of outstanding updates so hopefully you'll be able to reproduce (but perhaps you won't able to because I just rebooted the server which seems to temporarily make updates wotk again)
-
@nebulon true. It just happened to me again. Automatic updates don’t seem to run. Unless you pushed these updates after my scheduled auto updater. So i tried to update them manually and the app got stuck at cleaning up old install again. I restarted the server and it manually updated just fine. I’ll refrain from manually updating apps until i confirm the auto updates are working and will report back.
@jdaviescoates I’m with Contabo on an NVME plan. What’s yours?
@humptydumpty said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
@jdaviescoates I’m with Contabo on an NVME plan. What’s yours?
Netcup, same as @avatar1024
I never previously hit these issues with Hetzner
-
Yeah, I also think this post is mixing different situations. The "cleaning up old install" message is a known one in contabo. This is related to docker just getting stuck with those server. I don't know why it gets stuck and this is really hard to figure out without help from the VPS provider.
The other issue which @jdaviescoates is facing is backup not being mounted. This will make updates not work at all. Is there a notification in the eventlog about the failed update?
-
Yeah, I also think this post is mixing different situations. The "cleaning up old install" message is a known one in contabo. This is related to docker just getting stuck with those server. I don't know why it gets stuck and this is really hard to figure out without help from the VPS provider.
The other issue which @jdaviescoates is facing is backup not being mounted. This will make updates not work at all. Is there a notification in the eventlog about the failed update?
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
The "cleaning up old install" message is a known one in contabo.
@avatar1024 and I have both hit this on Netcup too
-
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
The "cleaning up old install" message is a known one in contabo.
@avatar1024 and I have both hit this on Netcup too
@jdaviescoates @girish me too
-
I have 3 apps that are stuck applying updates. The status of all 3 apps are identical in the stage of " Updating - Cleaning up old install ". What should I do now?
Cloudron v7.3.2 (Ubuntu 20.04.3 LTS)
The three sites are:
- ghost blog
- wordpress dev site
- listmonk
Here are the most recent logs under System Info > Logs
Nov 19 09:29:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:30:00 box:disks checkDiskSpace: checking disk space Nov 19 09:30:00 box:janitor Cleaning up expired tokens Nov 19 09:30:00 box:janitor Cleaned up 0 expired tokens Nov 19 09:30:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:30:00 box:disks checkDiskSpace: disk space checked. out of space: no Nov 19 09:30:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:30:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:30:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:30:40 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:30:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:31:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:31:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:31:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:31:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:31:40 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:31:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:32:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:32:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:32:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:32:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:32:36 box:locker Lock unreleased apptask Nov 19 09:32:40 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:32:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:33:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:33:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:33:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:33:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:33:40 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:33:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:34:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:34:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:34:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:34:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:34:40 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:34:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:35:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:35:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:35:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:35:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:35:40 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:35:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:36:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:36:08 box:updatechecker checkBoxUpdates: checking for updates Nov 19 09:36:08 box:updatechecker checkBoxUpdates: no updates Nov 19 09:36:08 box:updatechecker checkAppUpdates: checking for updates Nov 19 09:36:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:36:14 box:updatechecker checkBoxUpdates: checking for updates Nov 19 09:36:14 box:updatechecker checkBoxUpdates: no updates Nov 19 09:36:14 box:updatechecker checkAppUpdates: checking for updates Nov 19 09:36:19 box:updatechecker checkBoxUpdates: checking for updates Nov 19 09:36:19 box:updatechecker checkBoxUpdates: no updates Nov 19 09:36:19 box:updatechecker checkAppUpdates: checking for updates Nov 19 09:36:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:36:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:36:40 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:36:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:36:52 box:updatechecker checkBoxUpdates: checking for updates Nov 19 09:36:52 box:updatechecker checkBoxUpdates: no updates Nov 19 09:36:52 box:updatechecker checkAppUpdates: checking for updates Nov 19 09:36:57 box:updatechecker checkBoxUpdates: checking for updates Nov 19 09:36:58 box:updatechecker checkBoxUpdates: no updates Nov 19 09:36:58 box:updatechecker checkAppUpdates: checking for updates Nov 19 09:37:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:37:01 box:updatechecker checkBoxUpdates: checking for updates Nov 19 09:37:01 box:updatechecker checkBoxUpdates: no updates Nov 19 09:37:01 box:updatechecker checkAppUpdates: checking for updates Nov 19 09:37:06 box:updatechecker checkBoxUpdates: checking for updates Nov 19 09:37:06 box:updatechecker checkBoxUpdates: no updates Nov 19 09:37:06 box:updatechecker checkAppUpdates: checking for updates Nov 19 09:37:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:37:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:37:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:37:36 box:locker Lock unreleased apptask Nov 19 09:37:40 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:37:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:38:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:38:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:38:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:38:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:38:40 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:38:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:39:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:39:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:39:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:39:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:39:40 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:39:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:40:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:40:00 box:cloudron Getting logs for box as json Nov 19 09:40:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:40:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:40:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:40:40 box:apphealthmonitor app health: 21 alive / 4 dead.Note: It states 4 dead sites. One of them is a purposefully stopped app so please disregard that.
Services just keep on spinning for some:
Disk is not full
Boot is fine
@humptydumpty Hello, I've "Configuring - Cleaning up old install" problem too updating LAMPP
-
@humptydumpty Hello, I've "Configuring - Cleaning up old install" problem too updating LAMPP
-
@girish Hetzner VPS
Edit:
- Also "Recovering" backup it was impossible, always stucked on "Cleaning up old install".
So I rebooted while stucked on recovering old backup and recovering process was ok.
Now LAMPP is running with older version previous to 3.0.0 package
-
Ok, I think maybe this is storage box related.
@p44 @imc67 @jdaviescoates are you guys using storage box/ CIFS ?
-
Ok, I think maybe this is storage box related.
@p44 @imc67 @jdaviescoates are you guys using storage box/ CIFS ?
-
I imc67 referenced this topic on
-
Ok, I think maybe this is storage box related.
@p44 @imc67 @jdaviescoates are you guys using storage box/ CIFS ?
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
Ok, I think maybe this is storage box related.
@p44 @imc67 @jdaviescoates are you guys using storage box/ CIFS ?
Yes, indeed I am, @avatar1024 too, I think.
But I've been using it for ages before hitting this problem.
Saying that, previously it was a filesystem mount and not a CIFS.
Well, it was a CIFS, but not set-up by Cloudron, manually set-up by editing fstab following these instructions https://docs.hetzner.com/robot/storage-box/access/access-samba-cifs
Anyways, I only started hitting this issue once I moved from Hetzner Cloud VPS to Netcup vServer.
-
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
Ok, I think maybe this is storage box related.
@p44 @imc67 @jdaviescoates are you guys using storage box/ CIFS ?
Yes, indeed I am, @avatar1024 too, I think.
But I've been using it for ages before hitting this problem.
Saying that, previously it was a filesystem mount and not a CIFS.
Well, it was a CIFS, but not set-up by Cloudron, manually set-up by editing fstab following these instructions https://docs.hetzner.com/robot/storage-box/access/access-samba-cifs
Anyways, I only started hitting this issue once I moved from Hetzner Cloud VPS to Netcup vServer.
@jdaviescoates Initial hypothesis is that maybe hetzner storage box had a "reset" on their side . CIFS mounts are not auto-mounted (whether by Cloudron or not) because the mount system doesn't have such a feature. So, we were thinking the backups stopped working and when they stop working, updates were getting stuck.
-
@jdaviescoates Initial hypothesis is that maybe hetzner storage box had a "reset" on their side . CIFS mounts are not auto-mounted (whether by Cloudron or not) because the mount system doesn't have such a feature. So, we were thinking the backups stopped working and when they stop working, updates were getting stuck.
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
@jdaviescoates Initial hypothesis is that maybe hetzner storage box had a "reset" on their side . CIFS mounts are not auto-mounted (whether by Cloudron or not) because the mount system doesn't have such a feature. So, we were thinking the backups stopped working and when they stop working, updates were getting stuck.
Thanks for investigating, but I'm not sure that is it.
I only say that because right now I've got 3 apps stuck on "Updating - Cleaning up old install"
And yet the backup drive is mounted fine:
-
Thanks for @jdaviescoates for access to his system.
Basically, docker just stops working for some reason. The command below just hangs.
root@v2202211129685206445:/home/yellowtent/platformdata/logs/70b06ed9-1257-46d1-9fe9-1ffa7c722f21# docker stop d3ef55aa711f8d424f53bbef6a5ac813e9670cf89e77eaa171762dc524610ebcI see that docker is a few versions behind and there is a bunch of stuff related to locks and races, I have gone ahead and updated docker. @jdaviescoates Can you let me know if you hit this again? In the meantime, I will put the docker update in our ci for 7.3.3.
