Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task
-
@jdaviescoates I feel like this topic mixes different issues. If app tasks are queued, then this is mostly the case if the system backup or too many tasks are currently running.
@nebulon true. It just happened to me again. Automatic updates don’t seem to run. Unless you pushed these updates after my scheduled auto updater. So i tried to update them manually and the app got stuck at cleaning up old install again. I restarted the server and it manually updated just fine. I’ll refrain from manually updating apps until i confirm the auto updates are working and will report back.
@jdaviescoates I’m with Contabo on an NVME plan. What’s yours?
-
@jdaviescoates I feel like this topic mixes different issues. If app tasks are queued, then this is mostly the case if the system backup or too many tasks are currently running.
@nebulon no, that wasn't the reason for the queueing thing flashing up, it was because the backup destination was unmounted (as mentioned above)
This issue needs a proper investigation, I've emailed support@ and enabled SSH access for you on my server.
There are lots of outstanding updates so hopefully you'll be able to reproduce (but perhaps you won't able to because I just rebooted the server which seems to temporarily make updates wotk again)
-
@nebulon true. It just happened to me again. Automatic updates don’t seem to run. Unless you pushed these updates after my scheduled auto updater. So i tried to update them manually and the app got stuck at cleaning up old install again. I restarted the server and it manually updated just fine. I’ll refrain from manually updating apps until i confirm the auto updates are working and will report back.
@jdaviescoates I’m with Contabo on an NVME plan. What’s yours?
@humptydumpty said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
@jdaviescoates I’m with Contabo on an NVME plan. What’s yours?
Netcup, same as @avatar1024
I never previously hit these issues with Hetzner
-
Yeah, I also think this post is mixing different situations. The "cleaning up old install" message is a known one in contabo. This is related to docker just getting stuck with those server. I don't know why it gets stuck and this is really hard to figure out without help from the VPS provider.
The other issue which @jdaviescoates is facing is backup not being mounted. This will make updates not work at all. Is there a notification in the eventlog about the failed update?
-
Yeah, I also think this post is mixing different situations. The "cleaning up old install" message is a known one in contabo. This is related to docker just getting stuck with those server. I don't know why it gets stuck and this is really hard to figure out without help from the VPS provider.
The other issue which @jdaviescoates is facing is backup not being mounted. This will make updates not work at all. Is there a notification in the eventlog about the failed update?
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
The "cleaning up old install" message is a known one in contabo.
@avatar1024 and I have both hit this on Netcup too
-
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
The "cleaning up old install" message is a known one in contabo.
@avatar1024 and I have both hit this on Netcup too
@jdaviescoates @girish me too
-
I have 3 apps that are stuck applying updates. The status of all 3 apps are identical in the stage of " Updating - Cleaning up old install ". What should I do now?
Cloudron v7.3.2 (Ubuntu 20.04.3 LTS)
The three sites are:
- ghost blog
- wordpress dev site
- listmonk
Here are the most recent logs under System Info > Logs
Nov 19 09:29:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:30:00 box:disks checkDiskSpace: checking disk space Nov 19 09:30:00 box:janitor Cleaning up expired tokens Nov 19 09:30:00 box:janitor Cleaned up 0 expired tokens Nov 19 09:30:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:30:00 box:disks checkDiskSpace: disk space checked. out of space: no Nov 19 09:30:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:30:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:30:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:30:40 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:30:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:31:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:31:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:31:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:31:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:31:40 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:31:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:32:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:32:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:32:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:32:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:32:36 box:locker Lock unreleased apptask Nov 19 09:32:40 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:32:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:33:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:33:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:33:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:33:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:33:40 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:33:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:34:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:34:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:34:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:34:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:34:40 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:34:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:35:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:35:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:35:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:35:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:35:40 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:35:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:36:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:36:08 box:updatechecker checkBoxUpdates: checking for updates Nov 19 09:36:08 box:updatechecker checkBoxUpdates: no updates Nov 19 09:36:08 box:updatechecker checkAppUpdates: checking for updates Nov 19 09:36:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:36:14 box:updatechecker checkBoxUpdates: checking for updates Nov 19 09:36:14 box:updatechecker checkBoxUpdates: no updates Nov 19 09:36:14 box:updatechecker checkAppUpdates: checking for updates Nov 19 09:36:19 box:updatechecker checkBoxUpdates: checking for updates Nov 19 09:36:19 box:updatechecker checkBoxUpdates: no updates Nov 19 09:36:19 box:updatechecker checkAppUpdates: checking for updates Nov 19 09:36:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:36:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:36:40 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:36:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:36:52 box:updatechecker checkBoxUpdates: checking for updates Nov 19 09:36:52 box:updatechecker checkBoxUpdates: no updates Nov 19 09:36:52 box:updatechecker checkAppUpdates: checking for updates Nov 19 09:36:57 box:updatechecker checkBoxUpdates: checking for updates Nov 19 09:36:58 box:updatechecker checkBoxUpdates: no updates Nov 19 09:36:58 box:updatechecker checkAppUpdates: checking for updates Nov 19 09:37:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:37:01 box:updatechecker checkBoxUpdates: checking for updates Nov 19 09:37:01 box:updatechecker checkBoxUpdates: no updates Nov 19 09:37:01 box:updatechecker checkAppUpdates: checking for updates Nov 19 09:37:06 box:updatechecker checkBoxUpdates: checking for updates Nov 19 09:37:06 box:updatechecker checkBoxUpdates: no updates Nov 19 09:37:06 box:updatechecker checkAppUpdates: checking for updates Nov 19 09:37:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:37:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:37:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:37:36 box:locker Lock unreleased apptask Nov 19 09:37:40 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:37:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:38:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:38:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:38:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:38:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:38:40 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:38:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:39:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:39:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:39:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:39:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:39:40 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:39:50 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:40:00 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:40:00 box:cloudron Getting logs for box as json Nov 19 09:40:10 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:40:20 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:40:30 box:apphealthmonitor app health: 21 alive / 4 dead. Nov 19 09:40:40 box:apphealthmonitor app health: 21 alive / 4 dead.Note: It states 4 dead sites. One of them is a purposefully stopped app so please disregard that.
Services just keep on spinning for some:
Disk is not full
Boot is fine
@humptydumpty Hello, I've "Configuring - Cleaning up old install" problem too updating LAMPP
-
@humptydumpty Hello, I've "Configuring - Cleaning up old install" problem too updating LAMPP
@p44 where is your server hosted?
-
@girish Hetzner VPS
Edit:
- Also "Recovering" backup it was impossible, always stucked on "Cleaning up old install".
So I rebooted while stucked on recovering old backup and recovering process was ok.
Now LAMPP is running with older version previous to 3.0.0 package
-
Ok, I think maybe this is storage box related.
@p44 @imc67 @jdaviescoates are you guys using storage box/ CIFS ?
-
Ok, I think maybe this is storage box related.
@p44 @imc67 @jdaviescoates are you guys using storage box/ CIFS ?
-
I imc67 referenced this topic on
-
Ok, I think maybe this is storage box related.
@p44 @imc67 @jdaviescoates are you guys using storage box/ CIFS ?
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
Ok, I think maybe this is storage box related.
@p44 @imc67 @jdaviescoates are you guys using storage box/ CIFS ?
Yes, indeed I am, @avatar1024 too, I think.
But I've been using it for ages before hitting this problem.
Saying that, previously it was a filesystem mount and not a CIFS.
Well, it was a CIFS, but not set-up by Cloudron, manually set-up by editing fstab following these instructions https://docs.hetzner.com/robot/storage-box/access/access-samba-cifs
Anyways, I only started hitting this issue once I moved from Hetzner Cloud VPS to Netcup vServer.
-
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
Ok, I think maybe this is storage box related.
@p44 @imc67 @jdaviescoates are you guys using storage box/ CIFS ?
Yes, indeed I am, @avatar1024 too, I think.
But I've been using it for ages before hitting this problem.
Saying that, previously it was a filesystem mount and not a CIFS.
Well, it was a CIFS, but not set-up by Cloudron, manually set-up by editing fstab following these instructions https://docs.hetzner.com/robot/storage-box/access/access-samba-cifs
Anyways, I only started hitting this issue once I moved from Hetzner Cloud VPS to Netcup vServer.
@jdaviescoates Initial hypothesis is that maybe hetzner storage box had a "reset" on their side . CIFS mounts are not auto-mounted (whether by Cloudron or not) because the mount system doesn't have such a feature. So, we were thinking the backups stopped working and when they stop working, updates were getting stuck.
-
@jdaviescoates Initial hypothesis is that maybe hetzner storage box had a "reset" on their side . CIFS mounts are not auto-mounted (whether by Cloudron or not) because the mount system doesn't have such a feature. So, we were thinking the backups stopped working and when they stop working, updates were getting stuck.
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
@jdaviescoates Initial hypothesis is that maybe hetzner storage box had a "reset" on their side . CIFS mounts are not auto-mounted (whether by Cloudron or not) because the mount system doesn't have such a feature. So, we were thinking the backups stopped working and when they stop working, updates were getting stuck.
Thanks for investigating, but I'm not sure that is it.
I only say that because right now I've got 3 apps stuck on "Updating - Cleaning up old install"
And yet the backup drive is mounted fine:
-
Thanks for @jdaviescoates for access to his system.
Basically, docker just stops working for some reason. The command below just hangs.
root@v2202211129685206445:/home/yellowtent/platformdata/logs/70b06ed9-1257-46d1-9fe9-1ffa7c722f21# docker stop d3ef55aa711f8d424f53bbef6a5ac813e9670cf89e77eaa171762dc524610ebcI see that docker is a few versions behind and there is a bunch of stuff related to locks and races, I have gone ahead and updated docker. @jdaviescoates Can you let me know if you hit this again? In the meantime, I will put the docker update in our ci for 7.3.3.
-
Thanks for @jdaviescoates for access to his system.
Basically, docker just stops working for some reason. The command below just hangs.
root@v2202211129685206445:/home/yellowtent/platformdata/logs/70b06ed9-1257-46d1-9fe9-1ffa7c722f21# docker stop d3ef55aa711f8d424f53bbef6a5ac813e9670cf89e77eaa171762dc524610ebcI see that docker is a few versions behind and there is a bunch of stuff related to locks and races, I have gone ahead and updated docker. @jdaviescoates Can you let me know if you hit this again? In the meantime, I will put the docker update in our ci for 7.3.3.
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
Thanks for @jdaviescoates for access to his system.
You're welcome! Thanks for investigating.
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
Can you let me know if you hit this again?
Will do!
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
I will put the docker update in our ci for 7.3.3
Presumably until 7.3.3 is released and I upgrade I'll likely keep hitting this issue, right?
-
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
Thanks for @jdaviescoates for access to his system.
You're welcome! Thanks for investigating.
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
Can you let me know if you hit this again?
Will do!
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
I will put the docker update in our ci for 7.3.3
Presumably until 7.3.3 is released and I upgrade I'll likely keep hitting this issue, right?
@jdaviescoates I updated the docker on your server already . That's why, please let us know if you hit the issue again (which then means that docker upgrade didn't fix the issue).
But regardless, I will keep the docker update as part of 7.3.3 unless something else breaks. So, you won't have troubles after updating as well.
-
@jdaviescoates I updated the docker on your server already . That's why, please let us know if you hit the issue again (which then means that docker upgrade didn't fix the issue).
But regardless, I will keep the docker update as part of 7.3.3 unless something else breaks. So, you won't have troubles after updating as well.
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
@jdaviescoates I updated the docker on your server already . That's why, please let us know if you hit the issue again (which then means that docker upgrade didn't fix the issue).
Ah, right, got it, thanks! Will see what happens going forward...
-
Ok, I think maybe this is storage box related.
@p44 @imc67 @jdaviescoates are you guys using storage box/ CIFS ?
-
@girish I'm sorry to report that it doesn't seem outdated Docker was the issue because I noticed there were still outstanding updates to run and so clicked on one and the exact same thing has happened:
I've left it exactly as is if you want to go and have another poke around my server to see what you can discover...
