Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task

[nebulon]

@jdaviescoates I feel like this topic mixes different issues. If app tasks are queued, then this is mostly the case if the system backup or too many tasks are currently running.

[humptydumpty]

@nebulon true. It just happened to me again. Automatic updates don’t seem to run. Unless you pushed these updates after my scheduled auto updater. So i tried to update them manually and the app got stuck at cleaning up old install again. I restarted the server and it manually updated just fine. I’ll refrain from manually updating apps until i confirm the auto updates are working and will report back.

@jdaviescoates I’m with Contabo on an NVME plan. What’s yours?

[jdaviescoates]

@nebulon no, that wasn't the reason for the queueing thing flashing up, it was because the backup destination was unmounted (as mentioned above)

This issue needs a proper investigation, I've emailed support@ and enabled SSH access for you on my server.

There are lots of outstanding updates so hopefully you'll be able to reproduce (but perhaps you won't able to because I just rebooted the server which seems to temporarily make updates wotk again)

[jdaviescoates]

@humptydumpty said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

@jdaviescoates I’m with Contabo on an NVME plan. What’s yours?

Netcup, same as @avatar1024

I never previously hit these issues with Hetzner

[girish]

Yeah, I also think this post is mixing different situations. The "cleaning up old install" message is a known one in contabo. This is related to docker just getting stuck with those server. I don't know why it gets stuck and this is really hard to figure out without help from the VPS provider.

The other issue which @jdaviescoates is facing is backup not being mounted. This will make updates not work at all. Is there a notification in the eventlog about the failed update?

[jdaviescoates]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

The "cleaning up old install" message is a known one in contabo.

@avatar1024 and I have both hit this on Netcup too

[imc67]

@jdaviescoates @girish me too

[p44]

@humptydumpty Hello, I've "Configuring - Cleaning up old install" problem too updating LAMPP

[girish]

@p44 where is your server hosted?

[p44]

@girish Hetzner VPS

Edit:

• Also "Recovering" backup it was impossible, always stucked on "Cleaning up old install".

So I rebooted while stucked on recovering old backup and recovering process was ok.

Now LAMPP is running with older version previous to 3.0.0 package

[girish]

Ok, I think maybe this is storage box related.

@p44 @imc67 @jdaviescoates are you guys using storage box/ CIFS ?

[p44]

@girish Ok, yes I'm using storage box/CIFS with Hetzner Storage Box service

[jdaviescoates]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

Ok, I think maybe this is storage box related.

@p44 @imc67 @jdaviescoates are you guys using storage box/ CIFS ?

Yes, indeed I am, @avatar1024 too, I think.

But I've been using it for ages before hitting this problem.

Saying that, previously it was a filesystem mount and not a CIFS.

Well, it was a CIFS, but not set-up by Cloudron, manually set-up by editing fstab following these instructions https://docs.hetzner.com/robot/storage-box/access/access-samba-cifs

Anyways, I only started hitting this issue once I moved from Hetzner Cloud VPS to Netcup vServer.

[girish]

@jdaviescoates Initial hypothesis is that maybe hetzner storage box had a "reset" on their side . CIFS mounts are not auto-mounted (whether by Cloudron or not) because the mount system doesn't have such a feature. So, we were thinking the backups stopped working and when they stop working, updates were getting stuck.

[jdaviescoates]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

@jdaviescoates Initial hypothesis is that maybe hetzner storage box had a "reset" on their side . CIFS mounts are not auto-mounted (whether by Cloudron or not) because the mount system doesn't have such a feature. So, we were thinking the backups stopped working and when they stop working, updates were getting stuck.

Thanks for investigating, but I'm not sure that is it.

I only say that because right now I've got 3 apps stuck on "Updating - Cleaning up old install"

And yet the backup drive is mounted fine:

[girish]

Thanks for @jdaviescoates for access to his system.

Basically, docker just stops working for some reason. The command below just hangs.

root@v2202211129685206445:/home/yellowtent/platformdata/logs/70b06ed9-1257-46d1-9fe9-1ffa7c722f21# docker stop d3ef55aa711f8d424f53bbef6a5ac813e9670cf89e77eaa171762dc524610ebc

I see that docker is a few versions behind and there is a bunch of stuff related to locks and races, I have gone ahead and updated docker. @jdaviescoates Can you let me know if you hit this again? In the meantime, I will put the docker update in our ci for 7.3.3.

[jdaviescoates]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

Thanks for @jdaviescoates for access to his system.

You're welcome! Thanks for investigating.

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

Can you let me know if you hit this again?

Will do!

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

I will put the docker update in our ci for 7.3.3

Presumably until 7.3.3 is released and I upgrade I'll likely keep hitting this issue, right?

[girish]

@jdaviescoates I updated the docker on your server already . That's why, please let us know if you hit the issue again (which then means that docker upgrade didn't fix the issue).

But regardless, I will keep the docker update as part of 7.3.3 unless something else breaks. So, you won't have troubles after updating as well.

[jdaviescoates]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

@jdaviescoates I updated the docker on your server already . That's why, please let us know if you hit the issue again (which then means that docker upgrade didn't fix the issue).

Ah, right, got it, thanks! Will see what happens going forward...

[imc67]

@girish for backups yes