Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task

[girish]

Further, I am unable to remove the docker directory even after stopping and disabling docker...

root@vmi557975:/# sudo systemctl stop docker
Warning: Stopping docker.service, but it can still be activated by:
  docker.socket
root@vmi557975:/# rm -rf /var/lib/docker
rm: cannot remove '/var/lib/docker/overlay2/ff0c38183c43e7bcaaf9b564d44f762e0b22f4bd77592a1f0ddae1507dff138d/merged': Device or resource busy
rm: cannot remove '/var/lib/docker/overlay2/d19d5ab400f594ee0a9ce613fbd44ebdd98d72cfa7b0ef7afd1705189272636e/merged': Device or resource busy
rm: cannot remove '/var/lib/docker/overlay2/d8c1b7219fb875bed31be5c935420240e86a29ae38e42f06c547d8d20534e1c0/merged': Device or resource busy
root@vmi557975:/# ps aux | grep docker
root     1330267  0.0  0.0   9068  2260 pts/0    S+   13:58   0:00 grep --color=auto docker

Some dangling container is causing problems:

root@vmi557975:/# ps aux | grep container
root      304611  0.0  0.0 710832  8624 ?        Sl   Nov20   0:24 /usr/bin/containerd-shim-runc-v2 -namespace moby -id adf60365e7ebeed38c53723ed75725ed398a434fb7aaec805a910edcaa5ae901 -address /run/containerd/containerd.sock
root     1196183  0.0  0.0 710768  8464 ?        Sl   05:24   0:04 /usr/bin/containerd-shim-runc-v2 -namespace moby -id d4b36c14c6c2850853bd62513ff6203135f75047a8640001accadcf62ef496b1 -address /run/containerd/containerd.sock
root     1330278  0.0  0.0   9068  2200 pts/0    R+   13:59   0:00 grep --color=auto container

I guess this is why it's hanging, it's unable to remove images.

[humptydumpty]

@girish what are my options to get the server back up? snapshot restore? dashboard is up! apps are in "configuring - queued" state.

[girish]

@humptydumpty yes, I just removed the entire /var/lib/docker and re-creating all the apps (had to reboot to even remove it since it was locked by various processes).

[humptydumpty]

@girish they're "configuring" now. thanks!

[humptydumpty]

@girish excluding the time to install ubuntu which was around 10 mins at most, do you think i missed any incoming mail during the addon container recreation time?

[girish]

@humptydumpty practically, all email servers will retry sending email if they caught you in the downtime.

[jdaviescoates]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

I did this now on @jdaviescoates ' server a while ago to see if it helps as well.

Well, something is different. I see in your email that an n8n update successfully worked after the changes you made, but I just tried updating the outstanding WordPress update...

Now just seemingly stuck here:

So I tried clicking Restart App, which didn't work, so I tried Retry Task, Retry app restart, which did work but of course the app still isn't updated, but back where it was before I tried to update.

So, tried to update again, exact same thing. Just stuck on "Updating".

Sigh. I'm beginning to regret my server move (not only because of this, but also because I've been hitting email issues too - in part because my new IP was/ is blacklisted, but also because email keeps timing out which never used to happen before either)

Adding log:

All I could see when it first got stuck on "Updating" was this:

Nov 22 14:02:09 [POST] /backup
Nov 22 14:02:09 backing up
Nov 22 14:02:09 13:M 22 Nov 2022 14:02:09.480 * DB saved on disk

[girish]

@jdaviescoates I think this is something else, maybe I forgot something, let me check.

[jdaviescoates]

Then the restart, retry, attempt to update again logs (too long to paste here):

https://paste.uniteddiversity.coop/?0c9b9bec1cd7f806#5WH55kCQDv7pHUZtuzqSjS5EZCSDRqh5aL2WLvBbeKvA

[girish]

@jdaviescoates this was a bug with the patch I made yesterday to use docker cli instead of the docker API. I will revert back the code on your server to use the API.

[humptydumpty]

@girish do you still need SSH access for further investigation?

[girish]

@jdaviescoates reverted now.

[girish]

@humptydumpty done, thanks for the access. Let me know after a few updates how things are working out.

[jdaviescoates]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

@jdaviescoates reverted now.

Thanks, tried again but:

[girish]

@jdaviescoates Yeah, ok, I am out of ideas then.

Just tried a simple example:

root@v2202211129685206445:/home/cloudron-support# docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete 
Digest: sha256:faa03e786c97f07ef34423fccceeec2398ec8a5759259f94d99078f264e9d7af
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
... <stuff > ...
root@v2202211129685206445:/home/cloudron-support# docker rm -f 7b6984e77727
7b6984e77727
root@v2202211129685206445:/home/cloudron-support# docker rmi hello-world
<hangs>

[jdaviescoates]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

@jdaviescoates Yeah, ok, I am out of ideas then.

[girish]

@jdaviescoates I don't know how motivated you are to track this down, but either you can ask them for another server? I have the VPS 500 G10s iv in netcup .

Another idea is to go back to plain ubuntu and try to reproduce this issue with just docker since it's not Cloudron related. But not sure what you will gain with this...

[girish]

@jdaviescoates BTW, what "driver" did you select in Media in netcup ? I see that I have chosen VIRTIO even though SCSI is recommended.

[jdaviescoates]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

@jdaviescoates BTW, what "driver" did you select in Media in netcup ? I see that I have chosen VIRTIO even though SCSI is recommended.

Yeah mine is VIRTIO too, I didn't manually choose that though, that's just how it was already and I chose not to touch anything for fear or messing things up (but yeah, I also noticed it says SCSI is recommended)

[jdaviescoates]

@jdaviescoates said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

@jdaviescoates reverted now.

Thanks, tried again but:

Although, after a reboot I was just successfully able to run 3 updates without issue... let's see how long it lasts... hopefully forever and the mystery will never need solving!