Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task
-
I narrowed it down to image delete API call not getting a response. I wrote a simple example and that works
So, some strange issue in the docker-modem module. Unfortunately, there's a lot of code there to investigate.I do still wonder why this happens just now...!
'use strict'; const http = require('http'); const opts = { path: '/images/cloudron/org.moodle.cloudronapp:20220912-100537-1689a722e?force=false&noprune=false', method: 'DELETE', headers: {}, key: undefined, cert: undefined, ca: undefined, socketPath: '/var/run/docker.sock' }; const req = http.request(opts, function () { }); req.on('response', function (res) { console.log('got a response'); res.on('data', function () { console.log('data'); }); res.on('end', function () { console.log('ended'); }); }); req.on('error', function (error) { console.log(error); }); req.end(); -
It turns out it is docker issue after all. If I replace the module with calling docker CLI, it just hangs as well
It seems the command itself succeeds (i.e images gets deleted) but the command does not return. This happens across many commands, docker just hangs. No error in their logs.Restarting docker fixes the issue. So, not sure where we go from here.
-
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
I do still wonder why this happens just now...!
something deprecated upstream?
@jdaviescoates I wonder if upgrading to Ubuntu 22 helps. Do you want to try that? Since you are on 7.3, it should be safe to upgrade.
-
@jdaviescoates I wonder if upgrading to Ubuntu 22 helps. Do you want to try that? Since you are on 7.3, it should be safe to upgrade.
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
I wonder if upgrading to Ubuntu 22 helps.
Spin up a VPS and test?
-
@jdaviescoates I wonder if upgrading to Ubuntu 22 helps. Do you want to try that? Since you are on 7.3, it should be safe to upgrade.
@girish @jdaviescoates my home server v7.3.2 (Ubuntu 22.04.1 LTS) works just fine. I installed Searx and uninstalled it without an issue. I can probably try upgrading my Contabo server tonight. All I need to do is run the following command, right?
sudo do-release-upgrade -
@girish @jdaviescoates my home server v7.3.2 (Ubuntu 22.04.1 LTS) works just fine. I installed Searx and uninstalled it without an issue. I can probably try upgrading my Contabo server tonight. All I need to do is run the following command, right?
sudo do-release-upgrade -
So far, this is not reproducible on a netcup server. I left it overnight to make many updates and it seems to be doing fine.
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
So far, this is not reproducible on a netcup server. I left it overnight to make many updates and it seems to be doing fine.
Can confirm. Not reproducable on Netcup RS or VPS servers.
-
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
So far, this is not reproducible on a netcup server. I left it overnight to make many updates and it seems to be doing fine.
Can confirm. Not reproducable on Netcup RS or VPS servers.
@subven said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
So far, this is not reproducible on a netcup server. I left it overnight to make many updates and it seems to be doing fine.
Can confirm. Not reproducable on Netcup RS or VPS servers.
I wonder why so many (well, at least 3, me @avatar1024 @imc67 ) of us Netcup customers are hitting this issue.
I'm on a Netcup VPS 3000 G10.
-
Just to add some info:
All my three instances are on Netcup:
Two are RS on Cloudron 7.2.5 Ubuntu 20.04
One is a VPS on Cloudron 7.3.2 Ubuntu 22.04All are using a herzner storage box for backup mounted as SSHFS (not CIFS).
I only hit this issue on one instance on Cloudron 7.2.5. Rebooting the server solves it for a bit (I can update all apps manually and it works fine) but then the issue comes back at some point.
Everything in the update process works fine but it gets stuck on "Cleaning up old install". When I then cancel the task, go to the repair tab and click Restart task, the app just restart to a working state (not updated) but the update does not restart.
So what's weird is that it works completely fine on another instance that's exactly the same: same provider, same root server, same Cloudron/Ubuntu version.
Happy to give SSH access if useful.
-
@girish Updating the server was going smooth until I tried to recreate the addon containers. It's still adding new dots so I assume it's okay but has been running for over 6 1/2 hours! Is this normal?
Disk size is 420GB total w/ around 65GB used.
-
@girish Updating the server was going smooth until I tried to recreate the addon containers. It's still adding new dots so I assume it's okay but has been running for over 6 1/2 hours! Is this normal?
Disk size is 420GB total w/ around 65GB used.
@humptydumpty that should definitely not take so long. I assume you mean the running of
/home/yellowtent/box/scripts/recreate-containers? If you go Services view, are things running already? -
@humptydumpty that should definitely not take so long. I assume you mean the running of
/home/yellowtent/box/scripts/recreate-containers? If you go Services view, are things running already?@girish dashboard and all apps aren't loading .. said cloudron is offline in red then.. unable to connect error page.
-
@humptydumpty that should definitely not take so long. I assume you mean the running of
/home/yellowtent/box/scripts/recreate-containers? If you go Services view, are things running already? -
@girish I logged in in another putty window and ran
systemctl status box & systemctl status collectd -- both are active and running (green).
@humptydumpty yeah, I think what has happenned is that the docker is getting stuck again and thus addon containers are not getting created properly. Can you give me SSH access and drop a mail to support@ ? I can try to see if recreating the docker images solves anything.
-
@humptydumpty yeah, I think what has happenned is that the docker is getting stuck again and thus addon containers are not getting created properly. Can you give me SSH access and drop a mail to support@ ? I can try to see if recreating the docker images solves anything.
-
@girish email sent ..
btw should I close the addon container putty window?@humptydumpty thanks, just looking into this now. Indeed:
2022-11-22T05:48:22.494Z box:shell removeAllContainers exec: docker ps -qa --filter 'label=isCloudronManaged' | xargs --no-run-if-empty docker stopdocker is not responding
So, I will recreate the docker images completely . I did this now on @jdaviescoates ' server a while ago to see if it helps as well. -
Further, I am unable to remove the docker directory even after stopping and disabling docker...
root@vmi557975:/# sudo systemctl stop docker Warning: Stopping docker.service, but it can still be activated by: docker.socket root@vmi557975:/# rm -rf /var/lib/docker rm: cannot remove '/var/lib/docker/overlay2/ff0c38183c43e7bcaaf9b564d44f762e0b22f4bd77592a1f0ddae1507dff138d/merged': Device or resource busy rm: cannot remove '/var/lib/docker/overlay2/d19d5ab400f594ee0a9ce613fbd44ebdd98d72cfa7b0ef7afd1705189272636e/merged': Device or resource busy rm: cannot remove '/var/lib/docker/overlay2/d8c1b7219fb875bed31be5c935420240e86a29ae38e42f06c547d8d20534e1c0/merged': Device or resource busy root@vmi557975:/# ps aux | grep docker root 1330267 0.0 0.0 9068 2260 pts/0 S+ 13:58 0:00 grep --color=auto dockerSome dangling container is causing problems:
root@vmi557975:/# ps aux | grep container root 304611 0.0 0.0 710832 8624 ? Sl Nov20 0:24 /usr/bin/containerd-shim-runc-v2 -namespace moby -id adf60365e7ebeed38c53723ed75725ed398a434fb7aaec805a910edcaa5ae901 -address /run/containerd/containerd.sock root 1196183 0.0 0.0 710768 8464 ? Sl 05:24 0:04 /usr/bin/containerd-shim-runc-v2 -namespace moby -id d4b36c14c6c2850853bd62513ff6203135f75047a8640001accadcf62ef496b1 -address /run/containerd/containerd.sock root 1330278 0.0 0.0 9068 2200 pts/0 R+ 13:59 0:00 grep --color=auto containerI guess this is why it's hanging, it's unable to remove images.
-
Further, I am unable to remove the docker directory even after stopping and disabling docker...
root@vmi557975:/# sudo systemctl stop docker Warning: Stopping docker.service, but it can still be activated by: docker.socket root@vmi557975:/# rm -rf /var/lib/docker rm: cannot remove '/var/lib/docker/overlay2/ff0c38183c43e7bcaaf9b564d44f762e0b22f4bd77592a1f0ddae1507dff138d/merged': Device or resource busy rm: cannot remove '/var/lib/docker/overlay2/d19d5ab400f594ee0a9ce613fbd44ebdd98d72cfa7b0ef7afd1705189272636e/merged': Device or resource busy rm: cannot remove '/var/lib/docker/overlay2/d8c1b7219fb875bed31be5c935420240e86a29ae38e42f06c547d8d20534e1c0/merged': Device or resource busy root@vmi557975:/# ps aux | grep docker root 1330267 0.0 0.0 9068 2260 pts/0 S+ 13:58 0:00 grep --color=auto dockerSome dangling container is causing problems:
root@vmi557975:/# ps aux | grep container root 304611 0.0 0.0 710832 8624 ? Sl Nov20 0:24 /usr/bin/containerd-shim-runc-v2 -namespace moby -id adf60365e7ebeed38c53723ed75725ed398a434fb7aaec805a910edcaa5ae901 -address /run/containerd/containerd.sock root 1196183 0.0 0.0 710768 8464 ? Sl 05:24 0:04 /usr/bin/containerd-shim-runc-v2 -namespace moby -id d4b36c14c6c2850853bd62513ff6203135f75047a8640001accadcf62ef496b1 -address /run/containerd/containerd.sock root 1330278 0.0 0.0 9068 2200 pts/0 R+ 13:59 0:00 grep --color=auto containerI guess this is why it's hanging, it's unable to remove images.
