Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task
-
@avatar1024 this is not unmanaged WP related. We have been investigating this a lot last few days but it appears to happen very often in netcup servers at least.
The main issue is that runc is getting stuck (you can also do
ps auxfon the server):root 84188 0.0 0.0 712448 9880 ? Sl 11:09 0:01 /usr/bin/containerd-shim-runc-v2 -namespace moby -id afb25df337753f9944749bdba5bcb577ae58630d4529da3293ea3730c3164be4 -address /run/containerd/c root 84200 0.0 0.0 1230148 10704 ? Sl 11:09 0:00 \_ runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v2.task/moby/afb25df337753f9944749bdba5bcb577ae586 root 84210 0.0 0.0 1083864 10968 ? Ssl 11:09 0:00 \_ runc initThose runc init above is just hanging there waiting for something (you can strace and see it's waiting on some futex). It's unclear what the issue is so far.
-
@avatar1024 Yes I did reboot but nothing. No have any WP unmanaged on instance. Only two LAMP applications.
-
@avatar1024 this is not unmanaged WP related. We have been investigating this a lot last few days but it appears to happen very often in netcup servers at least.
The main issue is that runc is getting stuck (you can also do
ps auxfon the server):root 84188 0.0 0.0 712448 9880 ? Sl 11:09 0:01 /usr/bin/containerd-shim-runc-v2 -namespace moby -id afb25df337753f9944749bdba5bcb577ae58630d4529da3293ea3730c3164be4 -address /run/containerd/c root 84200 0.0 0.0 1230148 10704 ? Sl 11:09 0:00 \_ runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v2.task/moby/afb25df337753f9944749bdba5bcb577ae586 root 84210 0.0 0.0 1083864 10968 ? Ssl 11:09 0:00 \_ runc initThose runc init above is just hanging there waiting for something (you can strace and see it's waiting on some futex). It's unclear what the issue is so far.
-
@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
happen very often in netcup servers
I know you (or @nebulon) have an account at Netcups Forum. Maybe ask them for help?
-
So we had an instance which seems to not hit this issue anymore by making sure the kvm version is updated. On netcup this apparently means to shutdown and boot the instance through their control panel. A simple reboot/restart will not solve it. Maybe anyone here facing this and being on netcup could attempt this so we get more info.
-
So we had an instance which seems to not hit this issue anymore by making sure the kvm version is updated. On netcup this apparently means to shutdown and boot the instance through their control panel. A simple reboot/restart will not solve it. Maybe anyone here facing this and being on netcup could attempt this so we get more info.
@nebulon said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
Maybe anyone here facing this and being on netcup could attempt this so we get more info.
I just did that (a power cycle via their scp server control panel), let's see if it resolves the issue for me...
-
I also have a WordPress dev app on the cloudron that gets stuck and don't have it on the other instances. The only other app I have on it and not on the other ones is mattermost
PS: the instance I restarted earlier today and that was working now hangs again on updates. Restarted again and now working fine...for some time...
Weird, got three Netcup instances and only ever hit the issue one. Plus other people seem to be hitting it with other providers.
@avatar1024 said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
PS: the instance I restarted earlier today and that was working now hangs again on updates. Restarted again and now working fine...for some time...
Good news is that ever since the last reboot my instance stopped hanging (it's been 4 days). The only thing I did compared to the other time I rebooted is that I updated all apps manually to their latest version and so had no more update to do (the last app was WP unmanaged). Ever since updates have been working fine.
@nebulon said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:
Maybe anyone here facing this and being on netcup could attempt this so we get more info.
I had actually already done this from Netcup SCP the first time I hit the issue but it hadn't solved for me.
-
Ok, so one thing which I have noticed is common is that the kernel is 5.4.0-132-generic (output of
uname -narand Ubuntu 20.04.5 (lsb_release -a). Can others please confirm what kernel and ubuntu they are on? -
Ok, so one thing which I have noticed is common is that the kernel is 5.4.0-132-generic (output of
uname -narand Ubuntu 20.04.5 (lsb_release -a). Can others please confirm what kernel and ubuntu they are on?@girish
All three instances are on kernel 5.4.0-132-generic
Two instances (including the one that used to hang) are on Ubuntu 20.04.3
One instance is on 20.04.4 -
@girish is there a way to check what I was running prior to upgrading to22.04?
@humptydumpty no, I think not. Are you still facing this issue? What's your current kernel and ubuntu release?
-
@humptydumpty no, I think not. Are you still facing this issue? What's your current kernel and ubuntu release?
-
@humptydumpty no, I think not. Are you still facing this issue? What's your current kernel and ubuntu release?
@girish 100% fine since upgrading to 22.04 with the fixes you did when you SSH'd in.
Linux vmi5XXXX5.contaboserver.net 5.15.0-53-generic #59-Ubuntu SMP Mon Oct 17 18:53:30 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux No LSB modules are available. Distributor ID: Ubuntu Description: Ubuntu 22.04.1 LTS Release: 22.04 Codename: jammy -
So, the issue was just reproduced on our DigitalOcean demo cloudron. That is also running ubuntu 20 and 5.4.0-132-generic
@girish I'm in the clear but for others, what would be the best way to upgrade to 22.04 without running into docker hangs like I did?
-
Alright, this seems to be a kernel issue. Other programs are experiencing similar:
- https://github.com/prometheus/node_exporter/issues/2500
- https://bugs.launchpad.net/ubuntu/+source/containerd/+bug/1996678 (it's getting reverted at https://lists.ubuntu.com/archives/kernel-team/2022-November/135043.html)
- https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1990190
- https://www.mail-archive.com/kernel-packages@lists.launchpad.net/msg492057.html
- https://github.com/opencontainers/runc/issues/3641
GOMAXPROCS=1apparently alleviates the issue. So, I will try this one. -
G girish has marked this topic as solved on
-
G girish referenced this topic on
