Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task

[girish]

@jdaviescoates I don't know how motivated you are to track this down, but either you can ask them for another server? I have the VPS 500 G10s iv in netcup .

Another idea is to go back to plain ubuntu and try to reproduce this issue with just docker since it's not Cloudron related. But not sure what you will gain with this...

[girish]

@jdaviescoates BTW, what "driver" did you select in Media in netcup ? I see that I have chosen VIRTIO even though SCSI is recommended.

[jdaviescoates]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

@jdaviescoates BTW, what "driver" did you select in Media in netcup ? I see that I have chosen VIRTIO even though SCSI is recommended.

Yeah mine is VIRTIO too, I didn't manually choose that though, that's just how it was already and I chose not to touch anything for fear or messing things up (but yeah, I also noticed it says SCSI is recommended)

[jdaviescoates]

@jdaviescoates said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

@jdaviescoates reverted now.

Thanks, tried again but:

Although, after a reboot I was just successfully able to run 3 updates without issue... let's see how long it lasts... hopefully forever and the mystery will never need solving!

[jdaviescoates]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

you can ask them for another server?

I may well do that in attempt to solve this and also to get a clean IP that doesn't mess with my email...

But also a bit scared to do so given that my first server move resulted in this bizarre issue rearing it's head

[girish]

@jdaviescoates What's happenning is that some container is getting stuck and locking the image from getting removed. Actually, I want to investigate what that app might be. When this happens, can you please drop me a note? Maybe it's always the same app which is causing the problem (some cron job related thing maybe).

[jdaviescoates]

@girish will do: best place to drop you that note? in reply to the support email thread? on this thread? in chat on here? on matrix?

(my guess: WordPress Developer app)

[girish]

@jdaviescoates support@ is best

[jdaviescoates]

@girish fyi (added this bit with an edit so not sure you saw it?)

@jdaviescoates said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

(my guess: WordPress Developer app)

[humptydumpty]

@jdaviescoates I'm running WP dev too. Here's the full list of apps I'm using:

umami
ghost
wordpress dev
surfer
chatwoot
noco
element
vaultwarden
wekan
invoiceninja
etherpad
roundcube
snappymail
matrix
listmonk
paperless
freescout
alltube
kutt
snappymail

[avatar1024]

I also have a WordPress dev app on the cloudron that gets stuck and don't have it on the other instances. The only other app I have on it and not on the other ones is mattermost

PS: the instance I restarted earlier today and that was working now hangs again on updates. Restarted again and now working fine...for some time...

[subven]

@jdaviescoates said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

Yeah mine is VIRTIO too

I use SCSI on all my Netcup hosts. Not hitting the problem like I said before.

VPS 200 G8 v7.3.2 (clean) Ubuntu 20.04.2 LTS
RS 4000 G9 SE v7.3.2 (clean) Ubuntu 20.04.2 LTS

Stuff I have installed: Wordpress (managed), Matrix, Element, Bookstack, Freescout, LAMP (new and old ones), Gitea, Lychee, Roundcoube, Nextcloud, Statping, Collabora, Paperless, Surfer, Kimai, Jirafeau, Vaultwarden

[avatar1024]

So I guess it could well have to do with the WP unmanaged package as it seems like a common factor so far across all instances (of everyone in the thread) experiencing the issue and the singularity on my buggy instance instance compared to the others

[humptydumpty]

@girish the server is running fine after upgrading to 22.04. I manually updated one app yesterday and it updated just fine. A bunch of apps auto updated last night without any issues and I manually updated Wekan minutes ago. I hope I’m not jumping the gun here fingers crossed!

[p44]

@girish hello Girish, I'm trying to update a LAMP instance and problem still there, any ideas on how to proceed? Thank's

[girish]

@p44 can you write to support@ . I am afraid to give a suggestion without knowing the current state of the server.

[p44]

@girish Ok I'll do ASAP. thank's a lot

[avatar1024]

@p44 Have you tried to reboot your server and then do the update?

@p44 out of interest, do you have a WP unmanaged app installed on your instance?

[girish]

@avatar1024 this is not unmanaged WP related. We have been investigating this a lot last few days but it appears to happen very often in netcup servers at least.

The main issue is that runc is getting stuck (you can also do ps auxf on the server):

root       84188  0.0  0.0 712448  9880 ?        Sl   11:09   0:01 /usr/bin/containerd-shim-runc-v2 -namespace moby -id afb25df337753f9944749bdba5bcb577ae58630d4529da3293ea3730c3164be4 -address /run/containerd/c
root       84200  0.0  0.0 1230148 10704 ?       Sl   11:09   0:00  \_ runc --root /var/run/docker/runtime-runc/moby --log /run/containerd/io.containerd.runtime.v2.task/moby/afb25df337753f9944749bdba5bcb577ae586
root       84210  0.0  0.0 1083864 10968 ?       Ssl  11:09   0:00      \_ runc init

Those runc init above is just hanging there waiting for something (you can strace and see it's waiting on some futex). It's unclear what the issue is so far.

[robi]

@girish I wonder if sysbox would have the same issue.