Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task

[girish]

@jdaviescoates Initial hypothesis is that maybe hetzner storage box had a "reset" on their side . CIFS mounts are not auto-mounted (whether by Cloudron or not) because the mount system doesn't have such a feature. So, we were thinking the backups stopped working and when they stop working, updates were getting stuck.

[jdaviescoates]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

@jdaviescoates Initial hypothesis is that maybe hetzner storage box had a "reset" on their side . CIFS mounts are not auto-mounted (whether by Cloudron or not) because the mount system doesn't have such a feature. So, we were thinking the backups stopped working and when they stop working, updates were getting stuck.

Thanks for investigating, but I'm not sure that is it.

I only say that because right now I've got 3 apps stuck on "Updating - Cleaning up old install"

And yet the backup drive is mounted fine:

[girish]

Thanks for @jdaviescoates for access to his system.

Basically, docker just stops working for some reason. The command below just hangs.

root@v2202211129685206445:/home/yellowtent/platformdata/logs/70b06ed9-1257-46d1-9fe9-1ffa7c722f21# docker stop d3ef55aa711f8d424f53bbef6a5ac813e9670cf89e77eaa171762dc524610ebc

I see that docker is a few versions behind and there is a bunch of stuff related to locks and races, I have gone ahead and updated docker. @jdaviescoates Can you let me know if you hit this again? In the meantime, I will put the docker update in our ci for 7.3.3.

[jdaviescoates]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

Thanks for @jdaviescoates for access to his system.

You're welcome! Thanks for investigating.

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

Can you let me know if you hit this again?

Will do!

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

I will put the docker update in our ci for 7.3.3

Presumably until 7.3.3 is released and I upgrade I'll likely keep hitting this issue, right?

[girish]

@jdaviescoates I updated the docker on your server already . That's why, please let us know if you hit the issue again (which then means that docker upgrade didn't fix the issue).

But regardless, I will keep the docker update as part of 7.3.3 unless something else breaks. So, you won't have troubles after updating as well.

[jdaviescoates]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

@jdaviescoates I updated the docker on your server already . That's why, please let us know if you hit the issue again (which then means that docker upgrade didn't fix the issue).

Ah, right, got it, thanks! Will see what happens going forward...

[imc67]

@girish for backups yes

[jdaviescoates]

@girish I'm sorry to report that it doesn't seem outdated Docker was the issue because I noticed there were still outstanding updates to run and so clicked on one and the exact same thing has happened:

I've left it exactly as is if you want to go and have another poke around my server to see what you can discover...

[humptydumpty]

@girish I'm using Contabo's NVMe Storage Extension ( it's different than their Object storage) that doubles my storage. Do you think that has anything to do with it?

When I got it, I had to do the standard Ubuntu partition expansion commands to get it set up. What type of storage would this be? CIFS?

[jdaviescoates]

@humptydumpty said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

Do you think that has anything to do with it?

I don't think so because lots of us who aren't using Contabo at all are experiencing the same thing.

[humptydumpty]

@jdaviescoates You mentioned you're using storage box and I'm using some sort of storage expansion. Also, Girish mentioned it's a "known" Contabo issue. I'm interested to know if other Contabo users are also using storage expansion. Maybe that's the common factor/issue here?

[jdaviescoates]

@jdaviescoates said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

@girish I'm sorry to report that it doesn't seem outdated Docker was the issue because I noticed there were still outstanding updates to run and so clicked on one and the exact same thing has happened:

I've left it exactly as is if you want to go and have another poke around my server to see what you can discover...

Did you login to my server again and poke around with this? Because I note that that update has now completed (and I didn't intervene myself).

So, I figured I'd hit update on another app to see what happens, but same thing:

[girish]

@jdaviescoates yes, sorry, I am investigating on the server. I am able to reproduce this quite reliably, though I have no clue yet what is causing the issue.

[jdaviescoates]

@humptydumpty said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

Maybe that's the common factor/issue here?

That was an initial theory of @girish too, but e.g. @fbartels has also hit this issue and he isn't using external storage at all afaiu

[humptydumpty]

@jdaviescoates Oh, I see.

I just want to add that uninstalling apps gets stuck too. I installed Searx earlier and didn't like it and now I can't get rid of it (until tonight when I can restart the server).

It's stuck at " Uninstalling - Deleting image ".

Edit: Ending the task and retrying the uninstall actually deleted it without restarting the server.

[jdaviescoates]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

I am able to reproduce this quite reliably, though I have no clue yet what is causing the issue.

Fingers and toes crossed you'll be able to work it out!

[girish]

@humptydumpty said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

It's stuck at " Uninstalling - Deleting image ".

Yup, that's exactly where I have narrowed it down to. Something deep in dockerode a node module we use. Because docker itself is responding fine.

[girish]

I narrowed it down to image delete API call not getting a response. I wrote a simple example and that works So, some strange issue in the docker-modem module. Unfortunately, there's a lot of code there to investigate.

I do still wonder why this happens just now...!

'use strict';

const http = require('http');

const opts = {
  path: '/images/cloudron/org.moodle.cloudronapp:20220912-100537-1689a722e?force=false&noprune=false',
  method: 'DELETE',
  headers: {},
  key: undefined,
  cert: undefined,
  ca: undefined,
  socketPath: '/var/run/docker.sock'
};

const req = http.request(opts, function () { });
req.on('response', function (res) {
    console.log('got a response');
    res.on('data', function () { console.log('data'); });
    res.on('end', function () { console.log('ended'); });
});
req.on('error', function (error) {
    console.log(error);
});

req.end();

[jdaviescoates]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

I do still wonder why this happens just now...!

something deprecated upstream?

[girish]

It turns out it is docker issue after all. If I replace the module with calling docker CLI, it just hangs as well It seems the command itself succeeds (i.e images gets deleted) but the command does not return. This happens across many commands, docker just hangs. No error in their logs.

Restarting docker fixes the issue. So, not sure where we go from here.