Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task

[jdaviescoates]

@humptydumpty said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

Do you think that has anything to do with it?

I don't think so because lots of us who aren't using Contabo at all are experiencing the same thing.

[humptydumpty]

@jdaviescoates You mentioned you're using storage box and I'm using some sort of storage expansion. Also, Girish mentioned it's a "known" Contabo issue. I'm interested to know if other Contabo users are also using storage expansion. Maybe that's the common factor/issue here?

[jdaviescoates]

@jdaviescoates said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

@girish I'm sorry to report that it doesn't seem outdated Docker was the issue because I noticed there were still outstanding updates to run and so clicked on one and the exact same thing has happened:

I've left it exactly as is if you want to go and have another poke around my server to see what you can discover...

Did you login to my server again and poke around with this? Because I note that that update has now completed (and I didn't intervene myself).

So, I figured I'd hit update on another app to see what happens, but same thing:

[girish]

@jdaviescoates yes, sorry, I am investigating on the server. I am able to reproduce this quite reliably, though I have no clue yet what is causing the issue.

[jdaviescoates]

@humptydumpty said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

Maybe that's the common factor/issue here?

That was an initial theory of @girish too, but e.g. @fbartels has also hit this issue and he isn't using external storage at all afaiu

[humptydumpty]

@jdaviescoates Oh, I see.

I just want to add that uninstalling apps gets stuck too. I installed Searx earlier and didn't like it and now I can't get rid of it (until tonight when I can restart the server).

It's stuck at " Uninstalling - Deleting image ".

Edit: Ending the task and retrying the uninstall actually deleted it without restarting the server.

[jdaviescoates]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

I am able to reproduce this quite reliably, though I have no clue yet what is causing the issue.

Fingers and toes crossed you'll be able to work it out!

[girish]

@humptydumpty said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

It's stuck at " Uninstalling - Deleting image ".

Yup, that's exactly where I have narrowed it down to. Something deep in dockerode a node module we use. Because docker itself is responding fine.

[girish]

I narrowed it down to image delete API call not getting a response. I wrote a simple example and that works So, some strange issue in the docker-modem module. Unfortunately, there's a lot of code there to investigate.

I do still wonder why this happens just now...!

'use strict';

const http = require('http');

const opts = {
  path: '/images/cloudron/org.moodle.cloudronapp:20220912-100537-1689a722e?force=false&noprune=false',
  method: 'DELETE',
  headers: {},
  key: undefined,
  cert: undefined,
  ca: undefined,
  socketPath: '/var/run/docker.sock'
};

const req = http.request(opts, function () { });
req.on('response', function (res) {
    console.log('got a response');
    res.on('data', function () { console.log('data'); });
    res.on('end', function () { console.log('ended'); });
});
req.on('error', function (error) {
    console.log(error);
});

req.end();

[jdaviescoates]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

I do still wonder why this happens just now...!

something deprecated upstream?

[girish]

It turns out it is docker issue after all. If I replace the module with calling docker CLI, it just hangs as well It seems the command itself succeeds (i.e images gets deleted) but the command does not return. This happens across many commands, docker just hangs. No error in their logs.

Restarting docker fixes the issue. So, not sure where we go from here.

[girish]

@jdaviescoates I wonder if upgrading to Ubuntu 22 helps. Do you want to try that? Since you are on 7.3, it should be safe to upgrade.

[jdaviescoates]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

I wonder if upgrading to Ubuntu 22 helps.

Spin up a VPS and test?

[humptydumpty]

@girish @jdaviescoates my home server v7.3.2 (Ubuntu 22.04.1 LTS) works just fine. I installed Searx and uninstalled it without an issue. I can probably try upgrading my Contabo server tonight. All I need to do is run the following command, right?

sudo do-release-upgrade

[girish]

@humptydumpty This guide https://docs.cloudron.io/guides/upgrade-ubuntu-22/

[girish]

In the meantime, I have managed to resurrect our netcup account to see if this is reproducible there,

[girish]

So far, this is not reproducible on a netcup server. I left it overnight to make many updates and it seems to be doing fine.

[subven]

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

So far, this is not reproducible on a netcup server. I left it overnight to make many updates and it seems to be doing fine.

Can confirm. Not reproducable on Netcup RS or VPS servers.

[jdaviescoates]

@subven said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

@girish said in Apps Stuck Updating - Cleaning up old install - even after stopping/restarting task:

So far, this is not reproducible on a netcup server. I left it overnight to make many updates and it seems to be doing fine.

Can confirm. Not reproducable on Netcup RS or VPS servers.

I wonder why so many (well, at least 3, me @avatar1024 @imc67 ) of us Netcup customers are hitting this issue.

I'm on a Netcup VPS 3000 G10.

[avatar1024]

Just to add some info:

All my three instances are on Netcup:
Two are RS on Cloudron 7.2.5 Ubuntu 20.04
One is a VPS on Cloudron 7.3.2 Ubuntu 22.04

All are using a herzner storage box for backup mounted as SSHFS (not CIFS).

I only hit this issue on one instance on Cloudron 7.2.5. Rebooting the server solves it for a bit (I can update all apps manually and it works fine) but then the issue comes back at some point.

Everything in the update process works fine but it gets stuck on "Cleaning up old install". When I then cancel the task, go to the repair tab and click Restart task, the app just restart to a working state (not updated) but the update does not restart.

So what's weird is that it works completely fine on another instance that's exactly the same: same provider, same root server, same Cloudron/Ubuntu version.

Happy to give SSH access if useful.