Nextcloud App "not responding" after upgrading to Ubuntu 22.04

[andreasb]

Dear Cloudron Team

upgrading the instance from Ubuntu 20.04 to 22.04 following the corresponding tutorial worked as described. Since then however, Nextcloud App shows "not responding", apparently because the corresponding Redis Service doesn't start either. I tried restarting NC + Redis independently, as well as rebooting the server, all without success. I as well increased memory limit to Redis and put the service in "recovery mode". This without visible effect. The redis log shows the following message repeatedly:

Jun 22 21:02:25 cat: /sys/fs/cgroup/memory.max: No such file or directory

Any ideas, anybody?

[ApplegateR]

@andreasb we told that we can't upgrade 22.04 just yet because it wasn't supported.

[girish]

@andreasb Which cloudron version are you on?

(The issue has to do with cgroup v1 / v2 detection)

[jdaviescoates]

@ApplegateR indeed, https://www.cloudron.io/get.html still currently says:

Create a fresh Ubuntu Focal 20.04 x64 server and run these commands

But the linked to guide https://docs.cloudron.io/guides/upgrade-ubuntu-22/ does talk about upgrading. Although looks like it might be a work in progress as it also says e.g.

If you are still on Ubuntu 20, you must first upgrade to Ubuntu 20

Calling @staff to look into this and clarify

[girish]

@jdaviescoates Ubuntu 22 is mostly stable and working but really no rush to update. There is no active issue on Ubuntu 22, I am aware of.

[andreasb]

@girish Cloudron 7.2.4

[andreasb]

@ApplegateR didn't read that anywhere. Let me know if you'd like some information from that machine. If not, I will go back to the snapshot in that case later today.

[girish]

@andreasb thanks! I think the issue is that redis is not starting on ubuntu 22.04. This was already fixed . Can you please check docker images output and check for tag 3.3.0 of redis:

cloudron/redis                       3.3.0                       2187af39ff74   8 weeks ago    2.24GB

Then, docker ps -a should have something like:

2d3003643201   cloudron/redis:3.3.0                                           "/app/code/start.sh"     27 minutes ago   Up 7 minutes                                                                              redis-31ce4921-9b37-41bb-903c-aff587955c00

The 31ce... is the app id of nextcloud. And that redis is using 3.3.0..

[girish]

@andreasb I managed to reproduce this.

To fix this:

# sed -e 's/"version": ".*",/"version":"50.0.0",/' -i /home/yellowtent/platformdata/INFRA_VERSION
# systemctl restart box

The issue is that Ubuntu 22 uses cgroups v2 by default. Containers have to be re-recreated to recognize this change. Changing the infra version triggers a container re-create. Note that it will take a bit for the containers to get re-created. The UI will show "configuring" for all the apps and they will come up one by one.

I have fixed the guide accordingly.

[andreasb]

@girish thx for your support, rebuilding the containers did the job: after restarting, both Redis + NC are up and running again!

[humptydumpty]

@girish When does Cloudron support end for Ubuntu 20.04?

[girish]

@humptydumpty until canonical supports it. That will be for quite a while, see https://ubuntu.com/about/release-cycle

[necrevistonnezr]

@girish said in Nextcloud App "not responding" after upgrading to Ubuntu 22.04:

@andreasb I managed to reproduce this.

To fix this:

# sed -e 's/"version": ".*",/"version":"50.0.0",/' -i /home/yellowtent/platformdata/INFRA_VERSION
# systemctl restart box

The issue is that Ubuntu 22 uses cgroups v2 by default. Containers have to be re-recreated to recognize this change. Changing the infra version triggers a container re-create. Note that it will take a bit for the containers to get re-created. The UI will show "configuring" for all the apps and they will come up one by one.

I have fixed the guide accordingly.

Somehow, this doesn't persist. After a reboot, I see

{
    "version": "49.0.0",
    "baseImages": [
        {
            "repo": "cloudron/base",
            "tag": "cloudron/base:3.2.0@sha256:[REDACTED]"
        }
    ],

in /home/yellowtent/platformdata/INFRA_VERSION

And the mail service does not start.
The error log shows:

Jun 26 18:25:59 cat: /sys/fs/cgroup/memory/memory.limit_in_bytes: No such file or directory

[girish]

@necrevistonnezr It's not supposed to persist. It's just a "hack" to make the code recreate all the containers.

The mail container not starting is indeed a bug when solr/full text search is enabled. You can disable this temporarily.

[necrevistonnezr]

@girish How do I stop Solr in the terminal? I can't stop it in the dashboard - it just shows a spinning circle...

[girish]

@necrevistonnezr can you please try this one liner - https://git.cloudron.io/cloudron/box/-/commit/104bdaf76b1c66b6b7523e3bef0e61f8e18e86a4 and then systemctl restart box.

[necrevistonnezr]

@girish If that replacement should go into /home/yellowtent/platformdata/INFRA_VERSION - no, mail service still not working and showing the same error, even after a reboot.

Jun 27 05:50:14 cat: /sys/fs/cgroup/memory/memory.limit_in_bytes: No such file or directory
Jun 27 05:50:23 cat: /sys/fs/cgroup/memory/memory.limit_in_bytes: No such file or directory
Jun 27 05:50:38 cat: /sys/fs/cgroup/memory/memory.limit_in_bytes: No such file or directory

EDIT: I guess that replacement should go into /home/yellowtent/box/src/infra_version.js - but after a restart of box, still not running...

EDIT 2: After 30 minutes or so, mail service is running again!

[girish]

@necrevistonnezr said in Nextcloud App "not responding" after upgrading to Ubuntu 22.04:

EDIT: I guess that replacement should go into /home/yellowtent/box/src/infra_version.js - but after a restart of box, still not running...

Yes, that's the correct place to fix it!

I will add a warning to our docs that updating to ubuntu 22 when using FTS feature does not work out of the box / without patching until 7.3.