Nginx reverseproxy error after Cloudron upgrade to v8.3.1; Cloudron apps running but can't be stopped/restarted

[THI_Staff]

Hi All

Could you help with the following issue please?

Short version
Following a successful Cloudron platform upgrade, all our apps now display the following error:
Error : Nginx Error - Error reloading nginx: reverseproxy exited with code 1 signal null

The apps / services run ok; the only issues are 1) no backups were taken since the Cloudron platform upgrade, likely due to the app error status and 2) the apps can't be stopped / restarted likely due to the same error.

The underlying cause must be related to certificates, I would guess, so upon checking the renew certificates log I found these telling lines:

Apr 08 02:15:47 box:shell nginx: [emerg] cannot load certificate "/home/yellowtent/platformdata/nginx/cert/_.<our_website>.<our_cloudron_domain>.cert": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/home/yellowtent/platformdata/nginx/cert/_.<our_website>.<our_cloudron_domain>.cert','r') error:2006D080:BIO routines:BIO_new_file:no such file)
Apr 08 02:15:47 box:shell reverseproxy: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx errored BoxError: reverseproxy exited with code 1 signal null
Apr 08 02:15:47 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19)
Apr 08 02:15:47 at ChildProcess.emit (node:events:519:28)
Apr 08 02:15:47 at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
Apr 08 02:15:47 reason: 'Shell Error',
Apr 08 02:15:47 details: {},
Apr 08 02:15:47 code: 1,
Apr 08 02:15:47 signal: null
Apr 08 02:15:47 }
Apr 08 02:15:47 box:taskworker Task took 347.55 seconds
Apr 08 02:15:47 box:tasks setCompleted - 8714: {"result":null,"error":{"stack":"BoxError: Error reloading nginx: reverseproxy exited with code 1 signal null\n at reload (/home/yellowtent/box/src/reverseproxy.js:188:22)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async checkCerts (/home/yellowtent/box/src/reverseproxy.js:698:9)","name":"BoxError","reason":"Nginx Error","details":{},"message":"Error reloading nginx: reverseproxy exited with code 1 signal null"}}

Interestingly <our_website> is the only domain set up with manual DNS configuration in Cloudron, but this has been the case for years and it worked/works, until the latest Cloudron platform upgrade. But sure, the renew certificates log above is correct, there is no cert file at /home/yellowtent/platformdata/nginx/cert/_.<our_website>.<our_cloudron_domain>.cert but it should not be looking for one locally on our server, should it?

Longer version
The Cloudron platform upgrade from v8.2.4 to v8.3.1 was successful on 29/03/2025. Our last Cloudron app backups are from 29/03/2025 as well.

Our Cloudron apps now have a status of:
Error : Nginx Error - Error reloading nginx: reverseproxy exited with code 1 signal null

Starting up development instances of Cloudron apps (which are not running normally) gets a status of "Starting - Configuring reverse proxy", which after it sticks around for way too long, errors out with the same message as above. Trying to repair the app with the "Retry start app task" leads nowhere other than the same error being displayed again. And we can't stop any of these apps either, but they are nevertheless running.

All the renew certificates logs since 29/03/2025 have an error red x next to them; the relevant log output is mentioned above. Why is the reverse proxy getting stuck looking for the certificate of <our_website> which we have on manual DNS setup in Cloudron? And how do we tell it to not check this certificate any more? I'm guessing this is where the issue is.

Funny enough, according to the Cloudron event log, the certificate install for www.<our_website> succeeded on 01/04/2025 (proving the manual DNS setup has worked for years), this is 3 days after the Cloudron platform upgrade on 29/03/2025 but obviously the reverse proxy doesn't know / isn't happy with the certificate for <our_website> for whatever reason.

Can you advise how we fix this please? Thank you in advance.

THI Staff

[nebulon]

Can you run cloudron-support --troubleshoot via SSH on your server? It should suggest a fix.

[THI_Staff]

@nebulon Thank you for that.

Looks good to me. Though not sure about the 2 "To Be Filled By O.E.M.", should they be actual values?

root@server:~# cloudron-support --troubleshoot
Vendor: To Be Filled By O.E.M. Product: To Be Filled By O.E.M.
Linux: 5.4.0-148-generic
Ubuntu: focal 20.04
Processor: AMD Ryzen 5 5600X 6-Core Processor x 12
RAM: 65814680KB
Disk: /dev/md0        414G
[OK]    node version is correct
[OK]    IPv6 is enabled in kernel. No public IPv6 address
[OK]    docker is running
[OK]    docker version is correct
[OK]    MySQL is running
[OK]    nginx is running
[OK]    dashboard cert is valid
[OK]    dashboard is reachable via loopback
[OK]    box v8.3.1 is running
[OK]    netplan is good
[OK]    DNS is resolving via systemd-resolved
[OK]    Dashboard is reachable via domain name
        Domain <our_cloudron_domain> expiry check skipped because whois is not installed. Run 'apt install whois' to check
[OK]    unbound is running

After logging on via SSH I was greeted by this:

1 updates could not be installed automatically. For more details,
see /var/log/unattended-upgrades/unattended-upgrades.log

*** System restart required ***

That log file includes this interesting line:

2025-04-02 05:36:52,930 WARNING Could not figure out development release: Distribution data outdated. Please check for an update for distro-info-data. See /usr/share/doc/distro-info-data/README.Debian for details.

Forums on the net, like this one https://askubuntu.com/questions/1489441/unattended-upgrade-message-could-not-figure-out-development-release-distributi, cover the same point and suggest the following:

You need to upgrade just that one package distro-info-data, as in: sudo apt install --only-upgrade distro-info-data.

I encountered this too in a 22.04 server setup similar to yours (keeps up only with security updates). Turns out the warning message is exactly what it says, that is, we just needed to check for an update to distro-info-data and install it if available.

The issue, if you could call it that, was that the necessary update was only published in version 0.52ubuntu0.5 on Oct 24, after this question was asked, and two weeks or so after the warning started appearing in servers like yours and mine.

Is that worthwhile doing? I'm not a Ubuntu expert, but common sense says it can only help. Saying that our platform version is v8.3.1 (Ubuntu 20.04 TLS - not 22.04).

The 1 update from unattended-upgrades.log which could not be automatically installed could be this:

2025-04-10 06:15:21,756 INFO Package shim-signed is kept back because a related package is kept back or due to local apt_preferences(5).

Re: "System restart required", our server uptime is 2 years, so clearly I've not done a restart since taking over looking after this VPS (virtual private server).

Any thoughts please? Thank you again.

THI Staff

[THI_Staff]

PS: The log file /var/log/nginx/error.log has these 2 interesting lines:

2025/04/10 01:10:00 [emerg] 1665953#1665953: cannot load certificate "/home/yellowtent/platformdata/nginx/cert/_.<our_website>.<our_cloudron_domain>.cert": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/home/yellowtent/platformdata/nginx/cert/_.<our_website>.<our_cloudron_domain>.cert','r') error:2006D080:BIO routines:BIO_new_file:no such file)

2025/04/10 11:45:13 [error] 3689898#3689898: *2550325 open() "/home/yellowtent/box/dashboard/dist/.git/config" failed (2: No such file or directory), client: <IP_address>, server: my.<our_cloudron_domain>, request: "GET /.git/config HTTP/1.1", host: "my.<our_cloudron_domain>"

No other errors / entries to suggest why the reported error happens.

[girish]

@THI_Staff can you try this?

1. systemctl stop nginx
2. Then, cloudron-support --troubleshoot .

The troubleshoot can be a bit more smarter. I will fix the script.

[THI_Staff]

I can do @girish But what's the impact of that? Stopping nginx will presumably take all our websites down for a period? And after re-running cloundron-support --troubleshoot and seeing the new output, presumably I then run systemctl start nginx to restart all websites, right? Though there is some inherent risk here by stopping nginx because it's at server level. I need to check with the customer about the timing of this test / troubleshooting exercise; doing this at the weekend might not suit, Monday might be better

Thank you again; I appreciate your support and advice

[girish]

@THI_Staff it will only take down the apps for a very short time (assuming, --troubleshoot does it's job and brings it back up). But you are right, if the tool doesn't work then nginx will stay down until the issue is sorted out.

[THI_Staff]

Understood @girish I'll give that a try Monday morning UK time
I'm always cautious with these kind of things to avoid surprised, but sometimes you've just got to try things out and then work forward to fix things

[THI_Staff]

That worked @girish
cloudron-support --troubleshoot found a certificate which does not exist and so removed a conf file
nginx started promptly
After which the Cloudron apps could be restarted just fine
Thank you once again for your help