Bugreport - App in recovery mode crashing nginx - Cloudron frontend not accessible

[BrutalBirdie]

I just had an interesting support case.
A customer had a Cloudron that was non accessible via the frontend.

Looking at the system (and knowing no frontend at all) can only mean nginx issues.

journalctl -u nginx

Feb 23 05:18:47 my systemd[1]: Starting nginx - high performance web server...
Feb 23 05:18:52 my nginx[6671]: nginx: [emerg] cannot load certificate "/home/yellowtent/platformdata/nginx/cert/orga.domain.tld.cert": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen
Feb 23 05:18:52 my systemd[1]: nginx.service: Control process exited, code=exited status=1
Feb 23 05:18:52 my systemd[1]: nginx.service: Failed with result 'exit-code'.
Feb 23 05:18:52 my systemd[1]: Failed to start nginx - high performance web server.

Huh.
cd /etc/nginx/applications
grep -rin "orga" (searching for the subdomain of the app so I can get the nginx confile filename aka. appid.

f3a6649e-86c9-4fd0-ac97-a3675a36c19d.conf:16:    server_name  orga.sub.domain;
f3a6649e-86c9-4fd0-ac97-a3675a36c19d.conf:44:    server_name  orga.sub.domain;
f3a6649e-86c9-4fd0-ac97-a3675a36c19d.conf.off:50:    ssl_certificate      /home/yellowtent/platformdata/nginx/cert/orga.sub.domain.cert;
f3a6649e-86c9-4fd0-ac97-a3675a36c19d.conf:51:    ssl_certificate_key  /home/yellowtent/platformdata/nginx/cert/orga.sub.domain.key;

Moving the file to .off so nginx won't load it as config
mv f3a6649e-86c9-4fd0-ac97-a3675a36c19d.conf f3a6649e-86c9-4fd0-ac97-a3675a36c19d.conf.off
systemctl restart nginx.service

Now the frontend is accessible again.

HUH!
f3a6649e-86c9-4fd0-ac97-a3675a36c19d is a Redmine 4.2.1 => org.redmine.coudronapp@1.7.1 => Last Updated 10 months ago
and its labeled orga.sub.domain - broken and the app is in recovery mode.
Trying a force renew all certs:

Log:

Feb 23 12:24:32 box:reverseproxy expiryDate: /home/yellowtent/platformdata/nginx/cert/oldorga.sub.domain.cert notAfter=Sep 11 08:32:00 2021 GMT daysLeft=-165.11981987268518

What the..? oldorga.sub.domain?

@staff Could it be that an app in recovery mode does not get any new certs?
And simple the app had been so long in recovery mode the certs have not been renewed?
Why is there an oldorga.sub.domain which is getting renewed? There is no app with this location.

Something is fishy

[rmdes]

@BrutalBirdie from my epxerience, Apps in recovery mode or in any other state than "running" do not get certs or updates.

[girish]

@BrutalBirdie yes, this was a bug in 7.0.x. certificates of apps are "deleted" after 6 months or so. when this happens, the nginx config is left dangling. This is fixed in 7.1 with https://git.cloudron.io/cloudron/box/-/commit/5382e3d8321ddb96817f50ab94e9da56258b11e9