Disk is (suddenly) full on 1TB drive, can't access cloudron

[girish]

@shan Delete the nginx config files as well and then systemctl restart box. This will regenerate the nginx configs and cert files. After that, you will be able to access the dashboard. Go into each app's Location view and click save. That will regenerate the nginx config of each app.

(This tedious process is automated/fixed in next release.)

[shan]

@girish I've deleted the nginx conf file (home/yellowtent/platformdata/nginx/nginx.conf) and am encountering a new error. It seems systemctl restart box did not regenerate this.

[emerg] open() "/etc/nginx/nginx.conf" failed (2: No such file or directory)

[girish]

@shan Oh, my bad. I should have been clearer that only app configs have to be deleted. Anyway, please run /home/yellowtent/box/setup/start.sh which will create nginx config files.

[shan]

@girish that seems to have fixed all my problems! Can access the dashboard again. Looking forward to the next release when this is automated lol.

[nebulon]

As the root cause of this was, that the backup was continuing even though the backup disk was not mounted, we were now able to find the bug which caused this and possibly other similar issues.

So the check itself for the mountpoint was correct, but this result was just ignored by the code. This oversight will be fixed for the next release and should avoid such cases for mounted backup volumes in the future.

[shan]

@nebulon hey, not sure if you guys already implemented this fix, but I just had the exact same issue happen again this morning on a fully up-to-date Cloudron instance. Trying to remember wtf I did to fix this

[girish]

@shan yes, fix is coming in 7.3.3. What you have to do is first do journalctl -u nginx -fa. It will say some cert/conf file is bad. Just delete them and systemctl restart nginx and get it running. Then, systemctl restart box.

[shan]

@girish the error it gives is the same as this one above; it is not clear that I need to actually just delete the application certs ( just found their location again at nginx/applications )

nginx: [emerg] cannot load certificate key "/home/yellowtent/platformdata/nginx/cert/_.myserver.net.key": 
PEM_read_bio_PrivateKey() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: ANY PRIVATE KEY)

Deleting the application certs allowed nginx to restart but my webserver is still not running, gah

[girish]

@shan that message is saying that the cert cannot be loaded. Did you remove that file? (you have to also remove the .cert file along with the .key file). If you did remove that file, then go into /etc/nginx/applications and delete the conf files that reference the above cert.

[shan]

@girish I didn't remove that file, just the application certs and nginx is now running fine according to systemctl status nginx, just the webserver isn't loading. Unbound & box are fine too. Should I still delete that file even though nginx is running?

[girish]

@shan what do you mean by "webserver" ? Do you mean the dashboard? nginx is the webserver.

[shan]

@girish yeah the dashboard

[girish]

@shan you have to systemctl restart box, it will regenerate the nginx config needed for the dashboard. Then, if you refresh in browser, you might have to accept self signed certificate and login (that's OK). Then, go to Domains -> Renew Certs and you should be back.

BTW, it's safe to delete configs and certs because it's all in the database and code. Renew certs above does not get a new cert internally. It will sync the existing cert in db to disk.

[shan]

@girish alright was able to get the dashboard back, thanks! When can we expect that update with the fix?

[robi]

@shan 7.3.3 I believe.