Cloudron Forum

The issue is now resolved. The problem was with the GitLab database, where we found 160 GB of logs. The database dump was failing because there wasn’t enough free disk space. We have purged the database, and backups are now successful.

Thank you for your support.

@bennyv53 I recommend using ssh then (SSL support for internal postgres will break Cloudron's setup). Is the concern security related?

Turns out that Cloudron hit the same issue as https://forum.cloudron.io/topic/11276/cloudron-update-to-7-7-0-postgresql-does-not-start?_=1723244919333 and the fixed worked there also.

@girish perfect, thanks - appreciate it! I'm almost certain there could be ways to improve those queries, but it's always better to have a failsafe in place 😉

The current addon uses 14.9 so I guess you could run the corresponding image from https://hub.docker.com/_/postgres

@girish thank you, I tried it several times and it didn't worked. I tried it a last time and it worked.
I don't know why but thank you 🙂

This was a bug in 7.4.0 and is now fixed in 7.4.1.

We updated node to v18 (from v16). There was a subtle but impact change hidden... In node 16, the requestTimeout was 0. This meant unlimited time. In node 18, the requestTImeout is now 5 mins. This meant that large restores were failing if the upload didn't finish within 5 mins.

@roofboard said in Chat woot cannot uninstall after breaking the backup system.:

Mar 30 20:19:30 box:services Backing up postgresql
Mar 30 20:29:30 box:services pipeRequestToFile: timeout - connect or post-connect idle timeout

It seems somehow postgresql was not having a happy time. Maybe give it a bit more memory (Services -> PostgreSQL -> increase memory limit). The reboot probably just freed up some memory.

I have given these stable IPs now - https://git.cloudron.io/cloudron/box/-/commit/582994b9d646989b42e27dde1ae0c27bbb964834

@girish It's up now, and all the errors have disappeared.

Thanks, it works.

@girish many thanks !

Not sure why that slipped through. Is this sorted by now? We had a case where slow I/O caused the issue resulting in a timeout. Could this have been the case? https://forum.cloudron.io/topic/4780/postgresql-database-won-t-start/8

Sounds good. Yes, we're happy to help with query optimisation in apps and send feedback upstream. This is the quickest way we can do that. With some luck, we'll all end up with faster apps from this R&D too.

@girish I saw no red indicators on the nav bar. I guess that would have helped me towards the source of the problem. But I will keep an eye if a service gets into trouble again.

Ok, finally sorted after stopping queries that were locking old txnids and then doing a full vacuum. I suppose restarting the postgres container could have done that.

This was useful to find blocking queries, in addition to pg_terminate_backend:

SELECT pid, datname, usename, state, backend_xmin, query FROM pg_stat_activity WHERE backend_xmin IS NOT NULL ORDER BY age(backend_xmin) DESC;

Went from 88% full to 38% full in about 10 minutes.

@nebulon I didn't get any notifications in the dashboard for OOM which is why debugging was fairly unintuitive.

@girish If I encounter it again, I'll check dmesg and let you know.

@debossnow

Do I have to install a particular ADMIN PosgreSQL tool in my Cloudron application to see & administer this Mastadon database?

You can use the Weberminal to get shell like access.
On the top there will be button for Database access.

And to see all my files & their paths in a UI do I need to install a separate directory or file browser in my Cloudron application?

Like the Web Terminal you can use the File Manager to see your files.
The path in the File Manager is always /app/data/.

@privsec That worked, thanks!

@girish I use DBeaver with an SSH tunnel. I just found an option that I forgot to toggle on -- "Show all databases".

Now it shows all the databases!

On a side note, I did first try to configure an SSH tunnel and a post-command to execute "/usr/local/bin/docker exec -ti postgresql bash" and that worked as well.

But, only one tunnel is needed to the cloudron server to login as root that allows access to all databases.

I'm good to go. Thanks for the help!