Cloudron Forum

This was a bug in 7.4.0 and is now fixed in 7.4.1.

We updated node to v18 (from v16). There was a subtle but impact change hidden... In node 16, the requestTimeout was 0. This meant unlimited time. In node 18, the requestTImeout is now 5 mins. This meant that large restores were failing if the upload didn't finish within 5 mins.

@roofboard said in Chat woot cannot uninstall after breaking the backup system.:

Mar 30 20:19:30 box:services Backing up postgresql
Mar 30 20:29:30 box:services pipeRequestToFile: timeout - connect or post-connect idle timeout

It seems somehow postgresql was not having a happy time. Maybe give it a bit more memory (Services -> PostgreSQL -> increase memory limit). The reboot probably just freed up some memory.

I have given these stable IPs now - https://git.cloudron.io/cloudron/box/-/commit/582994b9d646989b42e27dde1ae0c27bbb964834

@girish It's up now, and all the errors have disappeared.

Thanks, it works.

@girish many thanks !

Not sure why that slipped through. Is this sorted by now? We had a case where slow I/O caused the issue resulting in a timeout. Could this have been the case? https://forum.cloudron.io/topic/4780/postgresql-database-won-t-start/8

Sounds good. Yes, we're happy to help with query optimisation in apps and send feedback upstream. This is the quickest way we can do that. With some luck, we'll all end up with faster apps from this R&D too.

@girish I saw no red indicators on the nav bar. I guess that would have helped me towards the source of the problem. But I will keep an eye if a service gets into trouble again.

Ok, finally sorted after stopping queries that were locking old txnids and then doing a full vacuum. I suppose restarting the postgres container could have done that.

This was useful to find blocking queries, in addition to pg_terminate_backend:

SELECT pid, datname, usename, state, backend_xmin, query FROM pg_stat_activity WHERE backend_xmin IS NOT NULL ORDER BY age(backend_xmin) DESC;

Went from 88% full to 38% full in about 10 minutes.

@nebulon I didn't get any notifications in the dashboard for OOM which is why debugging was fairly unintuitive.

@girish If I encounter it again, I'll check dmesg and let you know.

@debossnow

Do I have to install a particular ADMIN PosgreSQL tool in my Cloudron application to see & administer this Mastadon database?

You can use the Weberminal to get shell like access.
On the top there will be button for Database access.

And to see all my files & their paths in a UI do I need to install a separate directory or file browser in my Cloudron application?

Like the Web Terminal you can use the File Manager to see your files.
The path in the File Manager is always /app/data/.

@privsec That worked, thanks!

@girish I use DBeaver with an SSH tunnel. I just found an option that I forgot to toggle on -- "Show all databases".

Now it shows all the databases!

On a side note, I did first try to configure an SSH tunnel and a post-command to execute "/usr/local/bin/docker exec -ti postgresql bash" and that worked as well.

But, only one tunnel is needed to the cloudron server to login as root that allows access to all databases.

I'm good to go. Thanks for the help!

@murgero Thank you
Before seeing this, I decided to be a 'windows user' and reboot the cloudron server.
I guess same effect, but will definitely do your more limited kick next time.
And yes it is working again.
Many thanks

@girish thanks for the guidance, that's exactly what I was looking for

@girish No, I don't have a lot of experience with postgresql. The strategy you describe aligns with my understanding of mysql's philosophy, but postgresql seems to require admins be more explicit. E.g. granting permissions to an arbitrary quoted LIKE pattern is documented in mysql, but postgresql's GRANT docs don't mention anything of the like and look like they require the specific object to exist (by nature of being a valid identifier) before granting permissions. Though I could be mistaken.

@grienauer You can set this thread to "solved" yourself.

I finally managed to get to the bottom of this. There was a bug in the postgresql addon backup logic where connections were not closed properly. This resulted in periodic backups keeping lots of connections open. This is fixed in 6.3.

@amg-web I mean the recovery mode, which you can enable for the app in the "Repair" section on the app configure view in the Cloudron dashboard.