Backups failing (because of Mastodon, I think): Could not pipe....

[jdaviescoates]

I'm getting this error:

Could not pipe http://172.18.30.2:3000/databases/db00172a9d159d42c5b23058c4938f5821/backup?access_token=xxxxxxxxxxxxxxxe16654a341374b6f31866a7022b99ac30442ab1b5c060edd4a7e028f9e9d4ed42d5d83cb79d3353b9feb22184f59e60fd7d1edc3680cbe793450caadf1ee23b7ae4db6d193be6694f0d6e2161bd7c49b89ec28754dee59004151d0f8b364ce99243f34c1b8846db89478aa3bd48ed5a32989c8b3b9e4366dd to /home/yellowtent/appsdata/00172a9d-159d-42c5-b230-58c4938f5821/postgresqldump: socket hang up

I think it might be something to do with Mastodon because I can't update that either and see a very similar error there:

Could not pipe http://172.18.30.2:3000/databases/db00172a9d159d42c5b23058c4938f5821/backup?access_token=xxxxxxxxxxxxxxe16654a341374b6f31866a7022b99ac30442ab1b5c060edd4a7e028f9e9d4ed42d5d83cb79d3353b9feb22184f59e60fd7d1edc3680cbe793450caadf1ee23b7ae4db6d193be6694f0d6e2161bd7c49b89ec28754dee59004151d0f8b364ce99243f34c1b8846db89478aa3bd48ed5a32989c8b3b9e4366dd to /home/yellowtent/appsdata/00172a9d-159d-42c5-b230-58c4938f5821/postgresqldump: socket hang up
Could not pipe http://172.18.30.2:3000/databases/db00172a9d159d42c5b23058c4938f5821/backup?access_token=xxxxxxxxxxxxxxxxe16654a341374b6f31866a7022b99ac30442ab1b5c060edd4a7e028f9e9d4ed42d5d83cb79d3353b9feb22184f59e60fd7d1edc3680cbe793450caadf1ee23b7ae4db6d193be6694f0d6e2161bd7c49b89ec28754dee59004151d0f8b364ce99243f34c1b8846db89478aa3bd48ed5a32989c8b3b9e4366dd to /home/yellowtent/appsdata/00172a9d-159d-42c5-b230-58c4938f5821/postgresqldump: socket hang upe_text

[jdaviescoates]

Postgres did look like it was running low on memory, so I upped it.

It still seemed to be using quite a lot of memory.

Then I restarted it and the memory usage dropped right down again.

Then an instance of Discourse that was not responding came back up, the Mastodon updates worked and so did the backup.

In short: restarting postgresql in services fixed the issue.

[nebulon]

Is it possible that your postgres service is at limit memory wise? Were there any errors of the postgres service in its logs?

[jdaviescoates]

Postgres did look like it was running low on memory, so I upped it.

It still seemed to be using quite a lot of memory.

Then I restarted it and the memory usage dropped right down again.

Then an instance of Discourse that was not responding came back up, the Mastodon updates worked and so did the backup.

In short: restarting postgresql in services fixed the issue.

[nebulon]

Not sure what a good solution to this is, but one reason why we have a memory limit, is that such processes are tamed to a certain extent and don't start accumulating memory caches until they take resources away from other apps.