Mastodon instance ran out of storage, wont start back up.

[tymscar]

Hello. Ive been running a mastodon instance for a week and this morning it ran out of storage.

I increased the disk space and fought a bit with the DNS resolver but managed to get most of those errors out of the way but now its stuck on Not responding.

Logs seem fine, Im not sure what the issue might be.

[girish]

@tymscar are you able to visit mastodon directly i.e by not clicking on the dashboard icon? If so, usually systemctl restart box should fix it.

[tymscar]

In the meantime I have restored a backup from 2 days ago and that works fine. The one from yesterday however does not and I can confirm for sure that at the time of backup, it was working.

[girish]

@tymscar are you able to clone that backup into another location and see if that also results in not responding ?

[tymscar]

@girish I am only allowed to respond once every 6 minutes sadly so I will respond to both of your messages at once with more info!

So I cannot visit it locally or going straight to the ip. However my old old backup from 2 days ago seems to be fine. I would like to go to the more recent backup though.

I have in fact cloned it to see if that fixes the issue and sadly I can report back that it does not. Here is something weird though. I have 2 backups 5-10 minutes apart this morning, Those are the most recent backups. I suspect that might be when I ran out of storage, but why would that trigger two backups? are they both perhaps corrupted?

What would be a good avenue for me to try things?

[girish]

@tymscar Each update triggers a backup. Is the package different for those backups which are apart by 5-10 mins? Also, you cannot visit by IP, you can only visit by domain name because it's a virtual host setup - IP is not enough to figure out which app you are visiting.

[tymscar]

@girish so the backups that are a couple minutes apart are both on 1.8.4, but the backup that I can successfully run is 1.8.3.

I have also noticed an error on the 1.8.4 ones that I cant run, in the logs:
"Unknown primary key for table accounts in model Account."

I get this and what looks like a stacktrace in the logs of the app each time I visit the instance in my browser.

[girish]

@tymscar Seems like there is couple of reports upstream like this - https://github.com/mastodon/mastodon/issues/9953 and https://github.com/mastodon/mastodon/issues/16950 but they both mention maybe the database got corrupt. I wonder if the shortage of disk space made things corrupt since we started noticing it.

I would also check if Services -> PostgreSQL has enough memory moving forward.

[tymscar]

@girish One thing I noticed actually is that theres another 2 backups I cant go back to. All same error. The only one that I can is from 2 days ago. All have the same problem. And I am 100% sure I used it AFTER those