Baserow didn't let me access some tables then crashed when restarting it

[gabrielle]

Hello,

I've added a 70k records to a base in Baserow today. It was working alright (a bit slow, sure) but tonight it wouldn't let me access some tables and gave me error messages when trying to change rows in the tables I could access.

I tried increasing the memory with not much success. The graphs were showing I was quite under the limit but could not access a large part of my app.

When I tried restoring the back up, it gave me error 500, saying other users where accessing the app. Now it won't start, saying:

Nov 13 03:12:49 - ERROR (node:25) [DEP0040] DeprecationWarning: The punycode module is deprecated. Please use a userland alternative instead.
Nov 13 03:12:49 - (Use node --trace-deprecation ... to show where the warning was created)
Nov 13 03:12:50 => Healtheck error got response status 502

Can someone please help? I am out of my depth here

Thank you!

[nebulon]

Have you tried to restore it again? Also follow the restore logs in the logsviewer to maybe get more error info. The DeprecationWarning is likely irrelvant and something else does not startup, if you hit a 502 error via healthcheck.

[gabrielle]

I tried restoring it a number of times after giving more space to Postgresql and now it restarted again. For a while, Cloudron told me there was no DNS record pointing on the app (there was no modification there in the last months) and then it started working again.
What are the steps I can take so it doesn't happen again in the future?

[nebulon]

Hard to tell in hindsight in which state the app was but if adding more resources helped then this looks like the solution to ensure the system doesn't run out of disk space or maybe memory

[gabrielle]

I am running into a similar issue, there is 2/3 of the content of a table I can not access (record lines are displayed in white with no content and I can't click on them). I can not figure out what the issue is and it makes some view of the table crash (but not others).
In Cloudron, Baserow has an error saying: FileSystem Error: failed to register layer: write /app/code/web-frontend/node_modules/node-sass/build/Release/obj.target/binding.node: no space left on device
It's failing to update to 1.29.2 (currently at 1.29.0).
Weirdly, I have 4Go of space left on my server and the update is about 1Go.

Is there another device I should look at?

[nebulon]

Maybe other filesystems like /tmp or /run ran out of disk space temporarily? Can you open a webterminal into the app and run df right after the issue happens?

[gabrielle]

I've retried the update, df says:

Filesystem 1K-blocks Used Available Use% Mounted on
overlay 30297152 27992580 2288188 93% /
tmpfs 65536 0 65536 0% /dev
shm 65536 16 65520 1% /dev/shm
/dev/root 30297152 27992580 2288188 93% /tmp
tmpfs 1974552 0 1974552 0% /proc/acpi
tmpfs 1974552 0 1974552 0% /proc/scsi
tmpfs 1974552 0 1974552 0% /sys/firmware

[nebulon]

That seems okish. So rereading the previous messages here, just to clarify, so the original issue was about loading/displaying a huge table, while then the issue was to update the app which fails?

[gabrielle]

Yes exactly. Now, the app can also not create new backups. It seems to be keeping backups from Nov 15th onwards (when the backup policy is set to keep them 2 days only). I don't see how to delete specific backups to make more room nor why Cloudron doesn't manage to clear them.
Any chance we could hop on a call to figure this out?

[nebulon]

Cloudron will keep backups of the last update, regardless of the retention policy. Maybe this is what you see from the 15th. Generally if you try to make some room for backups, I assume you are having your backups on the local disk, which is strongly discouraged as if the server or the disk run into trouble you might lose those. Overall both issues might be connected if the system is running low on disk space, also /tmp will have trouble if (and we are not experts on baserow internals) the app needs to create large tmp files for big tables.

[gabrielle]

Good call on the backups saving location, I've set it up somewhere else yesterday.
Cloudron managed to create new backups last night and delete the old ones.
However, it's still failing to update to the next Baserow version. It seems that it creates a backup every time it tries, filling out the disk space and then it fails because there is not enough disk space.
Any idea of what I could do to fix this?

[nebulon]

I am confused, so have you configured backups to be on a remote storage or not?

[gabrielle]

Yes, the backups are on a remote storage.
The system now says:
Free space: 2.99 Go and /var/backups (Old Backups): 3.93 GB
I'd like to remove these old backups, how do I do it?

[nebulon]

If you have new backups made into the remote storage, you can safely just remove the local ones at /var/backups/ via SSH. Something like rm -rf /var/backups/* (to keep the folder itself)