Cloudron backup import failed - Not enough disk space, SSL certificate errors

[AmbroiseUnly]

Following https://docs.cloudron.io/backups/#restore-cloudron, I managed to import my Cloudron from a backup.

But I ran into several issues:

• Baserow App status went to "Error" mode
• Some Apps (n8n, Kutt, Metabase) status is "Running" but authentication is failing (basically, anything that requires a DB to work is failing)

After some digging, I noticed that:

• Baserow wasn't working because it's using a self-signing certificate

I tried to renew the certificate, but it failed with some "Disk full" error.

I tried to understand, because I've got 40GB and I thought that would be enough.
But it's not.

root@n8n-ubuntu-4gb-nbg1-2:~# df -H /dev/sda1
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 40G 39G 0 100% /

So, my understanding is:

• During the backup restore, the disk went full
• This caused various failures, including failure to create new SSL certificates (and, most likely, many other things I'm not aware)
• Which crashed some apps and rendered other unusable

At which point I wonder:

• How I redo the whole import from scratch (probably safer/faster than trying to fix everything manually)
• The Cloudron import mechanism should warn the root cause (it took me a while to understand the root issue, which seems to be the disk space)

[AmbroiseUnly]

I destroyed my server and created a new one.
I assigned 40Go additional Volume.

But I ran again into the same kind of issue:

Before restore

root@baserow-ubuntu-4gb-nbg1-4:~# df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 382M 920K 381M 1% /run
/dev/sda1 38G 13G 23G 37% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda15 253M 142K 252M 1% /boot/efi
tmpfs 382M 4.0K 382M 1% /run/user/0
/dev/sdb 40G 24K 38G 1% /mnt/HC_Volume_100833782

After restore

root@baserow-ubuntu-4gb-nbg1-4:~# df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 382M 1.9M 380M 1% /run
/dev/sda1 38G 36G 0 100% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda15 253M 142K 252M 1% /boot/efi
tmpfs 382M 4.0K 382M 1% /run/user/0
/dev/sdb 40G 24K 38G 1% /mnt/HC_Volume_100833782
overlay 38G 36G 0 100% /var/lib/docker/overlay2/081f6c51d17f0d75c649bcd7722a9c56f89e0c43b9f6b92e6533aa1663da1a4d/merged
overlay 38G 36G 0 100% /var/lib/docker/overlay2/0bd91f27d64f105d278c0e623117c11be28500555d6e40b07af059a4634b6d0d/merged
overlay 38G 36G 0 100% /var/lib/docker/overlay2/681dab328b3503c0f95005c4c1e71d399ba2234e2347f706ce9883da5bd33567/merged
overlay 38G 36G 0 100% /var/lib/docker/overlay2/0462fa83a2b80858c1f540379638cb24a8d05e46799467d9809c756a388b6100/merged
overlay 38G 36G 0 100% /var/lib/docker/overlay2/b0c4faf96bda5f865715897b09b3daac0d4cf0f448136383879fb3067547a88d/merged
overlay 38G 36G 0 100% /var/lib/docker/overlay2/b6ba8f206b97dddd57b97817110882f48ebaf676e516015260b8851343b33a78/merged
overlay 38G 36G 0 100% /var/lib/docker/overlay2/2d4aa43ac085b0574f973440ceff0b20508234d3790b90afbc57b19db8de774e/merged
overlay 38G 36G 0 100% /var/lib/docker/overlay2/0c6917967d7acce71017b18951c271ddf8271579e7d040813b9ca0d20d2e3803/merged
overlay 38G 36G 0 100% /var/lib/docker/overlay2/415cf7ee730face293fd4a6dca46d72eccd6102be2bad4ab28a292646b4016f2/merged
overlay 38G 36G 0 100% /var/lib/docker/overlay2/fb41f65fef4becd7981a3520e4af0f213542167c26a0f75308a388cbe91dd2ce/merged
overlay 38G 36G 0 100% /var/lib/docker/overlay2/7f7b3df11224aba7ea3edbb2e6e0fc372f90e134bfe1763be11a69436d0f641a/merged
overlay 38G 36G 0 100% /var/lib/docker/overlay2/0df9e1a0931108609c3b4338c395b873071eda76b2a68ceedbef6ce97d49fb70/merged
overlay 38G 36G 0 100% /var/lib/docker/overlay2/e7af40a8278f10f403937476b9295910aacdaa8e5e9b7bf9974af2be4c9d1ef6/merged
overlay 38G 36G 0 100% /var/lib/docker/overlay2/fe28db7407fa07e3bbaa9ea1ac31a812010c7b7849f631ec69aff3e78aae2080/merged

To my surprise, the restore process didn't use the space available on the other disk, and that caused the Baserow app to fail to install properly.

Although, this time, the SSL certificates were working, and my other apps are working. I guess it might be random based on which operations are done first.

Not the best DX. Ideally, Cloudron should tell me beforehand that the disk space is insufficient, and point me to the right direction as to how to work around that issue effectively.

[AmbroiseUnly]

I removed all the apps from my Cloudron server, but I see I still have 45% of space occupied (it was 37% before I restored the backup)

I wonder if there is a way to clean unused files?

root@baserow-ubuntu-4gb-nbg1-4:~# df -h
Filesystem Size Used Avail Use% Mounted on
tmpfs 382M 1.5M 381M 1% /run
/dev/sda1 38G 16G 20G 45% /
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
/dev/sda15 253M 142K 252M 1% /boot/efi
tmpfs 382M 4.0K 382M 1% /run/user/0
overlay 38G 16G 20G 45% /var/lib/docker/overlay2/081f6c51d17f0d75c649bcd7722a9c56f89e0c43b9f6b92e6533aa1663da1a4d/merged
overlay 38G 16G 20G 45% /var/lib/docker/overlay2/0bd91f27d64f105d278c0e623117c11be28500555d6e40b07af059a4634b6d0d/merged
overlay 38G 16G 20G 45% /var/lib/docker/overlay2/681dab328b3503c0f95005c4c1e71d399ba2234e2347f706ce9883da5bd33567/merged
overlay 38G 16G 20G 45% /var/lib/docker/overlay2/0462fa83a2b80858c1f540379638cb24a8d05e46799467d9809c756a388b6100/merged
overlay 38G 16G 20G 45% /var/lib/docker/overlay2/b0c4faf96bda5f865715897b09b3daac0d4cf0f448136383879fb3067547a88d/merged
overlay 38G 16G 20G 45% /var/lib/docker/overlay2/0c6917967d7acce71017b18951c271ddf8271579e7d040813b9ca0d20d2e3803/merged
overlay 38G 16G 20G 45% /var/lib/docker/overlay2/415cf7ee730face293fd4a6dca46d72eccd6102be2bad4ab28a292646b4016f2/merged
overlay 38G 16G 20G 45% /var/lib/docker/overlay2/f4eca66154862a166d325235632c791f7579b785e9b11cca25999a8cfea875fa/merged

---

Edit: I believe I should just run rm -rf /var/lib/docker once the installs are done.

In retrospection, I could have maybe done that during the install to save some space, but deleting stuff while other stuff is installing might also lead to troubles.

See https://docs.cloudron.io/storage/#docker-images

---

Edit 2: Well, all I achieved by stopping docker and removing the images is to completely crash my system.

Even going through all steps here didn't fix it. Restarting didn't work either. The only thing that didn't work from the doc was mysql, which container's doesn't exist.

root@baserow-ubuntu-4gb-nbg1-4:~# docker restart mysql
Error response from daemon: No such container: mysql

[AmbroiseUnly]

When restoring a backup for an app, the restoration tool should first check whether the backup is available online before doing all other operations, it would save a lot of time and trouble when giving the wrong path.

Also, having the ability to decide which apps to restore would be great.
I'm using Hetzner and its 40GB disk space is not enough, I'd have had a much easier time only installing a single app instead of installing them all.

Especially that I'm moving from DigitalOcean droplet to Hetzner because I want to split my servers and apps (make critical apps alone in a single server). Typically, I'm converting a 6 apps Cloudron install into:

• 1 server with 1 baserow app
• 1 server with 1 n8n app
• 1 server with the rest of my apps (non-critical ones)

The current import mechanism didn't help achieve this.

[girish]

@AmbroiseUnly said in Cloudron backup import failed - Not enough disk space, SSL certificate errors:

When restoring a backup for an app, the restoration tool should first check whether the backup is available online

When the backup is not found, the restore operation will fail and then you can try to restore again. But that's the theory... I guess it didn't work that way for you.

Are you still facing any issues. I lost track of the exact situation since it's across many threads.

[girish]

@AmbroiseUnly said in Cloudron backup import failed - Not enough disk space, SSL certificate errors:

How I redo the whole import from scratch (probably safer/faster than trying to fix everything manually)

FWIW, this is not possible. Once Cloudron is installed, there is no mechanism to restore it. We will have to implement this feature but it's not there because it also security implications (i.e REST routes have to be very carefully protected etc).

[AmbroiseUnly]

@girish said in Cloudron backup import failed - Not enough disk space, SSL certificate errors:

@AmbroiseUnly said in Cloudron backup import failed - Not enough disk space, SSL certificate errors:

When restoring a backup for an app, the restoration tool should first check whether the backup is available online

When the backup is not found, the restore operation will fail and then you can try to restore again. But that's the theory... I guess it didn't work that way for you.

That's how it works, it's just that it does numerous operations before fetching the backup, and thus failing a long time after beginning the import process, although it could fail right away, if it was fetching the backup earlier.

Are you still facing any issues. I lost track of the exact situation since it's across many threads.

Eventually, I imported it Dry Run mode and that worked well, I then updated the IP address myself and all is good now. The restore feature got broken because of some DNS stuff I still don't understand (was fetching DNS configuration from DigitalOcean instead of AWS Route53, for unknown reason)

[girish]

@AmbroiseUnly yeah, good idea. I will add it to our internal list to check for backup existence to give quicker feedback.