Why is my backup drive full?

[jdaviescoates]

My backups are failing.

It says:

Not enough disk space for backup. Needed: 21.6 GB Available: 7.13 GB

But my retention is only 2 days and I've clicked the Cleanup Backups button and the drive is 100GB...

So even if there were two days of backups kept at 22GB x 2 = 44GB there should still be plenty of space left over, not a mere 7.13GB!

What is going on?

Thanks!

[girish]

@jdaviescoates There are some quirks in the backup retention - https://docs.cloudron.io/backups/#retention-clean-up . Some backups like the recent 5.6.0, 5.6.1 backups might be retained for 3 weeks till they get cleaned up.

You can check the output of ls -l /var/backups. And just remove the corresponding timestamped directories entirely if you think the update went fine and you won't need those backups. This is safe. But don't remove the snapshot directory, this is required.

[jdaviescoates]

@girish said in Why is my backup drive full?:

ls -l /var/backups

I just ssh'd into my server and did that command.

It looks like I've got a load of backups from January and no recent ones?!?

Here is the output:

root@ubuntu-2gb-fsn1-1:~# ls -l /var/backups
total 1988
drwxr-xr-x 2 yellowtent yellowtent   4096 Jan 12  2020 2020-01-12-233001-480
drwxr-xr-x 2 yellowtent yellowtent   4096 Jan 13  2020 2020-01-13-233002-126
drwxr-xr-x 2 yellowtent yellowtent   4096 Jan 14  2020 2020-01-14-060000-963
drwxr-xr-x 2 yellowtent yellowtent   4096 Jan 15  2020 2020-01-15-060001-040
-rw-r--r-- 1 root       root        51200 Sep 18 06:25 alternatives.tar.0
-rw-r--r-- 1 root       root         2695 Jul 30 06:25 alternatives.tar.1.gz
-rw-r--r-- 1 root       root         2693 May  5 06:25 alternatives.tar.2.gz
-rw-r--r-- 1 root       root         2692 Mar 25  2020 alternatives.tar.3.gz
-rw-r--r-- 1 root       root         2689 Jan 28  2020 alternatives.tar.4.gz
-rw-r--r-- 1 root       root         2686 Jan 10  2020 alternatives.tar.5.gz
-rw-r--r-- 1 root       root         7663 Sep 24 06:08 apt.extended_states.0
-rw-r--r-- 1 root       root          946 Sep 23 06:42 apt.extended_states.1.gz
-rw-r--r-- 1 root       root          931 Sep 20 00:39 apt.extended_states.2.gz
-rw-r--r-- 1 root       root          926 Sep  9 06:57 apt.extended_states.3.gz
-rw-r--r-- 1 root       root          910 Sep  5 06:20 apt.extended_states.4.gz
-rw-r--r-- 1 root       root          927 Sep  4 06:21 apt.extended_states.5.gz
-rw-r--r-- 1 root       root          907 Jul 24 06:41 apt.extended_states.6.gz
-rw-r--r-- 1 root       root           11 Dec  8  2019 dpkg.arch.0
-rw-r--r-- 1 root       root           43 Dec  8  2019 dpkg.arch.1.gz
-rw-r--r-- 1 root       root           43 Dec  8  2019 dpkg.arch.2.gz
-rw-r--r-- 1 root       root           43 Dec  8  2019 dpkg.arch.3.gz
-rw-r--r-- 1 root       root           43 Dec  8  2019 dpkg.arch.4.gz
-rw-r--r-- 1 root       root           43 Dec  8  2019 dpkg.arch.5.gz
-rw-r--r-- 1 root       root           43 Dec  8  2019 dpkg.arch.6.gz
-rw-r--r-- 1 root       root          437 Dec  8  2019 dpkg.diversions.0
-rw-r--r-- 1 root       root          202 Dec  8  2019 dpkg.diversions.1.gz
-rw-r--r-- 1 root       root          202 Dec  8  2019 dpkg.diversions.2.gz
-rw-r--r-- 1 root       root          202 Dec  8  2019 dpkg.diversions.3.gz
-rw-r--r-- 1 root       root          202 Dec  8  2019 dpkg.diversions.4.gz
-rw-r--r-- 1 root       root          202 Dec  8  2019 dpkg.diversions.5.gz
-rw-r--r-- 1 root       root          202 Dec  8  2019 dpkg.diversions.6.gz
-rw-r--r-- 1 root       root          135 Dec  8  2019 dpkg.statoverride.0
-rw-r--r-- 1 root       root          129 Dec  8  2019 dpkg.statoverride.1.gz
-rw-r--r-- 1 root       root          129 Dec  8  2019 dpkg.statoverride.2.gz
-rw-r--r-- 1 root       root          129 Dec  8  2019 dpkg.statoverride.3.gz
-rw-r--r-- 1 root       root          129 Dec  8  2019 dpkg.statoverride.4.gz
-rw-r--r-- 1 root       root          129 Dec  8  2019 dpkg.statoverride.5.gz
-rw-r--r-- 1 root       root          129 Dec  8  2019 dpkg.statoverride.6.gz
-rw-r--r-- 1 root       root       673569 Sep 24 06:08 dpkg.status.0
-rw-r--r-- 1 root       root       185746 Sep 20 00:39 dpkg.status.1.gz
-rw-r--r-- 1 root       root       184937 Sep 18 06:10 dpkg.status.2.gz
-rw-r--r-- 1 root       root       184939 Sep  9 06:58 dpkg.status.3.gz
-rw-r--r-- 1 root       root       184403 Sep  5 06:20 dpkg.status.4.gz
-rw-r--r-- 1 root       root       184648 Sep  4 06:22 dpkg.status.5.gz
-rw-r--r-- 1 root       root       184322 Aug 23 06:36 dpkg.status.6.gz
-rw------- 1 root       root          780 Jul 19 22:20 group.bak
-rw------- 1 root       shadow        650 Jul 19 22:20 gshadow.bak
-rw------- 1 root       root         1725 Jul 19 22:20 passwd.bak
-rw------- 1 root       shadow        919 Jul 19 22:20 shadow.bak
drwxr-xr-x 2 yellowtent yellowtent   4096 Jan 15  2020 snapshot

But in Backups I see this one from 6 days ago listed:

So I don't understand why I can't see that in /var/backups ?

[jdaviescoates]

And somewhat bizarrely my latest notification about the backup failing says:

Backup failed: Not enough disk space for backup. Needed: 7.35 GB Available: 7.13 GB.

I've no idea why the needed space has dropped from 21.6GB to 7.35GB but I've definitely got more that 7.35GB on my Cloudron (my wife have 11.2GB on Nextcloud alone!)

[jdaviescoates]

Ah, looking at the logs, I think that 7.35GB needed is just for files.uniteddiversity.coop

Sep 25 23:00:01 box:settings initCache: pre-load settings
Sep 25 23:00:01 box:taskworker Starting task 2312. Logs are at /home/yellowtent/platformdata/logs/tasks/2312.log
Sep 25 23:00:01 box:tasks 2312: {"percent":2,"error":null}
Sep 25 23:00:01 box:tasks 2312: {"percent":1,"message":"Backing up files.uniteddiversity.coop"}
Sep 25 23:00:01 box:tasks 2312: {"percent":5,"message":"Snapshotting app files.uniteddiversity.coop"}
Sep 25 23:00:01 box:addons files.uniteddiversity.coop backupAddons
Sep 25 23:00:01 box:addons files.uniteddiversity.coop backupAddons: Backing up ["ldap","localstorage"]
Sep 25 23:00:01 box:backups files.uniteddiversity.coop snapshotApp: took 0.002 seconds
Sep 25 23:00:01 box:tasks 2312: {"percent":5,"message":"Uploading app snapshot files.uniteddiversity.coop"}
Sep 25 23:00:01 box:shell backup-snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb spawn: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb tgz {"localRoot":"/home/yellowtent/appsdata/08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb","layout":[]}
Sep 25 23:00:02 box:shell backup-snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb (stdout): 2020-09-25T22:00:02.751Z box:backupupload Backing up {"localRoot":"/home/yellowtent/appsdata/08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb","layout":[]} to snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb
Sep 25 23:00:02 box:shell backup-snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb (stdout): 2020-09-25T22:00:02.810Z box:settings initCache: pre-load settings
Sep 25 23:00:02 box:shell backup-snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb (stdout): 2020-09-25T22:00:02.924Z box:backups upload: id snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb format tgz dataLayout {"localRoot":"/home/yellowtent/appsdata/08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb","layout":[]}
Sep 25 23:00:03 box:shell backup-snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb (stdout): 2020-09-25T22:00:03.056Z box:storage/filesystem checkPreconditions: getting disk usage of /home/yellowtent/appsdata/08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb
Sep 25 23:00:03 box:shell backup-snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb (stdout): 2020-09-25T22:00:03.132Z box:storage/filesystem checkPreconditions: 6276598526 bytes
Sep 25 23:00:03 box:shell backup-snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb (stdout): 2020-09-25T22:00:03.194Z box:backupupload upload completed. error: { BoxError: Not enough disk space for backup. Needed: 7.35 GB Available: 7.13 GB
at /home/yellowtent/box/src/storage/filesystem.js:77:57
at process._tickCallback (internal/process/next_tick.js:68:7)
name: 'BoxError',
reason: 'FileSystem Error',
details: {},
message:
'Not enough disk space for backup. Needed: 7.35 GB Available: 7.13 GB' }
Sep 25 23:00:03 box:backups runBackupUpload: result - {"result":"Not enough disk space for backup. Needed: 7.35 GB Available: 7.13 GB"}
Sep 25 23:00:03 box:shell backup-snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb code: 50, signal: null
Sep 25 23:00:03 box:backups files.uniteddiversity.coop Unable to backup { BoxError: Not enough disk space for backup. Needed: 7.35 GB Available: 7.13 GB
at /home/yellowtent/box/src/backups.js:863:29
at f (/home/yellowtent/box/node_modules/once/once.js:25:25)
at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)
at ChildProcess.emit (events.js:198:13)
at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)
name: 'BoxError',
reason: 'External Error',
details: {},
message:
'Not enough disk space for backup. Needed: 7.35 GB Available: 7.13 GB' }
Sep 25 23:00:03 box:taskworker Task took 2.046 seconds
Sep 25 23:00:03 box:tasks setCompleted - 2312: {"result":null,"error":{"stack":"BoxError: Not enough disk space for backup. Needed: 7.35 GB Available: 7.13 GB\n at /home/yellowtent/box/src/backups.js:863:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:198:13)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"External Error","details":{},"message":"Not enough disk space for backup. Needed: 7.35 GB Available: 7.13 GB"}}
Sep 25 23:00:03 box:tasks 2312: {"percent":100,"result":null,"error":{"stack":"BoxError: Not enough disk space for backup. Needed: 7.35 GB Available: 7.13 GB\n at /home/yellowtent/box/src/backups.js:863:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:198:13)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"External Error","details":{},"message":"Not enough disk space for backup. Needed: 7.35 GB Available: 7.13 GB"}}

[jdaviescoates]

I just deleted the directories from Jan 12-14 (kept 15 for good measure) and then tried to run a backup.

Got this error:

Not enough disk space for backup. Needed: 2.18 GB Available: 2.17 GB

I'm seriously confused now.

Also, I note in System is showing 86.39GB of Backup data:

But I've no idea what nor where that is?!? Help!

[jdaviescoates]

I just had a look at whats inside /var/backups/snapshot and it looks like it's all from 15 Jan! Surely that can't be right?!?

root@ubuntu-2gb-fsn1-1:/var/backups/snapshot# ls -l
total 1218540
-rw-r--r-- 2 yellowtent yellowtent 1165420357 Jan 15  2020 app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb.tar.gz
-rw-r--r-- 2 yellowtent yellowtent       7667 Jan 15  2020 app_133a418f-dc94-4f4a-bd80-b29d6fe6d356.tar.gz
-rw-r--r-- 2 yellowtent yellowtent       8754 Jan 15  2020 app_2077efa7-c7a1-42fd-91f2-d528e594ee54.tar.gz
-rw-r--r-- 2 yellowtent yellowtent    7267103 Jan 15  2020 app_3eef7707-506f-4c12-b762-364589507136.tar.gz
-rw-r--r-- 2 yellowtent yellowtent       9855 Jan 15  2020 app_77519696-45f5-4bd8-bc28-78ebb2ba3664.tar.gz
-rw-r--r-- 2 yellowtent yellowtent      68475 Jan 15  2020 app_a0a9bc60-0693-4b86-bcd1-ce74c301ff3c.tar.gz
-rw-r--r-- 2 yellowtent yellowtent   74700646 Jan 15  2020 app_d0e0fa7c-e358-4d93-869b-e227feec349f.tar.gz
-rw-r--r-- 2 yellowtent yellowtent     278567 Jan 15  2020 box.tar.gz
root@ubuntu-2gb-fsn1-1:/var/backups/snapshot# 

[jdaviescoates]

I note that 15 Jan is actually the day I created the volume where my backups live:

[nebulon]

Just to get a clear picture here, can you double check your backup configuration to see the folder it actually is using for backups? Also check if you use tarball or rsync style backup.

Note that if you change the backup configuration, the old backups will remain on the disk, Cloudron does not do any purging of old backups. For example if you switch from tarball to rsync, the old tarballs will still exist on disk and have to be manually removed as of now.

[girish]

Also, make sure that the "Directory is external disk" option is checked. This will make sure that the external disk is actually mounted.

[jdaviescoates]

@nebulon said in Why is my backup drive full?:

Just to get a clear picture here, can you double check your backup configuration to see the folder it actually is using for backups?

Doh!

Yeah, I've got a volume mounted so it's not /var/backups/ it's at /mnt/backups

Here they all are!

root@ubuntu-2gb-fsn1-1:~# ls -l /mnt/backups
total 164
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep  6 22:30 2020-09-06-223002-163
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep  8 22:30 2020-09-08-223001-828
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep  8 22:30 2020-09-08-223002-037
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep  9 22:30 2020-09-09-223002-877
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep  9 22:38 2020-09-09-223003-092
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep  9 22:30 2020-09-09-223003-461
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep  9 22:30 2020-09-09-223048-638
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 11 22:30 2020-09-11-223001-919
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 11 22:30 2020-09-11-223002-397
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 12 22:40 2020-09-12-224039-652
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 13 22:41 2020-09-13-224129-874
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 13 22:45 2020-09-13-224527-403
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 13 22:47 2020-09-13-224555-257
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 13 22:47 2020-09-13-224713-002
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 13 22:48 2020-09-13-224757-613
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 14 02:30 2020-09-14-023003-201
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 14 02:31 2020-09-14-023003-213
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 14 02:30 2020-09-14-023003-224
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 14 02:31 2020-09-14-023050-510
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 15 22:40 2020-09-15-224017-990
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 18 22:40 2020-09-18-224017-550
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 19 00:57 2020-09-19-003001-934
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 20 00:39 2020-09-20-000010-527
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 20 22:30 2020-09-20-223002-164
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 21 00:56 2020-09-21-003001-880
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 21 22:30 2020-09-21-223003-597
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 21 22:30 2020-09-21-223003-806
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 21 22:30 2020-09-21-223003-870
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 23 22:30 2020-09-23-223001-540
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 24 00:30 2020-09-24-003001-489
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 24 22:11 2020-09-24-220003-529
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 25 13:37 2020-09-25-132557-773
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 25 22:30 2020-09-25-223001-288
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 25 22:30 2020-09-25-223001-426
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 26 02:30 2020-09-26-023001-266
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 26 08:36 2020-09-26-082946-798
drwx------ 2 root       root       16384 Jan 15  2020 lost+found
drwxr-xr-x 2 yellowtent yellowtent  4096 Sep 26 08:29 snapshot
root@ubuntu-2gb-fsn1-1:~# 

@nebulon said in Why is my backup drive full?:

Note that if you change the backup configuration, the old backups will remain on the disk, Cloudron does not do any purging of old backups. For example if you switch from tarball to rsync, the old tarballs will still exist on disk and have to be manually removed as of now.

That explains why the ancient ones (because I purchased and mounted a volume) were still in /var/backups

[girish]

@jdaviescoates said in Why is my backup drive full?:

/var/backups/

You can safely remove the files/backups here, will get back some space.

[jdaviescoates]

@girish said in Why is my backup drive full?:

You can safely remove the files/backups here, will get back some space.

Just to be clear, I'm assuming I can safely remove absolutely everything in there, right?

Perhaps even completely remove that folder?

[girish]

@jdaviescoates said in Why is my backup drive full?:

Perhaps even completely remove that folder?

Yes, you can remove it entirely.

[jdaviescoates]

@girish OK will do.

The error messages I'm getting are still really strange. Latest one is the strangest yet:

Not enough disk space for backup. Needed: 1.08 GB Available: 1.08 GB

Either way, guess I need to delete a few more old backups from /mnt/backups before backing up will work again...

[jdaviescoates]

In the end I just bit the bullet and doubled the size of my backup disk!

Still slightly confused/ bemused by all the strange random space errors I was getting though!

[ei8fdb]

@girish I had the same question as @jdaviescoates. I've moved my backups from internal same drive to an external USB drive.

I am using tarballs as backup format.

I've moved the backup directories from /var/backups/ to my external drive. If I want to restore from those backup directories, do I need the contents of the /var/backups/snapshot directory also?

If I don't want to restore from those older backups, can I confirm that it's OK to delete the contents of the /var/backups/snapshot directory?

Thanks for the helpful thread.

[avatar1024]

@jdaviescoates If you're using rsync, the space errors might not be so strange, as it would only need to add additional files, rather than backing up the whole thing again, hence the space required for a given backup might be quite random (depending what has changed since the last one). Just a suggestion though. And of course if you're using tarball then that is really weird!

[jdaviescoates]

@avatar1024 yeah, it was tarball.

[girish]

@ei8fdb said in Why is my backup drive full?:

I've moved the backup directories from /var/backups/ to my external drive. If I want to restore from those backup directories, do I need the contents of the /var/backups/snapshot directory also?

The snapshot directory is not required for restoring. BUT it's required for the actual the backups to work (so think of it as a working directory for backups). It's important to not remove that directory when doing backups! Since, Cloudron will use hard links from the actual backups to the snapshot directory, it's not really taking up extra space.