Why is my backup drive full?
-
My backups are failing.
It says:
Not enough disk space for backup. Needed: 21.6 GB Available: 7.13 GB
But my retention is only 2 days and I've clicked the Cleanup Backups button and the drive is 100GB...
So even if there were two days of backups kept at 22GB x 2 = 44GB there should still be plenty of space left over, not a mere 7.13GB!
What is going on?
Thanks!
-
@jdaviescoates There are some quirks in the backup retention - https://docs.cloudron.io/backups/#retention-clean-up . Some backups like the recent 5.6.0, 5.6.1 backups might be retained for 3 weeks till they get cleaned up.
You can check the output of
ls -l /var/backups. And just remove the corresponding timestamped directories entirely if you think the update went fine and you won't need those backups. This is safe. But don't remove thesnapshotdirectory, this is required. -
@girish said in Why is my backup drive full?:
ls -l /var/backups
I just ssh'd into my server and did that command.
It looks like I've got a load of backups from January and no recent ones?!?
Here is the output:
root@ubuntu-2gb-fsn1-1:~# ls -l /var/backups total 1988 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 12 2020 2020-01-12-233001-480 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 13 2020 2020-01-13-233002-126 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 14 2020 2020-01-14-060000-963 drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 15 2020 2020-01-15-060001-040 -rw-r--r-- 1 root root 51200 Sep 18 06:25 alternatives.tar.0 -rw-r--r-- 1 root root 2695 Jul 30 06:25 alternatives.tar.1.gz -rw-r--r-- 1 root root 2693 May 5 06:25 alternatives.tar.2.gz -rw-r--r-- 1 root root 2692 Mar 25 2020 alternatives.tar.3.gz -rw-r--r-- 1 root root 2689 Jan 28 2020 alternatives.tar.4.gz -rw-r--r-- 1 root root 2686 Jan 10 2020 alternatives.tar.5.gz -rw-r--r-- 1 root root 7663 Sep 24 06:08 apt.extended_states.0 -rw-r--r-- 1 root root 946 Sep 23 06:42 apt.extended_states.1.gz -rw-r--r-- 1 root root 931 Sep 20 00:39 apt.extended_states.2.gz -rw-r--r-- 1 root root 926 Sep 9 06:57 apt.extended_states.3.gz -rw-r--r-- 1 root root 910 Sep 5 06:20 apt.extended_states.4.gz -rw-r--r-- 1 root root 927 Sep 4 06:21 apt.extended_states.5.gz -rw-r--r-- 1 root root 907 Jul 24 06:41 apt.extended_states.6.gz -rw-r--r-- 1 root root 11 Dec 8 2019 dpkg.arch.0 -rw-r--r-- 1 root root 43 Dec 8 2019 dpkg.arch.1.gz -rw-r--r-- 1 root root 43 Dec 8 2019 dpkg.arch.2.gz -rw-r--r-- 1 root root 43 Dec 8 2019 dpkg.arch.3.gz -rw-r--r-- 1 root root 43 Dec 8 2019 dpkg.arch.4.gz -rw-r--r-- 1 root root 43 Dec 8 2019 dpkg.arch.5.gz -rw-r--r-- 1 root root 43 Dec 8 2019 dpkg.arch.6.gz -rw-r--r-- 1 root root 437 Dec 8 2019 dpkg.diversions.0 -rw-r--r-- 1 root root 202 Dec 8 2019 dpkg.diversions.1.gz -rw-r--r-- 1 root root 202 Dec 8 2019 dpkg.diversions.2.gz -rw-r--r-- 1 root root 202 Dec 8 2019 dpkg.diversions.3.gz -rw-r--r-- 1 root root 202 Dec 8 2019 dpkg.diversions.4.gz -rw-r--r-- 1 root root 202 Dec 8 2019 dpkg.diversions.5.gz -rw-r--r-- 1 root root 202 Dec 8 2019 dpkg.diversions.6.gz -rw-r--r-- 1 root root 135 Dec 8 2019 dpkg.statoverride.0 -rw-r--r-- 1 root root 129 Dec 8 2019 dpkg.statoverride.1.gz -rw-r--r-- 1 root root 129 Dec 8 2019 dpkg.statoverride.2.gz -rw-r--r-- 1 root root 129 Dec 8 2019 dpkg.statoverride.3.gz -rw-r--r-- 1 root root 129 Dec 8 2019 dpkg.statoverride.4.gz -rw-r--r-- 1 root root 129 Dec 8 2019 dpkg.statoverride.5.gz -rw-r--r-- 1 root root 129 Dec 8 2019 dpkg.statoverride.6.gz -rw-r--r-- 1 root root 673569 Sep 24 06:08 dpkg.status.0 -rw-r--r-- 1 root root 185746 Sep 20 00:39 dpkg.status.1.gz -rw-r--r-- 1 root root 184937 Sep 18 06:10 dpkg.status.2.gz -rw-r--r-- 1 root root 184939 Sep 9 06:58 dpkg.status.3.gz -rw-r--r-- 1 root root 184403 Sep 5 06:20 dpkg.status.4.gz -rw-r--r-- 1 root root 184648 Sep 4 06:22 dpkg.status.5.gz -rw-r--r-- 1 root root 184322 Aug 23 06:36 dpkg.status.6.gz -rw------- 1 root root 780 Jul 19 22:20 group.bak -rw------- 1 root shadow 650 Jul 19 22:20 gshadow.bak -rw------- 1 root root 1725 Jul 19 22:20 passwd.bak -rw------- 1 root shadow 919 Jul 19 22:20 shadow.bak drwxr-xr-x 2 yellowtent yellowtent 4096 Jan 15 2020 snapshotBut in Backups I see this one from 6 days ago listed:
So I don't understand why I can't see that in /var/backups ?
-
And somewhat bizarrely my latest notification about the backup failing says:
Backup failed: Not enough disk space for backup. Needed: 7.35 GB Available: 7.13 GB.
I've no idea why the needed space has dropped from 21.6GB to 7.35GB but I've definitely got more that 7.35GB on my Cloudron (my wife have 11.2GB on Nextcloud alone!)
-
Ah, looking at the logs, I think that 7.35GB needed is just for files.uniteddiversity.coop
Sep 25 23:00:01 box:settings initCache: pre-load settings Sep 25 23:00:01 box:taskworker Starting task 2312. Logs are at /home/yellowtent/platformdata/logs/tasks/2312.log Sep 25 23:00:01 box:tasks 2312: {"percent":2,"error":null} Sep 25 23:00:01 box:tasks 2312: {"percent":1,"message":"Backing up files.uniteddiversity.coop"} Sep 25 23:00:01 box:tasks 2312: {"percent":5,"message":"Snapshotting app files.uniteddiversity.coop"} Sep 25 23:00:01 box:addons files.uniteddiversity.coop backupAddons Sep 25 23:00:01 box:addons files.uniteddiversity.coop backupAddons: Backing up ["ldap","localstorage"] Sep 25 23:00:01 box:backups files.uniteddiversity.coop snapshotApp: took 0.002 seconds Sep 25 23:00:01 box:tasks 2312: {"percent":5,"message":"Uploading app snapshot files.uniteddiversity.coop"} Sep 25 23:00:01 box:shell backup-snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb spawn: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb tgz {"localRoot":"/home/yellowtent/appsdata/08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb","layout":[]} Sep 25 23:00:02 box:shell backup-snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb (stdout): 2020-09-25T22:00:02.751Z box:backupupload Backing up {"localRoot":"/home/yellowtent/appsdata/08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb","layout":[]} to snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb Sep 25 23:00:02 box:shell backup-snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb (stdout): 2020-09-25T22:00:02.810Z box:settings initCache: pre-load settings Sep 25 23:00:02 box:shell backup-snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb (stdout): 2020-09-25T22:00:02.924Z box:backups upload: id snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb format tgz dataLayout {"localRoot":"/home/yellowtent/appsdata/08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb","layout":[]} Sep 25 23:00:03 box:shell backup-snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb (stdout): 2020-09-25T22:00:03.056Z box:storage/filesystem checkPreconditions: getting disk usage of /home/yellowtent/appsdata/08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb Sep 25 23:00:03 box:shell backup-snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb (stdout): 2020-09-25T22:00:03.132Z box:storage/filesystem checkPreconditions: 6276598526 bytes Sep 25 23:00:03 box:shell backup-snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb (stdout): 2020-09-25T22:00:03.194Z box:backupupload upload completed. error: { BoxError: Not enough disk space for backup. Needed: 7.35 GB Available: 7.13 GB at /home/yellowtent/box/src/storage/filesystem.js:77:57 at process._tickCallback (internal/process/next_tick.js:68:7) name: 'BoxError', reason: 'FileSystem Error', details: {}, message: 'Not enough disk space for backup. Needed: 7.35 GB Available: 7.13 GB' } Sep 25 23:00:03 box:backups runBackupUpload: result - {"result":"Not enough disk space for backup. Needed: 7.35 GB Available: 7.13 GB"} Sep 25 23:00:03 box:shell backup-snapshot/app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb code: 50, signal: null Sep 25 23:00:03 box:backups files.uniteddiversity.coop Unable to backup { BoxError: Not enough disk space for backup. Needed: 7.35 GB Available: 7.13 GB at /home/yellowtent/box/src/backups.js:863:29 at f (/home/yellowtent/box/node_modules/once/once.js:25:25) at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9) at ChildProcess.emit (events.js:198:13) at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12) name: 'BoxError', reason: 'External Error', details: {}, message: 'Not enough disk space for backup. Needed: 7.35 GB Available: 7.13 GB' } Sep 25 23:00:03 box:taskworker Task took 2.046 seconds Sep 25 23:00:03 box:tasks setCompleted - 2312: {"result":null,"error":{"stack":"BoxError: Not enough disk space for backup. Needed: 7.35 GB Available: 7.13 GB\n at /home/yellowtent/box/src/backups.js:863:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:198:13)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"External Error","details":{},"message":"Not enough disk space for backup. Needed: 7.35 GB Available: 7.13 GB"}} Sep 25 23:00:03 box:tasks 2312: {"percent":100,"result":null,"error":{"stack":"BoxError: Not enough disk space for backup. Needed: 7.35 GB Available: 7.13 GB\n at /home/yellowtent/box/src/backups.js:863:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:198:13)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"External Error","details":{},"message":"Not enough disk space for backup. Needed: 7.35 GB Available: 7.13 GB"}} -
I just deleted the directories from Jan 12-14 (kept 15 for good measure) and then tried to run a backup.
Got this error:
Not enough disk space for backup. Needed: 2.18 GB Available: 2.17 GB
I'm seriously confused now.
Also, I note in System is showing 86.39GB of Backup data:
But I've no idea what nor where that is?!? Help!
-
I just had a look at whats inside /var/backups/snapshot and it looks like it's all from 15 Jan! Surely that can't be right?!?
root@ubuntu-2gb-fsn1-1:/var/backups/snapshot# ls -l total 1218540 -rw-r--r-- 2 yellowtent yellowtent 1165420357 Jan 15 2020 app_08388b1d-12c7-42c2-b6bd-a5c5be5cfcbb.tar.gz -rw-r--r-- 2 yellowtent yellowtent 7667 Jan 15 2020 app_133a418f-dc94-4f4a-bd80-b29d6fe6d356.tar.gz -rw-r--r-- 2 yellowtent yellowtent 8754 Jan 15 2020 app_2077efa7-c7a1-42fd-91f2-d528e594ee54.tar.gz -rw-r--r-- 2 yellowtent yellowtent 7267103 Jan 15 2020 app_3eef7707-506f-4c12-b762-364589507136.tar.gz -rw-r--r-- 2 yellowtent yellowtent 9855 Jan 15 2020 app_77519696-45f5-4bd8-bc28-78ebb2ba3664.tar.gz -rw-r--r-- 2 yellowtent yellowtent 68475 Jan 15 2020 app_a0a9bc60-0693-4b86-bcd1-ce74c301ff3c.tar.gz -rw-r--r-- 2 yellowtent yellowtent 74700646 Jan 15 2020 app_d0e0fa7c-e358-4d93-869b-e227feec349f.tar.gz -rw-r--r-- 2 yellowtent yellowtent 278567 Jan 15 2020 box.tar.gz root@ubuntu-2gb-fsn1-1:/var/backups/snapshot# -
I note that 15 Jan is actually the day I created the volume where my backups live:
-
Just to get a clear picture here, can you double check your backup configuration to see the folder it actually is using for backups? Also check if you use tarball or rsync style backup.
Note that if you change the backup configuration, the old backups will remain on the disk, Cloudron does not do any purging of old backups. For example if you switch from tarball to rsync, the old tarballs will still exist on disk and have to be manually removed as of now.
-
@nebulon said in Why is my backup drive full?:
Just to get a clear picture here, can you double check your backup configuration to see the folder it actually is using for backups?
Doh!
Yeah, I've got a volume mounted so it's not /var/backups/ it's at /mnt/backups
Here they all are!
root@ubuntu-2gb-fsn1-1:~# ls -l /mnt/backups total 164 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 6 22:30 2020-09-06-223002-163 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 8 22:30 2020-09-08-223001-828 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 8 22:30 2020-09-08-223002-037 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 9 22:30 2020-09-09-223002-877 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 9 22:38 2020-09-09-223003-092 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 9 22:30 2020-09-09-223003-461 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 9 22:30 2020-09-09-223048-638 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 11 22:30 2020-09-11-223001-919 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 11 22:30 2020-09-11-223002-397 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 12 22:40 2020-09-12-224039-652 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 13 22:41 2020-09-13-224129-874 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 13 22:45 2020-09-13-224527-403 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 13 22:47 2020-09-13-224555-257 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 13 22:47 2020-09-13-224713-002 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 13 22:48 2020-09-13-224757-613 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 14 02:30 2020-09-14-023003-201 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 14 02:31 2020-09-14-023003-213 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 14 02:30 2020-09-14-023003-224 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 14 02:31 2020-09-14-023050-510 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 15 22:40 2020-09-15-224017-990 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 18 22:40 2020-09-18-224017-550 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 19 00:57 2020-09-19-003001-934 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 20 00:39 2020-09-20-000010-527 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 20 22:30 2020-09-20-223002-164 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 21 00:56 2020-09-21-003001-880 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 21 22:30 2020-09-21-223003-597 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 21 22:30 2020-09-21-223003-806 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 21 22:30 2020-09-21-223003-870 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 23 22:30 2020-09-23-223001-540 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 24 00:30 2020-09-24-003001-489 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 24 22:11 2020-09-24-220003-529 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 25 13:37 2020-09-25-132557-773 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 25 22:30 2020-09-25-223001-288 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 25 22:30 2020-09-25-223001-426 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 26 02:30 2020-09-26-023001-266 drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 26 08:36 2020-09-26-082946-798 drwx------ 2 root root 16384 Jan 15 2020 lost+found drwxr-xr-x 2 yellowtent yellowtent 4096 Sep 26 08:29 snapshot root@ubuntu-2gb-fsn1-1:~#@nebulon said in Why is my backup drive full?:
Note that if you change the backup configuration, the old backups will remain on the disk, Cloudron does not do any purging of old backups. For example if you switch from tarball to rsync, the old tarballs will still exist on disk and have to be manually removed as of now.
That explains why the ancient ones (because I purchased and mounted a volume) were still in /var/backups
-
@jdaviescoates said in Why is my backup drive full?:
/var/backups/
You can safely remove the files/backups here, will get back some space.
-
@girish said in Why is my backup drive full?:
You can safely remove the files/backups here, will get back some space.
Just to be clear, I'm assuming I can safely remove absolutely everything in there, right?
Perhaps even completely remove that folder?
-
@jdaviescoates said in Why is my backup drive full?:
Perhaps even completely remove that folder?
Yes, you can remove it entirely.
-
@girish OK will do.
The error messages I'm getting are still really strange. Latest one is the strangest yet:
Not enough disk space for backup. Needed: 1.08 GB Available: 1.08 GB
Either way, guess I need to delete a few more old backups from /mnt/backups before backing up will work again...
-
In the end I just bit the bullet and doubled the size of my backup disk!
Still slightly confused/ bemused by all the strange random space errors I was getting though!
-
@girish I had the same question as @jdaviescoates. I've moved my backups from internal same drive to an external USB drive.
I am using tarballs as backup format.
I've moved the backup directories from /var/backups/ to my external drive. If I want to restore from those backup directories, do I need the contents of the /var/backups/snapshot directory also?
If I don't want to restore from those older backups, can I confirm that it's OK to delete the contents of the /var/backups/snapshot directory?
Thanks for the helpful thread.
-
@jdaviescoates If you're using rsync, the space errors might not be so strange, as it would only need to add additional files, rather than backing up the whole thing again, hence the space required for a given backup might be quite random (depending what has changed since the last one). Just a suggestion though. And of course if you're using tarball then that is really weird!
-
@avatar1024 yeah, it was tarball.
-
@ei8fdb said in Why is my backup drive full?:
I've moved the backup directories from /var/backups/ to my external drive. If I want to restore from those backup directories, do I need the contents of the /var/backups/snapshot directory also?
The snapshot directory is not required for restoring. BUT it's required for the actual the backups to work (so think of it as a working directory for backups). It's important to not remove that directory when doing backups! Since, Cloudron will use hard links from the actual backups to the snapshot directory, it's not really taking up extra space.
