Backup failing, Error when checking for disk space: Error running df: df exited with code null signal SIGTERM

[jayonrails]

Hi,

I had two backups failing with the following error message:

Error when checking for disk space: Error running df: df exited with code null signal SIGTERM

There is a longer log file available, I decided not to post it because it contains many personal data.

Manually running" df" works:

root@my:~# df -h
Filesystem                                Size  Used Avail Use% Mounted on
tmpfs                                     6.3G  5.0M  6.3G   1% /run
/dev/sda1                                 542G  154G  389G  29% /
tmpfs                                      32G     0   32G   0% /dev/shm
tmpfs                                     5.0M     0  5.0M   0% /run/lock
/dev/sda16                                881M  189M  631M  23% /boot
/dev/sda15                                105M  6.1M   99M   6% /boot/efi
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/5bfc5b4af703b35c3f90d5b25c3e9ff21f15e5d5ce621bdda4fdae95a9d70483/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/5c04b379d9defb1aab7a0fb23c41e05e59bd9214b0a7218a8ad7d054f0c19ebd/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/b092df4556b485d8ca33d9b7cdfb85d714e541ee412f5a068c8589b6993b57c3/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/820ba9d448424da8475ecc13d75ca9135651a60079858509bf6c47c738dcfc3e/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/4819e4856897fdb4a402a7540225f96c1eccde47430fae06e55cb3e62ae20f73/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/756986cae0363d1a99bad1c0414545476c0296638ee1a4ddf365e7036cbeabb5/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/896278f255d26cbc5aa6af49ca082044403bdbfc37ec9fd9d7a4931d16d4c648/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/ab6ab1ed3ca7105c02ccad86181837e4640c23f2330e1c0964d5212be06c4cc6/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/cb49570926eae6bde1c649854d4fde20e06f50129c532b2803ecbb20e2ab3a19/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/48e94a4d331dbcdee2fcce29c7432d1b9dad060d360cdce4cedb77cf37207abb/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/bf032ec6452b2eb4c8ff22ef82cc0b2373624f37cd7c91ef460f6b753fef7125/merged
u444094@u444094.your-storagebox.de:/home  1.0T   94G  931G  10% /mnt/cloudronbackup
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/5120d060ea583b21992fa8646e2ecddfd0b423f882a6d4a58468fafa5130ad68/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/38616af4f1bd995db643be3320bd1133cc44f202ce24ccde8155ca6aaaf3077b/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/e9e74e342a0ae84cc75bb66e006cc7e2028fd0a38199946992f9d26d6d84fc2c/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/97aeb95ba1bf882ad3385b31586ac0327fcd9d8a6af17c40f640fa8e3fc4bc4e/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/e769955350916a2813065b8689d2ec376b58309a969657981d08ec326c0c708a/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/96497cfe070e769a7483152e73068d4057613235ce4a971960a087ef5dbea3ca/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/4eba9b4664699e233d359b1027e325c06036f7aff03abca52e2718d5c0c74ba8/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/bc7ab1f83bcaffe1cf253d7ea660da4a2593f413506eb3159f249be9f0fa483d/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/45ccc493c4611b203d94bd92bec5275bb31a759941ce6899e0e30cd272a59120/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/136df1941cf7b4a5a2ebcccfc120fd3947e0e9faab209c302d242b73ed03bbcf/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/9481a8414742f8ea2cd6deb7a8cc681b180bbcb5eeefd7024842a456775eed97/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/f0c9041019437d4b8dfda8e8a452aa8dccdbfe232d3d1afb2eb0ec7296c251c3/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/2df56fc95b14d78174ae4d08cac74286d7980bd869ef493ac2d3ec0689727491/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/61bd9942b8907e3bc35ee036527388c0a475dc76248b7acdd3a26791923dafb8/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/9d9df6b80159995bd6180a03d851106a2c20e674e5c26e8a4a621f0fa2116ba4/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/c4fe70a7b14226b2ebe80208a57f4aaff7b936ee7d5359d51a37ad24a078978d/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/a7e91cb3158b191877758584b4347e8a971bd6b2f13d10664ac4b0c2f90ce8d3/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/20d2000198ed9839c17a6b3f19fcb3c068be7db0c7700837066f6c19901dde45/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/f3b17bcf2af309293fbf72dec721eb07588a03aab3db8177e30fc2e5fa54d755/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/8b896d9c33a3e2f1580fa1f2fb6d563c87b3e21c63cb7cf906e94cd67c08bdc5/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/13b1fb36de69d16d74f2b0586a2f5994162ad474e6ddbd92e825b770e000daac/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/438e6e14125d10ba6005509fde562e24461f905b7e7251eaee566f7708699cf8/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/6ada7feb1b0b54476bc7a00cb964a5105804d7f028a18d4714747cca4550be72/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/cca34a852250f97c7641c8c24257acab9fcc89ffeb3308e77fa99adbfefcd0f3/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/0cbdac8ad03fea9590aab73629f92c0e2d9a0d1dce619cad15c9ea72623d7509/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/b88dc85e813a93dcc5eeeb2942a6e908957ffdeacd81418c9bebecd13f9647d7/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/172c8fb0cf8ff6b6bae47934f1b17daf8f3fe4e954481f30f3f0ba592e462c85/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/519627495dfe6c1bc337eb7c192a548baafd3a17cbcc4a20e6d203d08e2fbb9f/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/8684fbff0facb34457506138ecb1209bb88a0d860b22fb9622529b60834723c9/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/4d8f54b378fecab054cdf1c5c611d17ce697b4367f6288a4b86008b924f5292e/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/0dc61a4ef7b019d2369ad3923012754bd4ae69b02e8598b9e07887fe397f135b/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/4c3bf9f4fce530434596076af09855b1572b644306682c976ae71c27f2d449fc/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/bf6598504ceb3848be1179f87f96feeea07b125f4a8fde950489cccec97e8442/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/cd098053e7a4822f019008353f92c27ee053f4a22359924f833842f61824acdd/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/eee59edad8a8c1085bea590f0efd4b664383588e55ccfb69abdfd9e9f0bd8276/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/21b8f3982c9191cf54aa2c4aadaf3e666101f47df4f28b1db269569232269ee2/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/0301c9b4a450aed2b26a6271b5a1c282c58616bdb38e268016d2c247b4414285/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/2118881573bbbc5b0eddcd6ccaf956fa5d9ca277e99386f2636f47e13629964e/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/c77f723445a5674d38d77b38d02bc101276da53eaaa7d8463ebce4c7970055fe/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/1132014c9618ab73577e4158ef8a3bbb1382371eab284fba202cc756319c7471/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/73e657750500f6df86fa5e6272f7f8d2564f6ea0bb747566f1193490e11bf841/merged
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/fa5e18ba7d507202835778f78463fad041332e1210f6e23767b1fba13fa4cc35/merged
tmpfs                                     6.3G   12K  6.3G   1% /run/user/0
overlay                                   542G  154G  389G  29% /var/lib/docker/overlay2/b6b60f320fcd2d38182cc82df5aff9bcb847c6e5c94c7dbf38e692339f3772f9/merged
root@my:~#

Same for lsblk:

root@my:~# lsblk
NAME    MAJ:MIN RM  SIZE RO TYPE MOUNTPOINTS
sda       8:0    0  560G  0 disk
├─sda1    8:1    0  559G  0 part /
├─sda14   8:14   0    4M  0 part
├─sda15   8:15   0  106M  0 part /boot/efi
└─sda16 259:0    0  913M  0 part /boot
sr0      11:0    1    4M  0 rom

I am running the following system:

Platform version is

v8.3.1 (Ubuntu 24.04.2 LTS)

cloudron-support --troubleshoot
Vendor: QEMU Product: Standard PC (i440FX + PIIX, 1996)
Linux: 6.8.0-56-generic
Ubuntu: noble 24.04
Processor: AMD EPYC 7B13 64-Core Processor
BIOS pc-i440fx-9.0  CPU @ 2.0GHz x 10
RAM: 65846508KB
Disk: /dev/sda1       389G
[OK]	node version is correct
[OK]	IPv6 is enabled and public IPv6 address is working
[OK]	docker is running
[OK]	docker version is correct
[OK]	MySQL is running
[OK]	nginx is running
[OK]	dashboard cert is valid
[OK]	dashboard is reachable via loopback
[OK]	box v8.3.1 is running
[OK]	netplan is good
[OK]	DNS is resolving via systemd-resolved
[OK]	Dashboard is reachable via domain name
[WARN]	Domain example.com expiry check skipped because whois does not have this information
[OK]	unbound is running

Is there anyone else with the same situation or has suggestions on which direction I might get into the problem itself?

Best
Jay

[BrutalBirdie]

@jayonrails said in Backup failing, Error when checking for disk space: Error running df: df exited with code null signal SIGTERM:

it contains many personal data

Then remove your hetzner storage box uid from the df -h output

---

But for real support the log is needed.
Domains and stuff can be search and replaced to anonymise the logfile.

[nebulon]

As @BrutalBirdie mentioned we would need any info from the logs (box.log) or maybe system logs around the time the process exited. Also is this reproducible easily for you?

[joseph]

I would also check dmesg output for any oom errors

[p44]

I had this issue on an instance just last night. I can't reproduce the issue. I rebooted the instance this morning. It's possible that the issue was related to Hetzner.

v8.3.1 (Ubuntu 22.04.5 LTS)
Product vServer
Storage provider Mountpoint (Via Hetzner Storage CIFS)

[jpavlovski]

Hi,

I have the exact same issue than described.
Seems that happened after the last security update.

This is easily reproductible as it happens each time I manually initiate a backup.

{
  "taskId": "229",
  "errorMessage": "Error when checking for disk space: Error running df: df exited with code 1 signal null",
  "timedOut": false,
  "backupId": null
}

Here are box.log logs when issue happens :

2025-04-04T19:10:38.879Z box:shell tasks /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 230 /home/yellowtent/platformdata/logs/tasks/230.log 15 1024 -999
2025-04-04T19:10:38.921Z box:shell Running as unit: box-task-230.service

2025-04-04T19:10:40.216Z box:apphealthmonitor app health: 1 running / 0 stopped / 0 unresponsive
2025-04-04T19:10:40.356Z box:shell Finished with result: exit-code
Main processes terminated with: code=exited/status=50
Service runtime: 1.413s
CPU time consumed: 1.891s
Memory peak: 3.2M
Memory swap peak: 0B

2025-04-04T19:10:40.356Z box:shell Service box-task-230 failed to run

2025-04-04T19:10:40.365Z box:shell Service box-task-230 finished with exit code 1

2025-04-04T19:10:40.367Z box:shell tasks: /usr/bin/sudo -S -E /home/yellowtent/box/src/scripts/starttask.sh 230 /home/yellowtent/platformdata/logs/tasks/230.log 15 1024 -999 errored BoxError: tasks exited with code 1 signal null
    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19)
    at ChildProcess.emit (node:events:519:28)
    at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
  reason: 'Shell Error',
  details: {},
  code: 1,
  signal: null
}

BR,

[nebulon]

Do you see anything in the dmesg / journalctl --system output related to this?

[jpavlovski]

Hi @nebulon,

Thank you for your last feedback.
I've finally found the issue.

While investigating further, I noticed that the rights on the backup folder were changed. (Probably during cleanup operations performed the week before the issue)
To resolve the issue, I changed the "prefix" in the backup settings so that a new backup folder has been created on the mountpoint with the good rights by the system.

It is now working as expected.

@jayonrails Maybe you can check the rights on the backup folder as well ?

Thank you again for your help.
Best Regards,

[p44]

@jpavlovski I confirm I had again three days ago