Full Cloudron Backups are failing on Cloudron version 7.7.0

[skeats]

Hello,

As of yesterday, when Cloudron updated to 7.7.0 now, my cifs backup of my Cloudron instance to my File Server is failing now with the following:

Error when checking for disk space: Error running df: file errored with code 1 message Command failed: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/cloudronbackup df: /mnt/cloudronbackup: Resource temporarily unavailable.

Here is the Log from this error:

Task 1278
[no timestamp]
[no timestamp] at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:110:19)
[no timestamp] at ChildProcess._handle.onexit (node:internal/child_process:291:12) {
[no timestamp] at ChildProcess.emit (node:events:513:28)
[no timestamp] at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:382:5)
[no timestamp] at async fullBackup (/home/yellowtent/box/src/backuptask.js:503:29)
[no timestamp] at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:360:5)
[no timestamp] at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[no timestamp] at runBackupUpload (/home/yellowtent/box/src/backuptask.js:165:15)
[no timestamp] code: 50,
[no timestamp] details: {},
[no timestamp] reason: 'Shell Error',
[no timestamp] signal: null
Apr 10 23:00:00 box:taskworker Starting task 1278. Logs are at /home/yellowtent/platformdata/logs/tasks/1278.log
Apr 10 23:00:00 box:tasks update 1278: {"percent":1,"message":"Backing up tourstest.experiencedmg.net (1/10)"}
Apr 10 23:00:00 box:tasks update 1278: {"percent":8.692307692307693,"message":"Snapshotting app tourstest.experiencedmg.net"}
Apr 10 23:00:00 box:services Backing up mysql
Apr 10 23:00:00 box:services backupAddons
Apr 10 23:00:00 box:services backupAddons: backing up ["mysql","localstorage","sendmail","redis","ldap"]
Apr 10 23:00:00 box:services pipeRequestToFile: connected with status code 200
Apr 10 23:00:00 box:services Backing up redis
Apr 10 23:00:00 box:backuptask snapshotApp: tourstest.experiencedmg.net took 0.075 seconds
Apr 10 23:00:00 box:services pipeRequestToFile: connected with status code 200
Apr 10 23:00:00 box:shell backup-snapshot/app_219ebf6d-360c-4dde-8348-72029e30949a /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_219ebf6d-360c-4dde-8348-72029e30949a tgz {"localRoot":"/home/yellowtent/appsdata/219ebf6d-360c-4dde-8348-72029e30949a","layout":[]}
Apr 10 23:00:00 box:tasks update 1278: {"percent":8.692307692307693,"message":"Uploading app snapshot tourstest.experiencedmg.net"}
Apr 10 23:00:00 box:backuptask runBackupUpload: result - {"result":"Error when checking for disk space: Error running df: file errored with code 1 message Command failed: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/cloudronbackup\ndf: /mnt/cloudronbackup: Resource temporarily unavailable\n"}
Apr 10 23:00:00 box:shell backup-snapshot/app_219ebf6d-360c-4dde-8348-72029e30949a: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_219ebf6d-360c-4dde-8348-72029e30949a tgz {"localRoot":"/home/yellowtent/appsdata/219ebf6d-360c-4dde-8348-72029e30949a","layout":[]} errored BoxError: backup-snapshot/app_219ebf6d-360c-4dde-8348-72029e30949a exited with code 50 signal null
Apr 10 23:00:00 box:tasks setCompleted - 1278: {"result":null,"error":{"stack":"BoxError: Error when checking for disk space: Error running df: file errored with code 1 message Command failed: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/cloudronbackup\ndf: /mnt/cloudronbackup: Resource temporarily unavailable\n\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:165:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:360:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:382:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:503:29)","name":"BoxError","reason":"External Error","details":{},"message":"Error when checking for disk space: Error running df: file errored with code 1 message Command failed: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/cloudronbackup\ndf: /mnt/cloudronbackup: Resource temporarily unavailable\n"}}
Apr 10 23:00:00 box:tasks update 1278: {"percent":100,"result":null,"error":{"stack":"BoxError: Error when checking for disk space: Error running df: file errored with code 1 message Command failed: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/cloudronbackup\ndf: /mnt/cloudronbackup: Resource temporarily unavailable\n\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:165:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:360:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:382:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:503:29)","name":"BoxError","reason":"External Error","details":{},"message":"Error when checking for disk space: Error running df: file errored with code 1 message Command failed: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/cloudronbackup\ndf: /mnt/cloudronbackup: Resource temporarily unavailable\n"}}
Apr 10 23:00:00 box:taskworker Task took 0.518 seconds
[no timestamp] Error when checking for disk space: Error running df: file errored with code 1 message Command failed: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/cloudronbackup
[no timestamp] /mnt/cloudronbackup: Resource temporarily unavailable

Any way we can get this fixed would be great!

[ChristopherMag]

I just started having the same issue today:

[no timestamp] 
[no timestamp]  at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:110:19)
[no timestamp]  at ChildProcess._handle.onexit (node:internal/child_process:291:12) {
[no timestamp]  at ChildProcess.emit (node:events:513:28)
[no timestamp]  at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:382:5)
[no timestamp]  at async fullBackup (/home/yellowtent/box/src/backuptask.js:503:29)
[no timestamp]  at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:360:5)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[no timestamp]  at runBackupUpload (/home/yellowtent/box/src/backuptask.js:165:15)
[no timestamp]  code: 50,
[no timestamp]  details: {},
[no timestamp]  reason: 'Shell Error',
[no timestamp]  signal: null
Apr 11 11:18:08 box:taskworker Starting task 12097. Logs are at /home/yellowtent/platformdata/logs/tasks/12097.log
Apr 11 11:18:08 box:tasks update 12097: {"percent":1,"message":"Backing up magnusonfinancialwebservices.cloudron.magnusonfinancial.com (1/19)"}
Apr 11 11:18:08 box:tasks update 12097: {"percent":5.545454545454546,"message":"Snapshotting app magnusonfinancialwebservices.cloudron.magnusonfinancial.com"}
Apr 11 11:18:08 box:services Backing up mongodb
Apr 11 11:18:08 box:services backupAddons
Apr 11 11:18:08 box:services backupAddons: backing up ["localstorage","mongodb","postgresql"]
Apr 11 11:18:09 box:services pipeRequestToFile: connected with status code 200
Apr 11 11:18:09 box:services Backing up postgresql
Apr 11 11:18:09 box:services pipeRequestToFile: connected with status code 200
Apr 11 11:18:14 box:backuptask snapshotApp: magnusonfinancialwebservices.cloudron.magnusonfinancial.com took 5.731 seconds
Apr 11 11:18:14 box:tasks update 12097: {"percent":5.545454545454546,"message":"Uploading app snapshot magnusonfinancialwebservices.cloudron.magnusonfinancial.com"}
Apr 11 11:18:14 box:backuptask runBackupUpload: adjusting heap size to 2048M
Apr 11 11:18:14 box:shell backup-snapshot/app_14252669-7b77-404a-b08a-500e405138aa /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_14252669-7b77-404a-b08a-500e405138aa tgz {"localRoot":"/home/yellowtent/appsdata/14252669-7b77-404a-b08a-500e405138aa","layout":[]}
Apr 11 11:18:15 box:backuptask runBackupUpload: result - {"result":"Error when checking for disk space: Error running df: file errored with code 1 message Command failed: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/cloudronbackup\ndf: /mnt/cloudronbackup: Resource temporarily unavailable\n"}
Apr 11 11:18:15 box:shell backup-snapshot/app_14252669-7b77-404a-b08a-500e405138aa: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_14252669-7b77-404a-b08a-500e405138aa tgz {"localRoot":"/home/yellowtent/appsdata/14252669-7b77-404a-b08a-500e405138aa","layout":[]} errored BoxError: backup-snapshot/app_14252669-7b77-404a-b08a-500e405138aa exited with code 50 signal null
Apr 11 11:18:15 box:taskworker Task took 6.184 seconds
Apr 11 11:18:15 box:tasks setCompleted - 12097: {"result":null,"error":{"stack":"BoxError: Error when checking for disk space: Error running df: file errored with code 1 message Command failed: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/cloudronbackup\ndf: /mnt/cloudronbackup: Resource temporarily unavailable\n\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:165:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:360:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:382:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:503:29)","name":"BoxError","reason":"External Error","details":{},"message":"Error when checking for disk space: Error running df: file errored with code 1 message Command failed: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/cloudronbackup\ndf: /mnt/cloudronbackup: Resource temporarily unavailable\n"}}
Apr 11 11:18:15 box:tasks update 12097: {"percent":100,"result":null,"error":{"stack":"BoxError: Error when checking for disk space: Error running df: file errored with code 1 message Command failed: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/cloudronbackup\ndf: /mnt/cloudronbackup: Resource temporarily unavailable\n\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:165:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:360:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:382:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:503:29)","name":"BoxError","reason":"External Error","details":{},"message":"Error when checking for disk space: Error running df: file errored with code 1 message Command failed: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/cloudronbackup\ndf: /mnt/cloudronbackup: Resource temporarily unavailable\n"}}
[no timestamp]  Error when checking for disk space: Error running df: file errored with code 1 message Command failed: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/cloudronbackup
[no timestamp]  /mnt/cloudronbackup: Resource temporarily unavailable
[no timestamp]  }

The same Error when checking for disk space: Error running df: file errored with code 1 message Command failed: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/cloudronbackup df: /mnt/cloudronbackup: Resource temporarily unavailable is listed in the web interface.

We are running cloudron 7.7.2 and Ubuntu 22.04.

I have rebooted, ssh'ed in and confirmed that I can access the backup mountpoint, list files, go into different backup folders, etc.

[nebulon]

Can you manually run df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/cloudronbackup then via SSH?

[ChristopherMag]

@nebulon I have run df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/cloudronbackup via SSH and get the same error:

cmagnuson@cloudron2:/mnt/cloudronbackup/2024-04-05-000000-649$ df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/cloudronbackup
df: /mnt/cloudronbackup: Resource temporarily unavailable

[nebulon]

hm so the mountpoint is not in full health then if such basic commands don't work. Do you see any kernel errors in the system logs itself?

[ChristopherMag]

I am still looking but after running sudo dmesg I did see the following:

[   17.363373] FS-Cache: Loaded
[   17.385124] FS-Cache: Netfs 'cifs' registered for caching
[   17.387237] Key type cifs.spnego registered
[   17.387248] Key type cifs.idmap registered
[   17.387569] CIFS: No dialect specified on mount. Default has changed to a more secure dialect, SMB2.1 or later (e.g. SMB3.1.1), from CIFS (SMB1). To use the less secure SMB1 dialect to access old servers which do not support SMB3.1.1 (or even SMB3 or SMB2.1) specify vers=1.0 on mount.
[   17.387571] CIFS: Attempting to mount \\actweb.magnusonfinancial.com\cloudronbackup
[   17.387634] CIFS: Attempting to mount \\actweb.magnusonfinancial.com\paperless-ngx
[   17.387651] CIFS: Attempting to mount \\actweb.magnusonfinancial.com\Files
[   17.388884] FS-Cache: Duplicate cookie detected
[   17.388999] FS-Cache: O-cookie c=00000003 [p=00000002 fl=222 nc=0 na=1]
[   17.389090] FS-Cache: O-cookie d=0000000061788d75{CIFS.server} n=0000000086d54d7c
[   17.389210] FS-Cache: O-key=[8] '020001bdc0a82034'
[   17.389312] FS-Cache: N-cookie c=00000004 [p=00000002 fl=2 nc=0 na=1]
[   17.389406] FS-Cache: N-cookie d=0000000061788d75{CIFS.server} n=00000000d0a48889
[   17.389515] FS-Cache: N-key=[8] '020001bdc0a82034'
[   17.389629] FS-Cache: Duplicate cookie detected
[   17.389720] FS-Cache: O-cookie c=00000003 [p=00000002 fl=222 nc=0 na=1]
[   17.389817] FS-Cache: O-cookie d=0000000061788d75{CIFS.server} n=0000000086d54d7c
[   17.389906] FS-Cache: O-key=[8] '020001bdc0a82034'
[   17.390004] FS-Cache: N-cookie c=00000005 [p=00000002 fl=2 nc=0 na=1]
[   17.390099] FS-Cache: N-cookie d=0000000061788d75{CIFS.server} n=0000000058b3c043
[   17.390190] FS-Cache: N-key=[8] '020001bdc0a82034'

[ChristopherMag]

Possibly related thread. I am currently on 5.15.0-102-generic kernel which is the same one others were pointing at having the same symptom of being able to mount network shares, see their contents, but get resource unavailable as an error via df.

[nebulon]

Can you then maybe check if /etc/systemd/system/mnt-cloudronbackup.mount misses some mount flags for your storage type? If so we can add this in our .mount template I guess

[skeats]

@nebulon I am getting all of the same results as @ChristopherMag and I have the following showing on my mnt-cloudronbackup.mount:

[Unit]
Description=cloudronbackup

Requires=unbound.service
After=unbound.service
Before=docker.service

[Mount]
What=//192.168.100.130/Backups/Backup/Cloudron Backup
Where=/mnt/cloudronbackup
Options=credentials=/home/yellowtent/platformdata/cifs/mnt-cloudronbackup.cred,rw,iocharset=utf8,file_mode=0666,dir_mode=0777,uid=yellowtent,gid=yellowtent
Type=cifs

[Install]
WantedBy=multi-user.target

That is the correct setup with my local Truenas File Server. I went onto my Truenas File Server and made sure SMB1 (CIFS) was enabled on the SAMBA service to rule that out. I did a reboot of my Cloudron server and tried again with the same issue. Weird thing, is that if I go to Backups and click the Remount Storage button, I still remounts and the green indicator is there.

[ChristopherMag]

This seems to be related to a kernel bug in 5.15.0-102.

I have run sudo apt-get update and then run sudo apt upgrade but didn't say yes just to see what packages would be upgraded and I didn't see anything that would be upgrading the kernel:

cmagnuson@cloudron2:~$ sudo apt upgrade
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
Get more security updates through Ubuntu Pro with 'esm-apps' enabled:
  libjs-jquery-ui python3-scipy
Learn more about Ubuntu Pro at https://ubuntu.com/pro
The following NEW packages will be installed:
  ubuntu-pro-client
The following packages will be upgraded:
  apt apt-utils collectd collectd-core collectd-utils coreutils ethtool firmware-sof-signed libapt-pkg6.0 libcollectdclient1 libldap-2.5-0 libldap-common libwbclient0 python3-update-manager snapd ubuntu-advantage-tools ubuntu-pro-client-l10n update-manager-core update-notifier-common
19 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 32.5 MB of archives.
After this operation, 1,268 kB disk space will be freed.
Do you want to continue? [Y/n] ^C

At this point it seems like a work around is needed to be able to bypass the df check or do something else to be able to get backups while waiting for ubuntu to release an update to the kernel they include.

I am going to try to reconfigured backups to go to local storage and them manually copy them over to the mountpoint so that at least I can get a backup for today.

@skeats Can you run uname -r to confirm what kernel version you are on?

[skeats]

@ChristopherMag I am on kernel version 5.15.0-102

[ChristopherMag]

I have made a /backups directory and run sudo chown yellowtent:yellowtent /backups and then configured the system to backup to that directory and am currently running a backup.

After that completes I will work on copying that over to /mnt/cloudronbackup

@skeats ps, if you want to paste logs I recommend putting ``` on a new line before the log output and ``` on the line after the log output and then the system will format it with a fixed width font and make it easier to read. Same thing when including commands inline by adding ` at the start of the command and ` at the end of the command so that ps -a turns into ps -a. You can try it out by editing your previous posts if you want to try it.

[necrevistonnezr]

If you’re on 22.04, you can safely update to the 6.x kernel: https://www.omgubuntu.co.uk/2023/08/ubuntu-22-04-linux-kernel-6-2

[ChristopherMag]

@nebulon I have added the previously used cloudron backup directory as a volume in cloudron and mounted it.

I am about to run cp -r /backups/2024-04-11-161325-365 /mnt/volumes/43cfcd99b751486ea8b2f56a194eb88b

Given that the snapshot directories are different in the local /backups folder vs the remote /mnt/volumes/43cfcd99b751486ea8b2f56a194eb88b is that going to cause an issue now or later when I switch back to using the original cifs share as the backup destination?

[ChristopherMag]

@necrevistonnezr The article made it sound like running sudo apt update && sudo apt full-upgrade would get the update and this was back in 2023 so wouldn't we already have it given that cloudron runs apt to do updates periodically?

I also didn't see anything that would update the kernel in the output of sudo apt upgrade listed in my post above. The article also indicated that this would be included in new ISO's but this server was just installed from a freshly downloaded iso about a month ago so it doesn't seem to be included in recent ISOs either.

[ChristopherMag]

@necrevistonnezr I think if I am understanding this correctly ubuntu has this notion of a HardWare Enablement (HWE) kernel that they maintain for older releases but it isn't the default. So in my case I believe I would need to run sudo apt install linux-generic-hwe-22.04 to get the HWE kernel which would be a 6.x kernel and should resolve this issue.

[ChristopherMag]

@skeats I have run sudo apt install linux-generic-hwe-22.04, rebooted, confirmed via uname -r that I am running 6.5.0-27-generic, reconfigured my backup settings (you probably wont have to do this as I was changing things to get a local backup), and now I can backup to the cifs share again.

@necrevistonnezr Thank you for pointing me in the right direction to be able to use the HWE kernel to bypass this bug.

[skeats]

@ChristopherMag thank you because I ran that command and it worked and my backups are working again with the 6.5.0-27-generic kernel!

[odie]

I had the same issue. For me, it worked to enable “use seal encryption” in the backup configuration. At least when I triggered a manual backup it worked. I’ll just have to wait to see if the scheduled backup runs as well.

[necrevistonnezr]

@ChristopherMag said in Full Cloudron Backups are failing on Cloudron version 7.7.0:

@necrevistonnezr The article made it sound like running sudo apt update && sudo apt full-upgrade would get the update and this was back in 2023 so wouldn't we already have it given that cloudron runs apt to do updates periodically?

apt full-upgrade is non-standard and not run by Cloudron, I believe.