Cloudron 9.0 (beta) bug reports

[WiseMetalhead]

@girish said in Cloudron 9.0 (beta) bug reports:

which service or set up specifically

It’s a local provider for me, and there’s no specific configuration for it in Cloudron.
For convenience, I can create a bucket and send you the access details by email so that you can try to reproduce the issue.
However, if the problem lies in how this provider implements S3 compatibility, I don’t think it’s worth spending too much time on it.
Initially, I thought the issue was related to the generic S3 configuration itself.

[girish]

@WiseMetalhead yeah, our CI tests object storages of DO, AWS, Vultr, Hetzner, minio .

For convenience, I can create a bucket and send you the access details by email so that you can try to reproduce the issue.

Yes please, if that's possible, would be great to understand where the problem is. I suspect this issue will get eventually hit by someone.

[WiseMetalhead]

@girish said in Cloudron 9.0 (beta) bug reports:

Yes please, if that's possible

I’ve sent the details from my email address vlishchenko****@gmail.com

[girish]

@jdaviescoates said in Cloudron 9.0 (beta) bug reports:

I wanted to turn of Automatic Platform updates, but leave App updates on:

Not yet, have to look into this in coming releases.

[robi]

@girish Can you simply add another column to the backups page that keeps track of how long each backup ran? Would be very useful!

[girish]

@Kubernetes said in Cloudron 9.0 (beta) bug reports:

After upgrading to Cloudron 9 I cannot send e-mails with Masquerading-Feature

Thanks for reporting! Indeed that flag is inverted! I have fixed the UI text.

Workaround: If you disable it there, I think you will see that masquerading works.

[girish]

@robi this info is there in Backup -> Info dialog (in the global backups). But just noticed it's missing in the App -> Backup -> Info dialog. Will add it there as well.

[robi]

@girish Oh, I would have never thought to look there. I think there are too many things hidden in the 3dots menus. I'd like to see the duration in the system backups list, we don't need the long form Date column there. Can we tighten that up?

[robi]

@girish what is the first column described by the file cabinet/box icon there? How is it sorting the list when clicked on? Should it even be there, or somewhere above the list?

[necrevistonnezr]

I get cp exited with code 1 errors on a backup via rsync to a locally attached USB-HDD:

Nov 11 00:16:14 box:backuptask upload: path snapshot/mail site 59742348-c3f1-490a-aa43-3dc1f1701f47 uploaded: {"transferred":24537785098,"size":24537785098,"fileCount":96084}
Nov 11 00:16:14 box:tasks updating task 21646 with: {"percent":91.47619047619045,"message":"Uploading integrity information to snapshot/mail.backupinfo (mail)"}
Nov 11 00:16:15 box:backupupload upload completed. error: null
Nov 11 00:16:15 box:backuptask runBackupUpload: result - {"result":{"stats":{"transferred":24537785098,"size":24537785098,"fileCount":96084},"integrity":{"signature":"72d32a4ba3105b1a7290323f430f85463992ce714e059553c3995f86bd8584364cf44b481149fe4731e4c2cde6174a7b2c07e19e2c938589594c04bae387550c"}}}
Nov 11 00:16:15 box:backuptask uploadMailSnapshot: took 671.27 seconds
Nov 11 00:16:15 box:backuptask backupMailWithTag: rotating mail snapshot of 59742348-c3f1-490a-aa43-3dc1f1701f47 to 2025-11-10-230001-644/mail_v9.0.7
Nov 11 00:16:15 box:tasks updating task 21646 with: {"percent":91.47619047619045,"message":"Copying /media/WD4TB/CloudronBackup/snapshot/mail to /media/WD4TB/CloudronBackup/2025-11-10-230001-644/mail_v9.0.7"}
Nov 11 00:16:15 box:shell filesystem: cp -aRl /media/WD4TB/CloudronBackup/snapshot/mail /media/WD4TB/CloudronBackup/2025-11-10-230001-644/mail_v9.0.7
Nov 11 00:16:17 box:shell filesystem: cp -aRl /media/WD4TB/CloudronBackup/snapshot/mail /media/WD4TB/CloudronBackup/2025-11-10-230001-644/mail_v9.0.7 errored BoxError: cp exited with code 1 signal null
Nov 11 00:16:17 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:82:23)
Nov 11 00:16:17 at ChildProcess.emit (node:events:519:28)
Nov 11 00:16:17 at maybeClose (node:internal/child_process:1101:16)
Nov 11 00:16:17 at ChildProcess._handle.onexit (node:internal/child_process:304:5) {
Nov 11 00:16:17 reason: 'Shell Error',
Nov 11 00:16:17 details: {},
Nov 11 00:16:17 stdout: <Buffer >,
Nov 11 00:16:17 stdoutLineCount: 0,
Nov 11 00:16:17 stderr: <Buffer 63 70 3a 20 63 61 6e 6e 6f 74 20 63 72 65 61 74 65 20 68 61 72 64 20 6c 69 6e 6b 20 27 2f 6d 65 64 69 61 2f 57 44 34 54 42 2f 43 6c 6f 75 64 72 6f 6e ... 210 more bytes>,
Nov 11 00:16:17 stderrLineCount: 1,
Nov 11 00:16:17 code: 1,
Nov 11 00:16:17 signal: null,
Nov 11 00:16:17 timedOut: false,
Nov 11 00:16:17 terminated: false
Nov 11 00:16:17 }
Nov 11 00:16:17 box:backuptask copy: copy to 2025-11-10-230001-644/mail_v9.0.7 errored. error: cp exited with code 1 signal null
Nov 11 00:16:17 box:tasks setCompleted - 21646: {"result":null,"error":{"message":"cp exited with code 1 signal null","reason":"External Error"},"percent":100}
Nov 11 00:16:17 box:tasks updating task 21646 with: {"completed":true,"result":null,"error":{"message":"cp exited with code 1 signal null","reason":"External Error"},"percent":100}
Nov 11 00:16:17 box:taskworker Task took 975.553 seconds
Nov 11 00:16:17 BoxError: cp exited with code 1 signal null
Nov 11 00:16:17 at copyInternal (/home/yellowtent/box/src/storage/filesystem.js:217:26)
Nov 11 00:16:17 at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
Nov 11 00:16:17 at async Object.copyDir (/home/yellowtent/box/src/storage/filesystem.js:235:12)
Nov 11 00:16:17 at async Object.copy (/home/yellowtent/box/src/backupformat/rsync.js:311:5)

EDIT: The error persists even after increasing RAM from 4 to 16 GB.

[girish]

That hex buffer translates to "cp: cannot create hard link '/media/WD4TB/Cloudron ... " .

@necrevistonnezr can you try running cp -aRl /media/WD4TB/CloudronBackup/snapshot/mail /media/WD4TB/CloudronBackup/2025-11-10-230001-644/mail_v9.0.7 manually on the server? It's failing for some reason. cp has a habit of failing when files disappear (as is the case with the mail directory and dovecot creating temp files). If that manual cp worked, can you try to do make a new backup one or two more times?

[nebulon]

@robi said in Cloudron 9.0 (beta) bug reports:

@girish what is the first column described by the file cabinet/box icon there? How is it sorting the list when clicked on? Should it even be there, or somewhere above the list?

This is the indicator if a backup is archived or not. Archived backups are kept despite the retention policy of a backup site. We removed quite a bit of tooltips, to be a bit more mobile friendly, I guess it isn't obivous enought though.

[necrevistonnezr]

@girish Thanks! After running
sudo -u yellowtent cp -aRlv /media/WD4TB/CloudronBackup/snapshot/mail /media/WD4TB/CloudronBackup/2025-11-10-230001-644/mail_v9.0.7
without error, backups went through without error, as well.
Is there something that can be done to prevent such an error?

[WiseMetalhead]

@girish said in Cloudron 9.0 (beta) bug reports:

which service or set up specifically?

It seems the issue isn’t with my local provider after all. As an experiment, I set up Garage on a server in my local network and configured it as an s3-v4–compatible provider for backups.

I adjusted the Memory Limit and Upload Part Size settings, then started a backup task. After that, I once again noticed in the logs that each part was 10 MB instead of 512 MB, as specified in the configuration. Additionally, I wasn’t able to set a custom S3 Region.
After saving the configuration, the region kept reverting to us-east-1, so I had to configure Garage to use that region identifier instead.

2025-11-11T14:57:13.843Z box:taskworker Starting task 2171. Logs are at /home/yellowtent/platformdata/logs/tasks/2171.log
2025-11-11T14:57:13.865Z box:taskworker Running task of type backup
2025-11-11T14:57:13.893Z box:tasks updating task 2171 with: {"percent":5.761904761904762,"message":"Backing up ***.***.ru (1/18). Waiting for lock"}
2025-11-11T14:57:13.899Z box:locks write: current locks: {"full_backup_task_a1cb8769-ccfb-4107-b574-ab4c7afef2b6":null,"app_backup_07fd2189-9378-4a35-b18d-5cef77461fb1":"2171"}
2025-11-11T14:57:13.899Z box:locks acquire: app_backup_07fd2189-9378-4a35-b18d-5cef77461fb1
2025-11-11T14:57:13.900Z box:tasks updating task 2171 with: {"percent":5.761904761904762,"message":"Snapshotting app ***.***.ru"}
2025-11-11T14:57:13.901Z box:services backupAddons
2025-11-11T14:57:13.901Z box:services backupAddons: backing up ["localstorage","postgresql","sendmail","oidc","redis"]
2025-11-11T14:57:13.902Z box:services Backing up postgresql
2025-11-11T14:57:14.139Z box:services pipeRequestToFile: connected with status code 200
2025-11-11T14:57:17.617Z box:services Backing up redis
2025-11-11T14:57:17.672Z box:services pipeRequestToFile: connected with status code 200
2025-11-11T14:57:17.682Z box:backuptask snapshotApp: ***.***.ru took 3.782 seconds
2025-11-11T14:57:17.697Z box:tasks updating task 2171 with: {"percent":5.761904761904762,"message":"Uploading app snapshot ***.***.ru"}
2025-11-11T14:57:17.697Z box:backuptask runBackupUpload: adjusting heap size to 2816M
2025-11-11T14:57:17.697Z box:shell backuptask: /usr/bin/sudo --non-interactive -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_07fd2189-9378-4a35-b18d-5cef77461fb1.tar.gz a1cb8769-ccfb-4107-b574-ab4c7afef2b6 {"localRoot":"/home/yellowtent/appsdata/07fd2189-9378-4a35-b18d-5cef77461fb1","layout":[{"localDir":"/mnt/md0/IApps/photos","remoteDir":"data"}]}
2025-11-11T14:57:18.216Z box:backupupload Backing up {"localRoot":"/home/yellowtent/appsdata/07fd2189-9378-4a35-b18d-5cef77461fb1","layout":[{"localDir":"/mnt/md0/IApps/photos","remoteDir":"data"}]} to snapshot/app_07fd2189-9378-4a35-b18d-5cef77461fb1.tar.gz
2025-11-11T14:57:18.218Z box:backuptask upload: path snapshot/app_07fd2189-9378-4a35-b18d-5cef77461fb1.tar.gz site a1cb8769-ccfb-4107-b574-ab4c7afef2b6 dataLayout {"localRoot":"/home/yellowtent/appsdata/07fd2189-9378-4a35-b18d-5cef77461fb1","layout":[{"localDir":"/mnt/md0/IApps/photos","remoteDir":"data"}]}
2025-11-11T14:57:18.357Z box:backuptask checkPreconditions: mount point status is {"state":"active"}
2025-11-11T14:57:18.357Z box:backuptask checkPreconditions: getting disk usage of /home/yellowtent/appsdata/07fd2189-9378-4a35-b18d-5cef77461fb1
2025-11-11T14:57:18.357Z box:shell backuptask: du --dereference-args --summarize --block-size=1 --exclude=*.lock --exclude=dovecot.list.index.log.* /home/yellowtent/appsdata/07fd2189-9378-4a35-b18d-5cef77461fb1
2025-11-11T14:57:18.365Z box:backuptask checkPreconditions: getting disk usage of /mnt/md0/IApps/photos
2025-11-11T14:57:18.365Z box:shell backuptask: du --dereference-args --summarize --block-size=1 --exclude=*.lock --exclude=dovecot.list.index.log.* /mnt/md0/IApps/photos
2025-11-11T14:57:19.311Z box:backuptask checkPreconditions: total required=125243674624 available=Infinity
2025-11-11T14:57:19.317Z box:backupformat/tgz upload: uploading to site a1cb8769-ccfb-4107-b574-ab4c7afef2b6 path snapshot/app_07fd2189-9378-4a35-b18d-5cef77461fb1.tar.gz (encrypted: false) dataLayout {"localRoot":"/home/yellowtent/appsdata/07fd2189-9378-4a35-b18d-5cef77461fb1","layout":[{"localDir":"/mnt/md0/IApps/photos","remoteDir":"data"}]}
2025-11-11T14:57:19.318Z box:tasks updating task 2171 with: {"percent":5.761904761904762,"message":"Uploading backup snapshot/app_07fd2189-9378-4a35-b18d-5cef77461fb1.tar.gz (***.***.ru)"}
2025-11-11T14:57:19.323Z box:backupformat/tgz tarPack: processing /home/yellowtent/appsdata/07fd2189-9378-4a35-b18d-5cef77461fb1
2025-11-11T14:57:19.330Z box:backupformat/tgz addToPack: added ./config.json file
2025-11-11T14:57:19.385Z box:backupformat/tgz addToPack: added ./dump.rdb file
2025-11-11T14:57:19.396Z box:backupformat/tgz addToPack: added ./fsmetadata.json file
2025-11-11T14:57:20.874Z box:storage/s3 Upload progress: {"loaded":10485760,"part":1,"Key":"snapshot/app_07fd2189-9378-4a35-b18d-5cef77461fb1.tar.gz","Bucket":"lvcl-backup"}
2025-11-11T14:57:22.826Z box:storage/s3 Upload progress: {"loaded":20971520,"part":2,"Key":"snapshot/app_07fd2189-9378-4a35-b18d-5cef77461fb1.tar.gz","Bucket":"lvcl-backup"}
2025-11-11T14:57:24.909Z box:storage/s3 Upload progress: {"loaded":31457280,"part":3,"Key":"snapshot/app_07fd2189-9378-4a35-b18d-5cef77461fb1.tar.gz","Bucket":"lvcl-backup"}
2025-11-11T14:57:26.524Z box:storage/s3 Upload progress: {"loaded":41943040,"part":4,"Key":"snapshot/app_07fd2189-9378-4a35-b18d-5cef77461fb1.tar.gz","Bucket":"lvcl-backup"}
2025-11-11T14:57:27.785Z box:storage/s3 Upload progress: {"loaded":52428800,"part":5,"Key":"snapshot/app_07fd2189-9378-4a35-b18d-5cef77461fb1.tar.gz","Bucket":"lvcl-backup"}
2025-11-11T14:57:29.329Z box:tasks updating task 2171 with: {"percent":5.761904761904762,"message":"Uploading backup 59M@6MBps (***.***.ru)"}
2025-11-11T14:57:29.652Z box:storage/s3 Upload progress: {"loaded":62914560,"part":6,"Key":"snapshot/app_07fd2189-9378-4a35-b18d-5cef77461fb1.tar.gz","Bucket":"lvcl-backup"}
2025-11-11T14:57:31.626Z box:storage/s3 Upload progress: {"loaded":73400320,"part":7,"Key":"snapshot/app_07fd2189-9378-4a35-b18d-5cef77461fb1.tar.gz","Bucket":"lvcl-backup"}
2025-11-11T14:57:33.614Z box:storage/s3 Upload progress: {"loaded":83886080,"part":8,"Key":"snapshot/app_07fd2189-9378-4a35-b18d-5cef77461fb1.tar.gz","Bucket":"lvcl-backup"}
2025-11-11T14:57:35.396Z box:storage/s3 Upload progress: {"loaded":94371840,"part":9,"Key":"snapshot/app_07fd2189-9378-4a35-b18d-5cef77461fb1.tar.gz","Bucket":"lvcl-backup"}

[girish]

@WiseMetalhead thanks, I didn't find the time today to debug, have to do it tomorrow.

[girish]

@necrevistonnezr cp is not robust enough against changing directories, so we have to rely on a retry (unless we go ahead and rewrite cp which is a biggish task). I could maybe wrap the cp logic in a retry loop for next release.