Unable to Backup Since Updating to 8.0.3

[Shaun Snapp]

Hello,

I have been unable to complete a backup since I upgraded to 8.0.3. I have daily backups set, and they have all failed since the 16th, which is when I upgraded to Cloudron 8.0.3

[joseph]

@Shaun-Snapp Can you check what error you hit? It should be in the backup logs. Backups -> Logs button is on the top left of the section in a dropdown.

[Shaun Snapp]

It seems to be self signed certificate error. All the backup logs since the 15th of August, so beginning with the 16th show this error. However, I don't see a self signed certificate in my setup, and I had no problems making backups until the 16th of August. I did not make any changes to my backup settings. So on the 15th there is no self signed certificate error, then on the 16th, after upgrading to 8.0.3 this self signed certificate error appears in the log of every backup that is attempted to be made.

Here is the log from a failed backup.

Task 12896
Aug 19 20:11:31 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/fonts.json position 3936 try 2
Aug 19 20:11:41 box:tasks update 12896: {"percent":12.11111111111111,"message":"Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/systemFonts.json (Try 2) (summitproducts.cloud)"}
Aug 19 20:11:41 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/systemFonts.json position 3938 try 2
Aug 19 20:11:41 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/transforms.js position 3939 try 2
Aug 19 20:19:54 box:backupformat/rsync Attempt 2 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:19:54 box:backupformat/rsync Attempt 2 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:20:04 box:backupformat/rsync Attempt 2 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:20:04 box:backupformat/rsync Attempt 2 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:20:14 box:tasks update 12896: {"percent":12.11111111111111,"message":"Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/init.php (Try 3) (summitproducts.cloud)"}
Aug 19 20:20:14 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/init.php position 3931 try 3
Aug 19 20:20:14 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/dist/blocks.style.build.css position 3930 try 3
Aug 19 20:20:14 box:backupformat/rsync Attempt 2 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:20:14 box:backupformat/rsync Attempt 2 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:20:24 box:tasks update 12896: {"percent":12.11111111111111,"message":"Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/blocks.js (Try 3) (summitproducts.cloud)"}
Aug 19 20:20:24 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/blocks.js position 3933 try 3
Aug 19 20:20:24 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/package-lock.json position 3932 try 3
Aug 19 20:20:24 box:backupformat/rsync Attempt 2 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:20:24 box:backupformat/rsync Attempt 2 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:20:34 box:tasks update 12896: {"percent":12.11111111111111,"message":"Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/common.scss (Try 3) (summitproducts.cloud)"}
Aug 19 20:20:34 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/common.scss position 3934 try 3
Aug 19 20:20:34 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/edit.js position 3935 try 3
Aug 19 20:20:34 box:backupformat/rsync Attempt 2 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:20:35 box:backupformat/rsync Attempt 2 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:20:44 box:tasks update 12896: {"percent":12.11111111111111,"message":"Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/index.js (Try 3) (summitproducts.cloud)"}
Aug 19 20:20:44 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/index.js position 3937 try 3
Aug 19 20:20:44 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/fonts.json position 3936 try 3
Aug 19 20:20:54 box:tasks update 12896: {"percent":12.11111111111111,"message":"Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/systemFonts.json (Try 3) (summitproducts.cloud)"}
Aug 19 20:20:54 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/systemFonts.json position 3938 try 3
Aug 19 20:20:55 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/transforms.js position 3939 try 3
Aug 19 20:29:07 box:backupformat/rsync Attempt 3 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:29:07 box:backupformat/rsync Attempt 3 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:29:17 box:backupformat/rsync Attempt 3 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:29:17 box:backupformat/rsync Attempt 3 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:29:27 box:tasks update 12896: {"percent":12.11111111111111,"message":"Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/init.php (Try 4) (summitproducts.cloud)"}
Aug 19 20:29:27 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/init.php position 3931 try 4
Aug 19 20:29:27 box:backupformat/rsync Attempt 3 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:29:27 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/dist/blocks.style.build.css position 3930 try 4
Aug 19 20:29:27 box:backupformat/rsync Attempt 3 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:29:37 box:tasks update 12896: {"percent":12.11111111111111,"message":"Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/blocks.js (Try 4) (summitproducts.cloud)"}
Aug 19 20:29:37 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/blocks.js position 3933 try 4
Aug 19 20:29:37 box:backupformat/rsync Attempt 3 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:29:37 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/package-lock.json position 3932 try 4
Aug 19 20:29:37 box:backupformat/rsync Attempt 3 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:29:47 box:tasks update 12896: {"percent":12.11111111111111,"message":"Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/edit.js (Try 4) (summitproducts.cloud)"}
Aug 19 20:29:47 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/edit.js position 3935 try 4
Aug 19 20:29:47 box:backupformat/rsync Attempt 3 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:29:47 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/common.scss position 3934 try 4
Aug 19 20:29:48 box:backupformat/rsync Attempt 3 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:29:57 box:tasks update 12896: {"percent":12.11111111111111,"message":"Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/fonts.json (Try 4) (summitproducts.cloud)"}
Aug 19 20:29:57 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/fonts.json position 3936 try 4
Aug 19 20:29:57 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/index.js position 3937 try 4
Aug 19 20:30:07 box:tasks update 12896: {"percent":12.11111111111111,"message":"Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/systemFonts.json (Try 4) (summitproducts.cloud)"}
Aug 19 20:30:07 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/systemFonts.json position 3938 try 4
Aug 19 20:30:08 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/transforms.js position 3939 try 4
Aug 19 20:38:20 box:backupformat/rsync Attempt 4 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:38:21 box:backupformat/rsync Attempt 4 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:38:30 box:backupformat/rsync Attempt 4 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:38:31 box:backupformat/rsync Attempt 4 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:38:40 box:tasks update 12896: {"percent":12.11111111111111,"message":"Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/init.php (Try 5) (summitproducts.cloud)"}
Aug 19 20:38:40 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/init.php position 3931 try 5
Aug 19 20:38:41 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/dist/blocks.style.build.css position 3930 try 5
Aug 19 20:38:41 box:backupformat/rsync Attempt 4 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:38:41 box:backupformat/rsync Attempt 4 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:38:50 box:tasks update 12896: {"percent":12.11111111111111,"message":"Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/package-lock.json (Try 5) (summitproducts.cloud)"}
Aug 19 20:38:50 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/package-lock.json position 3932 try 5
Aug 19 20:38:51 box:backupformat/rsync Attempt 4 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:38:51 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/blocks.js position 3933 try 5
Aug 19 20:38:51 box:backupformat/rsync Attempt 4 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:39:01 box:tasks update 12896: {"percent":12.11111111111111,"message":"Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/edit.js (Try 5) (summitproducts.cloud)"}
Aug 19 20:39:01 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/edit.js position 3935 try 5
Aug 19 20:39:01 box:backupformat/rsync Attempt 4 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:39:01 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/common.scss position 3934 try 5
Aug 19 20:39:01 box:backupformat/rsync Attempt 4 failed. Will retry: Upload error: self-signed certificate
Aug 19 20:39:11 box:tasks update 12896: {"percent":12.11111111111111,"message":"Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/fonts.json (Try 5) (summitproducts.cloud)"}
Aug 19 20:39:11 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/fonts.json position 3936 try 5
Aug 19 20:39:11 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/index.js position 3937 try 5
Aug 19 20:39:21 box:tasks update 12896: {"percent":12.11111111111111,"message":"Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/systemFonts.json (Try 5) (summitproducts.cloud)"}
Aug 19 20:39:21 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/systemFonts.json position 3938 try 5
Aug 19 20:39:21 box:backupformat/rsync Adding data/public/wp-content/plugins/olympus-google-fonts/blocks/src/google-fonts/transforms.js position 3939 try 5
Aug 19 20:47:34 box:syncer sync: done processing adds. error: BoxError: Upload error: self-signed certificate at Object.finish (/home/yellowtent/box/src/storage/s3.js:136:30) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async addFile (/home/yellowtent/box/src/backupformat/rsync.js:78:5) at async /home/yellowtent/box/src/backupformat/rsync.js:108:17 at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20) at async processTask (/home/yellowtent/box/src/backupformat/rsync.js:103:13) { reason: 'External Error', details: {} }
Aug 19 20:47:34 box:backupupload upload completed. error: BoxError: Upload error: self-signed certificate at Object.finish (/home/yellowtent/box/src/storage/s3.js:136:30) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async addFile (/home/yellowtent/box/src/backupformat/rsync.js:78:5) at async /home/yellowtent/box/src/backupformat/rsync.js:108:17 at async promiseRetry (/home/yellowtent/box/src/promise-retry.js:17:20) at async processTask (/home/yellowtent/box/src/backupformat/rsync.js:103:13) { reason: 'External Error', details: {} }
Aug 19 20:47:34 box:backuptask runBackupUpload: result - {"result":"Upload error: self-signed certificate"}
Aug 19 20:47:44 box:shell backup-snapshot/app_52e0bb95-8522-4cff-be7f-3ce6c0afeed5: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_52e0bb95-8522-4cff-be7f-3ce6c0afeed5 rsync {"localRoot":"/home/yellowtent/appsdata/52e0bb95-8522-4cff-be7f-3ce6c0afeed5","layout":[]} errored BoxError: backup-snapshot/app_52e0bb95-8522-4cff-be7f-3ce6c0afeed5 exited with code 50 signal null
[no timestamp] at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:122:19)
[no timestamp] at ChildProcess.emit (node:events:518:28)
[no timestamp] at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
[no timestamp] reason: 'Shell Error',
[no timestamp] details: {},
[no timestamp] code: 50,
[no timestamp] signal: null
[no timestamp] }
Aug 19 20:47:44 box:taskworker Task took 2863.724 seconds
Aug 19 20:47:44 box:tasks setCompleted - 12896: {"result":null,"error":{"stack":"BoxError: Upload error: self-signed certificate\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:166:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"External Error","details":{},"message":"Upload error: self-signed certificate"}}
Aug 19 20:47:44 box:tasks update 12896: {"percent":100,"result":null,"error":{"stack":"BoxError: Upload error: self-signed certificate\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:166:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"External Error","details":{},"message":"Upload error: self-signed certificate"}}
[no timestamp] Upload error: self-signed certificate
[no timestamp] at runBackupUpload (/home/yellowtent/box/src/backuptask.js:166:15)
[no timestamp] at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
[no timestamp] at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)
[no timestamp] at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)
[no timestamp] at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)

[nebulon]

The error might come from the self-signed certificate on your backup storage, not from your Cloudron.

Which backup storage are you using and does that have a valid cert?

[Shaun Snapp]

BTW, I just reset the certificates in Cloudron and retried a backup and got the same error.

I am using Digital Ocean. I can't find anything about certificates in my settings. And Digital Ocean would not have started asking for this just on the 16th. I am thinking that the 8.0.3 Cloudron update made a change to the backup such that a self signed certificate is necessary for the backup. I made no changes to anything in the backup and never even went into Digital Ocean to change anything.

[Shaun Snapp]

Notice this update note for 8.0.4

"ami: IMDv2 support
apps: do not log app logs to output
backupcleaner: do not remove the backup in progress
backups: make noop upload work again
dashboard: also render checklist items in apps.html
dashboard: remove mailbox import/export feature
ionos: add contract-owned eu-central-3
sftp: restore mode and owner
volumes: /mnt/volumes is reserved"

So it discusses making some things related to backups work again.

8.0.4 will not be stable for a week. So I should probably update to 8.0.4 when it is stable and then retry a backup.

[nebulon]

Well I don't think this is related to the certificates on your Cloudron, but the error indicates that during the backup the connection to the backup storage has issues and that your backup storage has invalid or rather self-signed certificates: Upload error: self-signed certificate

To be clear by saying you use Digital Ocean, does this refer to where the server is or the backup storage?

[Shaun Snapp]

Digital Ocean is where the Cloudron backup backs up to.

[Shaun Snapp]

Interestingly I just received this email from Amazon AWS telling me to upgrade to 8.0.4.

"Thank you for subscribing to Cloudron.

We are writing to inform you that Cloudron UG has added a new version to Cloudron on AWS Marketplace. As an existing customer, your subscription to the product, any running instances and access to previous versions are unaffected. However, Cloudron UG does recommend you to update to the latest version, Cloudron/8.0.4 by visiting https://aws.amazon.com/marketplace/pp/prodview-aep3gt5elubci.

For additional questions or upgrade information, please contact Cloudron UG directly. Visit the seller's profile page on AWS Marketplace."

However, as probably everyone knows, it is not yet stable.

[adnanjan.bcs]

@Shaun-Snapp same here too. how it cloud be solved?

[joseph]

@Shaun-Snapp If you go to Backups -> Configure -> Click Save without making any changes, does it work? I assume you mean you are using Digital Ocean Spaces?

[Shaun Snapp]

@joseph Yes, I can save the configuration in Cloudron Backups without making any changes. Yes I am using Digital Open Spaces. Here is there response to a ticket I opened with them.

"Please note that if your Space Bucket is generally working fine then this on-going issue might not be associated with that."

[Shaun Snapp]

@adnanjan-bcs Did this problem start after you updated to 8.0.3?

[joseph]

@Shaun-Snapp can you check if DNS is working fine on your server? Does host www.cloudron.io work on the server?

[Shaun Snapp]

@joseph I have not had any problems with my DNS. All of my sites are working fine. Except for the inability to make new backups, everything is working as it normally does.

[joseph]

@Shaun-Snapp thanks. I am short of ideas, can you contact us at support@cloudron.io and we can investigate further .

[Shaun Snapp]

I just updated to 8.0.4 and then tried a backup and I receive the same error of a self signed certificate.

[joseph]

Setting the nameservers to 8.8.8.8 and 1.1.1.1 resolved the issue.

[Shaun Snapp]

It did turn out to be a DNC issue which Joseph at Clourdon support fixed and then created a backup.