No backups because of NullPointerException

guyds

This morning I received the following error message from Cloudron:

Backup failed: Error copying cloudron/<mydomain.tld>/snapshot/app_3a528575-78a1-462f-a473-91c93871a359/Bt-sTAS-Jo9ep8OnqHMDOvgoe0qYa9iaP9DdcPZZ35I (4180 bytes): InternalError InternalError: unexpected: java.lang.NullPointerException. Logs are available here.

This is the log for that task:

retryable: true,
time: 2021-04-07T23:03:32.592Z,
statusCode: 500
Apr 08 01:03:32 box:shell backup-snapshot/app_3a528575-78a1-462f-a473-91c93871a359 (stdout): 2021-04-07T23:03:32.593Z box:backups Error uploading mongodbdump try 1: Error uploading cloudron/<mydomain.tld>/snapshot/app_3a528575-78a1-462f-a473-91c93871a359/kVoFT+4tn1jsWCd9Xz-rpaUQocY1jM71q9LF5JaNDe8. Message: Non-whitespace before first tag.
0
1
{ HTTP Code: XMLParserError
Apr 08 01:03:32 box:shell backup-snapshot/app_3a528575-78a1-462f-a473-91c93871a359 (stdout): 2021-04-07T23:03:32.738Z box:backupupload process: rss: 67387392 heapTotal: 27664384 heapUsed: 26014624 external: 2191265
Apr 08 01:03:32 box:backupupload v8 heap : used 26014848 total: 27664384 max: 2063859712
Apr 08 01:03:49 box:tasks 195: {"percent":26,"message":"Adding fsmetadata.json (Try 2) (chat.<mydomain.tld>)"}
Apr 08 01:03:49 box:shell backup-snapshot/app_3a528575-78a1-462f-a473-91c93871a359 (stdout): 2021-04-07T23:03:49.352Z box:backups Adding fsmetadata.json position 1 try 2
Apr 08 01:03:52 box:shell backup-snapshot/app_3a528575-78a1-462f-a473-91c93871a359 (stdout): 2021-04-07T23:03:52.595Z box:backups Adding mongodbdump position 2 try 2
Apr 08 01:04:02 box:shell backup-snapshot/app_3a528575-78a1-462f-a473-91c93871a359 (stdout): 2021-04-07T23:04:02.740Z box:backupupload process: rss: 67395584 heapTotal: 27664384 heapUsed: 25790640 external: 3138878
Apr 08 01:04:02 box:shell backup-snapshot/app_3a528575-78a1-462f-a473-91c93871a359 (stdout): 2021-04-07T23:04:02.741Z box:backupupload v8 heap : used 25790864 total: 27664384 max: 2063859712
Apr 08 01:04:32 box:shell backup-snapshot/app_3a528575-78a1-462f-a473-91c93871a359 (stdout): 2021-04-07T23:04:32.742Z box:backupupload process: rss: 67981312 heapTotal: 27664384 heapUsed: 26260744 external: 2577592
Apr 08 01:04:32 box:backupupload v8 heap : used 26260968 total: 27664384 max: 2063859712
Apr 08 01:04:55 box:shell backup-snapshot/app_3a528575-78a1-462f-a473-91c93871a359 (stdout): 2021-04-07T23:04:55.859Z box:storage/s3 Uploaded cloudron/<mydomain.tld>/snapshot/app_3a528575-78a1-462f-a473-91c93871a359/kVoFT+4tn1jsWCd9Xz-rpaUQocY1jM71q9LF5JaNDe8 with partSize 10485760: {"ETag":"\"831ea16fa4e7d88402bf80bb83330c16\"","VersionId":"4_z84a94f91bd0112697c82031c_f116a02a22fdbc58c_d20210407_m230455_c003_v0312004_t0017","Location":"https://<s3-bucket>.backblazeb2.com/cloudron/<mydomain.tld>/snapshot/app_3a528575-78a1-462f-a473-91c93871a359/kVoFT%2B4tn1jsWCd9Xz-rpaUQocY1jM71q9LF5JaNDe8","key":"cloudron/<mydomain.tld>/snapshot/app_3a528575-78a1-462f-a473-91c93871a359/kVoFT+4tn1jsWCd9Xz-rpaUQocY1jM71q9LF5JaNDe8","Key":"cloudron/<mydomain.tld>/snapshot/app_3a528575-78a1-462f-a473-91c93871a359/kVoFT+4tn1jsWCd9Xz-rpaUQocY1jM71q9LF5JaNDe8","Bucket":"<s3-bucket>"}
Apr 08 01:04:55 box:shell backup-snapshot/app_3a528575-78a1-462f-a473-91c93871a359 (stdout): 2021-04-07T23:04:55.860Z box:backups Uploaded mongodbdump
Apr 08 01:05:02 box:shell backup-snapshot/app_3a528575-78a1-462f-a473-91c93871a359 (stdout): 2021-04-07T23:05:02.743Z box:backupupload process: rss: 68530176 heapTotal: 27664384 heapUsed: 26670320 external: 2597597
Apr 08 01:05:02 box:shell backup-snapshot/app_3a528575-78a1-462f-a473-91c93871a359 (stdout): 2021-04-07T23:05:02.743Z box:backupupload v8 heap : used 26670776 total: 27664384 max: 2063859712
Apr 08 01:05:13 box:shell backup-snapshot/app_3a528575-78a1-462f-a473-91c93871a359 (stdout): 2021-04-07T23:05:13.120Z box:storage/s3 Uploaded cloudron/<mydomain.tld>/snapshot/app_3a528575-78a1-462f-a473-91c93871a359/AGChTUjiqYB5pGzHZ6v-StRfA5LNg0RDxICQ0H2ZpsY with partSize 10485760: {"ETag":"\"3a68b16d0b2305ff30bd492748c91079\"","VersionId":"4_z84a94f91bd0112697c82031c_f1194c7a7c62b59a7_d20210407_m230511_c003_v0312000_t0020","Location":"https://<s3-bucket>.backblazeb2.com/cloudron/<mydomain.tld>/snapshot/app_3a528575-78a1-462f-a473-91c93871a359/AGChTUjiqYB5pGzHZ6v-StRfA5LNg0RDxICQ0H2ZpsY","key":"cloudron/<mydomain.tld>/snapshot/app_3a528575-78a1-462f-a473-91c93871a359/AGChTUjiqYB5pGzHZ6v-StRfA5LNg0RDxICQ0H2ZpsY","Key":"cloudron/<mydomain.tld>/snapshot/app_3a528575-78a1-462f-a473-91c93871a359/AGChTUjiqYB5pGzHZ6v-StRfA5LNg0RDxICQ0H2ZpsY","Bucket":"<s3-bucket>"}
Apr 08 01:05:13 box:shell backup-snapshot/app_3a528575-78a1-462f-a473-91c93871a359 (stdout): 2021-04-07T23:05:13.121Z box:backups Uploaded fsmetadata.json
Apr 08 01:05:13 box:shell backup-snapshot/app_3a528575-78a1-462f-a473-91c93871a359 (stdout): 2021-04-07T23:05:13.122Z box:syncer Done processing adds null
Apr 08 01:05:13 box:shell backup-snapshot/app_3a528575-78a1-462f-a473-91c93871a359 (stdout): 2021-04-07T23:05:13.124Z box:backupupload upload completed. error: null
Apr 08 01:05:13 box:backups runBackupUpload: result - {"result":""}
Apr 08 01:05:13 box:backups chat.<mydomain.tld> uploadAppSnapshot: snapshot/app_3a528575-78a1-462f-a473-91c93871a359 done. 311.372 seconds
Apr 08 01:05:13 box:backups Rotating app backup of 3a528575-78a1-462f-a473-91c93871a359 to id 2021-04-07-230001-271/app_3a528575-78a1-462f-a473-91c93871a359_2021-04-07-230513-144_v2.15.0
Apr 08 01:05:13 box:tasks 195: {"percent":26,"message":"Copying files from 0-4 (chat.<mydomain.tld>)"}
Apr 08 01:05:13 box:tasks 195: {"percent":26,"message":"Copying AGChTUjiqYB5pGzHZ6v-StRfA5LNg0RDxICQ0H2ZpsY (chat.<mydomain.tld>)"}
Apr 08 01:05:13 box:tasks 195: {"percent":26,"message":"Copying Bt-sTAS-Jo9ep8OnqHMDOvgoe0qYa9iaP9DdcPZZ35I (chat.<mydomain.tld>)"}
Apr 08 01:05:13 box:tasks 195: {"percent":26,"message":"Copying dnPl31GL+kZJ36QC789XFNBai4edFLeiN9slYQCxrzw/mmuE7irLokki1+VTRq+LPwodJ4PC9yPhWpXl-b987Jo (chat.<mydomain.tld>)"}
Apr 08 01:05:13 box:tasks 195: {"percent":26,"message":"Copying kVoFT+4tn1jsWCd9Xz-rpaUQocY1jM71q9LF5JaNDe8 (chat.<mydomain.tld>)"}
Apr 08 01:05:15 box:tasks 195: {"percent":26,"message":"Retrying (1) copy of Bt-sTAS-Jo9ep8OnqHMDOvgoe0qYa9iaP9DdcPZZ35I. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:05:15 box:tasks 195: {"percent":26,"message":"Retrying (1) copy of dnPl31GL+kZJ36QC789XFNBai4edFLeiN9slYQCxrzw/mmuE7irLokki1+VTRq+LPwodJ4PC9yPhWpXl-b987Jo. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:05:15 box:tasks 195: {"percent":26,"message":"Retrying (1) copy of AGChTUjiqYB5pGzHZ6v-StRfA5LNg0RDxICQ0H2ZpsY. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:05:16 box:tasks 195: {"percent":26,"message":"Retrying (1) copy of kVoFT+4tn1jsWCd9Xz-rpaUQocY1jM71q9LF5JaNDe8. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:05:35 box:tasks 195: {"percent":26,"message":"Retrying (2) copy of dnPl31GL+kZJ36QC789XFNBai4edFLeiN9slYQCxrzw/mmuE7irLokki1+VTRq+LPwodJ4PC9yPhWpXl-b987Jo. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:05:35 box:tasks 195: {"percent":26,"message":"Retrying (2) copy of Bt-sTAS-Jo9ep8OnqHMDOvgoe0qYa9iaP9DdcPZZ35I. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:05:36 box:tasks 195: {"percent":26,"message":"Retrying (2) copy of kVoFT+4tn1jsWCd9Xz-rpaUQocY1jM71q9LF5JaNDe8. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:05:55 box:tasks 195: {"percent":26,"message":"Retrying (3) copy of Bt-sTAS-Jo9ep8OnqHMDOvgoe0qYa9iaP9DdcPZZ35I. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:05:56 box:tasks 195: {"percent":26,"message":"Retrying (3) copy of dnPl31GL+kZJ36QC789XFNBai4edFLeiN9slYQCxrzw/mmuE7irLokki1+VTRq+LPwodJ4PC9yPhWpXl-b987Jo. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:05:57 box:tasks 195: {"percent":26,"message":"Retrying (3) copy of kVoFT+4tn1jsWCd9Xz-rpaUQocY1jM71q9LF5JaNDe8. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:06:16 box:tasks 195: {"percent":26,"message":"Retrying (4) copy of Bt-sTAS-Jo9ep8OnqHMDOvgoe0qYa9iaP9DdcPZZ35I. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:06:16 box:tasks 195: {"percent":26,"message":"Retrying (4) copy of dnPl31GL+kZJ36QC789XFNBai4edFLeiN9slYQCxrzw/mmuE7irLokki1+VTRq+LPwodJ4PC9yPhWpXl-b987Jo. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:06:19 box:tasks 195: {"percent":26,"message":"Retrying (4) copy of kVoFT+4tn1jsWCd9Xz-rpaUQocY1jM71q9LF5JaNDe8. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:06:36 box:tasks 195: {"percent":26,"message":"Retrying (5) copy of Bt-sTAS-Jo9ep8OnqHMDOvgoe0qYa9iaP9DdcPZZ35I. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:06:37 box:tasks 195: {"percent":26,"message":"Retrying (5) copy of dnPl31GL+kZJ36QC789XFNBai4edFLeiN9slYQCxrzw/mmuE7irLokki1+VTRq+LPwodJ4PC9yPhWpXl-b987Jo. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:06:41 box:tasks 195: {"percent":26,"message":"Retrying (5) copy of kVoFT+4tn1jsWCd9Xz-rpaUQocY1jM71q9LF5JaNDe8. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:06:57 box:tasks 195: {"percent":26,"message":"Retrying (6) copy of Bt-sTAS-Jo9ep8OnqHMDOvgoe0qYa9iaP9DdcPZZ35I. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:06:57 box:tasks 195: {"percent":26,"message":"Retrying (6) copy of dnPl31GL+kZJ36QC789XFNBai4edFLeiN9slYQCxrzw/mmuE7irLokki1+VTRq+LPwodJ4PC9yPhWpXl-b987Jo. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:07:01 box:tasks 195: {"percent":26,"message":"Retrying (6) copy of kVoFT+4tn1jsWCd9Xz-rpaUQocY1jM71q9LF5JaNDe8. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:07:17 box:tasks 195: {"percent":26,"message":"Retrying (7) copy of Bt-sTAS-Jo9ep8OnqHMDOvgoe0qYa9iaP9DdcPZZ35I. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:07:18 box:tasks 195: {"percent":26,"message":"Retrying (7) copy of dnPl31GL+kZJ36QC789XFNBai4edFLeiN9slYQCxrzw/mmuE7irLokki1+VTRq+LPwodJ4PC9yPhWpXl-b987Jo. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:07:21 box:tasks 195: {"percent":26,"message":"Retrying (7) copy of kVoFT+4tn1jsWCd9Xz-rpaUQocY1jM71q9LF5JaNDe8. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:07:38 box:tasks 195: {"percent":26,"message":"Retrying (8) copy of Bt-sTAS-Jo9ep8OnqHMDOvgoe0qYa9iaP9DdcPZZ35I. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:07:39 box:tasks 195: {"percent":26,"message":"Retrying (8) copy of dnPl31GL+kZJ36QC789XFNBai4edFLeiN9slYQCxrzw/mmuE7irLokki1+VTRq+LPwodJ4PC9yPhWpXl-b987Jo. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:07:42 box:tasks 195: {"percent":26,"message":"Retrying (8) copy of kVoFT+4tn1jsWCd9Xz-rpaUQocY1jM71q9LF5JaNDe8. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:07:58 box:tasks 195: {"percent":26,"message":"Retrying (9) copy of Bt-sTAS-Jo9ep8OnqHMDOvgoe0qYa9iaP9DdcPZZ35I. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:07:59 box:tasks 195: {"percent":26,"message":"Retrying (9) copy of dnPl31GL+kZJ36QC789XFNBai4edFLeiN9slYQCxrzw/mmuE7irLokki1+VTRq+LPwodJ4PC9yPhWpXl-b987Jo. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:08:02 box:tasks 195: {"percent":26,"message":"Retrying (9) copy of kVoFT+4tn1jsWCd9Xz-rpaUQocY1jM71q9LF5JaNDe8. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:08:19 box:tasks 195: {"percent":26,"message":"Retrying (10) copy of Bt-sTAS-Jo9ep8OnqHMDOvgoe0qYa9iaP9DdcPZZ35I. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:08:20 box:tasks 195: {"percent":26,"message":"Retrying (10) copy of dnPl31GL+kZJ36QC789XFNBai4edFLeiN9slYQCxrzw/mmuE7irLokki1+VTRq+LPwodJ4PC9yPhWpXl-b987Jo. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:08:23 box:tasks 195: {"percent":26,"message":"Retrying (10) copy of kVoFT+4tn1jsWCd9Xz-rpaUQocY1jM71q9LF5JaNDe8. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:08:39 box:tasks 195: {"percent":26,"message":"Retrying (11) copy of Bt-sTAS-Jo9ep8OnqHMDOvgoe0qYa9iaP9DdcPZZ35I. Error: InternalError: unexpected: java.lang.NullPointerException 500 (chat.<mydomain.tld>)"}
Apr 08 01:08:39 box:storage/s3 copy: s3 copy error when copying cloudron/<mydomain.tld>/snapshot/app_3a528575-78a1-462f-a473-91c93871a359/Bt-sTAS-Jo9ep8OnqHMDOvgoe0qYa9iaP9DdcPZZ35I: InternalError: unexpected: java.lang.NullPointerException
Apr 08 01:08:39 box:tasks 195: {"percent":26,"message":"Copied 4 files with error: BoxError: Error copying cloudron/<mydomain.tld>/snapshot/app_3a528575-78a1-462f-a473-91c93871a359/Bt-sTAS-Jo9ep8OnqHMDOvgoe0qYa9iaP9DdcPZZ35I (4180 bytes): InternalError InternalError: unexpected: java.lang.NullPointerException (chat.<mydomain.tld>)"}
Apr 08 01:08:39 box:backups chat.<mydomain.tld> Unable to backup BoxError: Error copying cloudron/<mydomain.tld>/snapshot/app_3a528575-78a1-462f-a473-91c93871a359/Bt-sTAS-Jo9ep8OnqHMDOvgoe0qYa9iaP9DdcPZZ35I (4180 bytes): InternalError InternalError: unexpected: java.lang.NullPointerException
at Response.done (/home/yellowtent/box/src/storage/s3.js:287:52)
at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:369:18)
at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:14)
at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:690:12) {
reason: 'External Error',
details: {}
Apr 08 01:08:39 box:taskworker Task took 518.477 seconds
Apr 08 01:08:39 box:tasks setCompleted - 195: {"result":null,"error":{"stack":"BoxError: Error copying cloudron/<mydomain.tld>/snapshot/app_3a528575-78a1-462f-a473-91c93871a359/Bt-sTAS-Jo9ep8OnqHMDOvgoe0qYa9iaP9DdcPZZ35I (4180 bytes): InternalError InternalError: unexpected: java.lang.NullPointerException\n at Response.done (/home/yellowtent/box/src/storage/s3.js:287:52)\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:369:18)\n at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:14)\n at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)\n at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)\n at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:690:12)","name":"BoxError","reason":"External Error","details":{},"message":"Error copying cloudron/<mydomain.tld>/snapshot/app_3a528575-78a1-462f-a473-91c93871a359/Bt-sTAS-Jo9ep8OnqHMDOvgoe0qYa9iaP9DdcPZZ35I (4180 bytes): InternalError InternalError: unexpected: java.lang.NullPointerException"}}
Apr 08 01:08:39 box:tasks 195: {"percent":100,"result":null,"error":{"stack":"BoxError: Error copying cloudron/<mydomain.tld>/snapshot/app_3a528575-78a1-462f-a473-91c93871a359/Bt-sTAS-Jo9ep8OnqHMDOvgoe0qYa9iaP9DdcPZZ35I (4180 bytes): InternalError InternalError: unexpected: java.lang.NullPointerException\n at Response.done (/home/yellowtent/box/src/storage/s3.js:287:52)\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:369:18)\n at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)\n at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)\n at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:14)\n at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)\n at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)\n at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)\n at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:690:12)","name":"BoxError","reason":"External Error","details":{},"message":"Error copying cloudron/<mydomain.tld>/snapshot/app_3a528575-78a1-462f-a473-91c93871a359/Bt-sTAS-Jo9ep8OnqHMDOvgoe0qYa9iaP9DdcPZZ35I (4180 bytes): InternalError InternalError: unexpected: java.lang.NullPointerException"}}

I'm backing up to Backblaze B2 and encryption is enabled.

As a result of this error, not any backup was made last night (not for the apps and not for the cloudron itself).

Any ideas?

girish

@guyds This is a crash/external error from Backblaze B2. You can give them information about your account/bucket and the files and ask them to investigate the crash. Also, NullPointerException comes from java code, so it should be easy for them to fix hopefully.

guyds

@girish yes, I understand the error is from Backblaze, but the reason I reported the issue is because nothing is backed up at all because of this error. I wrongly assumed that backups are performed independently from each other.

Maybe you can consider implementing per app backups as well? And some retry logic to catch temporary issues?
Because right now it's an all or nothing approach: either everything is backed up or nothing at all, which is not a very reliable approach in my opinion.

girish

@guyds there is retry logic already (see the 'retry' string in the logs) but it seems it kept crashing on retry.

Backups are indeed independent and each app is backed up separately. If you go into App -> Config -> Backup -> Create Backup, it will create a backup just for a specific app. But if you go to Backup -> Create Backup, it creates a "full backup" which is essentially looping through each app and creating a backup and then linking them all together. If one of those app backups fail, the "full backup" fails. Not sure what else can be done here. Do we want to have "full backups" but which are not full because 1 or 2 apps backups inside failed?

guyds

@girish said in No backups because of NullPointerException:

@guyds there is retry logic already (see the 'retry' string in the logs) but it seems it kept crashing on retry.

I'm sorry, I can't believe I missed that, my bad!

Backups are indeed independent and each app is backed up separately.

That is how I understood it as well, however:

If you go into App -> Config -> Backup -> Create Backup, it will create a backup just for a specific app. But if you go to Backup -> Create Backup, it creates a "full backup" which is essentially looping through each app and creating a backup and then linking them all together.

This is for manual backups, not how the backup scheduling works, which is an all or nothing approach atm.

If one of those app backups fail, the "full backup" fails. Not sure what else can be done here.

Yes, so that's basically my concern: a failure with one app's backup results in a total backup failure for all apps.

Do we want to have "full backups" but which are not full because 1 or 2 apps backups inside failed?

Yes, maybe that's better than having no backup at all because then at least some of the apps might be recoverable from the partial backup.
But why not providing the option to store/upload the per app backups separately as well?
Or even better: provide the option to either store them as a separate backup ("per app backup"), as a "full backup" (current behavior) or both.
This way the backup of one app can fail (e.g. temp error with external storage provider) while the other apps are still being backed up correctly.

To be clear: I'm talking about scheduled backups here, not manual backups.

girish

While it's possible to implement, that's not how the backup system works right now. Internally, it behaves it pretty much as "separate" backup but we show it as a bundle because we thought that's the best way the user can easily understand.

But ultimately, we rely on the fact that most of the storage backend services are reasonably stable (which they are. I think the B2 service is quite new, so maybe they have some instabilities).

guyds

@girish said in No backups because of NullPointerException:

While it's possible to implement, that's not how the backup system works right now. Internally, it behaves it pretty much as "separate" backup but we show it as a bundle because we thought that's the best way the user can easily understand.

I understand that internally backups are actually already separate per app, but they aren't worth anything if the final bundle can't be stored / uploaded.

But ultimately, we rely on the fact that most of the storage backend services are reasonably stable (which they are. I think the B2 service is quite new, so maybe they have some instabilities).

In my experience (as a developer) it's very dangerous to rely on / asume that a(ny) service is always stable.
But that's just my experience of course

d19dotca

@guyds said in No backups because of NullPointerException:

In my experience (as a developer) it's very dangerous to rely on / asume that a(ny) service is always stable.

While that's definitely understandable, keep in mind that backups by their very nature (and to be of any value) need to be saved to external systems. Thus, Cloudron will always be at the mercy of external services for backups. So stability isn't something they can guarantee, and retry logic is already built in to avoid some of the smaller hiccups that can occur.

If you want true stability for your backups, I'd recommend using block storage instead of object storage services. Of course block storage is a bit more expensive, but far more reliable and far quicker too.

guyds

@d19dotca Yes, block storage can be more reliable, but my point is not regarding the backup technology being used for the backups.
I'm just suggesting that separating the backups on a per app basis can have some advantages as well: apps don't depend on each other for their backup to succeed, backups can be spread more easily, backups can be turned on/off on a per app basis, etc.

girish

The code is written with backup storage being reasonable reliable as @d19dotca mentioned. But I also see @guyds point as to why throw away incomplete backups. TBH, I have never thought about those.

Is there value in partial backups? Currently, these backups are tracked internally and cleaned up (since it was considered to have no value).

girish

Thinking about this a bit more:

As of now, a full box backup = backing up apps one by one and then box code. Maybe, we should reverse this order. i.e box code first and then apps (reason below).
We can show "partial" backups (i.e one or more app backups failed) in the Backup view with some UI hint/tooltip. If box backups itself fails, it won't appear at all. If box backup succeeded but one or more apps failed, then it shows a entry there with the list of apps that succeeded (like it does now). A box backup is required for the Cloudron "restore" process which is why I suggested the reorder in the first bullet.

Now, the question is if this will confuse people. Maybe with the right UI hint, we can make it clear that it's not a full backup but Cloudron backed up whatever it could.

d19dotca

@girish I like your suggestion to do the box data first (since that's probably the largest one in most cases I assume? I know it is by far the largest for me anyways) and then app backups after, failing entirely if the box backup doesn't succeed.

I'm a bit uneasy to the "partial" backup idea because I'd worry that may cause people to assume backups are working when they're not 100% "safe". Something symbolizing them as different would be a partial answer to that, but I would recommend the admin user gets an email notification (as we currently do) if backups fail, even if it's a partial fail. That way the admin user can quickly try to fix it without assuming it's all good still.

In other words, my concern would be in making sure admins aren't going to become "complacent" with their backups when partial backups succeed but still contain an error in them. I think as long as those partial backups still notify the admin user via an email alert, then that'd be a way to prevent complacency. Last thing you probably want is people who have a catastrophic failure and later realize their backups weren't 100% complete.

guyds

@d19dotca @girish As I see it there should be 2 backup "levels" and actually those are already in place, but for some reason they aren't treated like that in the end.
Let me clarify this...

The first level is the per app level. Right now, apps are already backed up one by one, but they are not stored nor reported individually. And this is the missing feature in my opinion.
So as I see it, once an app is backed up, it should be stored "as is" (as an individual backup) in the backup target location and reported in the backup tab of the app (where one can restore it, clone it or download it just as it is now, only it will be from the individual backup instead of the full box backup). If a backup fails for an app, then there will be no backup for that app at that date and preferably (optionally?) an email is sent to the admin.

Then the second level is combining those individual backups and add the box code and store it as the full box backup, exactly how it is done already. This full backup is accessible via the profile menu -> backups and should allow a user to restore the full cloudron to that particular state. If one of the app backups failed for whatever reason, then there will be no full backup available for that date and the admin should receive an email (= the current behavior).

So, again, I think the current issue is that everything is treated as a whole while it makes more sense in my opinion to treat each app individually and then in the end (optionally?) bundle the individual parts as a whole.

Does this make sense?
Or am I just freaking out here?

girish

@guyds said in No backups because of NullPointerException:

The first level is the per app level. Right now, apps are already backed up one by one, but they are not stored nor reported individually. And this is the missing feature in my opinion.

Ah no, they are listed in app -> backups. So, even if you do a full backup, each individual app backup will be listed in app -> backups. You can also use that backup to restore/clone the app.

So, again, I think the current issue is that everything is treated as a whole while it makes more sense in my opinion to treat each app individually and then in the end (optionally?) bundle the individual parts as a whole.

Yes, so they are treated individually, it's actually very close to what you want. The only issue is that when a full backup fails, those successful individual app backups that are part of a failed full backup will get removed/cleaned up every night.

I have made a task to fix the behavior, let's see.

guyds

@girish said in No backups because of NullPointerException:

@guyds said in No backups because of NullPointerException:

The first level is the per app level. Right now, apps are already backed up one by one, but they are not stored nor reported individually. And this is the missing feature in my opinion.

Ah no, they are listed in app -> backups. So, even if you do a full backup, each individual app backup will be listed in app -> backups.

Yes, they are listed at the app level, but there's no reporting at the app level because the backup succeeds or fails at the box level.

You can also use that backup to restore/clone the app.

Yes, but only if the backup succeeds for all apps.

So, again, I think the current issue is that everything is treated as a whole while it makes more sense in my opinion to treat each app individually and then in the end (optionally?) bundle the individual parts as a whole.

Yes, so they are treated individually, it's actually very close to what you want. The only issue is that when a full backup fails, those successful individual app backups that are part of a failed full backup will get removed/cleaned up every night.

Exactly, they are per app bot not treated like that in the end because success or failure is determined by the whole box.

I have made a task to fix the behavior, let's see.

Great, thanks!

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

No backups because of NullPointerException