issue with backups on Scaleway

[jeau]

For the past few days, I’ve been having a trouble with my automatic backup to Scaleway (in tgz format).

Manual backups work, but I may encounter the same issue with large backups (> 30 GB). The error displayed is FileSystem Error or Error: getaddrinfo EAI_AGAIN s3.fr-par.scw.cloud.

Jul 23 11:42:19 backupformat/tgz: addToPack: error adding ./data/wp-content/uploads/2024/04/Le-Clan-des-renard-10-72dpiweb.jpg file The operation was aborted
Jul 23 11:42:19 backupformat/tgz: BoxError: The operation was aborted
    at file:///home/yellowtent/box/src/backupformat/tgz.js:67:24
    at Sink._continuePack (/home/yellowtent/box/node_modules/tar-stream/pack.js:42:5)
    at Sink._destroy (/home/yellowtent/box/node_modules/tar-stream/pack.js:121:10)
    at WritableState.updateNonPrimary (/home/yellowtent/box/node_modules/streamx/index.js:213:16)
    at WritableState.update (/home/yellowtent/box/node_modules/streamx/index.js:195:72)
    at WritableState.updateWriteNT (/home/yellowtent/box/node_modules/streamx/index.js:564:10)
    at node:internal/process/task_queues:149:7
    at AsyncResource.runInAsyncScope (node:async_hooks:214:14)
    at AsyncResource.runMicrotask (node:internal/process/task_queues:146:8)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5) {
  reason: 'FileSystem Error',
  details: {}
}
Jul 23 11:42:19 backupformat/tgz: tarPack: packed 0 files
Jul 23 11:42:19 backupformat/tgz: AbortError: The operation was aborted
    at Object.destroyer (node:internal/streams/destroy:328:11)
    at createAsyncIterator (node:internal/streams/readable:1404:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5) {
  code: 'ABORT_ERR'
}
Jul 23 11:42:19 backupformat/tgz: Attempt 1 failed. Will retry: tarPack pipeline error: The operation was aborted 
Jan 01 01:00:00 /home/yellowtent/box/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:69
Jan 01 01:00:00                 throw this.decorateServiceException(Object.assign(new ErrorCtor({ name: errorName }), errorMetadata), dataObject);
Jan 01 01:00:00                                                                   ^
Jan 01 01:00:00 
Jan 01 01:00:00 InvalidArgument: Part number must be an integer between 1 and 1000, inclusive
Jan 01 01:00:00     at ProtocolLib.getErrorSchemaOrThrowBaseException (/home/yellowtent/box/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:69:67)
Jan 01 01:00:00     at AwsRestXmlProtocol.handleError (/home/yellowtent/box/node_modules/@aws-sdk/core/dist-cjs/submodules/protocols/index.js:1810:65)
Jan 01 01:00:00     at AwsRestXmlProtocol.deserializeResponse (/home/yellowtent/box/node_modules/@smithy/core/dist-cjs/submodules/protocols/index.js:334:24)
Jan 01 01:00:00     at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
Jan 01 01:00:00     at async /home/yellowtent/box/node_modules/@smithy/core/dist-cjs/submodules/schema/index.js:26:24
Jan 01 01:00:00     at async /home/yellowtent/box/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:385:20
Jan 01 01:00:00     at async /home/yellowtent/box/node_modules/@smithy/middleware-retry/dist-cjs/index.js:254:46
Jan 01 01:00:00     at async /home/yellowtent/box/node_modules/@aws-sdk/middleware-flexible-checksums/dist-cjs/index.js:243:24
Jan 01 01:00:00     at async /home/yellowtent/box/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:62:28
Jan 01 01:00:00     at async /home/yellowtent/box/node_modules/@aws-sdk/middleware-sdk-s3/dist-cjs/index.js:89:20
Jan 01 01:00:00     at async /home/yellowtent/box/node_modules/@aws-sdk/middleware-logger/dist-cjs/index.js:5:26
Jan 01 01:00:00     at async Upload.__doConcurrentUpload (/home/yellowtent/box/node_modules/@aws-sdk/lib-storage/dist-cjs/index.js:374:32)
Jan 01 01:00:00     at async Promise.all (index 0)
Jan 01 01:00:00     at async Upload.__doMultipartUpload (/home/yellowtent/box/node_modules/@aws-sdk/lib-storage/dist-cjs/index.js:419:9)
Jan 01 01:00:00     at async Upload.done (/home/yellowtent/box/node_modules/@aws-sdk/lib-storage/dist-cjs/index.js:238:16) {
Jan 01 01:00:00   '$fault': 'client',
Jan 01 01:00:00   '$retryable': undefined,
Jan 01 01:00:00   '$metadata': {
Jan 01 01:00:00     httpStatusCode: 400,
Jan 01 01:00:00     requestId: 'txg807b949ef8da446bbd29-006a61e1fb',
Jan 01 01:00:00     extendedRequestId: 'txg807b949ef8da446bbd29-006a61e1fb',
Jan 01 01:00:00     cfId: undefined,
Jan 01 01:00:00     attempts: 1,
Jan 01 01:00:00     totalRetryDelay: 0
Jan 01 01:00:00   },
Jan 01 01:00:00   Code: 'InvalidArgument',
Jan 01 01:00:00   RequestId: 'txg807b949ef8da446bbd29-006a61e1fb',
Jan 01 01:00:00   HostId: 'txg807b949ef8da446bbd29-006a61e1fb',
Jan 01 01:00:00   Resource: '/cloudron/snapshot/app_aa1ac9b6-3ba1-46f6-b4dc-fcfd3fe0b8a6.tar.gz'
Jan 01 01:00:00 }
Jan 01 01:00:00 
Jan 01 01:00:00 Node.js v24.13.0
Jul 23 11:42:25 shell: backuptask: /usr/bin/sudo --non-interactive -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_aa1ac9b6-3ba1-46f6-b4dc-fcfd3fe0b8a6.tar.gz ecbe2048-0622-4fe7-ab86-0a2712ae1c2b {"localRoot":"/home/yellowtent/appsdata/aa1ac9b6-3ba1-46f6-b4dc-fcfd3fe0b8a6","layout":[]} errored with code 1 and signal null timeout false terminated false - stdout: "" - stderr: ""
Jul 23 11:42:25 backuptask: runBackupUpload: backuptask crashed BoxError: /usr/bin/sudo exited with code 1 signal null
    at ChildProcess.<anonymous> (file:///home/yellowtent/box/src/shell.js:70:23)
    at ChildProcess.emit (node:events:508:28)
    at maybeClose (node:internal/child_process:1101:16)
    at ChildProcess._handle.onexit (node:internal/child_process:305:5) {
  reason: 'Shell Error',
  details: {},
  stdout: '',
  stdoutString: '',
  stdoutLineCount: 0,
  stderr: '',
  stderrString: '',
  stderrLineCount: 0,
  code: 1,
  signal: null,
  timedOut: false,
  terminated: false
}
Jul 23 11:42:25 backuptask: fullBackup: app emf.fr backup finished. Took 1091.923 seconds
Jul 23 11:42:25 locks: write: current locks: {"full_backup_task_ecbe2048-0622-4fe7-ab86-0a2712ae1c2b":null}
Jul 23 11:42:25 locks: release: app_backup_aa1ac9b6-3ba1-46f6-b4dc-fcfd3fe0b8a6
Jul 23 11:42:25 tasks: setCompleted - 27690: {"result":null,"error":{"message":"Backuptask crashed","reason":"Internal Error"},"percent":100}
Jul 23 11:42:25 tasks: updating task 27690 with: {"completed":true,"result":null,"error":{"message":"Backuptask crashed","reason":"Internal Error"},"percent":100}
Jul 23 11:42:25 taskworker: Task took 4824.905 seconds
Jul 23 11:42:25 BoxError: Backuptask crashed
    at runBackupUpload (file:///home/yellowtent/box/src/backuptask.js:190:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async uploadAppSnapshot (file:///home/yellowtent/box/src/backuptask.js:359:34)
    at async backupAppWithTag (file:///home/yellowtent/box/src/backuptask.js:382:26)
Jul 23 11:42:25 Exiting with code 0

[james]

Hello @jeau

Thanks for reporting, I think I see the issue.

InvalidArgument: Part number must be an integer between 1 and 1000, inclusive

This suggests this is Scaleway's Object Storage rejecting the 1001st part of a multipart upload, not a Cloudron bug in the strict sense, it's a limit collision.
See: https://www.scaleway.com/en/docs/object-storage/api-cli/multipart-uploads/

1 to 1000 parts per object
5 MB to 5 GB per part (except for the last one)
Each object stores up to 5 TB

The default configuration for a S3 based backup site is:

• Memory limit: 1 GiB
• Upload part size: 50 MiB
• Upload concurrency: 20
• Download concurrency: 30
• Copy concurrency: 500

So with your error, I would advise increasing the Upload part size to e.g.: 100 MiB that should cut the parts per object in half and could resolve your issue.

I am taking this topic also as feedback to improve the error message to make it more human readable.

[james]

Hello @jeau
Adding to my last message, when increasing the Upload part size you should also increase the Memory limit at least double or triple.
So when changing 50 MiB to 100 MiB the Memory limit should also go from 1 GiB to at least 2 GiB or better 3 GiB.

[jeau]

Many thanks @james
I’ve made these two changes to the settings
and I’m currently testing them;
I’ll also see how it goes when the automatic backup runs
I’ll keep you posted

[jeau]

Hi,
I have set the Memory limit to 16 GiB and the Upload part size to 512 MiB
It worked yesterday with the manual backups of individual applications and the whole system.
But last night’s automatic backup failed.
I suspect a network issue as I’m seeing this message Error: getaddrinfo EAI_AGAIN cloudron.s3.fr-par.scw.cloud, which I’ve seen before at other times.
Perhaps the problem comes from my hosting provider or from Scaleway, but it seems to only affect backups, which is strange.

Jul 23 23:49:34 services: Backing up mysql
Jul 23 23:49:34 pipeRequestToFile: connected with status code 200
Jul 23 23:49:34 pipeRequestToFile: success
Jul 23 23:53:35 shell: backuptask: /usr/bin/sudo --non-interactive -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_aa1ac9b6-3ba1-46f6-b4dc-fcfd3fe0b8a6.tar.gz ecbe2048-0622-4fe7-ab86-0a2712ae1c2b {"localRoot":"/home/yellowtent/appsdata/aa1ac9b6-3ba1-46f6-b4dc-fcfd3fe0b8a6","layout":[]}
Jul 23 23:53:35 backuptask: snapshotApp: emf.fr took 241.328 seconds
Jul 23 23:53:35 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading app snapshot emf.fr"}
Jul 23 23:53:35 backuptask: runBackupUpload: adjusting heap size to 8192M
Jul 23 23:53:36 backupupload: Backing up {"localRoot":"/home/yellowtent/appsdata/aa1ac9b6-3ba1-46f6-b4dc-fcfd3fe0b8a6","layout":[]} to snapshot/app_aa1ac9b6-3ba1-46f6-b4dc-fcfd3fe0b8a6.tar.gz
Jul 23 23:53:36 backuptask: upload: path snapshot/app_aa1ac9b6-3ba1-46f6-b4dc-fcfd3fe0b8a6.tar.gz site ecbe2048-0622-4fe7-ab86-0a2712ae1c2b dataLayout {"localRoot":"/home/yellowtent/appsdata/aa1ac9b6-3ba1-46f6-b4dc-fcfd3fe0b8a6","layout":[]}
Jul 23 23:53:37 backuptask: checkPreconditions: mount point status is {"state":"active","message":""}
Jul 23 23:53:37 backuptask: checkPreconditions: getting disk usage of /home/yellowtent/appsdata/aa1ac9b6-3ba1-46f6-b4dc-fcfd3fe0b8a6
Jul 23 23:53:37 shell: backuptask: du --dereference-args --summarize --block-size=1 --exclude=*.lock --exclude=dovecot.list.index.log.* /home/yellowtent/appsdata/aa1ac9b6-3ba1-46f6-b4dc-fcfd3fe0b8a6
Jul 23 23:53:43 backuptask: checkPreconditions: total required=45107490816 available=Infinity
Jul 23 23:53:43 backupformat/tgz: upload: uploading to site ecbe2048-0622-4fe7-ab86-0a2712ae1c2b path snapshot/app_aa1ac9b6-3ba1-46f6-b4dc-fcfd3fe0b8a6.tar.gz (encrypted: false) dataLayout {"localRoot":"/home/yellowtent/appsdata/aa1ac9b6-3ba1-46f6-b4dc-fcfd3fe0b8a6","layout":[]}
Jul 23 23:53:43 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup snapshot/app_aa1ac9b6-3ba1-46f6-b4dc-fcfd3fe0b8a6.tar.gz (emf.fr)"}
Jul 23 23:53:53 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 49M@5MBps (emf.fr)"}
Jul 23 23:54:03 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 97M@5MBps (emf.fr)"}
Jul 23 23:54:13 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 145M@5MBps (emf.fr)"}
Jul 23 23:54:23 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 191M@5MBps (emf.fr)"}
Jul 23 23:54:33 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 236M@5MBps (emf.fr)"}
Jul 23 23:54:43 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Still uploading backup (60s, 281M) (emf.fr)"}
Jul 23 23:54:53 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 325M@4MBps (emf.fr)"}
Jul 23 23:55:03 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 367M@4MBps (emf.fr)"}
Jul 23 23:55:13 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 410M@4MBps (emf.fr)"}
Jul 23 23:55:23 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 451M@4MBps (emf.fr)"}
Jul 23 23:55:33 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 498M@5MBps (emf.fr)"}
Jul 23 23:55:43 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Still uploading backup (120s, 540M) (emf.fr)"}
Jul 23 23:55:53 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 620M@8MBps (emf.fr)"}
Jul 23 23:56:03 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 713M@9MBps (emf.fr)"}
Jul 23 23:56:13 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 736M@2MBps (emf.fr)"}
Jul 23 23:56:23 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 761M@2MBps (emf.fr)"}
Jul 23 23:56:33 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 834M@7MBps (emf.fr)"}
Jul 23 23:56:43 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Still uploading backup (180s, 964M) (emf.fr)"}
Jul 23 23:56:53 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 1061M@10MBps (emf.fr)"}
Jul 23 23:57:03 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 1229M@17MBps (emf.fr)"}
Jul 23 23:57:13 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 1420M@19MBps (emf.fr)"}
Jul 23 23:57:23 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 1538M@12MBps (emf.fr)"}
Jul 23 23:57:33 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 1677M@14MBps (emf.fr)"}
Jul 23 23:57:43 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Still uploading backup (240s, 1840M) (emf.fr)"}
Jul 23 23:57:53 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 2016M@17MBps (emf.fr)"}
Jul 23 23:58:03 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 2119M@10MBps (emf.fr)"}
Jul 23 23:58:13 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 2299M@18MBps (emf.fr)"}
Jul 23 23:58:23 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 2459M@16MBps (emf.fr)"}
Jul 23 23:58:33 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 2567M@11MBps (emf.fr)"}
Jul 23 23:58:43 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Still uploading backup (300s, 2714M) (emf.fr)"}
Jul 23 23:58:53 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 2875M@16MBps (emf.fr)"}
Jul 23 23:59:04 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 3072M@20MBps (emf.fr)"}
Jul 23 23:59:14 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 3254M@18MBps (emf.fr)"}
Jul 23 23:59:24 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 3426M@17MBps (emf.fr)"}
Jul 23 23:59:34 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 3584M@16MBps (emf.fr)"}
Jul 23 23:59:43 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Still uploading backup (360s, 3718M) (emf.fr)"}
Jul 23 23:59:54 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 3923M@17MBps (emf.fr)"}
Jul 24 00:00:04 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 4089M@17MBps (emf.fr)"}
Jul 24 00:00:14 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 4181M@9MBps (emf.fr)"}
Jul 24 00:00:24 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 4349M@17MBps (emf.fr)"}
Jul 24 00:00:34 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 4514M@16MBps (emf.fr)"}
Jul 24 00:00:44 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Still uploading backup (421s, 4611M) (emf.fr)"}
Jul 24 00:00:54 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 4778M@16MBps (emf.fr)"}
Jul 24 00:01:04 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 4953M@17MBps (emf.fr)"}
Jul 24 00:01:16 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 5120M@17MBps (emf.fr)"}
Jul 24 00:01:26 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 5252M@13MBps (emf.fr)"}
Jul 24 00:01:36 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 5436M@18MBps (emf.fr)"}
Jul 24 00:01:44 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Still uploading backup (481s, 5609M) (emf.fr)"}
Jul 24 00:01:57 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 5779M@15MBps (emf.fr)"}
Jul 24 00:02:07 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 5966M@19MBps (emf.fr)"}
Jul 24 00:02:18 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 6144M@18MBps (emf.fr)"}
Jul 24 00:02:28 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 6315M@17MBps (emf.fr)"}
Jul 24 00:02:38 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 6498M@18MBps (emf.fr)"}
Jul 24 00:02:44 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Still uploading backup (541s, 6596M) (emf.fr)"}
Jul 24 00:02:59 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 6812M@16MBps (emf.fr)"}
Jul 24 00:03:09 tasks: updating task 27695 with: {"percent":61.606060606060616,"message":"Uploading backup 6992M@18MBps (emf.fr)"}
Jul 24 00:03:20 backupformat/tgz: addToPack: error adding ./data/wp-content/uploads/2021/07/AFF9AEV-BD-SCIENCES-1400x1980.jpg file The operation was aborted
Jul 24 00:03:20 backupformat/tgz: BoxError: The operation was aborted
    at file:///home/yellowtent/box/src/backupformat/tgz.js:67:24
    at Sink._continuePack (/home/yellowtent/box/node_modules/tar-stream/pack.js:42:5)
    at Sink._destroy (/home/yellowtent/box/node_modules/tar-stream/pack.js:121:10)
    at WritableState.updateNonPrimary (/home/yellowtent/box/node_modules/streamx/index.js:213:16)
    at WritableState.update (/home/yellowtent/box/node_modules/streamx/index.js:195:72)
    at WritableState.updateWriteNT (/home/yellowtent/box/node_modules/streamx/index.js:564:10)
    at node:internal/process/task_queues:149:7
    at AsyncResource.runInAsyncScope (node:async_hooks:214:14)
    at AsyncResource.runMicrotask (node:internal/process/task_queues:146:8)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5) {
  reason: 'FileSystem Error',
  details: {}
}
Jul 24 00:03:20 backupformat/tgz: tarPack: packed 0 files
Jul 24 00:03:20 backupformat/tgz: Attempt 1 failed. Will retry: tarPack pipeline error: The operation was aborted 
Jul 24 00:03:20 backupformat/tgz: AbortError: The operation was aborted
    at Object.destroyer (node:internal/streams/destroy:328:11)
    at createAsyncIterator (node:internal/streams/readable:1404:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5) {
  code: 'ABORT_ERR'
}
Jan 01 00:09:21 node:internal/process/promises:394
Jan 01 00:09:21     triggerUncaughtException(err, true /* fromPromise */);
Jan 01 00:09:21     ^
Jan 01 00:09:21 
Jan 01 00:09:21 Error: getaddrinfo EAI_AGAIN cloudron.s3.fr-par.scw.cloud
Jan 01 00:09:21     at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:122:26) {
Jan 01 00:09:21   errno: -3001,
Jan 01 00:09:21   code: 'EAI_AGAIN',
Jan 01 00:09:21   syscall: 'getaddrinfo',
Jan 01 00:09:21   hostname: 'cloudron.s3.fr-par.scw.cloud',
Jan 01 00:09:21   '$metadata': { attempts: 1, totalRetryDelay: 0 }
Jan 01 00:09:21 }
Jan 01 00:09:21 
Jan 01 00:09:21 Node.js v24.13.0
Jul 24 00:03:30 shell: backuptask: /usr/bin/sudo --non-interactive -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_aa1ac9b6-3ba1-46f6-b4dc-fcfd3fe0b8a6.tar.gz ecbe2048-0622-4fe7-ab86-0a2712ae1c2b {"localRoot":"/home/yellowtent/appsdata/aa1ac9b6-3ba1-46f6-b4dc-fcfd3fe0b8a6","layout":[]} errored with code 1 and signal null timeout false terminated false - stdout: "" - stderr: ""
Jul 24 00:03:30 backuptask: runBackupUpload: backuptask crashed BoxError: /usr/bin/sudo exited with code 1 signal null
    at ChildProcess.<anonymous> (file:///home/yellowtent/box/src/shell.js:70:23)
    at ChildProcess.emit (node:events:508:28)
    at maybeClose (node:internal/child_process:1101:16)
    at ChildProcess._handle.onexit (node:internal/child_process:305:5) {
  reason: 'Shell Error',
  details: {},
  stdout: '',
  stdoutString: '',
  stdoutLineCount: 0,
  stderr: '',
  stderrString: '',
  stderrLineCount: 0,
  code: 1,
  signal: null,
  timedOut: false,
  terminated: false
}
Jul 24 00:03:30 backuptask: fullBackup: app emf.fr backup finished. Took 836.467 seconds
Jul 24 00:03:30 locks: write: current locks: {"full_backup_task_ecbe2048-0622-4fe7-ab86-0a2712ae1c2b":null}
Jul 24 00:03:30 locks: release: app_backup_aa1ac9b6-3ba1-46f6-b4dc-fcfd3fe0b8a6
Jul 24 00:03:30 tasks: setCompleted - 27695: {"result":null,"error":{"message":"Backuptask crashed","reason":"Internal Error"},"percent":100}
Jul 24 00:03:30 tasks: updating task 27695 with: {"completed":true,"result":null,"error":{"message":"Backuptask crashed","reason":"Internal Error"},"percent":100}
Jul 24 00:03:30 taskworker: Task took 3808.784 seconds
Jul 24 00:03:30 BoxError: Backuptask crashed
    at runBackupUpload (file:///home/yellowtent/box/src/backuptask.js:190:15)
    at process.processTicksAndRejections (node:internal/process/task_queues:103:5)
    at async uploadAppSnapshot (file:///home/yellowtent/box/src/backuptask.js:359:34)
    at async backupAppWithTag (file:///home/yellowtent/box/src/backuptask.js:382:26)
Jul 24 00:03:30 Exiting with code 0

[james]

Hello @jeau
The following is an assumption.
If only the automated backup fails, at night, in this specific time frame could it be that the remote is simply overloaded and thus causing errors?
Since 00:00 o'clock is a widely used time slot for creating backups maybe shifting the time to e.g.: 03:00 o'clock might help.
Something similar also happened to Let's Encrypt with hard coded times for certificate renewals created a DDoS like behaviour impacting their service negatively.

Maybe worth a try?

[joseph]

Can you check how the DNS resolution is set up on your server? The default set up should use systemd-resolved . What nameservers are in /etc/resolve.conf (should be 127.0.0.53) . And then check /etc/systemd/resolved.conf and also resolvctl output

[jeau]

hi @james, hi @joseph

I’m going to change the schedule, but I don’t think that’s the problem – I’ve already tried different ones.

Here are the results of the commands:

client_XX@YZ:~$ cat /etc/resolv.conf
...

nameserver 127.0.0.53
options edns0 trust-ad
search netadmin.local

client_XX@YZ:~$ cat /etc/systemd/resolved.conf
...
[Resolve]
# Some examples of DNS servers which may be used for DNS= and FallbackDNS=:
# Cloudflare: 1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com
# Google:     8.8.8.8#dns.google 8.8.4.4#dns.google 2001:4860:4860::8888#dns.google 2001:4860:4860::8844#dns.google
# Quad9:      9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net
#DNS=
#FallbackDNS=
#Domains=
#DNSSEC=no
#DNSOverTLS=no
#MulticastDNS=no
#LLMNR=no
#Cache=no-negative
#CacheFromLocalhost=no
#DNSStubListener=yes
#DNSStubListenerExtra=
#ReadEtcHosts=yes
#ResolveUnicastSingleLabel=no

client_XX@YZ:~$ resolvectl
Global
       Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
resolv.conf mode: stub

Link 2 (eth0)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 109.69.184.202
       DNS Servers: 109.69.184.202 185.73.52.130
        DNS Domain: netadmin.local

Link 3 (ens19)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 4 (br-20b2daa3116f)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 5 (docker0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 7 (veth60f4143)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 9 (vethaa60690)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 10 (veth8a31d71)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 13 (vethaea343a)