Aws Backup error : EPIPE HTTP Code : NetworkingError

[CarbonBee]

Hi!

I recently setup my Cloudron server to backup with aws s3 everyday. The last few days, the backup ended when trying to upload my nextcloud backup (which weights more than 100GB).
The logs report the following (for security purpose, I removed the domain name, the application ID and the aws id) :

Oct 20 03:29:26 box:tasks 5308: {"percent":87.66666666666667,"message":"Uploading backup 392M@1MBps (cloud.mydomain.fr)"}
Oct 20 03:29:37 box:tasks 5308: {"percent":87.66666666666667,"message":"Uploading backup 104M@0MBps (cloud.mydomain.fr)"}
Oct 20 03:30:41 box:tasks 5308: {"percent":87.66666666666667,"message":"Uploading backup 392M@1MBps (cloud.mydomain.fr)"}
Oct 20 03:30:55 box:tasks 5308: {"percent":87.66666666666667,"message":"Uploading backup 1404M@2MBps (cloud.mydomain.fr)"}
Oct 20 03:34:06 box:tasks 5308: {"percent":87.66666666666667,"message":"Uploading backup 104M@0MBps (cloud.mydomain.fr)"}
Oct 20 03:34:15 box:tasks 5308: {"percent":87.66666666666667,"message":"Uploading backup 877M@1MBps (cloud.mydomain.fr)"}
Oct 20 03:34:37 box:tasks 5308: {"percent":87.66666666666667,"message":"Uploading backup 104M@0MBps (cloud.mydomain.fr)"}
Oct 20 03:37:31 box:shell backup-snapshot/app_<appId> (stdout): 2020-10-20T01:37:18.907Z box:storage/s3 Error uploading [snapshot/app_<appId>.tar.gz.enc]: s3 upload error. { Error: write EPIPE
at WriteWrap.afterWrite [as oncomplete] (net.js:789:14)
message: 'write EPIPE',
errno: 'EPIPE',
code: 'NetworkingError',
syscall: 'write',
region: 'eu-west-3',
hostname: '<s3Id>.s3.eu-west-3.amazonaws.com',
retryable: true,
time: 2020-10-20T01:36:48.195Z }
Oct 20 03:37:19 box:backupupload upload completed. error: { BoxError: Error uploading snapshot/app_<appId>.tar.gz.enc. Message: write EPIPE HTTP Code: NetworkingError
at ManagedUpload.callback (/home/yellowtent/box/src/storage/s3.js:130:33)
at ManagedUpload.cleanup (/home/yellowtent/box/node_modules/aws-sdk/lib/s3/managed_upload.js:629:10)
at Response.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/s3/managed_upload.js:566:28)
at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:364:18)
at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:683:14)
at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10
name: 'BoxError',
reason: 'External Error',
details: {},
message:
'Error uploading snapshot/app_<appId>.tar.gz.enc. Message: write EPIPE HTTP Code: NetworkingError' }
Oct 20 03:37:41 box:backups runBackupUpload: result - {"result":"Error uploading snapshot/app_<appId>.tar.gz.enc. Message: write EPIPE HTTP Code: NetworkingError"}
Oct 20 03:39:26 box:shell backup-snapshot/app_<appId> code: 50, signal: null
Oct 20 03:39:30 box:backups cloud.mydomain.fr Unable to backup { BoxError: Error uploading snapshot/app_<appId>.tar.gz.enc. Message: write EPIPE HTTP Code: NetworkingError
at /home/yellowtent/box/src/backups.js:863:29
at f (/home/yellowtent/box/node_modules/once/once.js:25:25)
at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)
at ChildProcess.emit (events.js:198:13)
at ChildProcess.EventEmitter.emit (domain.js:448:20)
at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)
name: 'BoxError',
reason: 'External Error',
details: {},
message:
'Error uploading snapshot/app_<appId>.tar.gz.enc. Message: write EPIPE HTTP Code: NetworkingError' }
Oct 20 03:39:31 box:taskworker Task took 16766.54 seconds
Oct 20 03:39:31 box:tasks setCompleted - 5308: {"result":null,"error":{"stack":"BoxError: Error uploading snapshot/app_<appId>.tar.gz.enc. Message: write EPIPE HTTP Code: NetworkingError\n at /home/yellowtent/box/src/backups.js:863:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:198:13)\n at ChildProcess.EventEmitter.emit (domain.js:448:20)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"External Error","details":{},"message":"Error uploading snapshot/app_<appId>.tar.gz.enc. Message: write EPIPE HTTP Code: NetworkingError"}}
Oct 20 03:39:32 box:tasks 5308: {"percent":100,"result":null,"error":{"stack":"BoxError: Error uploading snapshot/app_<appId>.tar.gz.enc. Message: write EPIPE HTTP Code: NetworkingError\n at /home/yellowtent/box/src/backups.js:863:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:198:13)\n at ChildProcess.EventEmitter.emit (domain.js:448:20)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:248:12)","name":"BoxError","reason":"External Error","details":{},"message":"Error uploading snapshot/app_<appId>.tar.gz.enc. Message: write EPIPE HTTP Code: NetworkingError"}}

I tried with 10MB and 100MB upload part size, but this does not seems to change anything.
Do you have any idea on what's going on?

Thanks!

[girish]

The issue was that we had a 5min upload timeout for each chunk in code. Removing the timeout made it work. Fix is in the next release.

[girish]

@CarbonBee Are you able to reproduce it all the time/frequently? It seems maybe the network on the server fluctuated.

[CarbonBee]

It's always doing that. I haven't been able to make a full backup for days. This begins to be critical.

[girish]

@CarbonBee Atleast the error seems to indicate that it's a networking error. Could it be that your VPS provider is throttling uploads of 100GB ?

[CarbonBee]

@girish I tried to backup only the nextcloud, which generated the same error.
I have no idea on any upload limitation. My server is hosted by Kimsufi, an OVH discounter.

[CarbonBee]

Well, a technician from Kimsufi just told me that there are no upload limit on my server. I don't really understand what's going on, because on more or less the same setup on a different server, it works fine.
Is there a way to track it down and find out where the issue come from?

[girish]

@CarbonBee I can't make out from the error what the error is. Are you able to contact us at support@cloudron.io and maybe give us ssh access?

[girish]

The issue was that we had a 5min upload timeout for each chunk in code. Removing the timeout made it work. Fix is in the next release.

[jordanurbs]

@girish is this fix and next release out yet? I have been consistently getting this error as of the last 2-3 weeks or so, out of the blue.

[girish]

@jordanurbs Yes, this is in 6.0.1. Are you in 6.0.1 ? https://git.cloudron.io/cloudron/box/-/commit/bedcd6fccf58830b316318699375bc1f582a5d7a has the fix. Do you have any error logs?