Backups failing with ERR_STREAM_WRITE_AFTER_END

[Peter Newman]

For the last 5 days, every backup is failing with ERR_STREAM_WRITE_AFTER_END. We are using the Linode backend, and are in communication with them as well to find out if anything changed on their end.

(stdout): 2021-03-21T06:14:46.894Z box:storage/s3 Error uploading [cloudron/snapshot/app_0443482b-6d44-49a5-b02c-92412307a711.tar.gz.enc]: s3 upload error. Error [ERR_STREAM_WRITE_AFTER_END]: write after end
at writeAfterEnd (_http_outgoing.js:668:15)
at ClientRequest.end (_http_outgoing.js:788:7)
at features.constructor.writeBody (/home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:137:14)
at ClientRequest.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:102:14)
at ClientRequest.emit (events.js:315:20)
at ClientRequest.EventEmitter.emit (domain.js:467:12)
at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:606:11)
at HTTPParser.parserOnHeadersComplete (_http_common.js:126:17)
at TLSSocket.socketOnData (_http_client.js:509:22)
at TLSSocket.emit (events.js:315:20) {
code: 'NetworkingError',
region: 'us-east-1',
hostname: '[redacted].us-east-1.linodeobjects.com',
retryable: true,
time: 2021-03-21T06:14:46.873Z

[nebulon]

@peter-newman this is maybe the same as already reported https://git.cloudron.io/cloudron/box/-/issues/779

[girish]

@peter-newman is this with linode object storage? I am trying to figure what the correct fix is with the upstream authors at https://github.com/aws/aws-sdk-js/issues/3404

[girish]

Alright, I found the issue.

Linode Object Storage has some strange behavior with Expect: 101-continue. You can see a more detailed analysis at https://www.linode.com/community/questions/21180/object-storage-upload-err_stream_write_after_end-write-after-end and https://github.com/aws/aws-sdk-js/issues/3404#issuecomment-805448190

At this point, I need either aws-js-sdk or linode guys to tell me who is wrong

[girish]

I submitted a PR at https://github.com/aws/aws-sdk-js/pull/3674

[Peter Newman]

@girish Ah, great - I'm one of the comments on that thread as well. That's an interesting problem.

[girish]

@peter-newman As a workaround, you can try this:

# cd /home/yellowtent/box
# sudo -u yellowtent npm install git+https://github.com/cloudron-io/aws-sdk-js.git#continue_once
# systemctl restart box

I think it will take a while for AWS or Linode to sort out the issue. If you try the workaround, let me know if it makes things work.

[girish]

I opened a linode support ticket #15378267 if anyone wants to take this up with linode.

[Peter Newman]

@girish said in Backups failing with ERR_STREAM_WRITE_AFTER_END:

npm install git+https://github.com/cloudron-io/aws-sdk-js.git#continue_once

This has worked for me.

Just as a note, I had to su yellowtent rather than sudo, as NPM kept complaining about root owned cache files (since HOME was still pointing at /root/). Just in case anyone else needs to follow these instructions.

[girish]

AWS team merged the PR https://github.com/aws/aws-sdk-js/pull/3674 . So, linode object storage should get fixed next release (6.3).

[girish]

For completeness, the working instructions is:

# cd /home/yellowtent/box
# sudo -u yellowtent npm install aws-sdk@2.879.0
# systemctl restart box