Backups failing with ERR_STREAM_WRITE_AFTER_END
-
For the last 5 days, every backup is failing with ERR_STREAM_WRITE_AFTER_END. We are using the Linode backend, and are in communication with them as well to find out if anything changed on their end.
(stdout): 2021-03-21T06:14:46.894Z box:storage/s3 Error uploading [cloudron/snapshot/app_0443482b-6d44-49a5-b02c-92412307a711.tar.gz.enc]: s3 upload error. Error [ERR_STREAM_WRITE_AFTER_END]: write after end at writeAfterEnd (_http_outgoing.js:668:15) at ClientRequest.end (_http_outgoing.js:788:7) at features.constructor.writeBody (/home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:137:14) at ClientRequest.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:102:14) at ClientRequest.emit (events.js:315:20) at ClientRequest.EventEmitter.emit (domain.js:467:12) at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:606:11) at HTTPParser.parserOnHeadersComplete (_http_common.js:126:17) at TLSSocket.socketOnData (_http_client.js:509:22) at TLSSocket.emit (events.js:315:20) { code: 'NetworkingError', region: 'us-east-1', hostname: '[redacted].us-east-1.linodeobjects.com', retryable: true, time: 2021-03-21T06:14:46.873Z@peter-newman this is maybe the same as already reported https://git.cloudron.io/cloudron/box/-/issues/779
-
For the last 5 days, every backup is failing with ERR_STREAM_WRITE_AFTER_END. We are using the Linode backend, and are in communication with them as well to find out if anything changed on their end.
(stdout): 2021-03-21T06:14:46.894Z box:storage/s3 Error uploading [cloudron/snapshot/app_0443482b-6d44-49a5-b02c-92412307a711.tar.gz.enc]: s3 upload error. Error [ERR_STREAM_WRITE_AFTER_END]: write after end at writeAfterEnd (_http_outgoing.js:668:15) at ClientRequest.end (_http_outgoing.js:788:7) at features.constructor.writeBody (/home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:137:14) at ClientRequest.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:102:14) at ClientRequest.emit (events.js:315:20) at ClientRequest.EventEmitter.emit (domain.js:467:12) at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:606:11) at HTTPParser.parserOnHeadersComplete (_http_common.js:126:17) at TLSSocket.socketOnData (_http_client.js:509:22) at TLSSocket.emit (events.js:315:20) { code: 'NetworkingError', region: 'us-east-1', hostname: '[redacted].us-east-1.linodeobjects.com', retryable: true, time: 2021-03-21T06:14:46.873Z@peter-newman is this with linode object storage? I am trying to figure what the correct fix is with the upstream authors at https://github.com/aws/aws-sdk-js/issues/3404
-
Alright, I found the issue.
Linode Object Storage has some strange behavior with
Expect: 101-continue. You can see a more detailed analysis at https://www.linode.com/community/questions/21180/object-storage-upload-err_stream_write_after_end-write-after-end and https://github.com/aws/aws-sdk-js/issues/3404#issuecomment-805448190At this point, I need either aws-js-sdk or linode guys to tell me who is wrong
-
Alright, I found the issue.
Linode Object Storage has some strange behavior with
Expect: 101-continue. You can see a more detailed analysis at https://www.linode.com/community/questions/21180/object-storage-upload-err_stream_write_after_end-write-after-end and https://github.com/aws/aws-sdk-js/issues/3404#issuecomment-805448190At this point, I need either aws-js-sdk or linode guys to tell me who is wrong
-
@girish Ah, great - I'm one of the comments on that thread as well. That's an interesting problem.
@peter-newman As a workaround, you can try this:
# cd /home/yellowtent/box # sudo -u yellowtent npm install git+https://github.com/cloudron-io/aws-sdk-js.git#continue_once # systemctl restart boxI think it will take a while for AWS or Linode to sort out the issue. If you try the workaround, let me know if it makes things work.
-
@peter-newman As a workaround, you can try this:
# cd /home/yellowtent/box # sudo -u yellowtent npm install git+https://github.com/cloudron-io/aws-sdk-js.git#continue_once # systemctl restart boxI think it will take a while for AWS or Linode to sort out the issue. If you try the workaround, let me know if it makes things work.
@girish said in Backups failing with ERR_STREAM_WRITE_AFTER_END:
npm install git+https://github.com/cloudron-io/aws-sdk-js.git#continue_once
This has worked for me.
Just as a note, I had to
su yellowtentrather thansudo, as NPM kept complaining about root owned cache files (sinceHOMEwas still pointing at/root/). Just in case anyone else needs to follow these instructions. -
@girish said in Backups failing with ERR_STREAM_WRITE_AFTER_END:
npm install git+https://github.com/cloudron-io/aws-sdk-js.git#continue_once
This has worked for me.
Just as a note, I had to
su yellowtentrather thansudo, as NPM kept complaining about root owned cache files (sinceHOMEwas still pointing at/root/). Just in case anyone else needs to follow these instructions.
