Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Solved Backups failing with ERR_STREAM_WRITE_AFTER_END

    Support
    backups
    3
    11
    430
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • P
      Peter Newman last edited by girish

      For the last 5 days, every backup is failing with ERR_STREAM_WRITE_AFTER_END. We are using the Linode backend, and are in communication with them as well to find out if anything changed on their end.

      (stdout): 2021-03-21T06:14:46.894Z box:storage/s3 Error uploading [cloudron/snapshot/app_0443482b-6d44-49a5-b02c-92412307a711.tar.gz.enc]: s3 upload error. Error [ERR_STREAM_WRITE_AFTER_END]: write after end
      at writeAfterEnd (_http_outgoing.js:668:15)
      at ClientRequest.end (_http_outgoing.js:788:7)
      at features.constructor.writeBody (/home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:137:14)
      at ClientRequest.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:102:14)
      at ClientRequest.emit (events.js:315:20)
      at ClientRequest.EventEmitter.emit (domain.js:467:12)
      at HTTPParser.parserOnIncomingClient [as onIncoming] (_http_client.js:606:11)
      at HTTPParser.parserOnHeadersComplete (_http_common.js:126:17)
      at TLSSocket.socketOnData (_http_client.js:509:22)
      at TLSSocket.emit (events.js:315:20) {
      code: 'NetworkingError',
      region: 'us-east-1',
      hostname: '[redacted].us-east-1.linodeobjects.com',
      retryable: true,
      time: 2021-03-21T06:14:46.873Z
      
      nebulon girish 2 Replies Last reply Reply Quote 0
      • nebulon
        nebulon Staff @Peter Newman last edited by

        @peter-newman this is maybe the same as already reported https://git.cloudron.io/cloudron/box/-/issues/779

        1 Reply Last reply Reply Quote 0
        • girish
          girish Staff @Peter Newman last edited by

          @peter-newman is this with linode object storage? I am trying to figure what the correct fix is with the upstream authors at https://github.com/aws/aws-sdk-js/issues/3404

          1 Reply Last reply Reply Quote 0
          • girish
            girish Staff last edited by

            Alright, I found the issue.

            Linode Object Storage has some strange behavior with Expect: 101-continue. You can see a more detailed analysis at https://www.linode.com/community/questions/21180/object-storage-upload-err_stream_write_after_end-write-after-end and https://github.com/aws/aws-sdk-js/issues/3404#issuecomment-805448190

            At this point, I need either aws-js-sdk or linode guys to tell me who is wrong 😉

            P 1 Reply Last reply Reply Quote 0
            • girish
              girish Staff last edited by

              I submitted a PR at https://github.com/aws/aws-sdk-js/pull/3674

              1 Reply Last reply Reply Quote 0
              • P
                Peter Newman @girish last edited by

                @girish Ah, great - I'm one of the comments on that thread as well. That's an interesting problem.

                girish 1 Reply Last reply Reply Quote 0
                • girish
                  girish Staff @Peter Newman last edited by girish

                  @peter-newman As a workaround, you can try this:

                  # cd /home/yellowtent/box
                  # sudo -u yellowtent npm install git+https://github.com/cloudron-io/aws-sdk-js.git#continue_once
                  # systemctl restart box
                  

                  I think it will take a while for AWS or Linode to sort out the issue. If you try the workaround, let me know if it makes things work.

                  P 1 Reply Last reply Reply Quote 0
                  • girish
                    girish Staff last edited by

                    I opened a linode support ticket #15378267 if anyone wants to take this up with linode.

                    1 Reply Last reply Reply Quote 0
                    • P
                      Peter Newman @girish last edited by Peter Newman

                      @girish said in Backups failing with ERR_STREAM_WRITE_AFTER_END:

                      npm install git+https://github.com/cloudron-io/aws-sdk-js.git#continue_once

                      This has worked for me.

                      Just as a note, I had to su yellowtent rather than sudo, as NPM kept complaining about root owned cache files (since HOME was still pointing at /root/). Just in case anyone else needs to follow these instructions.

                      girish 1 Reply Last reply Reply Quote 1
                      • girish
                        girish Staff @Peter Newman last edited by

                        AWS team merged the PR https://github.com/aws/aws-sdk-js/pull/3674 . So, linode object storage should get fixed next release (6.3).

                        1 Reply Last reply Reply Quote 0
                        • girish
                          girish Staff last edited by

                          For completeness, the working instructions is:

                          # cd /home/yellowtent/box
                          # sudo -u yellowtent npm install aws-sdk@2.879.0
                          # systemctl restart box
                          
                          1 Reply Last reply Reply Quote 3
                          • First post
                            Last post
                          Powered by NodeBB