Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Navigation

    Cloudron Forum
    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Minio Backup not working

    Support
    backups minio
    3
    9
    87
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • S
      savity last edited by girish

      Hi Cloudron Team
      since the last Update i having issues with backuping my apps to my Minio.
      Pleas find the Backup Logs from cloudron

      2021-03-30T00:03:48.564Z box:shell backup-snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839 (stdout): 2021-03-30T00:03:48.560Z box:storage/s3 Error uploading [myapplication/snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839.tar.gz.enc]: s3 upload error. Error [InternalError]: We encountered an internal error, please try again.: cause(open /home/pi/data/.minio.sys/multipart/57fac6043320d82c0dc8e5332bbf92a93d28feab70e7de8bfaf6a3ea333d99aa/74f2a374-0090-44b8-8126-1f35d785abb7/fs.json: operation not permitted)
      at Request.extractError (/home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:700:35)
      at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
      at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
      at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:14)
      at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)
      at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)
      at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10
      at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)
      at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:690:12)
      at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
      code: 'InternalError',
      region: null,
      time: 2021-03-30T00:03:48.551Z,
      requestId: '1670F5E49FD42D85',
      extendedRequestId: undefined,
      cfId: undefined,
      statusCode: 500,
      retryable: true
      }

      2021-03-30T00:11:38.029Z box:shell backup-snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839 (stdout): 2021-03-30T00:11:38.028Z box:backupupload v8 heap : used 49054720 total: 88043520 max: 4076863488

      2021-03-30T00:11:38.837Z box:tasks 3379: {"percent":51,"message":"Uploading backup 4867M@7MBps (myapplication.mydomain.com)"}
      2021-03-30T00:11:48.839Z box:tasks 3379: {"percent":51,"message":"Uploading backup 4956M@7MBps (myapplication.mydomain.com)"}
      2021-03-30T00:11:53.230Z box:shell backup-snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839 (stdout): 2021-03-30T00:11:53.228Z box:storage/s3 Error uploading [myapplication/snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839.tar.gz.enc]: s3 upload error. Error [InternalError]: We encountered an internal error, please try again.: cause(open /home/pi/data/.minio.sys/multipart/57fac6043320d82c0dc8e5332bbf92a93d28feab70e7de8bfaf6a3ea333d99aa/00ed00d6-0f64-4e24-b894-227bf502c053/fs.json: operation not permitted)
      at Request.extractError (/home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:700:35)
      at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
      at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
      at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:14)
      at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)
      at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)
      at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10
      at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)
      at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:690:12)
      at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
      code: 'InternalError',
      region: null,
      time: 2021-03-30T00:11:53.216Z,
      requestId: '1670F65577EB59C2',
      extendedRequestId: undefined,
      cfId: undefined,
      statusCode: 500,
      retryable: true
      }

      2021-03-30T00:20:11.937Z box:backups runBackupUpload: result - {"result":"Error uploading myapplication/snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839.tar.gz.enc. Message: We encountered an internal error, please try again.: cause(open /home/pi/data/.minio.sys/multipart/57fac6043320d82c0dc8e5332bbf92a93d28feab70e7de8bfaf6a3ea333d99aa/d279db31-dc24-4d74-959a-eb592d7915e0/fs.json: operation not permitted) HTTP Code: InternalError"}
      2021-03-30T00:20:12.008Z box:shell backup-snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839 code: 50, signal: null
      2021-03-30T00:20:12.010Z box:backups myapplication.mydomain.com Unable to backup BoxError: Error uploading myapplication/snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839.tar.gz.enc. Message: We encountered an internal error, please try again.: cause(open /home/pi/data/.minio.sys/multipart/57fac6043320d82c0dc8e5332bbf92a93d28feab70e7de8bfaf6a3ea333d99aa/d279db31-dc24-4d74-959a-eb592d7915e0/fs.json: operation not permitted) HTTP Code: InternalError
      at /home/yellowtent/box/src/backups.js:897:29
      at f (/home/yellowtent/box/node_modules/once/once.js:25:25)
      at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)
      at ChildProcess.emit (events.js:315:20)
      at ChildProcess.EventEmitter.emit (domain.js:467:12)
      at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12) {
      reason: 'External Error',
      details: {}
      }
      2021-03-30T00:20:12.011Z box:taskworker Task took 1211.065 seconds
      2021-03-30T00:20:12.011Z box:tasks setCompleted - 3379: {"result":null,"error":{"stack":"BoxError: Error uploading myapplication/snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839.tar.gz.enc. Message: We encountered an internal error, please try again.: cause(open /home/pi/data/.minio.sys/multipart/57fac6043320d82c0dc8e5332bbf92a93d28feab70e7de8bfaf6a3ea333d99aa/d279db31-dc24-4d74-959a-eb592d7915e0/fs.json: operation not permitted) HTTP Code: InternalError\n at /home/yellowtent/box/src/backups.js:897:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:315:20)\n at ChildProcess.EventEmitter.emit (domain.js:467:12)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)","name":"BoxError","reason":"External Error","details":{},"message":"Error uploading myapplication/snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839.tar.gz.enc. Message: We encountered an internal error, please try again.: cause(open /home/pi/data/.minio.sys/multipart/57fac6043320d82c0dc8e5332bbf92a93d28feab70e7de8bfaf6a3ea333d99aa/d279db31-dc24-4d74-959a-eb592d7915e0/fs.json: operation not permitted) HTTP Code: InternalError"}}
      2021-03-30T00:20:12.011Z box:tasks 3379: {"percent":100,"result":null,"error":{"stack":"BoxError: Error uploading myapplication/snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839.tar.gz.enc. Message: We encountered an internal error, please try again.: cause(open /home/pi/data/.minio.sys/multipart/57fac6043320d82c0dc8e5332bbf92a93d28feab70e7de8bfaf6a3ea333d99aa/d279db31-dc24-4d74-959a-eb592d7915e0/fs.json: operation not permitted) HTTP Code: InternalError\n at /home/yellowtent/box/src/backups.js:897:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:315:20)\n at ChildProcess.EventEmitter.emit (domain.js:467:12)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)","name":"BoxError","reason":"External Error","details":{},"message":"Error uploading myapplication/snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839.tar.gz.enc. Message: We encountered an internal error, please try again.: cause(open /home/pi/data/.minio.sys/multipart/57fac6043320d82c0dc8e5332bbf92a93d28feab70e7de8bfaf6a3ea333d99aa/d279db31-dc24-4d74-959a-eb592d7915e0/fs.json: operation not permitted) HTTP Code: InternalError"}

      S 1 Reply Last reply Reply Quote 0
      • S
        savity @savity last edited by

        @savity And intressting is i got data uploaded on my minio

        Its not all my data because it should be around 50GB

        88368fe8-57fb-4fac-b6f7-7210c1d726ef-grafik.png
        Snapshot:
        06ff9dfb-3483-44af-a6f2-ac9db421b491-grafik.png

        1 Reply Last reply Reply Quote 0
        • nebulon
          nebulon Staff last edited by

          From the error message, this looks like an issue encountered in the minio app itself. Please check if that instance has enough free space and also see if retrying the backup works and if it was only a temporary issue.

          S 2 Replies Last reply Reply Quote 0
          • S
            savity @nebulon last edited by

            @nebulon Got the Issues now 4 times a Row it got 1TB free i will check the logs

            1 Reply Last reply Reply Quote 0
            • S
              savity @nebulon last edited by

              @nebulon any clue on thath idk what it means sorry

              minio log:

              API: PutObjectPart(bucket=mybucket, object=myapp/snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839.tar.gz.enc)
              Time: 00:20:11 UTC 03/30/2021
              DeploymentID: d5aab97f-1878-40a6-a688-edf3b50b4960
              RequestID: 1670F6C994C249CD
              RemoteHost: 2.56.98.236
              UserAgent: aws-sdk-nodejs/2.850.0 linux/v14.15.4 callback
              Error: open /home/pi/data/.minio.sys/multipart/57fac6043320d82c0dc8e5332bbf92a93d28feab70e7de8bfaf6a3ea333d99aa/d279db31-dc24-4d74-959a-eb592d7915e0/fs.json: operation not permitted (*fs.PathError)
              4: cmd/api-errors.go:2009:cmd.toAPIErrorCode()
              3: cmd/api-errors.go:2034:cmd.toAPIError()
              2: cmd/object-handlers.go:2272:cmd.objectAPIHandlers.PutObjectPartHandler()
              1: net/http/server.go:2069:http.HandlerFunc.ServeHTTP()

              S 1 Reply Last reply Reply Quote 0
              • S
                savity @savity last edited by

                @nebulon I got the newest version and it looks like the issues is by uploading multiple files with gives a 500 back.

                Version:2021-03-26T00:00:41Z
                Version: go1.16.2 | CPUs: 4

                girish 1 Reply Last reply Reply Quote 0
                • girish
                  girish Staff @savity last edited by

                  @savity Are you running minio on the PI? (seems so from the path). Are you able to reproduce this issue when running minio on x86/some VPS? It seems to be something file system related with minio. Seems to me that some fs operation is not working with multipart uploads. Though it doesn't explain what the error is.

                  Maybe you can ask their issue tracker about it.

                  S 1 Reply Last reply Reply Quote 0
                  • girish
                    girish Staff last edited by

                    Trying to debug the backtrace a bit for fun:

                    https://github.com/minio/minio/blob/RELEASE.2021-03-26T00-00-41Z/cmd/object-handlers.go#L2272 seems to be last point of error. It seems somewhere in https://github.com/minio/minio/blob/7be71094710b966be624f10eeea25df02660493e/cmd/fs-v1-multipart.go#L369 failed

                    1 Reply Last reply Reply Quote 1
                    • S
                      savity @girish last edited by

                      @girish Yes it is a PI and i am always updates, i guess this is since the last minio update.
                      Sorry i dont have another vps 😄

                      1 Reply Last reply Reply Quote 0
                      • First post
                        Last post