Minio Backup not working

[savity]

Hi Cloudron Team
since the last Update i having issues with backuping my apps to my Minio.
Pleas find the Backup Logs from cloudron

2021-03-30T00:03:48.564Z box:shell backup-snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839 (stdout): 2021-03-30T00:03:48.560Z box:storage/s3 Error uploading [myapplication/snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839.tar.gz.enc]: s3 upload error. Error [InternalError]: We encountered an internal error, please try again.: cause(open /home/pi/data/.minio.sys/multipart/57fac6043320d82c0dc8e5332bbf92a93d28feab70e7de8bfaf6a3ea333d99aa/74f2a374-0090-44b8-8126-1f35d785abb7/fs.json: operation not permitted)
at Request.extractError (/home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:700:35)
at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:14)
at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:690:12)
at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
code: 'InternalError',
region: null,
time: 2021-03-30T00:03:48.551Z,
requestId: '1670F5E49FD42D85',
extendedRequestId: undefined,
cfId: undefined,
statusCode: 500,
retryable: true
}

2021-03-30T00:11:38.029Z box:shell backup-snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839 (stdout): 2021-03-30T00:11:38.028Z box:backupupload v8 heap : used 49054720 total: 88043520 max: 4076863488

2021-03-30T00:11:38.837Z box:tasks 3379: {"percent":51,"message":"Uploading backup 4867M@7MBps (myapplication.mydomain.com)"}
2021-03-30T00:11:48.839Z box:tasks 3379: {"percent":51,"message":"Uploading backup 4956M@7MBps (myapplication.mydomain.com)"}
2021-03-30T00:11:53.230Z box:shell backup-snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839 (stdout): 2021-03-30T00:11:53.228Z box:storage/s3 Error uploading [myapplication/snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839.tar.gz.enc]: s3 upload error. Error [InternalError]: We encountered an internal error, please try again.: cause(open /home/pi/data/.minio.sys/multipart/57fac6043320d82c0dc8e5332bbf92a93d28feab70e7de8bfaf6a3ea333d99aa/00ed00d6-0f64-4e24-b894-227bf502c053/fs.json: operation not permitted)
at Request.extractError (/home/yellowtent/box/node_modules/aws-sdk/lib/services/s3.js:700:35)
at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:14)
at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:690:12)
at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
code: 'InternalError',
region: null,
time: 2021-03-30T00:11:53.216Z,
requestId: '1670F65577EB59C2',
extendedRequestId: undefined,
cfId: undefined,
statusCode: 500,
retryable: true
}

2021-03-30T00:20:11.937Z box:backups runBackupUpload: result - {"result":"Error uploading myapplication/snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839.tar.gz.enc. Message: We encountered an internal error, please try again.: cause(open /home/pi/data/.minio.sys/multipart/57fac6043320d82c0dc8e5332bbf92a93d28feab70e7de8bfaf6a3ea333d99aa/d279db31-dc24-4d74-959a-eb592d7915e0/fs.json: operation not permitted) HTTP Code: InternalError"}
2021-03-30T00:20:12.008Z box:shell backup-snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839 code: 50, signal: null
2021-03-30T00:20:12.010Z box:backups myapplication.mydomain.com Unable to backup BoxError: Error uploading myapplication/snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839.tar.gz.enc. Message: We encountered an internal error, please try again.: cause(open /home/pi/data/.minio.sys/multipart/57fac6043320d82c0dc8e5332bbf92a93d28feab70e7de8bfaf6a3ea333d99aa/d279db31-dc24-4d74-959a-eb592d7915e0/fs.json: operation not permitted) HTTP Code: InternalError
at /home/yellowtent/box/src/backups.js:897:29
at f (/home/yellowtent/box/node_modules/once/once.js:25:25)
at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)
at ChildProcess.emit (events.js:315:20)
at ChildProcess.EventEmitter.emit (domain.js:467:12)
at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12) {
reason: 'External Error',
details: {}
}
2021-03-30T00:20:12.011Z box:taskworker Task took 1211.065 seconds
2021-03-30T00:20:12.011Z box:tasks setCompleted - 3379: {"result":null,"error":{"stack":"BoxError: Error uploading myapplication/snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839.tar.gz.enc. Message: We encountered an internal error, please try again.: cause(open /home/pi/data/.minio.sys/multipart/57fac6043320d82c0dc8e5332bbf92a93d28feab70e7de8bfaf6a3ea333d99aa/d279db31-dc24-4d74-959a-eb592d7915e0/fs.json: operation not permitted) HTTP Code: InternalError\n at /home/yellowtent/box/src/backups.js:897:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:315:20)\n at ChildProcess.EventEmitter.emit (domain.js:467:12)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)","name":"BoxError","reason":"External Error","details":{},"message":"Error uploading myapplication/snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839.tar.gz.enc. Message: We encountered an internal error, please try again.: cause(open /home/pi/data/.minio.sys/multipart/57fac6043320d82c0dc8e5332bbf92a93d28feab70e7de8bfaf6a3ea333d99aa/d279db31-dc24-4d74-959a-eb592d7915e0/fs.json: operation not permitted) HTTP Code: InternalError"}}
2021-03-30T00:20:12.011Z box:tasks 3379: {"percent":100,"result":null,"error":{"stack":"BoxError: Error uploading myapplication/snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839.tar.gz.enc. Message: We encountered an internal error, please try again.: cause(open /home/pi/data/.minio.sys/multipart/57fac6043320d82c0dc8e5332bbf92a93d28feab70e7de8bfaf6a3ea333d99aa/d279db31-dc24-4d74-959a-eb592d7915e0/fs.json: operation not permitted) HTTP Code: InternalError\n at /home/yellowtent/box/src/backups.js:897:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:315:20)\n at ChildProcess.EventEmitter.emit (domain.js:467:12)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)","name":"BoxError","reason":"External Error","details":{},"message":"Error uploading myapplication/snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839.tar.gz.enc. Message: We encountered an internal error, please try again.: cause(open /home/pi/data/.minio.sys/multipart/57fac6043320d82c0dc8e5332bbf92a93d28feab70e7de8bfaf6a3ea333d99aa/d279db31-dc24-4d74-959a-eb592d7915e0/fs.json: operation not permitted) HTTP Code: InternalError"}

[savity]

@savity And intressting is i got data uploaded on my minio

Its not all my data because it should be around 50GB

Snapshot:

[nebulon]

From the error message, this looks like an issue encountered in the minio app itself. Please check if that instance has enough free space and also see if retrying the backup works and if it was only a temporary issue.

[savity]

@nebulon Got the Issues now 4 times a Row it got 1TB free i will check the logs

[savity]

@nebulon any clue on thath idk what it means sorry

minio log:

API: PutObjectPart(bucket=mybucket, object=myapp/snapshot/app_5b67e84d-40da-4604-80ec-45e8d610c839.tar.gz.enc)
Time: 00:20:11 UTC 03/30/2021
DeploymentID: d5aab97f-1878-40a6-a688-edf3b50b4960
RequestID: 1670F6C994C249CD
RemoteHost: 2.56.98.236
UserAgent: aws-sdk-nodejs/2.850.0 linux/v14.15.4 callback
Error: open /home/pi/data/.minio.sys/multipart/57fac6043320d82c0dc8e5332bbf92a93d28feab70e7de8bfaf6a3ea333d99aa/d279db31-dc24-4d74-959a-eb592d7915e0/fs.json: operation not permitted (*fs.PathError)
4: cmd/api-errors.go:2009:cmd.toAPIErrorCode()
3: cmd/api-errors.go:2034:cmd.toAPIError()
2: cmd/object-handlers.go:2272:cmd.objectAPIHandlers.PutObjectPartHandler()
1: net/http/server.go:2069:http.HandlerFunc.ServeHTTP()

[savity]

@nebulon I got the newest version and it looks like the issues is by uploading multiple files with gives a 500 back.

Version:2021-03-26T00:00:41Z
Version: go1.16.2 | CPUs: 4

[girish]

@savity Are you running minio on the PI? (seems so from the path). Are you able to reproduce this issue when running minio on x86/some VPS? It seems to be something file system related with minio. Seems to me that some fs operation is not working with multipart uploads. Though it doesn't explain what the error is.

Maybe you can ask their issue tracker about it.

[girish]

Trying to debug the backtrace a bit for fun:

https://github.com/minio/minio/blob/RELEASE.2021-03-26T00-00-41Z/cmd/object-handlers.go#L2272 seems to be last point of error. It seems somewhere in https://github.com/minio/minio/blob/7be71094710b966be624f10eeea25df02660493e/cmd/fs-v1-multipart.go#L369 failed

[savity]

@girish Yes it is a PI and i am always updates, i guess this is since the last minio update.
Sorry i dont have another vps

[savity]

@girish said in Minio Backup not working:

Trying to debug the backtrace a bit for fun:
https://github.com/minio/minio/blob/RELEASE.2021-03-26T00-00-41Z/cmd/object-handlers.go#L2272 seems to be last point of error. It seems somewhere in https://github.com/minio/minio/blob/7be71094710b966be624f10eeea25df02660493e/cmd/fs-v1-multipart.go#L369 failed

Still have it in the newest Minio Version and Cloudron Version

Vaultwarden Backup Works, but Nextcloud not

[girish]

@savity is this rsync or tgz? If it's rsync, maybe try lowering all the limits in the Backups configure UI. Maybe the pi is unable to handle fast requests.

[robi]

Could this be a memory constraint issue on the Pi?

[CraigGoodspeed]

Random thought but check permissions on the folder(/home/pi/data/.minio.sys/multipart/57fac6043320d82c0dc8e5332bbf92a93d28feab70e7de8bfaf6a3ea333d99aa/74f2a374-0090-44b8-8126-1f35d785abb7/), I ran into the same issue today.

Turned out to be permissions issue.

To verify run minio with sudo, this is a quick fix and should not be left like this.

In my scenario i was running minio on a mount was formatted as NTFS and hence i could not set permissions on the disk.

[robi]

Perhaps there should be a permissions check script that puts warnings in the logs.

[savity]

@robi @CraigGoodspeed To laate After this here https://github.com/minio/minio/issues/12779 @girish
I decided to buy a synology make Raid change the Filesystem and do my backups on a Minio Docker.
Still don't get the answer thath it is the Harddrive why it was always possible making backups from "Vaultwarden" but not Nextcloud

[girish]

The upstream answer is good to know that minio won't work on FUSE.