Minio Client

[RoboMod]

Are there any plans to use the minio client for minio backups? I'm asking because I have trouble uploading a "larger" dataset of about 10 GB. The problem is that the backup always runs to a certain file because it uploads every single file again even if there is no change. The minio client provides a "mirror" function which would deal better with that using differential uploads.

[nebulon]

Never heard of the minio client mirror function, but the cloudron backup rsyncer would also only upload the changed files, however only once an initial full backup is finished completely, which probably is then not the case for you. Are there any errors reported for the backup?

[RoboMod]

@nebulon On Cloudron's side I get the error: 'Connection timed out after 300000ms HTTP Code: TimeoutError'. Looking on my minio server I get: 'msg="Unable to create object part." cause="#has incomplete body"' and 'msg="Failed to create object <filename>" cause="unexpected EOF"'.

[nebulon]

That looks like a bug itself and should be fixed instead of moving to some other syncing approach. Is there anything suspicious about this file (size, filename, ...)? Also does this happen always to the same file?

[RoboMod]

@nebulon It seems to happen on different files but till now always in the same directory. Right now, I'm running an app backup only of this nextcloud instance. I'll report back when this is finished.

[RoboMod]

Okay. So, now I let cloudron try to updated and thereby backup this nextcloud installation again and again. I always get the following error for many files:

2019-03-03T18:35:47.350Z box:shell backup-snapshot/app_836a8eb8-4f45-4099-aac5-83d6c25884c2 (stdout): 2019-03-03T18:35:47.348Z box:storage/s3 Error uploading [snapshot/app_836a8eb8-4f45-4099-aac5-83d6c25884c2/data/<path>]: s3 upload error. { TimeoutError: Connection timed out after 300000ms
    at ClientRequest.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:83:34)
    at Object.onceWrapper (events.js:313:30)
    at emitNone (events.js:106:13)
    at ClientRequest.emit (events.js:208:7)
    at Socket.emitTimeout (_http_client.js:706:34)
    at Object.onceWrapper (events.js:313:30)
    at emitNone (events.js:106:13)
    at Socket.emit (events.js:208:7)
    at Socket._onTimeout (net.js:410:8)
    at ontimeout (timers.js:498:11)
  message: 'Connection timed out after 300000ms',
  code: 'TimeoutError',
  time: 2019-03-03T18:35:47.342Z,
  region: 'us-east-1',
  hostname: 'localhost',
  retryable: true }

Then I get these errors for only a file:

2019-03-03T18:50:06.563Z box:shell backup-snapshot/app_836a8eb8-4f45-4099-aac5-83d6c25884c2 (stdout): 2019-03-03T18:50:06.286Z box:syncer Done processing adds { BackupsError: Error uploading snapshot/app_836a8eb8-4f45-4099-aac5-83d6c25884c2/data/<path>. Message: Connection timed out after 300000ms HTTP Code: TimeoutError
    at ManagedUpload.callback (/home/yellowtent/box/src/storage/s3.js:151:33)
    at Response.finishSinglePart (/home/yellowtent/box/node_modules/aws-sdk/lib/s3/managed_upload.js:661:28)
    at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:364:18)
    at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
    at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
    at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:683:14)
    at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)
  name: 'BackupsError',
  reason: 'external error',
  message: 'Error uploading snapshot/app_836a8eb8-4f45-4099-aac5-83d6c25884c2/data/<path>. Message: Connection timed out after 300000ms HTTP Code: TimeoutError' }

2019-03-03T18:50:06.568Z box:shell backup-snapshot/app_836a8eb8-4f45-4099-aac5-83d6c25884c2 (stdout): 2019-03-03T18:50:06.564Z box:backupupload upload completed with error { BackupsError: Error uploading snapshot/app_836a8eb8-4f45-4099-aac5-83d6c25884c2/data/<path>. Message: Connection timed out after 300000ms HTTP Code: TimeoutError
    at /home/yellowtent/box/src/backups.js:373:36
    at /home/yellowtent/box/src/syncer.js:166:31
    at /home/yellowtent/box/node_modules/async/dist/async.js:473:16
    at iterateeCallback (/home/yellowtent/box/node_modules/async/dist/async.js:988:17)
    at /home/yellowtent/box/node_modules/async/dist/async.js:969:16
    at /home/yellowtent/box/node_modules/async/dist/async.js:4617:26
    at f (/home/yellowtent/box/node_modules/once/once.js:25:25)
    at /home/yellowtent/box/src/backups.js:368:21
    at ManagedUpload.callback (/home/yellowtent/box/src/storage/s3.js:151:24)
    at Response.finishSinglePart (/home/yellowtent/box/node_modules/aws-sdk/lib/s3/managed_upload.js:661:28)
  name: 'BackupsError',
  reason: 'external error',
  message: 'Error uploading snapshot/app_836a8eb8-4f45-4099-aac5-83d6c25884c2/data/<path>. Message: Connection timed out after 300000ms HTTP Code: TimeoutError' }

And in the end the backup crashed with:

2019-03-03T18:50:06.569Z box:shell backup-snapshot/app_836a8eb8-4f45-4099-aac5-83d6c25884c2 (stdout): 2019-03-03T18:50:06.569Z box:backupupload upload completed

2019-03-03T18:50:06.706Z box:backups runBackupUpload: result - [object Object]
2019-03-03T18:50:06.783Z box:shell backup-snapshot/app_836a8eb8-4f45-4099-aac5-83d6c25884c2 code: 50, signal: null
2019-03-03T18:50:06.895Z box:apptask nextcloud.xeac.de update aborted because backup failed { BackupsError: Error uploading snapshot/app_836a8eb8-4f45-4099-aac5-83d6c25884c2/data/<path>. Message: Connection timed out after 300000ms HTTP Code: TimeoutError
    at /home/yellowtent/box/src/backups.js:682:29
    at f (/home/yellowtent/box/node_modules/once/once.js:25:25)
    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:66:9)
    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
  name: 'BackupsError',
  reason: 'external error',
  message: 'Error uploading snapshot/app_836a8eb8-4f45-4099-aac5-83d6c25884c2/data/<path>. Message: Connection timed out after 300000ms HTTP Code: TimeoutError',
  backupError: true }
2019-03-03T18:50:06.896Z box:apptask nextcloud.xeac.de updating app with values: {"installationState":"installed","installationProgress":"","health":null,"updateConfig":null}
2019-03-03T18:50:07.095Z box:apptask Apptask completed with error { BackupsError: Error uploading snapshot/app_836a8eb8-4f45-4099-aac5-83d6c25884c2/data/<path>. Message: Connection timed out after 300000ms HTTP Code: TimeoutError
    at /home/yellowtent/box/src/backups.js:682:29
    at f (/home/yellowtent/box/node_modules/once/once.js:25:25)
    at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:66:9)
    at emitTwo (events.js:126:13)
    at ChildProcess.emit (events.js:214:7)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:198:12)
  name: 'BackupsError',
  reason: 'external error',
  message: 'Error uploading snapshot/app_836a8eb8-4f45-4099-aac5-83d6c25884c2/data/<path>. Message: Connection timed out after 300000ms HTTP Code: TimeoutError',
  backupError: true }
2019-03-03T18:50:07.096Z box:apptask Apptask completed for 836a8eb8-4f45-4099-aac5-83d6c25884c2

[girish]

@RoboMod Sure, sorry for the no response. From the logs in the other post, the error is Message: Connection timed out after 300000ms HTTP Code: TimeoutError. Maybe the server becomes unreachable because of some network error? Is that possible in your setup?