SOLVED Backup timing out

[eyecreate]

My backups in the past week have been unable to fully complete. The Nextcloud app backup process always gets stuck. Did I just reach a resource limit(ex. it was taking close to too long before and more data added recently put it into timeout territory)? What can I configure to fix this? I don't see any errors in cloudron, but the minio backup server on same network shows this:

Error: read tcp <ip>:9000-><ip>:35152: read: connection reset by peer
       4: github.com/minio/minio@/cmd/api-errors.go:1880:cmd.toAPIErrorCode()
       3: github.com/minio/minio@/cmd/api-errors.go:1905:cmd.toAPIError()
       2: github.com/minio/minio@/cmd/object-handlers.go:2246:cmd.objectAPIHandlers.PutObjectPartHandler()
       1: net/http/server.go:2036:http.HandlerFunc.ServeHTTP()

[girish]

@eyecreate Which Cloudron version are you on? Did this happen even in Cloudron 6.1 ? There is currently a backup issue (networking error) related issue we are investigating at https://github.com/aws/aws-sdk-js/issues/3404 . Seems to be some issue related with nodejs and the aws-sdk library version.

[eyecreate]

@girish This happened on 6.1 and I updated to 6.2 to see if it changed anything.(it didn't)

What I see doesn't initially seem to be related to that issue, but that is based on me not finding anything like that so far in the backup logs.

What is the set value for timing out a backup process?
Could it be running out of memory?
What would cause the backup process to not move on to another item?

I've attached below edited logs from the backup process starting with last found upload messages and ending with last lines of log.

2021-03-16T08:05:53.288Z box:backups Uploaded postgresqldump

2021-03-16T08:06:05.182Z box:shell backup-snapshot/app_d32f7472-a8f5-4909-b190-af5e3457aecb (stdout): 2021-03-16T08:06:05.181Z box:backupupload process: rss: 377643008 heapTotal: 38162432 heapUsed: 31268320 external: 201905702
2021-03-16T08:06:05.182Z box:backupupload v8 heap : used 31272552 total: 38162432 max: 2197815296

2021-03-16T08:06:10.390Z box:shell backup-snapshot/app_d32f7472-a8f5-4909-b190-af5e3457aecb (stdout): 2021-03-16T08:06:10.390Z box:storage/s3 Uploaded snapshot/app_d32f7472-a8f5-4909-b190-af5e3457aecb/data/<user>/uploads/e1af8eccb773d34b10671a3bc5ae8462/0000000010240000-0000000014948188 with partSize 33554432: {"ETag":"\"47baf95e2236fd790d11da409611c1f6\"","Location":"http://<ip>:9000/<bucket>/snapshot/app_d32f7472-a8f5-4909-b190-af5e3457aecb/data/<user>/uploads/e1af8eccb773d34b10671a3bc5ae8462/0000000010240000-0000000014948188","key":"snapshot/app_d32f7472-a8f5-4909-b190-af5e3457aecb/data/<user>/uploads/e1af8eccb773d34b10671a3bc5ae8462/0000000010240000-0000000014948188","Key":"snapshot/app_d32f7472-a8f5-4909-b190-af5e3457aecb/data/<user>/uploads/e1af8eccb773d34b10671a3bc5ae8462/0000000010240000-0000000014948188","Bucket":"<bucket>"}

2021-03-16T08:06:10.391Z box:shell backup-snapshot/app_d32f7472-a8f5-4909-b190-af5e3457aecb (stdout): 2021-03-16T08:06:10.390Z box:backups Uploaded data/<user>/uploads/e1af8eccb773d34b10671a3bc5ae8462/0000000010240000-0000000014948188

2021-03-16T08:06:35.183Z box:shell backup-snapshot/app_d32f7472-a8f5-4909-b190-af5e3457aecb (stdout): 2021-03-16T08:06:35.182Z box:backupupload process: rss: 377774080 heapTotal: 38162432 heapUsed: 31339352 external: 201906008
2021-03-16T08:06:35.182Z box:backupupload v8 heap : used 31339576 total: 38162432 max: 2197815296

end

2021-03-17T01:41:06.599Z box:shell backup-snapshot/app_d32f7472-a8f5-4909-b190-af5e3457aecb (stdout): 2021-03-17T01:41:06.599Z box:backupupload process: rss: 378445824 heapTotal: 38162432 heapUsed: 32285704 external: 201905090
2021-03-17T01:41:06.599Z box:backupupload v8 heap : used 32285944 total: 38162432 max: 2197815296

[eyecreate]

It seems that switching the backup format from rsync to tgz has allowed the backup to complete again.

[girish]

I will mark this as solved for now but also ack that there is possibly some bug either on minio side or Cloudron side.