Minio backup fails for no reason

thibaud

Hello,

I have configured backups with Minio and it worked perfectly for weeks.

But it does no longer work. I think the problem started after an update, but I am not sure. As far as I understand from the logs (both from Cloudron and Minio sides) Cloudron tries to copy to Minio bucket a multipart archive and it fails. And Minio log says Cloudron tries to rename and/or copy a file that does not exist (yet, I guess).

Abstract from Cloudron log :

Unable to backup { BoxError: Error copying crsav_/snapshot/app_f0770611-2970-4e86-9047-0260e0297b79.tar.gz.enc (35961955060 bytes): XMLParserError XMLParserError: Unexpected close tag\nLine: 5\nColumn: 7\nChar: >' }

Abstract from Minio log :

Error: rename /data/.minio.sys/tmp/057fd8e5-234e-4e99-abe9-4ddb96796b34/e4b6158e-b4f5-4afc-8bb2-08e63228e618.2ff401ac-895f-43f9-aa2a-e40e9b862aef.9 /data/.minio.sys/multipart/caa122a8292342e6d4f7592bf3aed821c15f88ae471d458b70474d192b99be02/e4b6158e-b4f5-4afc-8bb2-08e63228e618/00009.0d55bc9f9c910f5b64049580ba387d34.1073741824: no such file or directory

       5: github.com/minio/minio@/cmd/fs-v1-helpers.go:397:cmd.fsSimpleRenameFile()

       4: github.com/minio/minio@/cmd/fs-v1-multipart.go:331:cmd.(*FSObjects).PutObjectPart()

       3: github.com/minio/minio@/cmd/fs-v1-multipart.go:259:cmd.(*FSObjects).CopyObjectPart()

       2: github.com/minio/minio@/cmd/object-handlers.go:1892:cmd.objectAPIHandlers.CopyObjectPartHandler()

       1: net/http/server.go:2007:http.HandlerFunc.ServeHTTP()

Thank you,
Thibaud

girish

@thibaud Do you have a guesstimate of what is the size of that file? Also, in the backup configuration, do you minio or s3 compatible as the provider? Can you also post a screenshot of Backups -> configure -> advanced settings (just the bottom part of the advanced settings screenshot).

thibaud

@girish I have no idea. I understand the backup fails when it comes to Nextcloud. So basically, the file is large.

Backup conf is on Minio.

Are those settings new ? I am pretty sure I have never seen them. I guess they have something to do with my problem...

girish

@thibaud said in Minio backup fails for no reason:

So basically, the file is large

You can check this from the graphs maybe as to how much the app consumes? It's under System view.

Are those settings new ?

Yes, new in 5.4 and 5.5

thibaud

@girish

You can check this from the graphs maybe as to how much the app consumes? It's under System view.

Yes. As expected its huge as it is a Nextcloud. The entire app consumes 36 GB.

Yes, new in 5.4 and 5.5

And I think my problem appeared after 5.4 update. So basically I guess I need to increase memory limit? Up to... how much?

robi

@thibaud until it works

girish

@thibaud Yes, I would start with say 1GB or so and see if it works. Note that this memory is just the "max" and also only for the duration of the backup itself.

thibaud

Thank you all.

@girish I increased the parameters step by step and tried to backup for each (until the max i.e. 16 go). I still have the same error message. Any help?

girish

@thibaud Are you able to mail the entire backup log to support@cloudron.io ? Let's see if it's something obvious from the logs.

thibaud

@girish Done!

girish

@thibaud Thanks. It seems multi-part copies are failing altogether. As in, even the first part is failing. I thought it was failing much later.

Retrying (1) multipart copy of crsav_/snapshot/app_f0770611-2970-4e86-9047-0260e0297b79.tar.gz.enc. Error: XMLParserError: Unexpected close tag\nLine: 5\nColumn: 7\nChar: > 405 (

Let me quickly test this and see if it's easy to reproduce.

girish

@thibaud Mmm, I cannot reproduce this. I tried to backup and it succeeded over ~40 parts (https://paste.cloudron.io/ihulamulow.apache). So, to take a step back, it failed in "rename" . Is there something special about the filesystem where minio is located? Is it ext4? What is the underlying filesytem of Cloudron itself?

thibaud

@girish

My Minio is self-hosted in a Docker on a Synology NAS which underlying filesystem is proprietary (btfrs). I already checked if there are any known issue reported with Minio. I will double check.

My Cloudron is hosted on Ubuntu / ext4.

imc67

@thibaud said in Minio backup fails for no reason:

Minio is self-hosted in a Docker on a Synology NAS which underlying filesystem is proprietary (btfrs)

I have exactly the same situation on two Cloudrons to Minio's on two NAS's, only difference is that my backups are rsync and not tar. I experience no single issue and with Watchtower in Docker Minio is automatically updated to the latest version within 24 hours.

thibaud

@imc67 said in Minio backup fails for no reason:

my backups are rsync and not tar

Thanks for the tip. I switched to rsync and I was full of hope. Unfortunately it also fails.

@girish I send you Minio full log by mail.

girish

@thibaud I replied to you on support@ but the issue is that there is some long file name. The current rsync+encryption backup has some file name length limitation - https://docs.cloudron.io/backups/#encryption . There is a feature request at https://forum.cloudron.io/topic/3057/consider-improvements-to-the-backup-experience-to-support-long-filenames-directory-names .

Run the following command in /home/yellowtent/appsdata to find the large filenames:

find . -type f -printf "%f\n" | awk '{ print length(), $0 | "sort -rn" }' | less

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Solved Minio backup fails for no reason