Backups are failing

[LoudLemur]

Backup failed: Error copying ****/snapshot/app_84cb668f-5d43-48b2-8500-a24907a49d55.tar.gz (1838629651 bytes): XMLParserError XMLParserError: Unexpected close tag Line: 5 Column: 7 Char: >. Logs are available here.

I have tried doing all I can, except reinstalling Cloudron and starting from scratch. I don't know how to fix this.

Should I email the logs to Cloudron?

[jdaviescoates]

@LoudLemur yes email support@cloudron.io

[jdaviescoates]

also, it looks like @thibaud and @scooke have had the same XmlParserError and that both were backing up to Minio.

Are you also backing up to Minio?

[girish]

These errors are to do with some error in the remote endpoint.

Saw your email on support, looks like you are on Vultr. Let me see if I can reproduce it.

[LoudLemur]

@jdaviescoates said in Backups are failing:

also, it looks like @thibaud and @scooke have had the same XmlParserError and that both were backing up to Minio.

Are you also backing up to Minio?

This is on Vultr.

[girish]

@LoudLemur I can actually reproduce this in Vultr's SJ region

[girish]

I can reproduce this in NJ region as we well.

[girish]

There is a general issue in Vultr multipart copy.

# ./rclone ls vultr://
5316608876 cloudron-backups2/2023-10-26-062728-820/app_xc.smartserver.io_v4.20.3.tar.gz
5316608876 cloudron-backups2/snapshot/app_410ea704-713a-4cf0-bb9d-e59aa14ab56b.tar.gz

Then, if I try to copy, it just hangs (I started this 25 mins ago):

# time ./rclone copy vultr://cloudron-backups2/snapshot/app_410ea704-713a-4cf0-bb9d-e59aa14ab56b.tar.gz vultr://cloudron-backups2/timeit

[girish]

Found a crash in rclone, reported https://github.com/rclone/rclone/issues/7391

[girish]

I couldn't figure how to make rclone copy the file but I fixed Cloudron code.

@LoudLemur You can patch this single line - https://git.cloudron.io/cloudron/box/-/commit/934c701be266f8cf095a841c4d69a0dc2770ca2c . The file on Cloudron itself is /home/yellowtent/box/src/storage/s3.js

[LoudLemur]

Thank you, Girish, for diagnosing and solving this bug so quickly! Backups are working again.

If somebody else has this issue and needs to do the patch, you would go to the operating system for your server and find the file there and edit it. Cloudron also have a support tool inside Cloudron which enables you to temporarily grant ssh access to your server, which would let Cloudron's brilliant team patch it for you.