Rotating backup parse error

[tradenet]

I don't know if this is a coincidence after upgrading to v7.3.5
A backup returned a parse error after rotating backup to minio storage.
With this error:

Error: XMLParserError: Unexpected close tag\nLine: 5\nColumn: 7\nChar: > 504
```"

at Response.done (/home/yellowtent/box/src/storage/s3.js:348:48)
at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:367:18)
at Request.callListeners (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:686:14)
at Request.transition (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:22:10)
at AcceptorStateMachine.runTo (/home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:14:12)
at /home/yellowtent/box/node_modules/aws-sdk/lib/state_machine.js:26:10
at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:38:9)
at Request.<anonymous> (/home/yellowtent/box/node_modules/aws-sdk/lib/request.js:688:12)

[girish]

@tradenet which s3 provider are you using?

[tradenet]

@girish I am using minio on my server. It's been working fine up until I upgraded to v7.3.5 unless it's a coincidence. I rebooted the Mastadon instance yesterday. Today I get the same error.
Interestingly enough, I do see the backup for yesterday in the minio bucket for yesterday this time around.

[girish]

@tradenet do you see any crashes on the minio side?

Also, can you check if the bucket name is not included in the endpoint url? That's the only check that was added in the patch release.

[tradenet]

@girish I am not entirely sure how to check this. However, I don't see the bucket name in the endpoint ie.
Bucket name is "disk1"
I see:

box:storage/s3 Uploaded maple/snapshot/app_5c2458fc-73dd-4f1a-8536-4387d9eb08de.tar.gz

Where the folder in the bucket is "maple/".
However, I am also noticing some postgres memory errors a few hours after the backup failure notice.

[klawitterb]

I’m also experiencing the same problem, it always seems to fail on large files or when a lot of files are backed up. In my case it’s a Nextcloud backup which is failing. Smaller apps with less data work just fine.
Using a similar setup: incremental backup with minio/s3

[girish]

@klawitterb @tradenet If you could give us access to your setup, can you please write to support@cloudron.io . Happy to debug further.

We use minio as well for some of our backups and haven't faced any issue as such.

[eddiiii]

I get the same error when I try to create a backup of my Nextcloud. The second app, Collabora, can be backed up flawlessly. I'm using a self-hosted minio instance.

[girish]

@eddiiii We haven't gotten a way to debug this. If you can give us access to your setup to debug, please send us a mail at support@cloudron.io .

BTW, are you able to backup other apps apart from collabora?

[eddiiii]

@girish I currently only use Nextcloud and Collabora. I'm writing an email to support.

[eddiiii]

@eddiiii i wrote a message to support@cloudron.io but i get no answer since 6 days. My Nextcloud app is not backuped yet. That is not a good situation.

[girish]

@eddiiii This got lost . So, I debugged this a bit and the error is coming from minio. Minio is either crashing or responding incorrectly. Can you tell us how you have deployed minio ? I also see that you have deployed minio on http. There is a related issue (https://github.com/minio/minio/issues/8050) which talks about how running minio on port 80 acts strangely with similar XML errors. Can you deploy minio with TLS ?

[LoudLemur]

This error is happening with Vultr Object storage, too. Regularly scheduled backups fail.

XMLParserError: Unexpected close tag Line: 5 Column: 7 Char: > 504

Also, the pretty colour coded vertical bars (red for backup fail) aren't showing in notifications. I thought this recently requested new feature was implemented.

[girish]

@LoudLemur said in Rotating backup parse error:

Also, the pretty colour coded vertical bars (red for backup fail) aren't showing in notifications. I thought this recently requested new feature was implemented.

Something for @nebulon

[nebulon]

The color coding is fixed with https://git.cloudron.io/cloudron/box/-/commit/7d731d76008fedcd55e47688090ad55156dbef63

[girish]

So, these errors are caused by the remote server. The XML error is returned by the AWS SDK in response to some remote error. Unfortunately, there is nothing much we do on Cloudron side to fix this.