My Mastodon backups are failing now, nevermind not even pretending to be successful!

[jdaviescoates]

@scooke said in My Mastodon backups are failing now, nevermind not even pretending to be successful!:

What is going on???

I don't know, but perhaps share more info about your set-up.

What back-up format? backed-up to where? with what sort of mount? etc.

[nebulon]

This does seem like the storage backend/service you are using is responding wrongly. Which one are you using there?

[girish]

I have seen this happen when upstream does not support multipart copy correctly. Which provider is this again? Can you go into snapshots/ as well (upstream) and check what is the size of your app backup ?

[scooke]

@girish This is really starting to get frustrating. I removed Mastodon from the auto backups because that seemed to be the app causing a problem, likely some weird bit of media in it (seeing as it's Mastodon). Tehn, Peertube starting acting up - I had changed buckets and changed mode from tgz to rsyn - then it failed with a video file. Ok, fine, rsync can't do videos.
I reverted back to the original bucket which has been working fine up until 2 weeks ago, changed back to tgz, and removed Peertube too. Tried manually backing up the rest - result:

Error uploading snapshot/app_040b01cf-5c14-4db5-ba54-83f3356cb466.tar.gz. Message: Unexpected close tag Line: 5 Column: 7 Char: > HTTP Code: XMLParserError

It's the tailend that keeps pooping up: "Message: Unexpected close tag Line: 5 Column: 7 Char: > HTTP Code: XMLParserError" This time it was an unmanaged WP site that's never given me problems.

The entire time I'm using my self-hosted Minio instance on a CapRover installation. It's been working fine up until 2 weeks ago. Something changed. I don't know what.

[scooke]

I've rechecked the bucket, made sure it was public (it had been public but had changed to Custom), nothing else is turned on (like versioning), it has a specific Identity (User), and I'm retrying (without mastodon or peertube) using rsync again, with the following:

[jdaviescoates]

@scooke said in My Mastodon backups are failing now, nevermind not even pretending to be successful!:

CapRover

As you yourself have frequently pointed out on this very forum, CapRover isn't really suitable for production as it doesn't Just Work like Cloudron does.

Given you've said this, I surprised to learn you are entrusting your backups to it.

Edit: but looking here it looks like the API endpoint got tweaked 4 days ago, could that be it?

https://github.com/caprover/one-click-apps/blob/master/public/v4/apps/minio.yml

[scooke]

@jdaviescoates Could be. I will look that up. Suffice to say, it seems the federated apps using Minio as a backend are the ones that have been choking. The backup is running fine thus far (about 25%).

Looks like this is the big difference:

            **Dashboard**: https://$$cap_appname.$$cap_root_domain
            **S3 API Endpoint**: https://$$cap_appname-api.$$cap_root_domain

where the S3 API Endpoint was https://$$cap_appname-s3.$$cap_root_domain before.

[girish]

@scooke are there any logs in minio/caprover ?

[scooke]

@girish Not sure where to look. But I removed both Peertube and Mastodon from the Cloudron backup and the most recent back up succeeded. I'll try to locate where the logs are and see whats up.

My guess is that since both of those were ALSO using Minio as their own storage there was some kind of conflict or overload (even though their respective destinations were different).

@jdaviescoates I do see that the Minio version has updated on CapRover, but my instance is still running the previous, so there isn't any problem with the url change.

[scooke]

@girish I found where the logs are, but the backup process for the Mastodon instance took very very long, and I went to bed. I checked this morning: the bucket which Mastodon uses for its own media is working fine, but there was initially no sign of the app backup. In the app's Cloudron dashboard there was no notice that it failed, but there was no record of the backup either. I eventually found it in the /snapshots folder in the bucket for the main Cloudron backup. I downloaded the backup configuration for the previous Mastodon app backup, 2 weeks ago, and everything matches up EXCEPT the region in that file differs from the region set in Minio. Could this be messing things up?

[nebulon]

Since some time back, you also indicated that at some point you saw the same error with a Wordpress app instance, I guess this is all not too related to a specific app but maybe more like something choking on maybe low resources somewhere. For example can you check if that minio instance may restart due to running out of memory and thus responding with broken xml temporarily? Or also if those backups now take so long, is there a cron job running on the minio instance server which may restart services related here? I don't know caprover but just to rule those things out.

[scooke]

@nebulon Aa far as I can tell, there is no restart or low resources.

This time the backup of the Mastodon seems to have completed, using rsync, so it took about 5 hours. I say seems because it at least appears in the destination bucket BUT not in the list of backups in the apps Cloudron dashboard.

This might be good enough for me... assuming that I could, for example, download one of the previously listed Backup Configurations, such as app-backup-config-2023-02-08-xx-xx-xx (my.mastodonapp.com).json and then tweak it so that when I then Import a Backup, I could choose this edited one pointing to the correct place in my Minio instance... right?

Why would it not be listed as complete though??

[nebulon]

Most likely the backup then has not correctly succeeded, otherwise it would be listed. Maybe we have to debug this on your instance directly then, also potentially having to see what the minio endpoint does.

[nebulon]

Seems like this may have the same root cause as https://forum.cloudron.io/topic/8588/rotating-backup-parse-error/6

[girish]

@scooke If you write to support@cloudron.io, we can look into it.

[scooke]

@girish I think I will. I've looked over the logs but I don't think I'm recognizing something important. You all have that eye, that gift!

[LoudLemur]

I still have the same problem. I tried creating a new object storage bucket. That didn't work. I tried starting all of the inactive applications and manually updating them till they were all current. Then I restarted the server and tried to backup again. That didn't work, either.

I might try deleting some applications to see if I can isolate the problem.
That didn't work either.

I have noticed that the backup process indicates the number of applications to backup, however it counts 24 instead of 25. I deleted a few more applications and restarted the server. When I tried the backup again, the number of applications did correctly tally with the number Cloudron thought it was going to backup, however, the backup still failed.

The logs seemed to indicate that the problem seemed to be with Immich. I decided to backup the data and deleted immich. The backup still didn't work!

[girish]

Please follow this thread at https://forum.cloudron.io/topic/10266/backups-are-failing