My Mastodon backups are failing now, nevermind not even pretending to be successful!
-
@scooke said in My Mastodon backups are failing now, nevermind not even pretending to be successful!:
What is going on???
I don't know, but perhaps share more info about your set-up.
What back-up format? backed-up to where? with what sort of mount? etc.
-
-
@girish This is really starting to get frustrating. I removed Mastodon from the auto backups because that seemed to be the app causing a problem, likely some weird bit of media in it (seeing as it's Mastodon). Tehn, Peertube starting acting up - I had changed buckets and changed mode from tgz to rsyn - then it failed with a video file. Ok, fine, rsync can't do videos.
I reverted back to the original bucket which has been working fine up until 2 weeks ago, changed back to tgz, and removed Peertube too. Tried manually backing up the rest - result:Error uploading snapshot/app_040b01cf-5c14-4db5-ba54-83f3356cb466.tar.gz. Message: Unexpected close tag Line: 5 Column: 7 Char: > HTTP Code: XMLParserErrorIt's the tailend that keeps pooping up: "Message: Unexpected close tag Line: 5 Column: 7 Char: > HTTP Code: XMLParserError" This time it was an unmanaged WP site that's never given me problems.
The entire time I'm using my self-hosted Minio instance on a CapRover installation. It's been working fine up until 2 weeks ago. Something changed. I don't know what.
-
I've rechecked the bucket, made sure it was public (it had been public but had changed to Custom), nothing else is turned on (like versioning), it has a specific Identity (User), and I'm retrying (without mastodon or peertube) using rsync again, with the following:
-
@scooke said in My Mastodon backups are failing now, nevermind not even pretending to be successful!:
CapRover
As you yourself have frequently pointed out on this very forum, CapRover isn't really suitable for production as it doesn't Just Work like Cloudron does.
Given you've said this, I surprised to learn you are entrusting your backups to it.
Edit: but looking here it looks like the API endpoint got tweaked 4 days ago, could that be it?
https://github.com/caprover/one-click-apps/blob/master/public/v4/apps/minio.yml
-
@jdaviescoates Could be. I will look that up. Suffice to say, it seems the federated apps using Minio as a backend are the ones that have been choking. The backup is running fine thus far (about 25%).
Looks like this is the big difference:
**Dashboard**: https://$$cap_appname.$$cap_root_domain **S3 API Endpoint**: https://$$cap_appname-api.$$cap_root_domainwhere the S3 API Endpoint was
https://$$cap_appname-s3.$$cap_root_domainbefore. -
-
@girish Not sure where to look. But I removed both Peertube and Mastodon from the Cloudron backup and the most recent back up succeeded. I'll try to locate where the logs are and see whats up.
My guess is that since both of those were ALSO using Minio as their own storage there was some kind of conflict or overload (even though their respective destinations were different).
@jdaviescoates I do see that the Minio version has updated on CapRover, but my instance is still running the previous, so there isn't any problem with the url change.
-
@girish I found where the logs are, but the backup process for the Mastodon instance took very very long, and I went to bed. I checked this morning: the bucket which Mastodon uses for its own media is working fine, but there was initially no sign of the app backup. In the app's Cloudron dashboard there was no notice that it failed, but there was no record of the backup either. I eventually found it in the /snapshots folder in the bucket for the main Cloudron backup. I downloaded the backup configuration for the previous Mastodon app backup, 2 weeks ago, and everything matches up EXCEPT the region in that file differs from the region set in Minio. Could this be messing things up?
-
Since some time back, you also indicated that at some point you saw the same error with a Wordpress app instance, I guess this is all not too related to a specific app but maybe more like something choking on maybe low resources somewhere. For example can you check if that minio instance may restart due to running out of memory and thus responding with broken xml temporarily? Or also if those backups now take so long, is there a cron job running on the minio instance server which may restart services related here? I don't know caprover but just to rule those things out.
-
@nebulon Aa far as I can tell, there is no restart or low resources.
This time the backup of the Mastodon seems to have completed, using rsync, so it took about 5 hours. I say seems because it at least appears in the destination bucket BUT not in the list of backups in the apps Cloudron dashboard.
This might be good enough for me... assuming that I could, for example, download one of the previously listed Backup Configurations, such as
app-backup-config-2023-02-08-xx-xx-xx (my.mastodonapp.com).jsonand then tweak it so that when I then Import a Backup, I could choose this edited one pointing to the correct place in my Minio instance... right?Why would it not be listed as complete though??
-
-
-
I still have the same problem. I tried creating a new object storage bucket. That didn't work. I tried starting all of the inactive applications and manually updating them till they were all current. Then I restarted the server and tried to backup again. That didn't work, either.
I might try deleting some applications to see if I can isolate the problem.
That didn't work either.I have noticed that the backup process indicates the number of applications to backup, however it counts 24 instead of 25. I deleted a few more applications and restarted the server. When I tried the backup again, the number of applications did correctly tally with the number Cloudron thought it was going to backup, however, the backup still failed.
The logs seemed to indicate that the problem seemed to be with Immich. I decided to backup the data and deleted immich. The backup still didn't work!
-
G girish marked this topic as a question on
-
G girish has marked this topic as solved on
