My Mastodon backups are failing now, nevermind not even pretending to be successful!
-
@jdaviescoates Could be. I will look that up. Suffice to say, it seems the federated apps using Minio as a backend are the ones that have been choking. The backup is running fine thus far (about 25%).
Looks like this is the big difference:
**Dashboard**: https://$$cap_appname.$$cap_root_domain **S3 API Endpoint**: https://$$cap_appname-api.$$cap_root_domainwhere the S3 API Endpoint was
https://$$cap_appname-s3.$$cap_root_domainbefore. -
@girish Not sure where to look. But I removed both Peertube and Mastodon from the Cloudron backup and the most recent back up succeeded. I'll try to locate where the logs are and see whats up.
My guess is that since both of those were ALSO using Minio as their own storage there was some kind of conflict or overload (even though their respective destinations were different).
@jdaviescoates I do see that the Minio version has updated on CapRover, but my instance is still running the previous, so there isn't any problem with the url change.
-
@girish I found where the logs are, but the backup process for the Mastodon instance took very very long, and I went to bed. I checked this morning: the bucket which Mastodon uses for its own media is working fine, but there was initially no sign of the app backup. In the app's Cloudron dashboard there was no notice that it failed, but there was no record of the backup either. I eventually found it in the /snapshots folder in the bucket for the main Cloudron backup. I downloaded the backup configuration for the previous Mastodon app backup, 2 weeks ago, and everything matches up EXCEPT the region in that file differs from the region set in Minio. Could this be messing things up?
-
Since some time back, you also indicated that at some point you saw the same error with a Wordpress app instance, I guess this is all not too related to a specific app but maybe more like something choking on maybe low resources somewhere. For example can you check if that minio instance may restart due to running out of memory and thus responding with broken xml temporarily? Or also if those backups now take so long, is there a cron job running on the minio instance server which may restart services related here? I don't know caprover but just to rule those things out.
-
Since some time back, you also indicated that at some point you saw the same error with a Wordpress app instance, I guess this is all not too related to a specific app but maybe more like something choking on maybe low resources somewhere. For example can you check if that minio instance may restart due to running out of memory and thus responding with broken xml temporarily? Or also if those backups now take so long, is there a cron job running on the minio instance server which may restart services related here? I don't know caprover but just to rule those things out.
@nebulon Aa far as I can tell, there is no restart or low resources.
This time the backup of the Mastodon seems to have completed, using rsync, so it took about 5 hours. I say seems because it at least appears in the destination bucket BUT not in the list of backups in the apps Cloudron dashboard.
This might be good enough for me... assuming that I could, for example, download one of the previously listed Backup Configurations, such as
app-backup-config-2023-02-08-xx-xx-xx (my.mastodonapp.com).jsonand then tweak it so that when I then Import a Backup, I could choose this edited one pointing to the correct place in my Minio instance... right?Why would it not be listed as complete though??
-
Most likely the backup then has not correctly succeeded, otherwise it would be listed. Maybe we have to debug this on your instance directly then, also potentially having to see what the minio endpoint does.
-
@scooke If you write to support@cloudron.io, we can look into it.
-
I still have the same problem. I tried creating a new object storage bucket. That didn't work. I tried starting all of the inactive applications and manually updating them till they were all current. Then I restarted the server and tried to backup again. That didn't work, either.
I might try deleting some applications to see if I can isolate the problem.
That didn't work either.I have noticed that the backup process indicates the number of applications to backup, however it counts 24 instead of 25. I deleted a few more applications and restarted the server. When I tried the backup again, the number of applications did correctly tally with the number Cloudron thought it was going to backup, however, the backup still failed.
The logs seemed to indicate that the problem seemed to be with Immich. I decided to backup the data and deleted immich. The backup still didn't work!
-
G girish marked this topic as a question on
-
G girish has marked this topic as solved on
