Mastodon backups listed in a Cloudron-wide backup, but aren't actually backed up!?

[scooke]

I'm putting this in Support, rather than Mastodon, because it is quite strange.

My Mastodon instance was blank this evening, and giving me some kind of https error. Instead of trying (to figure out how) to renew the cert, I just thought "I'll restore to a backup, easy-peasy". The version listed was 1.10.0, so I chose to restore to a 1.9.1, just a few days ago. Part way through an error message popped up saying the backup can't be found. ?

SO I checked the Minio instance, the bucket where the backups are going, and sure enough there is the item, like "2022-12-23-210007-476", and inside this are all the other apps... but no Mastodon app.

When I click on the backups shown in the Backup tab in the Cloudron-wide Dashboard, where it says how many apps are backed up, it will show that the Mastodon app WAS included... but then in the Minio bucket, it isn't.

What in the world? It isn't in the list of excluded apps, it IS shown as being backed up, but when I checked every single Cloudron-wide backup, it isn't there (even if the backup app list says it was).

So as it stands, I now have lost my Mastodon instance. This is not good! Because even the app-specific backup, 2 days ago, is listed as backed up, but IT IS NOT.

Any ideas?? At the very least, can I somehow get the not-running-correctly-but-at-least-it-was-here Mastodon running? Or would the current docker images/containers have been deleted already when I tried to Restore?

EDIT: This may be dumb, but I am using the webhost's "Server Backup" to restore the entire thing. If that works, I will investigate what is going on with Mastodon not being included in the backups. What other apps are likewise being excluded??

[murgero]

@scooke In the backup tab you can download the backup for the specific app, what happens when you click that..?

[scooke]

@scooke UPDATE: The whole-server Restore by my webhost worked.

As I poked around I realized that in the env.production file of the Mastodon app it is set to use the same Minio instance, but a different bucket, for backups. The Cloudron-facing backup in the app dashboard DOES NOT indicate to WHERE the backup is happening. I assumed it was to the Cloudron-wide set destination.

Is this the cause? Even when the entire CLoudron is being backed up, does Cloudron reference the Mastodon's env.production app and thus backs it up but to the other bucket??

I might just edit that env.production file to have the Mastodon app use the same bucket for its data. But, then that bucket will get very large.

[scooke]

@scooke If I understand you, when I do that it brings me to the app's dashboard, at which point I have chosen which back up I want to restore to, the restore begins, then fails because it can't find the backup. It's listed in the full app backup, but isn't in the bucket used for that.

All I can think is that the Mastodon app is being backed up at the same time as all the others, but to the bucket specified in the env.production list. Which, I guess makes sense, but I thought that setting was for the data, not the app.

[scooke]

It seems to have hung now for awhile, showing,
Jan 13 22:32:34 box:tasks update 13910: {"percent":30,"message":"Uploading backup 722M@0MBps (social.futurnumerique.com)"}

It would be helpful to see WHERE it is being backed up to. Neither of the two possible destination buckets indicate a change.

[jdaviescoates]

@scooke said in Mastodon backups listed in a Cloudron-wide backup, but aren't actually backed up!?:

All I can think is that the Mastodon app is being backed up at the same time as all the others, but to the bucket specified in the env.production list.

I'm not sure, but I think and presume that when you edit your mastodon env.production to store data on an external S3 bucket then Cloudron is no longer backing up that stored data at all, but just the rest of the app code etc.

(I'd guess it's the same for your Peertube given you done something similar with that)

Hopefully @staff can confirm one way or another.

[scooke]

@jdaviescoates That is what I assumed too, but there is no sign of the Mastodon app in the full Cloudron backup bucket, despite it being in the list of backed up apps shown in the Backup Dashboard of Cloudron.

In fact, since I've been trying to poke around, it appears that my MAstodon app stopped using the Minio destination and is instead backing up, or at least accumulating, data on the server that my Cloudron is on. The last files uploaded to the bucket was weeks ago, according to the Minio bucket. And nothing from Mastodon is showing up in the Cloudron bucket.... what is happening?

[jdaviescoates]

@scooke Oh, that sounds weird. Good luck sorting it out!

[girish]

@scooke Did you get this sorted out?

[scooke]

@girish No, actually, I was just looking, saw that this app is listed in my full cloudron backup, but isn't present in the actual Minio bucket. I downloaded the individual app's backup configuration which also says it was backed up to that bucket. When I then choose to download the backup (from the apps dashboard) it starts something, then fails, and actually a get a humungusly long url on a page that says This app is currently not responding. Please try refreshing the page in a few minutes..

What in the world? Why is this app NOT getting backed up, while all the records say it is?? It can't be the S3_ENABLED=true in the env file because when i check that bucket (same Minio server) there is only the data, not the app.

Where did the app go?? If I hadn't had a full disk backup at SSDNodes I would have lost this app, and who knows what else.

[girish]

@scooke Interesting situation... I don't think it's the S3 flag either that is causing problems. Can you do an individual backup (app -> backup) and check the app logs and minio ?

[girish]

Makes me wonder if you hit a bug that I fixed last week related to s3 api copy errors. The copy fails but the backup code incorrectly flags it as a success. (https://git.cloudron.io/cloudron/box/-/commit/40bcfdba76bde24b7e9e50613b99c51889b30d00 is the fix).

[scooke]

@girish So I just copy and paste those lines into the appropriate files?

[girish]

@scooke before that, we should determine if that is actually the bug.

Do you see the app backed up under snapshot/ ? My fix was for the situation where the upload itself succeeds (into snapshot/) but the remote copy fails (thus no timestamped directories).

[scooke]

@girish Is this /snapshot/, or /home/yellow/snapshot/? Thank you.

Yes, it is in /snapshot (in the Minio bucket). Is this good?

[girish]

@scooke Right, in the minio snapshot directory. So, is it also correct that there are no backups of any app, not just mastodon ?

The issue (I fixed) is that the minio URL entered in backups -> configure is incorrect. Did you put the bucket name at the end of the "minio url" ?

[scooke]

@girish All the other apps backed up. That's the strange thing... only Mastodon is not backed up.

I'll check up on the other thing.

[scooke]

@girish the bucket name has its own box in which to enter. So no, i didn't add the bucket name as part of the url. But since everything else is backing up, how would this affect just the one app?

[girish]

@scooke you are right, then the issue is something else if only one app is not backing up. Can you update to 7.3.5 and see if you can reproduce the issue? The fix I was talking about is in 7.3.5.