App Backups Not Happening?

[doodlemania2]

I have my apps backing up:

But when I go to download one, I get an error:

A quick look at my S3 endpoint (idrive) and the files are indeeed not there:

Since snapshots are there, I checked the full system backups and they seem fine:

and:

It's like the app specific backups that happen when I update an app are just ... not happening - file not found style.

This bit me today when attempting to roll baserow back because "something" changed and my webhooks stopped working. Since none of the listed backups were actually present on idrive, I assume I have data loss, except for the snapshot.

So, two questions - can I restore a single app from a snapshot of the system backup and...how can I figure out what's cooking with my app backups?

[girish]

@doodlemania2 If I understand your screenshots correctly, the files are in Snapshot directory but there are no other files ? i.e there are no top level timestamped directories alongside the snapshot directory. This feels like the "copy object" is not working. Can you confirm? Seems quite serious if that's the case.

[doodlemania2]

@girish indeed, there are no timestamped app folders. Wondering if idrive is being weird? Here's the scrubbed config I'm using:

I had to delete the app I was messing with as it was in a bad state after attempting to restore to an earlier version and then backup not found. Assuming the Box backups are working (feels like they are given items in the snapshot folder), the data should be in there somewhere, but again, yes, nothing at the root of the bucket other than 'snapshots.'

[girish]

@doodlemania2 said in App Backups Not Happening?:

Assuming the Box backups are working (feels like they are given items in the snapshot folder), the data should be in there somewhere, but again, yes, nothing at the root of the bucket other than 'snapshots.'

Unfortunately, this means that backups are not working. There must be timestamped directories at the root of the bucket. Let me test idrive e2 and get back (we did add this in 7.4 already).

[doodlemania2]

@girish better me than a customer! bummed about the data loss but i can rebuild it. Let me know if you need a test bed to poke on things.

[doodlemania2]

@girish one thing to check though would be why it was (continues to) reporting success. Surely, it's throwing an error in there somewhere!

[girish]

@doodlemania2 that sounds like a bug which we fixed in 7.3. Are you on 7.3.6 ? Specifically https://git.cloudron.io/cloudron/box/-/commit/ea441d0b4b34830d24e1e7b7c32615d9ec93eefd

[doodlemania2]

@girish was on 7.3.4 - updating now! Will confirm back.

[doodlemania2]

@girish - got to 7.3.5 but stuck getting to 7.3.6 with:

"taskId": "12895",
"errorMessage": "Error listing objects in snapshot/app_19ee5686-c489-4855-b5b1-dcd62b3d287b.tar.gz.enc. Message: The specified key does not exist. HTTP Code: NoSuchKey",
"timedOut": false
}

[girish]

@doodlemania2 the bug was fixed in 7.3.5. Can you try a backup ? Does it fail in copy?

[doodlemania2]

@girish Okay, so, something else might be going on here. Refer back to my backup config pointing at cloudron-backup-dallas bucket. Snapshots are landing in there and look good.

Now, recently, I had migrated the server to a new host, and the original bucket was named cloudron-backup-migrate. I kept that bucket around just in case I needed to get an old copy back.

I ran several app updates this morning to test as requested. Guess where the app backups are landing?

So, it seems there's an artifact somewhere still holding on to the old bucket? (am currently on 7.3.5)

[girish]

@doodlemania2 In the Backup config, do you happen to have the bucket name in the endpoint by any chance ?

[doodlemania2]

@girish shit... yep - removed the bucket name and all works fine now. good lord. sorry about that - quite a chase

[girish]

@doodlemania2 that's exactly what the change fixes! it won't let you save that invalid config anymore (i.e putting a bucket name in the endpoint)