Backups still not verified or what?

[joseph]

@philkunz also, none of the app backups (i.e backup of other apps) work and not just a specific one, correct? If not, can you please try to clone a couple of other apps, just to rule out some obvious guesses.

[philkunz]

I needed exactly 2 backups to work from 2 different apps in the last 3 month. Both did not work. The question is: What exactly are you hashing? Where are you creating the hash? How are you creating the hash? A hash is simply a function that can tell you with a certain probability, that a certain input is equal to another input if the hash matches. It tells you nothing about wether the backup works or not. It might increase your odds, but nothing more.

Just tested with another app -> also failing. And yes, there are hash mismatches. Just tested a few. Hashes do not match.

[philkunz]

So essentially, you are not even comparing backup hashes by reading back the file you just stored, or why is there no warning prior to needing a backup? How can this be considered "verifying" backups?

[philkunz]

Its like you roll the dice with your customers.

[philkunz]

Like take invoiceninja: What do you tell someone who relies on your cloudron solution and who has not taken appropriate measures to save financial data otherwise, because he thought: "I have backups."

[philkunz]

Even for my usecase, mostly gitea instances, it would be shitty to loose data. Good thing I save everything on various levels...

[philkunz]

To put it bluntly: Working Backups are more important than a refined UI, new features, or a new app. If mismatching hashes are not even detected automatically, there is work to be done before doing anything else. Otherwise this in my view is unresponsible negligence for a product like this.

[philkunz]

@joseph @girish If you guys need help, want to hop on a call to see whats going on, let me know. I love Cloudron, but this kind of thing needs to work. Given the indications above it seems clear to me, that Cloudron can be improved on this front.

[girish]

@philkunz the hash is the hash of the file Cloudron saved to the storage. If they are not matching, then it either means the initial upload by Cloudron was wrong or something is off with the CIFS storage.

I can't reproduce this with hetzner CIFS at least which is to say it's not a general problem. Can you get in touch at support@cloudron.io and we can debug this further . Thanks!

[philkunz]

@girish -> The question is: How many people using CIFS actually try to use backups? And how many are complaining? I'll write an email, just one quick question -> Would it be possible to have an option to read back the file after backup to compare it with the hash then? How else would one detect a backup problem at scale? Cause my understanding is, right now the hash does nothing, except show wether a backup is broken when I need it (which is too late to do anything about it, causing potential data loss)

[philkunz]

The only reason I can think of, why this is not done already by default would be bandwidth and s3 egress cost, otherwise it should be the default tbh, to at least read back what was stored.

[nebulon]

So to clarify a bit further, there are two related things mixed I feel. So the latest version does calculate the hashes of the files it expects to be on the backup storage. What is not yet added is the part where the user gets notified or warned if the file on the remote storage does not pass the validation. This will come though.

The second is the actual restore issue you are facing and not really related as such to a verification issue, but the situation as such is why we introduced it, it is just not all finished (missing UI mostly)

[girish]

@philkunz hetzner storage box (CIFS) is widely used. I don't have numbers since we don't collect them. I don't have much idea about Synology/CIFS .

Would it be possible to have an option to read back the file after backup to compare it with the hash then?

yes, that's exactly the eventual idea. I think maybe we misled you with the "What's coming in 9.0" post. I make those posts much in advance to collect feedback etc (that post was made a year ago). It doesn't mean everything there got implemented. I usually go back and edit them to indicate what got implemented and what got moved etc. I guess it's a balance between whether to communicate what's coming etc or not. We keep the development open so such inconsistencies are unavoidable.

Specifically for integrity, we are still working on this - some technical notes here and here . It's still a WIP and not exposed to user until we are sure that the integrity is reliable. It would be even worse if we assured the user that things are good when they aren't .

[philkunz]

Interesting part is -> I tested with another cloudron instance that runs on another company account, but uses the same Synology backup target -> that one works. Really strange. Yet there is another user in the linked thread that has the same problem...

[girish]

Just checked my Synology (but mine is a router and not a DS), it only has SMB 2 . @philkunz the DS has SMB3 correct?

[philkunz]

Yes, smb3. recreating the site config does not help. Then errors out as Password/Mac mismatch on restoration.

[philkunz]

I think it has something to do with bigger backups like upwards 20GB of data

[philkunz]

Has to be a cifs problem related to large backups. NFS works, Otherwise same setup.

[philkunz]

But that points to how important the e2e read back step is.

[girish]

@philkunz said in Backups still not verified or what?:

Has to be a cifs problem related to large backups. NFS works, Otherwise same setup.

that is valuable information. @luckow has a synology and volunteered to help us testing this . Will wait to check if he can reproduce this. He already tested with small backups and that worked .

@luckow we need to test with large backup sizes > 20GB .