Backups still not verified or what?

[philkunz]

https://forum.cloudron.io/post/114219 -> Here it is specifically said v9 verifies backups. So is it being done? Or not? Another user posted problems with CIFS. Either way -> I doubt you are verifying backups if this still happens and there is no error otherwise.

[philkunz]

Again ProxMox saves the day... Its a bad feeling knowing I can't rely on Cloudron to have my back at the moment.

[joseph]

@philkunz the verification signature is now saved but there is no UI yet .

Can you check if you have a .backupinfo file next to your .tar.gz ? Inside .backupinfo there will be a sha256 checksum. Can you check if sha256 sum matches with the .tar.gz file ?

[philkunz]

Wait, you are simply creating a hash? Verification means you create a snapshot of the live filesystem, then actually unpack the tar and check wether that matches, after each backup. Simply saving a hash does nothing.

[philkunz]

There is indication now, that CIFS mounts might a have a problem of some sorts.

[philkunz]

My CIFS mount is on a Synology RS2423RP+ rack mounted unit with ECC usinf Btrfs... I doubt that Synology messes up the file with something essential like this, when this is the basic thing they sell to SMEs.

[joseph]

Not at all. Hashing is very well suited for data integrity. Not sure why live snapshots of file system is needed . one two three etc. Plenty of articles to read about this.

I think but for a start we need to know if what Cloudron thinks it saved on the backup disk matches what is currently on the disk. This is what computing a checksum will tell us.

[joseph]

@philkunz also, none of the app backups (i.e backup of other apps) work and not just a specific one, correct? If not, can you please try to clone a couple of other apps, just to rule out some obvious guesses.

[philkunz]

I needed exactly 2 backups to work from 2 different apps in the last 3 month. Both did not work. The question is: What exactly are you hashing? Where are you creating the hash? How are you creating the hash? A hash is simply a function that can tell you with a certain probability, that a certain input is equal to another input if the hash matches. It tells you nothing about wether the backup works or not. It might increase your odds, but nothing more.

Just tested with another app -> also failing. And yes, there are hash mismatches. Just tested a few. Hashes do not match.

[philkunz]

So essentially, you are not even comparing backup hashes by reading back the file you just stored, or why is there no warning prior to needing a backup? How can this be considered "verifying" backups?

[philkunz]

Its like you roll the dice with your customers.

[philkunz]

Like take invoiceninja: What do you tell someone who relies on your cloudron solution and who has not taken appropriate measures to save financial data otherwise, because he thought: "I have backups."

[philkunz]

Even for my usecase, mostly gitea instances, it would be shitty to loose data. Good thing I save everything on various levels...

[philkunz]

To put it bluntly: Working Backups are more important than a refined UI, new features, or a new app. If mismatching hashes are not even detected automatically, there is work to be done before doing anything else. Otherwise this in my view is unresponsible negligence for a product like this.

[philkunz]

@joseph @girish If you guys need help, want to hop on a call to see whats going on, let me know. I love Cloudron, but this kind of thing needs to work. Given the indications above it seems clear to me, that Cloudron can be improved on this front.

[girish]

@philkunz the hash is the hash of the file Cloudron saved to the storage. If they are not matching, then it either means the initial upload by Cloudron was wrong or something is off with the CIFS storage.

I can't reproduce this with hetzner CIFS at least which is to say it's not a general problem. Can you get in touch at support@cloudron.io and we can debug this further . Thanks!

[philkunz]

@girish -> The question is: How many people using CIFS actually try to use backups? And how many are complaining? I'll write an email, just one quick question -> Would it be possible to have an option to read back the file after backup to compare it with the hash then? How else would one detect a backup problem at scale? Cause my understanding is, right now the hash does nothing, except show wether a backup is broken when I need it (which is too late to do anything about it, causing potential data loss)

[philkunz]

The only reason I can think of, why this is not done already by default would be bandwidth and s3 egress cost, otherwise it should be the default tbh, to at least read back what was stored.

[nebulon]

So to clarify a bit further, there are two related things mixed I feel. So the latest version does calculate the hashes of the files it expects to be on the backup storage. What is not yet added is the part where the user gets notified or warned if the file on the remote storage does not pass the validation. This will come though.

The second is the actual restore issue you are facing and not really related as such to a verification issue, but the situation as such is why we introduced it, it is just not all finished (missing UI mostly)

[girish]

@philkunz hetzner storage box (CIFS) is widely used. I don't have numbers since we don't collect them. I don't have much idea about Synology/CIFS .

Would it be possible to have an option to read back the file after backup to compare it with the hash then?

yes, that's exactly the eventual idea. I think maybe we misled you with the "What's coming in 9.0" post. I make those posts much in advance to collect feedback etc (that post was made a year ago). It doesn't mean everything there got implemented. I usually go back and edit them to indicate what got implemented and what got moved etc. I guess it's a balance between whether to communicate what's coming etc or not. We keep the development open so such inconsistencies are unavoidable.

Specifically for integrity, we are still working on this - some technical notes here and here . It's still a WIP and not exposed to user until we are sure that the integrity is reliable. It would be even worse if we assured the user that things are good when they aren't .