Backup Improvements: Restic Backend

[girish]

Not yet but I would like to discuss one thing here. Backups are crucial and loss of data for us implies loss of business and money quite literally. This is why we wrote the backup code ourselves a while ago. Also, why we create our own packages - it's all about data integrity and loss of data === loss of trust in product.

Initially, before we wrote our own backup stuff, I remember we used duplicati and btrfs etc . We faced various issues and there was essentially no help from upstream. Now, restic I am sure is great but if there is some corruption or issue, our customers will look to us to solve this. So, this is a tricky situation for us Maybe we can do some restic integration with lots of warnings? End user also has to know what to do if there is restic corruption and other issues. Keep in mind restic is also not 1.0 yet . They say "Once version 1.0.0 is released, we guarantee backward compatibility of all repositories within one major version; ...".

Any suggestions?

[girish]

this also reminded me of @nebulon 's suggestion of having more provider specific tweaking.

@imc67 are you using hetzner storage box or similar for backups ? what's the plan ? For storage backends that have rsync running, we can do lot better than now.

[imc67]

@girish yes indeed I use Hetzner Storagebox and use tgz over CIFS. Is there a better/smarter/quicker way?

[girish]

@imc67 yes, Hetzner Storagebox has rsync . So, we can be muuuch more efficient when calculating changes. I will investigate how much better it can get.

[imc67]

Curious for which settings do you recommend for backups to Storagebox and how much better it can get

[imc67]

@girish still very curious for the results of your investigation? Everything that's more efficient and quicker than tgz over CIFS more than 200GB to a Storagebox is very very welcome

[girish]

@imc67 still working through the backups rewrite, will leave a note here once I have something.

[imc67]

@girish after 4 months I'm still curious for the results of the "backups rewrite" as our daily backup now takes >5 hours

[girish]

@imc67 the rewrite hasn't happened I think this will only happen over xmas and new year vacation when support is low.

[imc67]

Any update on the backup methodology? I read a lot of issues in the forum about crashing backups due to several reasons. Maybe this can help?

[girish]

@imc67 we have not started working on a restic backend, if that's what you are asking.

I think most of the issues are just because of using a large variety of s3 providers. There's way too many of them. Made worse by people choosing the cheapest/low end providers who don't provide any support either. That and SSHFS/CIFS flakiness. I doubt moving to restic solves any of this because they are not related to backup code itself but infrastructure related. Just my thoughts though.

Now instead of dealing with our backup issues, we would be dealing with issue of another backend like say 1, 2 and 3 - these are just restic issues of last week. It would be worse because we have not much idea on how to deal with restic bugs.

[robi]

@girish It could also be the opposite experience, since restic would be handling all the issues, meaning less work for you with custom one-off fixes as they crop up.

[imc67]

FWIW I have a RaspberryPi for 3 years with a Restic daily backup to a Cloudron Minio S3 and 2 weeks ago I needed a restore for the first time. It took me the most time to rediscover how I configured it 3 years ago and how to restore but it worked flawless! I know it’s just n=1 but it’s worth to have a look at it.

[girish]

@robi yes definitely.

I do know that people want urgent fixes when backups and restores are not working, I just don't know how we can take up this responsibility. If my production site is down and it doesn't restore, what now? This will cost us real money because people will ask for a refund.

[fbartels]

@girish I think something that could be interesting, is the ability to use the cloudron filesystem dump and then have a "hook" that could post process the data. Possibly with capturing whatever that hook produces in a cloudron notification (to be natively informed if the post processing fails).

I have set up a few cloudrons in the last week with a combination of local disk rsync in cloudron (with local retention and hardlinks) and then using autorestic and Cron to push the cloudron backup via sftp to a hetzner storageboxes (optionally as a subuser). Have even performed a restore test, for which I first restored the previous state of /var/backups before running cloudrons restore.

Autorestic is taking care of retention and streamlining the sftp backend use. I even use it to make regular snapshots of the appsdata and boxdata directories in the yellowtent home.

The benefit of the hook would be the extended piece of mind that as soon als cloudron has finished the backup, it is pushed to its remote location.

@girish said in Backup Improvements: Restic Backend:

This will cost us real money because people will ask for a refund

Yes, restore of backups is crucial. Luckily so far cloudron has not let me down and overall it has been a very stable experience.

[necrevistonnezr]

@fbartels said in Backup Improvements: Restic Backend:

I have set up a few cloudrons in the last week with a combination of local disk rsync in cloudron (with local retention and hardlinks) and then using autorestic and Cron to push the cloudron backup via sftp to a hetzner storageboxes (optionally as a subuser). Have even performed a restore test, for which I first restored the previous state of /var/backups before running cloudrons restore.

@fbartels Exactly my setup for a couple of years: https://forum.cloudron.io/topic/6928/tutorial-remote-backup-of-local-cloudron-backup-snapshots-with-restic-rclone/
What‘s nice is that you can e.g. check and restore files on a different machine running restic & rclone.

[robi]

@girish said in Backup Improvements: Restic Backend:

I do know that people want urgent fixes when backups and restores are not working,

Yes, however be aware of fear based thinking here. It's not the end of the world, previous backups exist and the situation is temporary.

I just don't know how we can take up this responsibility.

It isn't your responsibility, even though you empathize and do a lot for us to make it easier. External backups are an external responsibility and best effort support here, which you do well. So, no fear.

If my production site is down and it doesn't restore, what now? This will cost us real money because people will ask for a refund.

It's like any other issue, you and we will find a way. This is the way.
Again, avoid decisions out of fear. And no, it will not cost you money (you're not hurting there), as it's not a Cloudron issue, but an external one.

You can relax and trust your loyal community to show you the way forward, as @fbartels and @necrevistonnezr already have.

[girish]

@necrevistonnezr that's an excellent post which I forgot about! I just proceeded to set up a backup of my laptop with restic+minio. Let's see how it goes!

[girish]

After a year, prune fails randomly

The index references 1 needed pack files which are missing from the repository:
  64c2b4cd29f07b6d64f4d0ede2509fd680597b16db65b77e14e8575e1265f1bf
Fatal: packs from index missing in repo

[necrevistonnezr]

https://forum.restic.net/t/fatal-packs-from-index-missing-in-repo/4869 --> https://github.com/restic/restic/issues/828#issuecomment-706186047 --> https://restic.readthedocs.io/en/stable/077_troubleshooting.html

Judging from the background story in Very slow restic prune , my guess would be that the S3 bucket listing is incomplete from time to time. So, a quite likely explanation is that the list of pack files which prune gets in incomplete and hence the missing file errors. As far as I remember, minio has options to ensure a consistent file listing or maybe you’re encountering some timeout.