Disaster recovery - Restoring a backup

[makemrproper]

Hi

I can't understand how I as an operator shall use the backup system Cloudron offers.

Yesterday I made a mistake and deleted an app I shouldn't have deleted (root cause).

Question time:
Given that I only remember the name of the container, how can I restore the backup of the container?
I don't have the UUID, I don't have the BackupID. I have full access to the object storage, api keys, secrets and encryption key.

My object storage is teeming with backups, Cloudron is set to do it every 6 hours. And this succeeds day in and day out. Great!

However It is not clear to me how to restore these backups.

In my Object storage I have backup archives like this

They seem to follow the naming convention of the uuid for the contiainer, OK that makes sense.

When I am going to restore my backups I have to provide these fields:

As far as I can tell, as an operator I need to keep track of what was the UUID of the container I deleted, in order to restore it. In addition I must also know what is the BackupID. More on that.

When I upload one of my backup config files, there seems to be reference to a BackupID ?
For one of my backups, this is the BackupID
2020-06-13-064435-141/box_2020-06-13-065422-378_v5.2.4 <--- If I don't have the Backup Config File how can I generate this ID? Or how to I find the ID of the container I accidentally deleted?

---

The only surefire way I can think of is to download the backup configuration file for every container and keep them in a folder on my onedrive for whenever disaster strikes such as now. Is this the way every one else are doing it?

Is there a place I can get an overview and piece together the variables and be able to do a restore of the container?

in order not to get stuck in the future, I now will need to do the following when I put an application in production (business critical):

• Create a spreadsheet containing a list of all my containers.
• Keep track of the UUID for each container.
• Download the backup configuration file for each container.
• Explicitly store the BackupID in the spreadsheet per application.
• Maybe there are more fields I must keep track of as well (???).
• Randomly test the restore capability of each container monthly.

Unless there are some elegant solutions here?

Guys this was my first brush with disaster recovery for an application I host on Cloudron and it not go down well.
If it wasn't for my random DigitalOcean Droplet snapshot(backups) and DigitalOcean scheduled VM backups, I would have been screwed here. I was not able to find the right backup archive and the right values to enter to correctly restore backup.

If anything this is a defect toward the documentation, because I cannot figure it out by reading this: https://cloudron.io/documentation/backups/ document. All of the great backup functionality in Cloudron didn't help me.

In Risk Management we use the Bowtie risk causal model and focus on our Barrier Management. The backups of Cloudron was supposed to be a recovery barrier.

This is not uncommon in IT, disaster recovery is a key think to test out. Unfortunately for me I hadn't done all the testing necessary as I didn't see myself ending up in this scenario (making a mistake and deleting a container I actually needed).

[girish]

@makemrproper said in Disaster recovery - Restoring a backup:

As far as I can tell, as an operator I need to keep track of what was the UUID of the container I deleted, in order to restore it.

Sorry for much confusion. Let me first try to explain how to import.

Finding the App's ID

Each Cloudron app has a unique ID. This is separate from the internal docker container id and you should never have to use any docker container id related stuff,

Now the question is how to figure out this unique ID of an uninstalled app. One way is to look into the event log. Select the uninstalled apps as the filter:

Then when you click on the event, you will see the app id:

Locating the backup

The backups are stored with the filename timestamp/app_<id>_v<packageversion>.tar.gz (for tgz backups) or timestamp/app_<id>_v<packageversion> (for rsync backups). The package version is needed in the case where you have a old version of an app and you want to restore to that old version.

Install an empty app

We have to next install the app for appstore. For this go to the appstore and install the app.

Notice the package version in the URL bar. Above, I am trying to install package v2.0.5 of wordpress. This version must match the version in the backup we located above.

Import app

Finally, go to app -> configure -> import

OK, all that said, what can be improved here. I see that the way to locate an uninstalled app's id is missing from the docs. I can add that part. Is there anything else we can fix on our side? Maybe we can keep the app's fqdn -> uuid listing alongside the backups?

[cbeams]

@girish There appears to be a bug when installing older, non-current versions by modifying the query string as mentioned above.

This is what it looks like when I try to install an older version of Ghost (v3.139.4) when the current version is 3.40.5:

Notice how the dialog tells me I'm going to be installing the current version versus the one I specified. I thought that this meant that the query string technique didn't (any longer) work, so I tried a bogus version number (v42.0.0) and got the following error message:

Then I figured it's just a display bug, and went ahead with the installation. The desired older version (v3.139.4) was indeed installed correctly, but the process was pretty confusing.

Perhaps more importantly, I would have had no way of knowing how to install an older version of an app if I hadn't seen this forum post and the tip above about changing the query string. I would have expected there to be a drop down, or at least text field available in the installation workflow to allow me to specify a version other than the most current.

I've since searched the docs as well, and I couldn't find anything on this feature. In any case, I'm glad it's there, and the process of testing out my remote backups against this older app version was incredibly smooth!

[nebulon]

@cbeams If I understand you correctly, I think the confusion comes from package version vs upstream app version. Those are basically never the same.

[girish]

@cbeams Yeah, this confusion comes up now and then. There is first the app's version - this is the upstream app version. Then there is the Cloudron package version. These two are totally separate. In the example above:

• This app packages Ghost 3.40.5. Here, 3.40.5 refers to the upstream Ghost version. This also means that should you want to move out (or into) of Cloudron, then you must be using Ghost 3.40.5.
• The package version is 3.139.4. This is what you see in the URL bar.

This has come up atleast a couple of other times in the past and that too with the very same app Ghost. This is because the package version and upstream version resemble each other. In fact, the last time someone complained, I randomly bumped up the package version by 100, in the hope of reducing this confusion (This is why it is 3.139.4. If I hadn't bumped it, it would be 3.39.4...)

If you have any suggestions on how to reduce this confusion, we are open to ideas. Also, if you go to an App's Update section, it will say more clearly the package version and upstream version. The main restriction we have is the package versions have to be semver. We rely on it heavily for the update mechanism. (so a complex debian style packaging version is quite hard to implement for us)

[cbeams]

@girish said in Disaster recovery - Restoring a backup:

If you have any suggestions on how to reduce this confusion, we are open to ideas.

Thanks for the explanation. Indeed, the visually similar version numbers did throw me off here. Perhaps one way to make things clearer would be to explicitly mention both app version and package version in the installation dialog. Then the reader can't miss the distinction.

@cbeams wrote:

Perhaps more importantly, I would have had no way of knowing how to install an older version of an app if I hadn't seen this forum post and the tip above about changing the query string. I would have expected there to be a drop down, or at least text field available in the installation workflow to allow me to specify a version other than the most current.

Just re-iterating this here, as I think it's actually the more important improvement request. Shouldn't there be a more first-class way to discover and install older app/package versions? This is an important use case for anyone doing disaster recovery for an app older than the latest version.

[girish]

@cbeams said in Disaster recovery - Restoring a backup:

Just re-iterating this here, as I think it's actually the more important improvement request

Yes, I think that's a good idea. Some drop down where we can select the package version instead of having to rely on the URL bar. Are you able to open a request here - https://forum.cloudron.io/category/97/feature-requests ?

think it requires some appstore changes since we also need some API on the appstore side to query all the available package version.

[cbeams]

@girish wrote:

Are you able to open a request [for this]?

Done, thanks: https://forum.cloudron.io/topic/4306/provide-ui-support-for-choosing-app-package-version-during-installation