No backups because of NullPointerException
-
@guyds there is retry logic already (see the 'retry' string in the logs) but it seems it kept crashing on retry.
Backups are indeed independent and each app is backed up separately. If you go into App -> Config -> Backup -> Create Backup, it will create a backup just for a specific app. But if you go to Backup -> Create Backup, it creates a "full backup" which is essentially looping through each app and creating a backup and then linking them all together. If one of those app backups fail, the "full backup" fails. Not sure what else can be done here. Do we want to have "full backups" but which are not full because 1 or 2 apps backups inside failed?
-
@girish said in No backups because of NullPointerException:
@guyds there is retry logic already (see the 'retry' string in the logs) but it seems it kept crashing on retry.
I'm sorry, I can't believe I missed that, my bad!
Backups are indeed independent and each app is backed up separately.
That is how I understood it as well, however:
If you go into App -> Config -> Backup -> Create Backup, it will create a backup just for a specific app. But if you go to Backup -> Create Backup, it creates a "full backup" which is essentially looping through each app and creating a backup and then linking them all together.
This is for manual backups, not how the backup scheduling works, which is an all or nothing approach atm.
If one of those app backups fail, the "full backup" fails. Not sure what else can be done here.
Yes, so that's basically my concern: a failure with one app's backup results in a total backup failure for all apps.
Do we want to have "full backups" but which are not full because 1 or 2 apps backups inside failed?
Yes, maybe that's better than having no backup at all because then at least some of the apps might be recoverable from the partial backup.
But why not providing the option to store/upload the per app backups separately as well?
Or even better: provide the option to either store them as a separate backup ("per app backup"), as a "full backup" (current behavior) or both.
This way the backup of one app can fail (e.g. temp error with external storage provider) while the other apps are still being backed up correctly.To be clear: I'm talking about scheduled backups here, not manual backups.
-
While it's possible to implement, that's not how the backup system works right now. Internally, it behaves it pretty much as "separate" backup but we show it as a bundle because we thought that's the best way the user can easily understand.
But ultimately, we rely on the fact that most of the storage backend services are reasonably stable (which they are. I think the B2 service is quite new, so maybe they have some instabilities).
-
@girish said in No backups because of NullPointerException:
While it's possible to implement, that's not how the backup system works right now. Internally, it behaves it pretty much as "separate" backup but we show it as a bundle because we thought that's the best way the user can easily understand.
I understand that internally backups are actually already separate per app, but they aren't worth anything if the final bundle can't be stored / uploaded.
But ultimately, we rely on the fact that most of the storage backend services are reasonably stable (which they are. I think the B2 service is quite new, so maybe they have some instabilities).
In my experience (as a developer) it's very dangerous to rely on / asume that a(ny) service is always stable.
But that's just my experience of course
-
@guyds said in No backups because of NullPointerException:
In my experience (as a developer) it's very dangerous to rely on / asume that a(ny) service is always stable.
While that's definitely understandable, keep in mind that backups by their very nature (and to be of any value) need to be saved to external systems. Thus, Cloudron will always be at the mercy of external services for backups. So stability isn't something they can guarantee, and retry logic is already built in to avoid some of the smaller hiccups that can occur.
If you want true stability for your backups, I'd recommend using block storage instead of object storage services. Of course block storage is a bit more expensive, but far more reliable and far quicker too.
-
@d19dotca Yes, block storage can be more reliable, but my point is not regarding the backup technology being used for the backups.
I'm just suggesting that separating the backups on a per app basis can have some advantages as well: apps don't depend on each other for their backup to succeed, backups can be spread more easily, backups can be turned on/off on a per app basis, etc. -
The code is written with backup storage being reasonable reliable as @d19dotca mentioned. But I also see @guyds point as to why throw away incomplete backups. TBH, I have never thought about those.
Is there value in partial backups? Currently, these backups are tracked internally and cleaned up (since it was considered to have no value).
-
Thinking about this a bit more:
- As of now, a full box backup = backing up apps one by one and then box code. Maybe, we should reverse this order. i.e box code first and then apps (reason below).
- We can show "partial" backups (i.e one or more app backups failed) in the Backup view with some UI hint/tooltip. If box backups itself fails, it won't appear at all. If box backup succeeded but one or more apps failed, then it shows a entry there with the list of apps that succeeded (like it does now). A box backup is required for the Cloudron "restore" process which is why I suggested the reorder in the first bullet.
Now, the question is if this will confuse people. Maybe with the right UI hint, we can make it clear that it's not a full backup but Cloudron backed up whatever it could.
-
@girish I like your suggestion to do the box data first (since that's probably the largest one in most cases I assume? I know it is by far the largest for me anyways) and then app backups after, failing entirely if the box backup doesn't succeed.
I'm a bit uneasy to the "partial" backup idea because I'd worry that may cause people to assume backups are working when they're not 100% "safe". Something symbolizing them as different would be a partial answer to that, but I would recommend the admin user gets an email notification (as we currently do) if backups fail, even if it's a partial fail. That way the admin user can quickly try to fix it without assuming it's all good still.
In other words, my concern would be in making sure admins aren't going to become "complacent" with their backups when partial backups succeed but still contain an error in them. I think as long as those partial backups still notify the admin user via an email alert, then that'd be a way to prevent complacency. Last thing you probably want is people who have a catastrophic failure and later realize their backups weren't 100% complete.
-
@d19dotca @girish As I see it there should be 2 backup "levels" and actually those are already in place, but for some reason they aren't treated like that in the end.
Let me clarify this...The first level is the per app level. Right now, apps are already backed up one by one, but they are not stored nor reported individually. And this is the missing feature in my opinion.
So as I see it, once an app is backed up, it should be stored "as is" (as an individual backup) in the backup target location and reported in the backup tab of the app (where one can restore it, clone it or download it just as it is now, only it will be from the individual backup instead of the full box backup). If a backup fails for an app, then there will be no backup for that app at that date and preferably (optionally?) an email is sent to the admin.Then the second level is combining those individual backups and add the box code and store it as the full box backup, exactly how it is done already. This full backup is accessible via the profile menu -> backups and should allow a user to restore the full cloudron to that particular state. If one of the app backups failed for whatever reason, then there will be no full backup available for that date and the admin should receive an email (= the current behavior).
So, again, I think the current issue is that everything is treated as a whole while it makes more sense in my opinion to treat each app individually and then in the end (optionally?) bundle the individual parts as a whole.
Does this make sense?
Or am I just freaking out here?
-
@guyds said in No backups because of NullPointerException:
The first level is the per app level. Right now, apps are already backed up one by one, but they are not stored nor reported individually. And this is the missing feature in my opinion.
Ah no, they are listed in app -> backups. So, even if you do a full backup, each individual app backup will be listed in app -> backups. You can also use that backup to restore/clone the app.
So, again, I think the current issue is that everything is treated as a whole while it makes more sense in my opinion to treat each app individually and then in the end (optionally?) bundle the individual parts as a whole.
Yes, so they are treated individually, it's actually very close to what you want. The only issue is that when a full backup fails, those successful individual app backups that are part of a failed full backup will get removed/cleaned up every night.
I have made a task to fix the behavior, let's see.
-
@girish said in No backups because of NullPointerException:
@guyds said in No backups because of NullPointerException:
The first level is the per app level. Right now, apps are already backed up one by one, but they are not stored nor reported individually. And this is the missing feature in my opinion.
Ah no, they are listed in app -> backups. So, even if you do a full backup, each individual app backup will be listed in app -> backups.
Yes, they are listed at the app level, but there's no reporting at the app level because the backup succeeds or fails at the box level.
You can also use that backup to restore/clone the app.
Yes, but only if the backup succeeds for all apps.
So, again, I think the current issue is that everything is treated as a whole while it makes more sense in my opinion to treat each app individually and then in the end (optionally?) bundle the individual parts as a whole.
Yes, so they are treated individually, it's actually very close to what you want. The only issue is that when a full backup fails, those successful individual app backups that are part of a failed full backup will get removed/cleaned up every night.
Exactly, they are per app bot not treated like that in the end because success or failure is determined by the whole box.
I have made a task to fix the behavior, let's see.
Great, thanks!
