Stopped apps not getting backed up, is this intentional?
-
@girish Yes, that'd be perfect. Thank you. If it's not a defect for anyone else, then I guess I'm alone for some reason. haha. I could use your help either way though, so if you want remote access just let me know. I can say that my backups are scheduled to run every 6 hours and are successful as I can see all the backups on all other apps that are running. But the stopped app so far today still has the latest backup as "yesterday", and that was a manual one I took. There's basically been no backups on it at all that were auto-created.
wrote on Apr 29, 2020, 10:28 AM last edited by -
I forgot to update this topic yesterday. Yes, I can reproduce this myself as well. Looking into if this is a UI regression or something in the backend. It's most likely a UI issue, because we have not intentionally changed anything on this part of code.
-
@d19dotca @jdaviescoates This was indeed a regression. And one that would have bit us hard when we update to Focal (since we rely on backups to upgrade). Thanks for reporting, I have made a fix for the next release. From what I can make out, it's been broken since 5.0 (so past 3 months or so).
In the meantime, you can apply this yourself if you like, it's a one-liner (just remove the
app.health === apps.HEALTH_HEALTHYpart) in/home/yellowtent/box/src/backups.jsand thensystemctl restart box.https://git.cloudron.io/cloudron/box/-/commit/73d83ec57ef83a29543cda61e2e4bf7bc4b30f9a
-
@d19dotca @jdaviescoates This was indeed a regression. And one that would have bit us hard when we update to Focal (since we rely on backups to upgrade). Thanks for reporting, I have made a fix for the next release. From what I can make out, it's been broken since 5.0 (so past 3 months or so).
In the meantime, you can apply this yourself if you like, it's a one-liner (just remove the
app.health === apps.HEALTH_HEALTHYpart) in/home/yellowtent/box/src/backups.jsand thensystemctl restart box.https://git.cloudron.io/cloudron/box/-/commit/73d83ec57ef83a29543cda61e2e4bf7bc4b30f9a
-
@girish I'm not sure what Focal is, but you're welcome I guess? haha. Thanks for fixing it so quickly!
-
@d19dotca @jdaviescoates This was indeed a regression. And one that would have bit us hard when we update to Focal (since we rely on backups to upgrade). Thanks for reporting, I have made a fix for the next release. From what I can make out, it's been broken since 5.0 (so past 3 months or so).
In the meantime, you can apply this yourself if you like, it's a one-liner (just remove the
app.health === apps.HEALTH_HEALTHYpart) in/home/yellowtent/box/src/backups.jsand thensystemctl restart box.https://git.cloudron.io/cloudron/box/-/commit/73d83ec57ef83a29543cda61e2e4bf7bc4b30f9a
wrote on May 31, 2020, 7:37 AM last edited by d19dotca May 31, 2020, 7:39 AM@girish Hi Girish, it seems that stopped apps still aren't being backed up (at least not automatically). Was this supposed to be fixed by now? I assumed it would have been but maybe not?
For example, one of my WordPress Unmanaged sites was on package 2.2.0-1 the other day, but no backups for it in the list. Then when I updated today to 2.3.0, there was a backup for 2.2.0-1 (because of the update) but nothing else for 2.2.0-1 when my Cloudron is being backed up every 6 hours and I had updated it on Thursday or Friday I believe to 2.2.0-1 at that time. I start it, update, then stop it again as it's a template app for me so I don't want it constantly running, trying to save resources.
I'm on Cloudron 5.2.4 right now.
-
@d19dotca Indeed, there is some change in behavior in 5.2 (and I think I found a bug as well with your report).
We create an app backup by telling each db addon to create a "dump". The system addons like MySQL/Mongo etc are always running. In the case of redis, there is a new redis instance per-app. In previous releases, even when the app was stopped, we used to leave redis running.
In 5.2, now that we exposed redis in Services view, we thought people will surely complain that redis is now running and wasting resources despite the app being stopped. So, we decided to stop all app addons as well.
This has the side-effect that we cannot take fresh "dump". So, it was decided that if an app is stopped, then instead of taking a "fresh" backup, we will simply "reuse" the last known good backup of an app.
This means that:
- If an app has no backup to start with and you stop the app, this app is never backed up.
- If you create a backup, then stop the app. All is good. Cloudron will re-use this backup for future backups.
- If you create a backup, then make changes in the app and just stop the app, then the changes you made are not part of the backup because we are only reusing the last known good backup and the changes are not part of a backup.
Finally, I tried to investigate how one could ever end up with no backups at all (like in your other post). I think I found a bug in our backup cleanup logic. This backup cleanup logic deletes app backups that are not referenced by anything. I will get this fixed in the next release. But a workaround is this:
- Start the app
- Take a full box backup. ie Backups -> Create Backup (i.e full Cloudron backup). This creates the app backup as well as creates the necessary reference which will keep cleanup logic happy.
- After this, the stopped app backup will be preserved.
Can you let me know if the above works?
-
@d19dotca Indeed, there is some change in behavior in 5.2 (and I think I found a bug as well with your report).
We create an app backup by telling each db addon to create a "dump". The system addons like MySQL/Mongo etc are always running. In the case of redis, there is a new redis instance per-app. In previous releases, even when the app was stopped, we used to leave redis running.
In 5.2, now that we exposed redis in Services view, we thought people will surely complain that redis is now running and wasting resources despite the app being stopped. So, we decided to stop all app addons as well.
This has the side-effect that we cannot take fresh "dump". So, it was decided that if an app is stopped, then instead of taking a "fresh" backup, we will simply "reuse" the last known good backup of an app.
This means that:
- If an app has no backup to start with and you stop the app, this app is never backed up.
- If you create a backup, then stop the app. All is good. Cloudron will re-use this backup for future backups.
- If you create a backup, then make changes in the app and just stop the app, then the changes you made are not part of the backup because we are only reusing the last known good backup and the changes are not part of a backup.
Finally, I tried to investigate how one could ever end up with no backups at all (like in your other post). I think I found a bug in our backup cleanup logic. This backup cleanup logic deletes app backups that are not referenced by anything. I will get this fixed in the next release. But a workaround is this:
- Start the app
- Take a full box backup. ie Backups -> Create Backup (i.e full Cloudron backup). This creates the app backup as well as creates the necessary reference which will keep cleanup logic happy.
- After this, the stopped app backup will be preserved.
Can you let me know if the above works?
wrote on Jun 2, 2020, 1:28 AM last edited by@girish hmm, okay thank you for the detailed reply. I think this makes sense to me. The only concern I have or am left wondering is what happens to apps that are almost always stopped with regards to backup data retention policies? For example, I have a Wordpress app I use as a template to clone from so that it already has all the standard plugins and settings I like to use as the site admin. And it’s basically always stopped unless I’m making updates. What happens to it if it’s stopped for a month for example; and the backup data retention policy is set to a week? Does the backup for it get removed still or is it marked special and never deleted?
-
@d19dotca As long as the app backup is part of a full cloudron (box) backup, it will be retained since box backup and all it's containing apps are retained per the policy.
The above is the behavior in 5.2. Currently, only the box backups determine whether a backup is kept or not. I think the retention policies should be applied to app backups as well. This way, if you made an app backup, it won't get cleaned out immediately... (since no box backup is referencing it). I will get this fixed for next release.
-
@d19dotca As long as the app backup is part of a full cloudron (box) backup, it will be retained since box backup and all it's containing apps are retained per the policy.
The above is the behavior in 5.2. Currently, only the box backups determine whether a backup is kept or not. I think the retention policies should be applied to app backups as well. This way, if you made an app backup, it won't get cleaned out immediately... (since no box backup is referencing it). I will get this fixed for next release.
wrote on Jun 4, 2020, 6:23 PM last edited by@girish So that makes it sound like it won't actually be retained though, no? If the box backups are stored per the policy and it includes that app that's stopped, then if I made a backup a month ago it'd be removed per the policy if it's set to a week, right? Or did I misunderstand? Are you saying the individual app backups (not part of the system-wide box backup) are permanent, no retaining policy applies to those ones?
-
@d19dotca Just wanted to leave a note that there are some discrepancies in code. I will get it sorted out and give a concrete reply - https://git.cloudron.io/cloudron/box/-/issues/692