Latest backup of stopped app must be retained

[girish]

I tried to reproduce this.

• I installed some apps.
• I took a full backup - this is required! Without this, the stopped app below won't have backups. We don't take backups of stopped apps.
• I stopped one app.
• Took another backup and downloaded the backup.
• Restored full server from the backup

Recently on a Cloudron server migration I discovered that a stopped app was not migrated. It was not a big thing so I left it that way.

The stopped app was migrated for me. It downloaded the backup and left the app in stopped state.

Now I noticed that a stopped app isn't back upped but also the existing backups are ALL deleted by the system wide backup retention settings.

The backups were not deleted for me (since it restored fine).

It seems the core issue is that the backup disappeared (which is probably why it didn't migrate either). Were the backups of the stopped app disabled by any chance? Otherwise, if you can tell me how to reproduce, I can try this again.

[robi]

@girish Set the retention to something low and see if the stopped app is part of the backups after a cleanup.

[girish]

@robi yes, I have checked that as well. The original report was about migration though and not cleanup related.

[robi]

@girish I can't tell if you're following the logic here..

..if the cleanup removed the backup, it would not be present after migration.

[scooke]

@robi But even following the logic, it makes sense. Cloudron did what it was asked if there was a short rentention of backups and one app's backup was cleaned up.. it's gone. This is not a problem of Cloudron. Now, the OP was saying that the ideal conditions were met (at least, they haven't confirmed what the stopped apps Backup settings were) - app stopped, long rentention of backups, Cloudron then migrated, and now no more stopped app nor its backup(s).

As I check my own setup, I often have some apps turned off, and they appear to be included in the Cloudron-wide backup, each time the entire Cloudron has been backed up. So somehow OP's entire Cloudron backup did not include that stopped app. Mine do.

Sorry for the rambling answer: I reread the post and OP isn't trying to find or restore that lost stopped app from the previous Cloudron. They are noting that stopped apps on the new Cloudron are not getting backed up as part of the entire Cloudron backup, and if an app had been running, then stopped, then all of its existing backups are removed at the next entire Cloudron backup. THIS is totally different from what we've been trying to address. THIS is only about backups of stopped apps on a Cloudron when an entire Cloudron backup event is triggered. Ignore "migration".

[imc67]

@robi @girish sorry for late response but I also suspect this behavior. It looked like the backup of the stopped app was removed and thus did not migrated.

That’s why my statement was like in the subject.

I still have one of my Cloudron’s having a stopped app without a “latest” backup but it was stopped in July so I wonder if there are still loggings?

[girish]

@imc67 yeah, the logs are probably not there anymore. One thing is if you go to the Eventlog section of the app, do you see if a backup was ever made?

I can confirm there is this quirk: If an app is stopped and backup was never created, then we don't have backups for this app. Maybe we have to inform the user about this somewhere.

[daixiwen]

Hi! I thought I'd write here instead of creating a new thread because I have a very similar problem. I have 3 stopped apps on my system and discovered all of a sudden that none of them have a backup any more. I wonder if there is a problem with the backup retention code. I managed to reproduce the problem this way:

• put the retention policy to something very low (like 2 days)
• stop an app and force a full system backup. The backup contains all the apps, including the stopped one.
• let the system do scheduled backups for a few days. The following backups don't contain the stopped app, since there wasn't any change.
• when the number of backups exceed the retention policy, the automatic clean up process removes the initial complete backup, leaving no backups of the stopped apps

I don't think this is expected behavior, the documentation does say that Cloudron should always keep at least one backup of a stopped app. I'll now try having at least one manual backup set up as persistent and see in two says if it's still there, but ideally this should occur automatically. I can see from this thread here that is was a problem in 2020, could it be a bug that was reintroduced later in Cloudron?

[scooke]

I hear you @daixiwen, but it also makes sense-- "exceed the retention policy", if there is no backup that means the app isn't being used, and so Cloudron, I think, rightly deletes the old backup of an unused app. I think a current attempt to avoid this is to archive an app, but to be honest I don't know what that means - the stopped, archived app IS ALSO the backup?

It is working the way it's intended, I think. You'll find plenty of other posts too where someone stopped an app, let it rest for a long time, and then when they decide to start it up again they discover that it is waaayy behind in Updates. Then they discover it is so far behind updates that they missed breaking changes, and now they can't update but have to start fresh. They blame Cloudron for that. Or, if Cloudron didn't delete a backup, another scenario happens where the user's disk fills up due to all these backups and stopped apps. Again, they blame Cloudron. I know, I know, Cloudron has removed sooo much friction and does pretty much everything perfectly that when something like this pops up, it's a shock, and it's Cloudron's fault.

When I realized this, several iterations ago, I realized that I needed to be more proactive in managing my server and Cloudron. If I stopped an app with the intention of leaving it for a while, I made sure to download the corresponding Backup Configuration, AND the actual backup from wherever it was. I stored them either on my local machine, or another server. This saved me a few times. I also ended up paying for a server with more storage so that I wouldn't accidentally run out of space, and then paid for a 3rd backup destination (either another VPS, or S3, B2, Minio, Scaleway storage, etc.).

Maybe the Cloudron devs can code something that works the way you think it should. Maybe they can't/won't. Is it as simple as some checkbox option that says, "When this box is checked, never delete this backup"? But then they'd have to code that for every app's backup. I am unaware of the complexities of the magic the Cloudron team pulls off. What I do know is ultimately, my data and my machine are under my care, and even though Cloudron makes me an Alita Battle Angel techwise, it's still my responsibility.

[daixiwen]

@scooke I can understand the logic, and I think I can fix this by making a manual backup persistent, but the documentation says here:

For installed apps and box backups, the latest backup is always retained regardless of the policy. This ensures that even if all the backups are outside of the retention policy, there is still at least one backup preserved. This change also ensure that the latest backup of stopped apps is preserved when not referenced by any box backup.

So either this is not working as intended, or the documentation needs to be updated, I think.

[scooke]

@daixiwen Even better, they HAVE already allowed for specific backup persistence:

I'm going to go try that out.

The wording definitely makes it sound like at least one backup, "the latest backup", should be available, but my point still stands. As perfect as Cloudron is, I've bumped into hiccups with backups myself, too. And so I pay close attention to them. In the situation you mentioned, you can either let your stopped app backups keep disappearing while you wait for a solution, or you can realize it needs a bit of your help. EDIT: I see you wrote " I'll now try having at least one manual backup set up as persistent and see in two says if it's still there, but ideally this should occur automatically." so good on you for taking it into your own hands. Let us know what happens!