Cloudron not removing expired backups from deleted apps

[d19dotca]

I'm not sure if this is intentional or not, but it seems "wrong" to me so wanted to raise this. I noticed in reviewing my backups recently that it still had many from way past the retention period and the commonality in them all were they were apps that had been removed (i.e. running for a week or so to test things then deleted from Cloudron), so it seems like Cloudron is not deleting backups after the retention policy expires for previously deleted applications.

[girish]

@d19dotca Thanks for reporting. This seems like a bug/regression because all app backups are cleaned up per backup policy and the code doesn't depend on installation state.

Let me test this and get back.

[girish]

@d19dotca I tested this out and it seems they are getting removed for me. If you run the 'Cleanup backups' manually, can you check the logs why those specific backups are getting retained ?

Note that if an app was updated and then later deleted, the quirk of the backup system is that the backups that got created during the app backup are still retained for 3 weeks. Could this be the case?

[d19dotca]

@girish said in Cloudron not removing expired backups from deleted apps:

Note that if an app was updated and then later deleted, the quirk of the backup system is that the backups that got created during the app backup are still retained for 3 weeks. Could this be the case?

I don't think it'd have been that because the backups went as far back as November 2022, well over the 3 weeks.

@girish said in Cloudron not removing expired backups from deleted apps:

If you run the 'Cleanup backups' manually, can you check the logs why those specific backups are getting retained ?

This may be difficult as I had run through and manually removed them when I reported the issue (sorry about that). I did it now though and don't see ay errors, but may be because I already removed a lot of the outdated ones earlier.

[girish]

Alright, so if someone else hits this please leave a note here.

[d19dotca]

@girish Somewhat related, I decided to clear out all my 2022 backups, and I then ran the Cleanup Backups option in Cloudron to keep things in sync. Unfortunately it still shows a backup from November 30th 2022 in the list of Box Backups. I got the cleanup logs from before the deletion of the 2022 backups and then the cleanup logs from after the deletion of 2022 backups. There seems to be something broken at least in my environment with regards to the cleanup functionality in Cloudron backups.

Currently I'm not seeing any real differences when comparing them. Here's the end of the logs for the latest run, doesn't seem like anything was removed but not sure if I'm reading it correctly:

2023-02-09T05:59:29.699Z box:backupcleaner cleanupAppBackups: done
2023-02-09T05:59:29.699Z box:tasks update 19006: {"percent":70,"message":"Checking storage backend and removing stale entries in database"}
2023-02-09T05:59:41.204Z box:backupcleaner cleanupMissingBackups: done
2023-02-09T05:59:41.204Z box:tasks update 19006: {"percent":90,"message":"Cleaning snapshots"}
2023-02-09T05:59:41.316Z box:backupcleaner cleanupSnapshots: done
2023-02-09T05:59:41.316Z box:taskworker Task took 11.845 seconds
2023-02-09T05:59:41.316Z box:tasks setCompleted - 19006: {"result":{"removedBoxBackupPaths":[],"removedMailBackupPaths":[],"removedAppBackupPaths":[],"missingBackupPaths":[]},"error":null}
2023-02-09T05:59:41.316Z box:tasks update 19006: {"percent":100,"result":{"removedBoxBackupPaths":[],"removedMailBackupPaths":[],"removedAppBackupPaths":[],"missingBackupPaths":[]},"error":null}

[girish]

@d19dotca Did you change the backup provider in the past? Currently, if you switch things, we lose track of the backups on the remote (Even if you switch it back)

[d19dotca]

@girish I’ve changed backup providers before but not since October sometime. It’s been the same for a few months now. But even if I had changed backup providers, isn’t the cleanup supposed to remove any backups from Cloudron that it doesn’t see in the current setup? Maybe I’ve misunderstood that part.

[girish]

@d19dotca said in Cloudron not removing expired backups from deleted apps:

But even if I had changed backup providers, isn’t the cleanup supposed to remove any backups from Cloudron that it doesn’t see in the current setup?

What I meant is:

1. You had backups in service A
2. You changed backups to service B
3. You changed backups back to service A

In the above situation, Cloudron loses track of backups in service A after step 2. When the backup cleaner runs, it will see that backup entries in it's database don't exist in the remote (now B) anymore and it will remove them from it's database. When you then do step 3, it won't delete those old backups because it has lost track of them.

This may not apply to you, but just something to keep in mind.

[d19dotca]

@girish said in Cloudron not removing expired backups from deleted apps:

@d19dotca said in Cloudron not removing expired backups from deleted apps:

But even if I had changed backup providers, isn’t the cleanup supposed to remove any backups from Cloudron that it doesn’t see in the current setup?

What I meant is:

1. You had backups in service A
2. You changed backups to service B
3. You changed backups back to service A

In the above situation, Cloudron loses track of backups in service A after step 2. When the backup cleaner runs, it will see that backup entries in it's database don't exist in the remote (now B) anymore and it will remove them from it's database. When you then do step 3, it won't delete those old backups because it has lost track of them.

This may not apply to you, but just something to keep in mind.

Oh I see what you mean now. Okay yeah anytime I've changed providers I've also done that "reset" thing you taught me by changing Backups to no-op, running the Clean Backups part, and then back again to the new provider settings to avoid all those linking issues.

[d19dotca]

@girish - it seems this is still an issue. I checked today and in my IDrive e2 storage bucket, I see the oldest bucket of March 20, 2023 and it's only for one app in particular, so I assumed it was one that I saved or something, however it doesn't appear to be the case.

What I noticed is Cloudron doesn't have any instances of it in the Backups list for the app, however I do see a difference in the package version so I assume this might be why, though I thought that only still stayed around for 3 weeks or something like that. The package version in the expired backup is 3.0.3, but Cloudron shows only up to 3.0.4 in the app-level backup for this particular one.

Currently running Cloudron v7.4.2. I haven't touched my backup settings (apart from maybe the retention period, but not the provider), so I don't think there should be any linking issues. Unclear what's going on here. Since I'm in the middle of a few web design projects with active work, I am wanting to avoid changing any backup settings if I can. I know in the latest Cloudron there's an actual IDrive e2 storage option but I was using it before that was provided so I have it set to the "s3-compatible" option, in case that is making any difference here.

[girish]

@d19dotca said in Cloudron not removing expired backups from deleted apps:

What I noticed is Cloudron doesn't have any instances of it in the Backups list for the app

mmm, without inspecting logs, it's hard to know. You have to fish for errors in the logs, maybe there is something in the event log ? But only the current release keeps events for 3 months. It seems that Cloudron thinks it deleted the backup but IDrive e2 did not. One thing is because IDrive is relative new (to us), we don't know how stable that service is. Maybe there are some special network errors/up time issues that we need to handle.

[d19dotca]

@girish I definitely don't see any errors in the logs when I looked before, but it's pretty chatty so maybe I missed something. I'm happy to let you inside the server if needed, but it's certainly not a high priority and I've got a few projects on the go where I really don't want to mess up the backups (can't afford to lose/reset them currently). So maybe I can tackle this again in a couple of months when the backups are slightly less critical, if you prefer that, that works for me too.

[girish]

@d19dotca Yes, that sounds good. We also fixed the logs view now in 7.5 to give a history of past logs. The current one is hard to work with when debugging such things.