Deleting Backups - unexpected behaviour

[luckow]

Me - on the mission to have a better understanding of Cloudrons backup workflow.
Make every day a Backup at 11PM. Total of backups: two.
Ok. Listing shows three backups, because of the update to 5.6.0 today. It will be 2 after 11PM.

Ok. Try it now. Clicking "Cleanup Backups". Expected behaviour: 0 backups in the listing. Wrong. Nothing changed. Looking into Event Log shows: 0 removed backups.

Ok. Let's compare it with the external backup storage (minio on a different Cloudron instance).

Ok. This needs more testing.

FYI: I have no idea if I changed anything in the backup settings in the last 10 days. my next testcase is to delete the whole endpoint, spin up a new minio instance. connect the Cloudron to the new storage endpoint and change nothing in the backup settings for the next 7 days. Let's see if everything works like expected.

[girish]

I forgot I documented this - https://docs.cloudron.io/backups/#retention-clean-up

[girish]

@luckow If you click on Clean up Backup and then click the Show Logs button, it will have a log of why each backup is preserved or deleted. It's a bit arcane to understand but if you paste the parts which say preservexxx , I can help decipher.

[luckow]

It doesn't look like preserveSecs makes sense

Sep 11 22:00:16 box:settings initCache: pre-load settings
Sep 11 22:00:16 box:taskworker Starting task 641. Logs are at /home/yellowtent/platformdata/logs/tasks/641.log
Sep 11 22:00:16 box:tasks 641: {"percent":2,"error":null}
Sep 11 22:00:16 box:tasks 641: {"percent":10,"message":"Cleaning box backups"}
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-10-230012-415/box_2020-09-10-230044-903_v5.5.0 box keepWithinSecs
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-10-210000-798/box_2020-09-10-210036-557_v5.5.0 box keepWithinSecs
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-09-210000-933/box_2020-09-09-210041-129_v5.5.0 box keepWithinSecs
Sep 11 22:00:16 box:backups cleanupBoxBackups: done
Sep 11 22:00:16 box:tasks 641: {"percent":40,"message":"Cleaning app backups"}
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-10-230012-415/app_e9f85919-502d-4b59-97ae-d073f2e54fff_2020-09-10-230042-063_v1.0.2 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-10-210000-798/app_e9f85919-502d-4b59-97ae-d073f2e54fff_2020-09-10-210033-848_v1.0.2 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-09-210000-933/app_e9f85919-502d-4b59-97ae-d073f2e54fff_2020-09-09-210038-556_v1.0.2 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-08-28-213000-891/app_e9f85919-502d-4b59-97ae-d073f2e54fff_2020-08-28-213001-983_v1.0.1 app preserveSecs
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-10-230012-415/app_af18b149-703c-429b-96fc-8cb0f2a5561c_2020-09-10-230040-949_v1.4.5 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-10-210000-798/app_af18b149-703c-429b-96fc-8cb0f2a5561c_2020-09-10-210032-175_v1.4.5 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-09-210000-933/app_af18b149-703c-429b-96fc-8cb0f2a5561c_2020-09-09-210036-955_v1.4.5 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-02-213000-992/app_af18b149-703c-429b-96fc-8cb0f2a5561c_2020-09-02-213002-048_v1.4.4 app preserveSecs
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-10-230012-415/app_a2d58898-6c99-48f3-bd91-84a73524721b_2020-09-10-230039-574_v1.7.0 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-10-210000-798/app_a2d58898-6c99-48f3-bd91-84a73524721b_2020-09-10-210030-626_v1.7.0 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-09-210000-933/app_a2d58898-6c99-48f3-bd91-84a73524721b_2020-09-09-210035-485_v1.7.0 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-10-230012-415/app_941bc6c6-72cb-416b-ba3f-9a112ad2d7c4_2020-09-10-230034-424_v1.1.0 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-10-210000-798/app_941bc6c6-72cb-416b-ba3f-9a112ad2d7c4_2020-09-10-210026-587_v1.1.0 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-09-210000-933/app_941bc6c6-72cb-416b-ba3f-9a112ad2d7c4_2020-09-09-210030-626_v1.1.0 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-10-230012-415/app_93a78c89-d2c2-46a1-b57e-2b8bf518d916_2020-09-10-230022-647_v1.0.2 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-10-210000-798/app_93a78c89-d2c2-46a1-b57e-2b8bf518d916_2020-09-10-210013-071_v1.0.2 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-09-210000-933/app_93a78c89-d2c2-46a1-b57e-2b8bf518d916_2020-09-09-210015-685_v1.0.2 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-08-31-213000-909/app_93a78c89-d2c2-46a1-b57e-2b8bf518d916_2020-08-31-213002-135_v1.0.1 app preserveSecs
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-08-25-033000-866/app_93a78c89-d2c2-46a1-b57e-2b8bf518d916_2020-08-25-033001-932_v1.0.0 app preserveSecs
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-10-230012-415/app_8e64a058-8d8c-41c4-8550-1b4019006fcc_2020-09-10-230021-162_v1.2.1 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-10-210000-798/app_8e64a058-8d8c-41c4-8550-1b4019006fcc_2020-09-10-210011-822_v1.2.1 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-09-210000-933/app_8e64a058-8d8c-41c4-8550-1b4019006fcc_2020-09-09-210014-183_v1.2.1 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-10-230012-415/app_75ad1fbb-f09a-46f3-8414-addfba6da593_2020-09-10-230019-641_v0.5.0 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-10-210000-798/app_75ad1fbb-f09a-46f3-8414-addfba6da593_2020-09-10-210008-519_v0.5.0 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-09-210000-933/app_75ad1fbb-f09a-46f3-8414-addfba6da593_2020-09-09-210011-099_v0.5.0 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-10-230012-415/app_6f4c222e-1adc-4b47-ad9e-9c5d40be46da_2020-09-10-230017-561_v0.7.0 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-10-210000-798/app_6f4c222e-1adc-4b47-ad9e-9c5d40be46da_2020-09-10-210005-865_v0.7.0 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-09-210000-933/app_6f4c222e-1adc-4b47-ad9e-9c5d40be46da_2020-09-09-210008-008_v0.7.0 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-03-213000-893/app_6f4c222e-1adc-4b47-ad9e-9c5d40be46da_2020-09-03-213005-022_v0.6.0 app preserveSecs
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-08-24-213000-924/app_6f4c222e-1adc-4b47-ad9e-9c5d40be46da_2020-08-24-213004-190_v0.5.0 app preserveSecs
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-10-230012-415/app_40bdc701-b03e-4989-858d-056b31fc3eb2_2020-09-10-230014-114_v1.9.1 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-10-210000-798/app_40bdc701-b03e-4989-858d-056b31fc3eb2_2020-09-10-210002-569_v1.9.1 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-09-210000-933/app_40bdc701-b03e-4989-858d-056b31fc3eb2_2020-09-09-210002-875_v1.9.1 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-08-27-213001-082/app_40bdc701-b03e-4989-858d-056b31fc3eb2_2020-08-27-213002-811_v1.9.0 app preserveSecs
Sep 11 22:00:16 box:backups cleanupAppBackups: done
Sep 11 22:00:16 box:tasks 641: {"percent":90,"message":"Cleaning snapshots"}
Sep 11 22:00:16 box:backups cleanupSnapshots: done
Sep 11 22:00:16 box:taskworker Task took 0.103 seconds
Sep 11 22:00:16 box:tasks setCompleted - 641: {"result":{"removedBoxBackupIds":[],"removedAppBackupIds":[]},"error":null}
Sep 11 22:00:16 box:tasks 641: {"percent":100,"result":{"removedBoxBackupIds":[],"removedAppBackupIds":[]},"error":null}

[girish]

@luckow One of the cases where preserveSecs is set is when an update happens. For example,

Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-09-210000-933/app_af18b149-703c-429b-96fc-8cb0f2a5561c_2020-09-09-210036-955_v1.4.5 app reference
Sep 11 22:00:16 box:backups applyBackupRetentionPolicy: 2020-09-02-213000-992/app_af18b149-703c-429b-96fc-8cb0f2a5561c_2020-09-02-213002-048_v1.4.4 app preserveSecs

See the version changed from 1.4.4 to 1.4.5. We keep the backups that were created as a result of an app update for 3 weeks. This gives the user up to 3 weeks to determine any regressions in the newer version and recover data.

The reference means that it's referenced by a box backup.

[girish]

I forgot I documented this - https://docs.cloudron.io/backups/#retention-clean-up

[luckow]

Reading documentation is sysadmins nightmare
Thank you for the clarification. That makes sense and it looks like a clever solution.

I guess that in case of a Cloudron trivia night, the question "How many backups do you expect in your storage location?" will be answered in many different ways according to the "important rules" from the documentation