Cloudron Forum

@timconsidine said in Moving Cloudron to new server : stop apps?: that I didn't set the PTR record in the new VPS provider until I started the migration I ran into exactly this problem the last time as well

To conclude this, it was a memory issue. The instance as a whole was a bit overcommited. If the backup task is idle, it won't consume any memory. Also Cloudron does not reserve memory based on the limits set, neither for backup nor for apps. The limit is just to avoid rouge apps or the backup task to bring kill other apps.

@leggias hm getting a SIGSEGV is odd here. If you like you can enable remote SSH support and send us a mail to support@cloudron.io mentioning your dashboard domain. Then we can take a direct look at your setup to get down to the issue.

@fbartels said in Backup Preference: Snapshot vs S3?: Hi @jordanurbs, the viability of snapshots for backup has for example been discussed in https://forum.cloudron.io/topic/4161/disc-snapshots?_=1628139201934. Yes, and to quote what @nebulon pointed out there: @nebulon said in Disc snapshots: Disk snapshots or VPS snapshots don't work well with Cloudron, since backups are per-app and not per-server on Cloudron. This is to be able to rollback/restore individual apps without interfering with the system or other apps running on it. Still for a secondary fallback backup solution this can still be recommended. That's what I do. I do Cloudron backups to a Hetzner Storage Box, plus pay Hetzner for their automated backups, plus occasionally do a snapshot too. Although really I ought to backup to another provider too, to avoid the potential issues @girish once had with DO. I'll probably choose another eco VPS provider to do that at some point https://ud.coop/ecovps

@michaelf I put some notes on how to read the logs at https://docs.cloudron.io/backups/#cleanup-backups and also replied to your support ticket. Do you see any backups in linode object storage that are not listed in the logs? Otherwise, the logs do look fine to me.

@robi there is a already bunch of "workarounds" for rsync. Empty directories, executable bit of files cannot be stored in most object storage. So, there is fsmetadata.json file that stores this information outside of the files. When restoring, we use that file to restore back the state. I guess we can extend that file to also save and restore timestamps. If anyone wants this leave a note and I can look into it in the future.

@rmdes I ended up setting up a local backup for Cloudron and use RClone to handle the backup. I've used RClone for years with no issue, and since going this route, no issues with my Cloudron backup.

Thanks a bunch @girish worked like a charm

@girish Ahhh I see. Thankfully right now backups only take 3-4 hours. I'm hoping once I'm at gigabit I can speed that up. My network is definitely a bottleneck rn.

@mazarian that's correct, the permission is the thing I fixed in 6.3.5. Glad it's working now and thanks for reporting back!

@girish do you need anything from me or can you replicate that by yourself?

@girish that worked. Thanks

@d19dotca said in Backups failing on same app each time: By the way, since there's a ton of backup crashing that seem memory related... is it possible to improve the logging so that there's something in there saying "OutOfMemory" or something like that so poeple know why it's crashing? Yeah, I would love this as well. Sadly, there seems to be no way on linux to know this reliably. A process just gets killed unceremoniously with SIGKILL and that's that If anyone know how this can be detected would be great.

We are using minIO for it. s3 compatible. the Service on our side was down… and then we also have a problem with backup size. but now it works! thx!

@dylightful if you click the 'cleanup backups' button in the Backups view, is there anything interesting in the logs?

@nebulon you would need a list of app backups that are allowed to fail (app store apps) and ones that aren't (system services). Partial backups are only ok if an app store app gets an error.

@privsec Note, just because it's "running" doesn't mean it's running well.

Not sure what was going on before but things appear to be working fine now. Thank you @girish for the clarification on the different types of backups referenced in the log and the explanation that preserveSecs will remain for 3 weeks regardless of the normal backup retenetion period. cmagnuson@cloudron:/mnt/cloudronbackup$ du -h --max-depth=1 /mnt/cloudronbackup/ du: cannot read directory '/mnt/cloudronbackup/.SecuriSyncFileServer': Permission denied 0 /mnt/cloudronbackup/.SecuriSyncFileServer 0 /mnt/cloudronbackup/2021-05-09-100000-534 13K /mnt/cloudronbackup/2021-05-12-090222-976 116M /mnt/cloudronbackup/2021-05-13-090000-459 76M /mnt/cloudronbackup/2021-05-15-090000-481 3.9M /mnt/cloudronbackup/2021-05-18-090000-491 18K /mnt/cloudronbackup/2021-05-18-110000-453 2.3M /mnt/cloudronbackup/2021-05-19-090000-454 34M /mnt/cloudronbackup/2021-05-20-090000-492 25M /mnt/cloudronbackup/2021-05-20-090000-509 120M /mnt/cloudronbackup/2021-05-20-090000-510 5.1G /mnt/cloudronbackup/2021-05-20-110000-521 4.5K /mnt/cloudronbackup/2021-05-21-090140-989 337K /mnt/cloudronbackup/2021-05-21-090141-009 65M /mnt/cloudronbackup/2021-05-21-110141-085 13K /mnt/cloudronbackup/2021-05-22-090000-495 66M /mnt/cloudronbackup/2021-05-22-110000-663 66M /mnt/cloudronbackup/2021-05-23-110000-475 65M /mnt/cloudronbackup/2021-05-24-110000-457 68M /mnt/cloudronbackup/2021-05-25-110000-458 5.5K /mnt/cloudronbackup/2021-05-26-090000-515 5.5M /mnt/cloudronbackup/2021-05-26-090000-541 331M /mnt/cloudronbackup/2021-05-26-110000-804 28K /mnt/cloudronbackup/snapshot 6.1G /mnt/cloudronbackup/ 2021-05-26T16:16:31.424Z box:settings initCache: pre-load settings 2021-05-26T16:16:31.433Z box:taskworker Starting task 2042. Logs are at /home/yellowtent/platformdata/logs/tasks/2042.log 2021-05-26T16:16:31.435Z box:tasks 2042: {"percent":2,"error":null} 2021-05-26T16:16:31.440Z box:tasks 2042: {"percent":10,"message":"Cleaning box backups"} 2021-05-26T16:16:31.443Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/box_2021-05-26-110120-087_v6.2.8 box keepWithinSecs 2021-05-26T16:16:31.443Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/box_2021-05-25-110030-827_v6.2.8 box keepWithinSecs 2021-05-26T16:16:31.443Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/box_2021-05-24-110030-725_v6.2.8 box keepWithinSecs 2021-05-26T16:16:31.443Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/box_2021-05-23-110032-528_v6.2.8 box keepWithinSecs 2021-05-26T16:16:31.443Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/box_2021-05-22-110031-980_v6.2.8 box keepWithinSecs 2021-05-26T16:16:31.443Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/box_2021-05-21-110211-336_v6.2.8 box keepWithinSecs 2021-05-26T16:16:31.443Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/box_2021-05-20-110030-471_v6.2.8 box keepWithinSecs 2021-05-26T16:16:31.443Z box:backups cleanupBoxBackups: done 2021-05-26T16:16:31.443Z box:tasks 2042: {"percent":40,"message":"Cleaning app backups"} 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_d2fb1510-462d-45d0-84b8-8143184152fc_2021-05-26-110118-719_v2.1.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_d2fb1510-462d-45d0-84b8-8143184152fc_2021-05-25-110029-455_v2.1.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_d2fb1510-462d-45d0-84b8-8143184152fc_2021-05-24-110029-345_v2.1.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_d2fb1510-462d-45d0-84b8-8143184152fc_2021-05-23-110031-187_v2.1.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_d2fb1510-462d-45d0-84b8-8143184152fc_2021-05-22-110030-638_v2.1.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_d2fb1510-462d-45d0-84b8-8143184152fc_2021-05-21-110209-942_v2.1.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_d2fb1510-462d-45d0-84b8-8143184152fc_2021-05-20-110029-188_v2.1.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_ef14e130-abca-436c-9dd9-70748cf0a9a6_2021-05-26-110119-468_v1.4.7 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_ef14e130-abca-436c-9dd9-70748cf0a9a6_2021-05-25-110030-222_v1.4.7 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_ef14e130-abca-436c-9dd9-70748cf0a9a6_2021-05-24-110030-120_v1.4.7 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_ef14e130-abca-436c-9dd9-70748cf0a9a6_2021-05-23-110031-935_v1.4.7 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_ef14e130-abca-436c-9dd9-70748cf0a9a6_2021-05-22-110031-388_v1.4.7 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_ef14e130-abca-436c-9dd9-70748cf0a9a6_2021-05-21-110210-722_v1.4.7 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-21-090141-009/app_ef14e130-abca-436c-9dd9-70748cf0a9a6_2021-05-21-090141-659_v1.4.6 app preserveSecs 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_ef14e130-abca-436c-9dd9-70748cf0a9a6_2021-05-20-110029-891_v1.4.6 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-13-090000-459/app_ef14e130-abca-436c-9dd9-70748cf0a9a6_2021-05-13-090001-079_v1.4.5 app preserveSecs 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_cb6982cb-7f7b-492b-a25f-bb1c443c320f_2021-05-26-110115-857_v1.11.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_cb6982cb-7f7b-492b-a25f-bb1c443c320f_2021-05-25-110026-458_v1.11.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_cb6982cb-7f7b-492b-a25f-bb1c443c320f_2021-05-24-110026-430_v1.11.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_cb6982cb-7f7b-492b-a25f-bb1c443c320f_2021-05-23-110027-861_v1.11.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_cb6982cb-7f7b-492b-a25f-bb1c443c320f_2021-05-22-110027-450_v1.11.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_cb6982cb-7f7b-492b-a25f-bb1c443c320f_2021-05-21-110206-980_v1.11.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_cb6982cb-7f7b-492b-a25f-bb1c443c320f_2021-05-20-110026-387_v1.11.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_c6169a84-917b-4f21-9ac4-2d3178f2b61e_2021-05-26-110114-671_v1.8.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_c6169a84-917b-4f21-9ac4-2d3178f2b61e_2021-05-25-110025-294_v1.8.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_c6169a84-917b-4f21-9ac4-2d3178f2b61e_2021-05-24-110025-242_v1.8.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_c6169a84-917b-4f21-9ac4-2d3178f2b61e_2021-05-23-110026-686_v1.8.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_c6169a84-917b-4f21-9ac4-2d3178f2b61e_2021-05-22-110026-219_v1.8.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-090000-495/app_c6169a84-917b-4f21-9ac4-2d3178f2b61e_2021-05-22-090001-059_v1.7.2 app preserveSecs 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_c6169a84-917b-4f21-9ac4-2d3178f2b61e_2021-05-21-110205-780_v1.7.2 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_c6169a84-917b-4f21-9ac4-2d3178f2b61e_2021-05-20-110025-278_v1.7.2 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_c95e1f50-f42d-4e18-b646-935a62a8eb63_2021-05-26-110115-279_v0.2.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_c95e1f50-f42d-4e18-b646-935a62a8eb63_2021-05-25-110025-891_v0.2.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_c95e1f50-f42d-4e18-b646-935a62a8eb63_2021-05-24-110025-848_v0.2.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_c95e1f50-f42d-4e18-b646-935a62a8eb63_2021-05-23-110027-294_v0.2.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_c95e1f50-f42d-4e18-b646-935a62a8eb63_2021-05-22-110026-870_v0.2.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_c95e1f50-f42d-4e18-b646-935a62a8eb63_2021-05-21-110206-378_v0.2.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_c95e1f50-f42d-4e18-b646-935a62a8eb63_2021-05-20-110025-848_v0.2.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_ab9e5ab3-7cd6-4da1-8e23-b723ac717ebe_2021-05-26-110113-910_v1.6.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_ab9e5ab3-7cd6-4da1-8e23-b723ac717ebe_2021-05-25-110024-539_v1.6.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_ab9e5ab3-7cd6-4da1-8e23-b723ac717ebe_2021-05-24-110024-365_v1.6.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_ab9e5ab3-7cd6-4da1-8e23-b723ac717ebe_2021-05-23-110025-901_v1.6.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_ab9e5ab3-7cd6-4da1-8e23-b723ac717ebe_2021-05-22-110025-349_v1.6.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_ab9e5ab3-7cd6-4da1-8e23-b723ac717ebe_2021-05-21-110205-018_v1.6.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_ab9e5ab3-7cd6-4da1-8e23-b723ac717ebe_2021-05-20-110024-571_v1.6.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-090000-509/app_ab9e5ab3-7cd6-4da1-8e23-b723ac717ebe_2021-05-20-090001-147_v1.6.0 app preserveSecs 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-15-090000-481/app_ab9e5ab3-7cd6-4da1-8e23-b723ac717ebe_2021-05-15-090000-993_v1.5.0 app preserveSecs 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_a2f4a9ba-b070-4939-8e22-269432d2e4da_2021-05-26-110113-139_v1.7.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_a2f4a9ba-b070-4939-8e22-269432d2e4da_2021-05-25-110023-794_v1.7.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_a2f4a9ba-b070-4939-8e22-269432d2e4da_2021-05-24-110023-646_v1.7.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_a2f4a9ba-b070-4939-8e22-269432d2e4da_2021-05-23-110025-177_v1.7.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_a2f4a9ba-b070-4939-8e22-269432d2e4da_2021-05-22-110024-619_v1.7.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_a2f4a9ba-b070-4939-8e22-269432d2e4da_2021-05-21-110204-329_v1.7.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_a2f4a9ba-b070-4939-8e22-269432d2e4da_2021-05-20-110023-939_v1.7.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_8b586a7b-abd0-431e-a2dd-89b4af61da65_2021-05-26-110112-480_v1.10.9 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_8b586a7b-abd0-431e-a2dd-89b4af61da65_2021-05-25-110023-135_v1.10.9 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_8b586a7b-abd0-431e-a2dd-89b4af61da65_2021-05-24-110023-070_v1.10.9 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_8b586a7b-abd0-431e-a2dd-89b4af61da65_2021-05-23-110024-589_v1.10.9 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_8b586a7b-abd0-431e-a2dd-89b4af61da65_2021-05-22-110024-049_v1.10.9 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_8b586a7b-abd0-431e-a2dd-89b4af61da65_2021-05-21-110203-767_v1.10.9 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-090140-989/app_8b586a7b-abd0-431e-a2dd-89b4af61da65_2021-05-21-090141-414_v1.10.8 app preserveSecs 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_8b586a7b-abd0-431e-a2dd-89b4af61da65_2021-05-20-110023-388_v1.10.8 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-12-090222-976/app_8b586a7b-abd0-431e-a2dd-89b4af61da65_2021-05-12-090223-373_v1.10.7 app preserveSecs 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_846af4f9-e4c2-4b03-9368-b0b7fbfb71c8_2021-05-26-110101-363_v4.9.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-090000-541/app_846af4f9-e4c2-4b03-9368-b0b7fbfb71c8_2021-05-26-090001-770_v4.9.0 app preserveSecs 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_846af4f9-e4c2-4b03-9368-b0b7fbfb71c8_2021-05-25-110010-985_v4.9.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_846af4f9-e4c2-4b03-9368-b0b7fbfb71c8_2021-05-24-110010-783_v4.9.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_846af4f9-e4c2-4b03-9368-b0b7fbfb71c8_2021-05-23-110010-882_v4.9.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_846af4f9-e4c2-4b03-9368-b0b7fbfb71c8_2021-05-22-110010-896_v4.9.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_846af4f9-e4c2-4b03-9368-b0b7fbfb71c8_2021-05-21-110151-243_v4.9.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_846af4f9-e4c2-4b03-9368-b0b7fbfb71c8_2021-05-20-110010-777_v4.9.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_7541955b-ee8d-4327-a473-9fa434803fb3_2021-05-26-110009-866_v1.9.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_7541955b-ee8d-4327-a473-9fa434803fb3_2021-05-25-110009-766_v1.9.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_7541955b-ee8d-4327-a473-9fa434803fb3_2021-05-24-110009-564_v1.9.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_7541955b-ee8d-4327-a473-9fa434803fb3_2021-05-23-110009-614_v1.9.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_7541955b-ee8d-4327-a473-9fa434803fb3_2021-05-22-110009-611_v1.9.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_7541955b-ee8d-4327-a473-9fa434803fb3_2021-05-21-110149-971_v1.9.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_7541955b-ee8d-4327-a473-9fa434803fb3_2021-05-20-110009-618_v1.9.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-090000-510/app_7541955b-ee8d-4327-a473-9fa434803fb3_2021-05-20-090001-479_v1.9.2 app preserveSecs 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_67716680-7709-4d68-afbb-5694dbb0ddb4_2021-05-26-110008-925_v1.2.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-090000-515/app_67716680-7709-4d68-afbb-5694dbb0ddb4_2021-05-26-090000-976_v1.1.0 app preserveSecs 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_67716680-7709-4d68-afbb-5694dbb0ddb4_2021-05-25-110008-828_v1.1.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_67716680-7709-4d68-afbb-5694dbb0ddb4_2021-05-24-110008-632_v1.1.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_67716680-7709-4d68-afbb-5694dbb0ddb4_2021-05-23-110008-655_v1.1.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_67716680-7709-4d68-afbb-5694dbb0ddb4_2021-05-22-110008-672_v1.1.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_67716680-7709-4d68-afbb-5694dbb0ddb4_2021-05-21-110149-034_v1.1.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_67716680-7709-4d68-afbb-5694dbb0ddb4_2021-05-20-110008-346_v1.1.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_65700d32-b168-4258-83dc-183579e4c46b_2021-05-26-110008-454_v1.3.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_65700d32-b168-4258-83dc-183579e4c46b_2021-05-25-110008-361_v1.3.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_65700d32-b168-4258-83dc-183579e4c46b_2021-05-24-110008-182_v1.3.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_65700d32-b168-4258-83dc-183579e4c46b_2021-05-23-110008-207_v1.3.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_65700d32-b168-4258-83dc-183579e4c46b_2021-05-22-110008-208_v1.3.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_65700d32-b168-4258-83dc-183579e4c46b_2021-05-21-110148-582_v1.3.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_65700d32-b168-4258-83dc-183579e4c46b_2021-05-20-110007-925_v1.3.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-090000-492/app_65700d32-b168-4258-83dc-183579e4c46b_2021-05-20-090000-952_v1.3.2 app preserveSecs 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_50239bb3-42d5-4982-a8ff-40246d31ae30_2021-05-26-110007-852_v1.14.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_50239bb3-42d5-4982-a8ff-40246d31ae30_2021-05-25-110007-751_v1.14.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_50239bb3-42d5-4982-a8ff-40246d31ae30_2021-05-24-110007-616_v1.14.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_50239bb3-42d5-4982-a8ff-40246d31ae30_2021-05-23-110007-640_v1.14.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_50239bb3-42d5-4982-a8ff-40246d31ae30_2021-05-22-110007-595_v1.14.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_50239bb3-42d5-4982-a8ff-40246d31ae30_2021-05-21-110148-013_v1.14.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_50239bb3-42d5-4982-a8ff-40246d31ae30_2021-05-20-110007-358_v1.14.1 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_29472923-d304-4e03-90d7-f873e7911302_2021-05-26-110006-674_v1.1.0 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_29472923-d304-4e03-90d7-f873e7911302_2021-05-25-110006-560_v1.1.0 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_29472923-d304-4e03-90d7-f873e7911302_2021-05-24-110006-429_v1.1.0 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_29472923-d304-4e03-90d7-f873e7911302_2021-05-23-110006-429_v1.1.0 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_29472923-d304-4e03-90d7-f873e7911302_2021-05-22-110006-412_v1.1.0 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_29472923-d304-4e03-90d7-f873e7911302_2021-05-21-110146-851_v1.1.0 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_29472923-d304-4e03-90d7-f873e7911302_2021-05-20-110006-227_v1.1.0 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_3d6a5f8f-b500-4536-90b3-41d1c5844a37_2021-05-26-110007-205_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_3d6a5f8f-b500-4536-90b3-41d1c5844a37_2021-05-25-110007-114_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_3d6a5f8f-b500-4536-90b3-41d1c5844a37_2021-05-24-110006-983_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_3d6a5f8f-b500-4536-90b3-41d1c5844a37_2021-05-23-110006-998_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_3d6a5f8f-b500-4536-90b3-41d1c5844a37_2021-05-22-110006-947_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_3d6a5f8f-b500-4536-90b3-41d1c5844a37_2021-05-21-110147-384_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_3d6a5f8f-b500-4536-90b3-41d1c5844a37_2021-05-20-110006-729_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_1d0eb4b7-a363-49ee-b0f3-0f4166a58ec1_2021-05-26-110005-683_v1.51.2 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_1d0eb4b7-a363-49ee-b0f3-0f4166a58ec1_2021-05-25-110005-441_v1.51.2 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_1d0eb4b7-a363-49ee-b0f3-0f4166a58ec1_2021-05-24-110005-393_v1.51.2 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_1d0eb4b7-a363-49ee-b0f3-0f4166a58ec1_2021-05-23-110005-341_v1.51.2 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_1d0eb4b7-a363-49ee-b0f3-0f4166a58ec1_2021-05-22-110005-405_v1.51.2 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_1d0eb4b7-a363-49ee-b0f3-0f4166a58ec1_2021-05-21-110145-839_v1.51.2 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_1d0eb4b7-a363-49ee-b0f3-0f4166a58ec1_2021-05-20-110005-229_v1.51.2 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-18-090000-491/app_1d0eb4b7-a363-49ee-b0f3-0f4166a58ec1_2021-05-18-090003-169_v1.51.1 app preserveSecs 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_18286784-d392-46d3-a037-4b766a670308_2021-05-26-110002-987_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_18286784-d392-46d3-a037-4b766a670308_2021-05-25-110002-740_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_18286784-d392-46d3-a037-4b766a670308_2021-05-24-110002-732_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_18286784-d392-46d3-a037-4b766a670308_2021-05-23-110002-673_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_18286784-d392-46d3-a037-4b766a670308_2021-05-22-110002-791_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_18286784-d392-46d3-a037-4b766a670308_2021-05-21-110143-226_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_18286784-d392-46d3-a037-4b766a670308_2021-05-20-110002-686_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_162736d0-c899-4722-a15b-8ba186f53427_2021-05-26-110001-245_v1.23.1 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_162736d0-c899-4722-a15b-8ba186f53427_2021-05-25-110000-881_v1.23.1 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_162736d0-c899-4722-a15b-8ba186f53427_2021-05-24-110000-883_v1.23.1 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_162736d0-c899-4722-a15b-8ba186f53427_2021-05-23-110000-898_v1.23.1 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_162736d0-c899-4722-a15b-8ba186f53427_2021-05-22-110001-093_v1.23.1 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_162736d0-c899-4722-a15b-8ba186f53427_2021-05-21-110141-523_v1.23.1 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_162736d0-c899-4722-a15b-8ba186f53427_2021-05-20-110000-944_v1.23.1 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-19-090000-454/app_162736d0-c899-4722-a15b-8ba186f53427_2021-05-19-090000-847_v1.23.0 app preserveSecs 2021-05-26T16:16:31.459Z box:backups cleanupAppBackups: done 2021-05-26T16:16:31.459Z box:tasks 2042: {"percent":70,"message":"Cleaning missing backups"} 2021-05-26T16:16:31.518Z box:tasks 2042: {"percent":90,"message":"Cleaning snapshots"} 2021-05-26T16:16:31.573Z box:backups cleanupSnapshots: done 2021-05-26T16:16:31.573Z box:taskworker Task took 0.169 seconds 2021-05-26T16:16:31.573Z box:tasks setCompleted - 2042: {"result":{"removedBoxBackupIds":[],"removedAppBackupIds":[],"missingBackupIds":[]},"error":null} 2021-05-26T16:16:31.573Z box:tasks 2042: {"percent":100,"result":{"removedBoxBackupIds":[],"removedAppBackupIds":[],"missingBackupIds":[]},"error":null}

@d19dotca Yes, I think you're right, thank you. I need to look into that. Or move some long term storage files ("archive") into Minio. That might just be moving the problem, but dividing the problem might be a partial solution.

@girish Thanks for looking at this. Just triggered another backup with all apps active and it succeeded. I guess then it's finally time to move to the new hardware (plus making sure that the new system is using the c locale by default)