@michaelf Got the logs! Still clearing out my back log, will get back tomorrow for sure.

@robi there is a already bunch of "workarounds" for rsync. Empty directories, executable bit of files cannot be stored in most object storage. So, there is fsmetadata.json file that stores this information outside of the files. When restoring, we use that file to restore back the state. I guess we can extend that file to also save and restore timestamps.

If anyone wants this leave a note and I can look into it in the future.

@girish I just restarted my server after some security updates went through and SSHFS stopped working for me as well. I deleted the backup destination and added it again and it started working. I've been on 6.3.5 for a long while (perhaps since it's release) for reference...

Thanks a bunch @girish worked like a charm

@girish Ahhh I see. Thankfully right now backups only take 3-4 hours. I'm hoping once I'm at gigabit I can speed that up. My network is definitely a bottleneck rn.

@mazarian that's correct, the permission is the thing I fixed in 6.3.5. Glad it's working now and thanks for reporting back!

@jdaviescoates There's two levels of retry - things like network errors, transient api errors etc are retried immediately. The other retry is in the next scheduled backup time. There is no other retry between scheduled backup time.

@girish that worked. Thanks

@d19dotca said in Backups failing on same app each time:

By the way, since there's a ton of backup crashing that seem memory related... is it possible to improve the logging so that there's something in there saying "OutOfMemory" or something like that so poeple know why it's crashing?

Yeah, I would love this as well. Sadly, there seems to be no way on linux to know this reliably. A process just gets killed unceremoniously with SIGKILL and that's that 😕 If anyone know how this can be detected would be great.

We are using minIO for it. s3 compatible. the Service on our side was down… and then we also have a problem with backup size. but now it works! thx!

@dylightful if you click the 'cleanup backups' button in the Backups view, is there anything interesting in the logs?

@nebulon you would need a list of app backups that are allowed to fail (app store apps) and ones that aren't (system services).

Partial backups are only ok if an app store app gets an error.

@privsec Note, just because it's "running" doesn't mean it's running well. 😉

Not sure what was going on before but things appear to be working fine now.

Thank you @girish for the clarification on the different types of backups referenced in the log and the explanation that preserveSecs will remain for 3 weeks regardless of the normal backup retenetion period.

cmagnuson@cloudron:/mnt/cloudronbackup$ du -h --max-depth=1 /mnt/cloudronbackup/ du: cannot read directory '/mnt/cloudronbackup/.SecuriSyncFileServer': Permission denied 0 /mnt/cloudronbackup/.SecuriSyncFileServer 0 /mnt/cloudronbackup/2021-05-09-100000-534 13K /mnt/cloudronbackup/2021-05-12-090222-976 116M /mnt/cloudronbackup/2021-05-13-090000-459 76M /mnt/cloudronbackup/2021-05-15-090000-481 3.9M /mnt/cloudronbackup/2021-05-18-090000-491 18K /mnt/cloudronbackup/2021-05-18-110000-453 2.3M /mnt/cloudronbackup/2021-05-19-090000-454 34M /mnt/cloudronbackup/2021-05-20-090000-492 25M /mnt/cloudronbackup/2021-05-20-090000-509 120M /mnt/cloudronbackup/2021-05-20-090000-510 5.1G /mnt/cloudronbackup/2021-05-20-110000-521 4.5K /mnt/cloudronbackup/2021-05-21-090140-989 337K /mnt/cloudronbackup/2021-05-21-090141-009 65M /mnt/cloudronbackup/2021-05-21-110141-085 13K /mnt/cloudronbackup/2021-05-22-090000-495 66M /mnt/cloudronbackup/2021-05-22-110000-663 66M /mnt/cloudronbackup/2021-05-23-110000-475 65M /mnt/cloudronbackup/2021-05-24-110000-457 68M /mnt/cloudronbackup/2021-05-25-110000-458 5.5K /mnt/cloudronbackup/2021-05-26-090000-515 5.5M /mnt/cloudronbackup/2021-05-26-090000-541 331M /mnt/cloudronbackup/2021-05-26-110000-804 28K /mnt/cloudronbackup/snapshot 6.1G /mnt/cloudronbackup/ 2021-05-26T16:16:31.424Z box:settings initCache: pre-load settings 2021-05-26T16:16:31.433Z box:taskworker Starting task 2042. Logs are at /home/yellowtent/platformdata/logs/tasks/2042.log 2021-05-26T16:16:31.435Z box:tasks 2042: {"percent":2,"error":null} 2021-05-26T16:16:31.440Z box:tasks 2042: {"percent":10,"message":"Cleaning box backups"} 2021-05-26T16:16:31.443Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/box_2021-05-26-110120-087_v6.2.8 box keepWithinSecs 2021-05-26T16:16:31.443Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/box_2021-05-25-110030-827_v6.2.8 box keepWithinSecs 2021-05-26T16:16:31.443Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/box_2021-05-24-110030-725_v6.2.8 box keepWithinSecs 2021-05-26T16:16:31.443Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/box_2021-05-23-110032-528_v6.2.8 box keepWithinSecs 2021-05-26T16:16:31.443Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/box_2021-05-22-110031-980_v6.2.8 box keepWithinSecs 2021-05-26T16:16:31.443Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/box_2021-05-21-110211-336_v6.2.8 box keepWithinSecs 2021-05-26T16:16:31.443Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/box_2021-05-20-110030-471_v6.2.8 box keepWithinSecs 2021-05-26T16:16:31.443Z box:backups cleanupBoxBackups: done 2021-05-26T16:16:31.443Z box:tasks 2042: {"percent":40,"message":"Cleaning app backups"} 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_d2fb1510-462d-45d0-84b8-8143184152fc_2021-05-26-110118-719_v2.1.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_d2fb1510-462d-45d0-84b8-8143184152fc_2021-05-25-110029-455_v2.1.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_d2fb1510-462d-45d0-84b8-8143184152fc_2021-05-24-110029-345_v2.1.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_d2fb1510-462d-45d0-84b8-8143184152fc_2021-05-23-110031-187_v2.1.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_d2fb1510-462d-45d0-84b8-8143184152fc_2021-05-22-110030-638_v2.1.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_d2fb1510-462d-45d0-84b8-8143184152fc_2021-05-21-110209-942_v2.1.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_d2fb1510-462d-45d0-84b8-8143184152fc_2021-05-20-110029-188_v2.1.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_ef14e130-abca-436c-9dd9-70748cf0a9a6_2021-05-26-110119-468_v1.4.7 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_ef14e130-abca-436c-9dd9-70748cf0a9a6_2021-05-25-110030-222_v1.4.7 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_ef14e130-abca-436c-9dd9-70748cf0a9a6_2021-05-24-110030-120_v1.4.7 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_ef14e130-abca-436c-9dd9-70748cf0a9a6_2021-05-23-110031-935_v1.4.7 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_ef14e130-abca-436c-9dd9-70748cf0a9a6_2021-05-22-110031-388_v1.4.7 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_ef14e130-abca-436c-9dd9-70748cf0a9a6_2021-05-21-110210-722_v1.4.7 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-21-090141-009/app_ef14e130-abca-436c-9dd9-70748cf0a9a6_2021-05-21-090141-659_v1.4.6 app preserveSecs 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_ef14e130-abca-436c-9dd9-70748cf0a9a6_2021-05-20-110029-891_v1.4.6 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-13-090000-459/app_ef14e130-abca-436c-9dd9-70748cf0a9a6_2021-05-13-090001-079_v1.4.5 app preserveSecs 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_cb6982cb-7f7b-492b-a25f-bb1c443c320f_2021-05-26-110115-857_v1.11.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_cb6982cb-7f7b-492b-a25f-bb1c443c320f_2021-05-25-110026-458_v1.11.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_cb6982cb-7f7b-492b-a25f-bb1c443c320f_2021-05-24-110026-430_v1.11.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_cb6982cb-7f7b-492b-a25f-bb1c443c320f_2021-05-23-110027-861_v1.11.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_cb6982cb-7f7b-492b-a25f-bb1c443c320f_2021-05-22-110027-450_v1.11.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_cb6982cb-7f7b-492b-a25f-bb1c443c320f_2021-05-21-110206-980_v1.11.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_cb6982cb-7f7b-492b-a25f-bb1c443c320f_2021-05-20-110026-387_v1.11.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_c6169a84-917b-4f21-9ac4-2d3178f2b61e_2021-05-26-110114-671_v1.8.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_c6169a84-917b-4f21-9ac4-2d3178f2b61e_2021-05-25-110025-294_v1.8.0 app reference 2021-05-26T16:16:31.457Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_c6169a84-917b-4f21-9ac4-2d3178f2b61e_2021-05-24-110025-242_v1.8.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_c6169a84-917b-4f21-9ac4-2d3178f2b61e_2021-05-23-110026-686_v1.8.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_c6169a84-917b-4f21-9ac4-2d3178f2b61e_2021-05-22-110026-219_v1.8.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-090000-495/app_c6169a84-917b-4f21-9ac4-2d3178f2b61e_2021-05-22-090001-059_v1.7.2 app preserveSecs 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_c6169a84-917b-4f21-9ac4-2d3178f2b61e_2021-05-21-110205-780_v1.7.2 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_c6169a84-917b-4f21-9ac4-2d3178f2b61e_2021-05-20-110025-278_v1.7.2 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_c95e1f50-f42d-4e18-b646-935a62a8eb63_2021-05-26-110115-279_v0.2.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_c95e1f50-f42d-4e18-b646-935a62a8eb63_2021-05-25-110025-891_v0.2.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_c95e1f50-f42d-4e18-b646-935a62a8eb63_2021-05-24-110025-848_v0.2.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_c95e1f50-f42d-4e18-b646-935a62a8eb63_2021-05-23-110027-294_v0.2.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_c95e1f50-f42d-4e18-b646-935a62a8eb63_2021-05-22-110026-870_v0.2.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_c95e1f50-f42d-4e18-b646-935a62a8eb63_2021-05-21-110206-378_v0.2.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_c95e1f50-f42d-4e18-b646-935a62a8eb63_2021-05-20-110025-848_v0.2.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_ab9e5ab3-7cd6-4da1-8e23-b723ac717ebe_2021-05-26-110113-910_v1.6.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_ab9e5ab3-7cd6-4da1-8e23-b723ac717ebe_2021-05-25-110024-539_v1.6.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_ab9e5ab3-7cd6-4da1-8e23-b723ac717ebe_2021-05-24-110024-365_v1.6.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_ab9e5ab3-7cd6-4da1-8e23-b723ac717ebe_2021-05-23-110025-901_v1.6.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_ab9e5ab3-7cd6-4da1-8e23-b723ac717ebe_2021-05-22-110025-349_v1.6.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_ab9e5ab3-7cd6-4da1-8e23-b723ac717ebe_2021-05-21-110205-018_v1.6.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_ab9e5ab3-7cd6-4da1-8e23-b723ac717ebe_2021-05-20-110024-571_v1.6.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-090000-509/app_ab9e5ab3-7cd6-4da1-8e23-b723ac717ebe_2021-05-20-090001-147_v1.6.0 app preserveSecs 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-15-090000-481/app_ab9e5ab3-7cd6-4da1-8e23-b723ac717ebe_2021-05-15-090000-993_v1.5.0 app preserveSecs 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_a2f4a9ba-b070-4939-8e22-269432d2e4da_2021-05-26-110113-139_v1.7.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_a2f4a9ba-b070-4939-8e22-269432d2e4da_2021-05-25-110023-794_v1.7.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_a2f4a9ba-b070-4939-8e22-269432d2e4da_2021-05-24-110023-646_v1.7.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_a2f4a9ba-b070-4939-8e22-269432d2e4da_2021-05-23-110025-177_v1.7.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_a2f4a9ba-b070-4939-8e22-269432d2e4da_2021-05-22-110024-619_v1.7.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_a2f4a9ba-b070-4939-8e22-269432d2e4da_2021-05-21-110204-329_v1.7.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_a2f4a9ba-b070-4939-8e22-269432d2e4da_2021-05-20-110023-939_v1.7.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_8b586a7b-abd0-431e-a2dd-89b4af61da65_2021-05-26-110112-480_v1.10.9 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_8b586a7b-abd0-431e-a2dd-89b4af61da65_2021-05-25-110023-135_v1.10.9 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_8b586a7b-abd0-431e-a2dd-89b4af61da65_2021-05-24-110023-070_v1.10.9 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_8b586a7b-abd0-431e-a2dd-89b4af61da65_2021-05-23-110024-589_v1.10.9 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_8b586a7b-abd0-431e-a2dd-89b4af61da65_2021-05-22-110024-049_v1.10.9 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_8b586a7b-abd0-431e-a2dd-89b4af61da65_2021-05-21-110203-767_v1.10.9 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-090140-989/app_8b586a7b-abd0-431e-a2dd-89b4af61da65_2021-05-21-090141-414_v1.10.8 app preserveSecs 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_8b586a7b-abd0-431e-a2dd-89b4af61da65_2021-05-20-110023-388_v1.10.8 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-12-090222-976/app_8b586a7b-abd0-431e-a2dd-89b4af61da65_2021-05-12-090223-373_v1.10.7 app preserveSecs 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_846af4f9-e4c2-4b03-9368-b0b7fbfb71c8_2021-05-26-110101-363_v4.9.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-090000-541/app_846af4f9-e4c2-4b03-9368-b0b7fbfb71c8_2021-05-26-090001-770_v4.9.0 app preserveSecs 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_846af4f9-e4c2-4b03-9368-b0b7fbfb71c8_2021-05-25-110010-985_v4.9.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_846af4f9-e4c2-4b03-9368-b0b7fbfb71c8_2021-05-24-110010-783_v4.9.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_846af4f9-e4c2-4b03-9368-b0b7fbfb71c8_2021-05-23-110010-882_v4.9.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_846af4f9-e4c2-4b03-9368-b0b7fbfb71c8_2021-05-22-110010-896_v4.9.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_846af4f9-e4c2-4b03-9368-b0b7fbfb71c8_2021-05-21-110151-243_v4.9.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_846af4f9-e4c2-4b03-9368-b0b7fbfb71c8_2021-05-20-110010-777_v4.9.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_7541955b-ee8d-4327-a473-9fa434803fb3_2021-05-26-110009-866_v1.9.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_7541955b-ee8d-4327-a473-9fa434803fb3_2021-05-25-110009-766_v1.9.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_7541955b-ee8d-4327-a473-9fa434803fb3_2021-05-24-110009-564_v1.9.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_7541955b-ee8d-4327-a473-9fa434803fb3_2021-05-23-110009-614_v1.9.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_7541955b-ee8d-4327-a473-9fa434803fb3_2021-05-22-110009-611_v1.9.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_7541955b-ee8d-4327-a473-9fa434803fb3_2021-05-21-110149-971_v1.9.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_7541955b-ee8d-4327-a473-9fa434803fb3_2021-05-20-110009-618_v1.9.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-090000-510/app_7541955b-ee8d-4327-a473-9fa434803fb3_2021-05-20-090001-479_v1.9.2 app preserveSecs 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_67716680-7709-4d68-afbb-5694dbb0ddb4_2021-05-26-110008-925_v1.2.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-090000-515/app_67716680-7709-4d68-afbb-5694dbb0ddb4_2021-05-26-090000-976_v1.1.0 app preserveSecs 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_67716680-7709-4d68-afbb-5694dbb0ddb4_2021-05-25-110008-828_v1.1.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_67716680-7709-4d68-afbb-5694dbb0ddb4_2021-05-24-110008-632_v1.1.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_67716680-7709-4d68-afbb-5694dbb0ddb4_2021-05-23-110008-655_v1.1.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_67716680-7709-4d68-afbb-5694dbb0ddb4_2021-05-22-110008-672_v1.1.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_67716680-7709-4d68-afbb-5694dbb0ddb4_2021-05-21-110149-034_v1.1.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_67716680-7709-4d68-afbb-5694dbb0ddb4_2021-05-20-110008-346_v1.1.0 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_65700d32-b168-4258-83dc-183579e4c46b_2021-05-26-110008-454_v1.3.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_65700d32-b168-4258-83dc-183579e4c46b_2021-05-25-110008-361_v1.3.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_65700d32-b168-4258-83dc-183579e4c46b_2021-05-24-110008-182_v1.3.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_65700d32-b168-4258-83dc-183579e4c46b_2021-05-23-110008-207_v1.3.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_65700d32-b168-4258-83dc-183579e4c46b_2021-05-22-110008-208_v1.3.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_65700d32-b168-4258-83dc-183579e4c46b_2021-05-21-110148-582_v1.3.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_65700d32-b168-4258-83dc-183579e4c46b_2021-05-20-110007-925_v1.3.3 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-090000-492/app_65700d32-b168-4258-83dc-183579e4c46b_2021-05-20-090000-952_v1.3.2 app preserveSecs 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_50239bb3-42d5-4982-a8ff-40246d31ae30_2021-05-26-110007-852_v1.14.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_50239bb3-42d5-4982-a8ff-40246d31ae30_2021-05-25-110007-751_v1.14.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_50239bb3-42d5-4982-a8ff-40246d31ae30_2021-05-24-110007-616_v1.14.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_50239bb3-42d5-4982-a8ff-40246d31ae30_2021-05-23-110007-640_v1.14.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_50239bb3-42d5-4982-a8ff-40246d31ae30_2021-05-22-110007-595_v1.14.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_50239bb3-42d5-4982-a8ff-40246d31ae30_2021-05-21-110148-013_v1.14.1 app reference 2021-05-26T16:16:31.458Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_50239bb3-42d5-4982-a8ff-40246d31ae30_2021-05-20-110007-358_v1.14.1 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_29472923-d304-4e03-90d7-f873e7911302_2021-05-26-110006-674_v1.1.0 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_29472923-d304-4e03-90d7-f873e7911302_2021-05-25-110006-560_v1.1.0 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_29472923-d304-4e03-90d7-f873e7911302_2021-05-24-110006-429_v1.1.0 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_29472923-d304-4e03-90d7-f873e7911302_2021-05-23-110006-429_v1.1.0 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_29472923-d304-4e03-90d7-f873e7911302_2021-05-22-110006-412_v1.1.0 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_29472923-d304-4e03-90d7-f873e7911302_2021-05-21-110146-851_v1.1.0 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_29472923-d304-4e03-90d7-f873e7911302_2021-05-20-110006-227_v1.1.0 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_3d6a5f8f-b500-4536-90b3-41d1c5844a37_2021-05-26-110007-205_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_3d6a5f8f-b500-4536-90b3-41d1c5844a37_2021-05-25-110007-114_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_3d6a5f8f-b500-4536-90b3-41d1c5844a37_2021-05-24-110006-983_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_3d6a5f8f-b500-4536-90b3-41d1c5844a37_2021-05-23-110006-998_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_3d6a5f8f-b500-4536-90b3-41d1c5844a37_2021-05-22-110006-947_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_3d6a5f8f-b500-4536-90b3-41d1c5844a37_2021-05-21-110147-384_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_3d6a5f8f-b500-4536-90b3-41d1c5844a37_2021-05-20-110006-729_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_1d0eb4b7-a363-49ee-b0f3-0f4166a58ec1_2021-05-26-110005-683_v1.51.2 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_1d0eb4b7-a363-49ee-b0f3-0f4166a58ec1_2021-05-25-110005-441_v1.51.2 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_1d0eb4b7-a363-49ee-b0f3-0f4166a58ec1_2021-05-24-110005-393_v1.51.2 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_1d0eb4b7-a363-49ee-b0f3-0f4166a58ec1_2021-05-23-110005-341_v1.51.2 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_1d0eb4b7-a363-49ee-b0f3-0f4166a58ec1_2021-05-22-110005-405_v1.51.2 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_1d0eb4b7-a363-49ee-b0f3-0f4166a58ec1_2021-05-21-110145-839_v1.51.2 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_1d0eb4b7-a363-49ee-b0f3-0f4166a58ec1_2021-05-20-110005-229_v1.51.2 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-18-090000-491/app_1d0eb4b7-a363-49ee-b0f3-0f4166a58ec1_2021-05-18-090003-169_v1.51.1 app preserveSecs 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_18286784-d392-46d3-a037-4b766a670308_2021-05-26-110002-987_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_18286784-d392-46d3-a037-4b766a670308_2021-05-25-110002-740_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_18286784-d392-46d3-a037-4b766a670308_2021-05-24-110002-732_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_18286784-d392-46d3-a037-4b766a670308_2021-05-23-110002-673_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_18286784-d392-46d3-a037-4b766a670308_2021-05-22-110002-791_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_18286784-d392-46d3-a037-4b766a670308_2021-05-21-110143-226_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_18286784-d392-46d3-a037-4b766a670308_2021-05-20-110002-686_v4.12.7 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-26-110000-804/app_162736d0-c899-4722-a15b-8ba186f53427_2021-05-26-110001-245_v1.23.1 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-25-110000-458/app_162736d0-c899-4722-a15b-8ba186f53427_2021-05-25-110000-881_v1.23.1 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-24-110000-457/app_162736d0-c899-4722-a15b-8ba186f53427_2021-05-24-110000-883_v1.23.1 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-23-110000-475/app_162736d0-c899-4722-a15b-8ba186f53427_2021-05-23-110000-898_v1.23.1 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-22-110000-663/app_162736d0-c899-4722-a15b-8ba186f53427_2021-05-22-110001-093_v1.23.1 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-21-110141-085/app_162736d0-c899-4722-a15b-8ba186f53427_2021-05-21-110141-523_v1.23.1 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-20-110000-521/app_162736d0-c899-4722-a15b-8ba186f53427_2021-05-20-110000-944_v1.23.1 app reference 2021-05-26T16:16:31.459Z box:backups applyBackupRetentionPolicy: 2021-05-19-090000-454/app_162736d0-c899-4722-a15b-8ba186f53427_2021-05-19-090000-847_v1.23.0 app preserveSecs 2021-05-26T16:16:31.459Z box:backups cleanupAppBackups: done 2021-05-26T16:16:31.459Z box:tasks 2042: {"percent":70,"message":"Cleaning missing backups"} 2021-05-26T16:16:31.518Z box:tasks 2042: {"percent":90,"message":"Cleaning snapshots"} 2021-05-26T16:16:31.573Z box:backups cleanupSnapshots: done 2021-05-26T16:16:31.573Z box:taskworker Task took 0.169 seconds 2021-05-26T16:16:31.573Z box:tasks setCompleted - 2042: {"result":{"removedBoxBackupIds":[],"removedAppBackupIds":[],"missingBackupIds":[]},"error":null} 2021-05-26T16:16:31.573Z box:tasks 2042: {"percent":100,"result":{"removedBoxBackupIds":[],"removedAppBackupIds":[],"missingBackupIds":[]},"error":null}

@d19dotca Yes, I think you're right, thank you.
I need to look into that.
Or move some long term storage files ("archive") into Minio.
That might just be moving the problem, but dividing the problem might be a partial solution.

@girish Thanks for looking at this. Just triggered another backup with all apps active and it succeeded. I guess then it's finally time to move to the new hardware (plus making sure that the new system is using the c locale by default)

@girish haha, yeah that's fair. Deleting backups is probably one of those tasks that should intentionally be difficult to do to avoid any issues (i.e. they need to delete the backups manually).

Only thing I can suggest maybe is a little icon designation either beside each of the backups that is past the retention policy (or part of a version upgrade backup) that explains it quickly and links to the docs or something, or maybe just one overall info button in the corner of the backups page or something. Not sure which is most feasible. I'll file a feature request formally for that though in a moment.

EDIT: I filed the feature request here: https://forum.cloudron.io/topic/4890/improve-the-backup-page-specifically-retention-related-items-to-explain-retention-policies-and-why-some-backups-exist-beyond-them

@girish After rebooting the machine it has been working for three continuous backups - I will report back if this changes. Thanks for your help.

@girish no, they're pretty useless. Their web UI S3 console is such crap it can't handle the chatty API requests and keeps timing out. Also I may be wrong that multiple directories are because of failures and restarts. It just looks like multiple changed apps per day get a new dir.

So I am attempting other workarounds. Like creating a new bucket and just nuking the old one.

rsync isn't great for object store backups as it makes a ton of small files.
tgz isn't great as it's a lot of repeated information.

We need something hybrid that is the best of both.
Something like backing up to a local Minio much more quickly then doing an object to object store transfer offsite, which is much more efficient. This may also offer an opportunity to dedupe and further optimize.

@girish said in No backups because of NullPointerException:

@guyds said in No backups because of NullPointerException:

The first level is the per app level. Right now, apps are already backed up one by one, but they are not stored nor reported individually. And this is the missing feature in my opinion.

Ah no, they are listed in app -> backups. So, even if you do a full backup, each individual app backup will be listed in app -> backups.

Yes, they are listed at the app level, but there's no reporting at the app level because the backup succeeds or fails at the box level.

You can also use that backup to restore/clone the app.

Yes, but only if the backup succeeds for all apps.

So, again, I think the current issue is that everything is treated as a whole while it makes more sense in my opinion to treat each app individually and then in the end (optionally?) bundle the individual parts as a whole.

Yes, so they are treated individually, it's actually very close to what you want. The only issue is that when a full backup fails, those successful individual app backups that are part of a failed full backup will get removed/cleaned up every night.

Exactly, they are per app bot not treated like that in the end because success or failure is determined by the whole box.

I have made a task to fix the behavior, let's see.

Great, thanks!

Perhaps there should be a permissions check script that puts warnings in the logs.

@girish Thank's! Good the idea to end DST...

@girish Awesome thank you, I appreciate the advice. All seems to be working great now.

Also thanks @marcusquinn

@martinkbs it will be good to have documented the tested rclone setup as part of the documentation!

Ok, thanks

For completeness, the working instructions is:

# cd /home/yellowtent/box # sudo -u yellowtent npm install aws-sdk@2.879.0 # systemctl restart box

@brutalbirdie Thank you, I'll give that a shot!

@imc67 said in Updating FreeScout failed:

@girish said in Updating FreeScout failed:

@imc67 there is a bug here . Can you apply this one liner and then set the memory limit https://git.cloudron.io/cloudron/box/-/commit/750f313c6a36d8d961892df4a41cb7c41faa1f1b ? You have systemctl restart box after you change that line in /home/yellowtent/box/src/apps.js.

Will try this in a few hours

@girish this solved the bug, thanks!

@nebulon I enabled remote debugging and emailed, look forward to hearing back

I will mark this as solved for now but also ack that there is possibly some bug either on minio side or Cloudron side.