"Backup Failed" since the last Cloudron Update
-
@joseph Hello,
Since the Cloudron Update, it fails every time. Everyday I have the cloudron notification saying "Backup Failed" with similar logs.
I use AWS S3 for backups for now.
Today's logs are there :
Aug 11 05:31:29 box:tasks update 3071: {"percent":50.01960784313726,"message":"Finishing multipart copy - 2024-08-11-020000-466/app_ia.example.com_v2.3.11.tar.gz.enc"} Aug 11 05:31:29 box:tasks update 3071: {"percent":50.01960784313726,"message":"Copied 1 files with error: null"} Aug 11 05:31:29 box:backuptask copy: copied successfully to 2024-08-11-020000-466/app_ia.example.com_v2.3.11. Took 671.94 seconds Aug 11 05:31:29 box:taskworker Task took 5488.952 seconds Aug 11 05:31:29 box:tasks setCompleted - 3071: {"result":null,"error":{"stack":"BoxError: Backup not found\n at Object.setState (/home/yellowtent/box/src/backups.js:237:42)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async rotateAppBackup (/home/yellowtent/box/src/backuptask.js:304:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:384:12)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Not found","details":{},"message":"Backup not found"}} Aug 11 05:31:29 box:tasks update 3071: {"percent":100,"result":null,"error":{"stack":"BoxError: Backup not found\n at Object.setState (/home/yellowtent/box/src/backups.js:237:42)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async rotateAppBackup (/home/yellowtent/box/src/backuptask.js:304:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:384:12)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Not found","details":{},"message":"Backup not found"}} [no timestamp] Backup not found [no timestamp] at Object.setState (/home/yellowtent/box/src/backups.js:237:42) [no timestamp] at process.processTicksAndRejections (node:internal/process/task_queues:95:5) [no timestamp] at async rotateAppBackup (/home/yellowtent/box/src/backuptask.js:304:5) [no timestamp] at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:384:12) [no timestamp] at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)
-
Hi,
I have the same issue. I use Cloudflare R2 for backups. I checked the files on backup backend- the apps data are uploaded, but missing "box" and "mail" file. If I run the backup manually, all files are present.
Aug 12 01:28:26 box:tasks update 807: {"percent":26,"message":"Copying part 5 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=4294967296-5368709119"} Aug 12 01:28:26 box:tasks update 807: {"percent":26,"message":"Copying part 1 - Etag: \"33b868e12c0a1a8af8b44a2cde45a093\""} Aug 12 01:28:26 box:tasks update 807: {"percent":26,"message":"Copying part 6 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=5368709120-6442450943"} Aug 12 01:28:43 box:tasks update 807: {"percent":26,"message":"Copying part 4 - Etag: \"01bb95a9d8f3f5fd56e7ab8043f102cb\""} Aug 12 01:28:43 box:tasks update 807: {"percent":26,"message":"Copying part 7 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=6442450944-7516192767"} Aug 12 01:28:50 box:tasks update 807: {"percent":26,"message":"Copying part 6 - Etag: \"c0b28dc1aa75b88ccfb001e204a446eb\""} Aug 12 01:28:50 box:tasks update 807: {"percent":26,"message":"Copying part 8 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=7516192768-8589934591"} Aug 12 01:28:59 box:tasks update 807: {"percent":26,"message":"Copying part 5 - Etag: \"fa5db189a1e003859392f2e65ab1037c\""} Aug 12 01:28:59 box:tasks update 807: {"percent":26,"message":"Copying part 9 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=8589934592-9663676415"} Aug 12 01:29:09 box:tasks update 807: {"percent":26,"message":"Copying part 7 - Etag: \"962eb50a4efcb9b4f807d41f9d43192e\""} Aug 12 01:29:09 box:tasks update 807: {"percent":26,"message":"Copying part 10 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=9663676416-10737418239"} Aug 12 01:29:14 box:tasks update 807: {"percent":26,"message":"Copying part 8 - Etag: \"4be980ac4ffb101b7976aa9fef9db8db\""} Aug 12 01:29:14 box:tasks update 807: {"percent":26,"message":"Copying part 11 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=10737418240-11811160063"} Aug 12 01:29:29 box:tasks update 807: {"percent":26,"message":"Copying part 9 - Etag: \"2bf02638777130e223237cab2fcf0a77\""} Aug 12 01:29:29 box:tasks update 807: {"percent":26,"message":"Copying part 12 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=11811160064-12884901887"} Aug 12 01:29:34 box:tasks update 807: {"percent":26,"message":"Copying part 10 - Etag: \"6e1184a760e94378ec2537522680c39b\""} Aug 12 01:29:34 box:tasks update 807: {"percent":26,"message":"Copying part 13 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=12884901888-13958643711"} Aug 12 01:29:36 box:tasks update 807: {"percent":26,"message":"Copying part 11 - Etag: \"3d6efeccd5b0da0f00b5732134f7022d\""} Aug 12 01:29:36 box:tasks update 807: {"percent":26,"message":"Copying part 14 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=13958643712-15032385535"} Aug 12 01:29:54 box:tasks update 807: {"percent":26,"message":"Copying part 12 - Etag: \"dea273f8059489ac8ad0432ee01df7cc\""} Aug 12 01:29:54 box:tasks update 807: {"percent":26,"message":"Copying part 15 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=15032385536-16106127359"} Aug 12 01:29:58 box:tasks update 807: {"percent":26,"message":"Copying part 13 - Etag: \"9693e089e8d419b2dd4f8f22097d0d08\""} Aug 12 01:29:58 box:tasks update 807: {"percent":26,"message":"Copying part 16 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=16106127360-17179869183"} Aug 12 01:29:59 box:tasks update 807: {"percent":26,"message":"Copying part 14 - Etag: \"084a28eeacf7e0283ba857fbcd5c698d\""} Aug 12 01:29:59 box:tasks update 807: {"percent":26,"message":"Copying part 17 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=17179869184-18253611007"} Aug 12 01:30:20 box:tasks update 807: {"percent":26,"message":"Copying part 15 - Etag: \"b9e20236f9775acc4ba95beae6a6bbe9\""} Aug 12 01:30:20 box:tasks update 807: {"percent":26,"message":"Copying part 18 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=18253611008-19327352831"} Aug 12 01:30:24 box:tasks update 807: {"percent":26,"message":"Copying part 17 - Etag: \"f1f821d2e88e4f668b3187e6c010d39f\""} Aug 12 01:30:24 box:tasks update 807: {"percent":26,"message":"Copying part 19 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=19327352832-20401094655"} Aug 12 01:30:26 box:tasks update 807: {"percent":26,"message":"Copying part 16 - Etag: \"815bfca5b54a8f18500b40a934056683\""} Aug 12 01:30:26 box:tasks update 807: {"percent":26,"message":"Copying part 20 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=20401094656-21474836479"} Aug 12 01:30:46 box:tasks update 807: {"percent":26,"message":"Copying part 19 - Etag: \"e8377d62367350b86523e1ac4a5592cd\""} Aug 12 01:30:46 box:tasks update 807: {"percent":26,"message":"Copying part 21 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=21474836480-22548578303"} Aug 12 01:30:48 box:tasks update 807: {"percent":26,"message":"Copying part 18 - Etag: \"4afe748647398ea10fb988cf4245fd9a\""} Aug 12 01:30:48 box:tasks update 807: {"percent":26,"message":"Copying part 22 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=22548578304-23622320127"} Aug 12 01:30:53 box:tasks update 807: {"percent":26,"message":"Copying part 20 - Etag: \"fe2b1e53c801f24a542d1a110cbd97a6\""} Aug 12 01:30:53 box:tasks update 807: {"percent":26,"message":"Copying part 23 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=23622320128-24696061951"} Aug 12 01:31:15 box:tasks update 807: {"percent":26,"message":"Copying part 21 - Etag: \"6833536e5d01a8f37e8c495fe2ff9fe5\""} Aug 12 01:31:15 box:tasks update 807: {"percent":26,"message":"Copying part 24 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=24696061952-25769803775"} Aug 12 01:31:20 box:tasks update 807: {"percent":26,"message":"Copying part 22 - Etag: \"42d3741db4d8954e25b7e2bfdb055761\""} Aug 12 01:31:20 box:tasks update 807: {"percent":26,"message":"Copying part 25 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=25769803776-26843545599"} Aug 12 01:31:27 box:tasks update 807: {"percent":26,"message":"Copying part 23 - Etag: \"ec99fdd8532734c70b26dff5dc8d1878\""} Aug 12 01:31:27 box:tasks update 807: {"percent":26,"message":"Copying part 26 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=26843545600-27917287423"} Aug 12 01:31:44 box:tasks update 807: {"percent":26,"message":"Copying part 24 - Etag: \"ada05bf27416a40c96b39d6bae214286\""} Aug 12 01:31:44 box:tasks update 807: {"percent":26,"message":"Copying part 27 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=27917287424-28991029247"} Aug 12 01:31:44 box:tasks update 807: {"percent":26,"message":"Copying part 25 - Etag: \"886a5b95657bec703890efea71fc2cb4\""} Aug 12 01:31:44 box:tasks update 807: {"percent":26,"message":"Copying part 28 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=28991029248-30064771071"} Aug 12 01:31:53 box:tasks update 807: {"percent":26,"message":"Copying part 26 - Etag: \"cfda9f13b0781a5e8617c9d0772bdf12\""} Aug 12 01:31:53 box:tasks update 807: {"percent":26,"message":"Copying part 29 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=30064771072-31138512895"} Aug 12 01:32:06 box:tasks update 807: {"percent":26,"message":"Copying part 27 - Etag: \"e14692a5d40c41dde51c3a4b2f679d74\""} Aug 12 01:32:06 box:tasks update 807: {"percent":26,"message":"Copying part 30 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=31138512896-32212254719"} Aug 12 01:32:17 box:tasks update 807: {"percent":26,"message":"Copying part 29 - Etag: \"26a4e2ab629de27e0eaf15f7d12a23bf\""} Aug 12 01:32:17 box:tasks update 807: {"percent":26,"message":"Copying part 31 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=32212254720-33285996543"} Aug 12 01:32:28 box:tasks update 807: {"percent":26,"message":"Copying part 30 - Etag: \"6ff04fb1897a060f2360f3d70a8ef64e\""} Aug 12 01:32:28 box:tasks update 807: {"percent":26,"message":"Copying part 32 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=33285996544-34359738367"} Aug 12 01:32:32 box:tasks update 807: {"percent":26,"message":"Copying part 28 - Etag: \"bd5141885323b7ae657b29b69339e025\""} Aug 12 01:32:32 box:tasks update 807: {"percent":26,"message":"Copying part 33 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=34359738368-35433480191"} Aug 12 01:32:43 box:tasks update 807: {"percent":26,"message":"Copying part 31 - Etag: \"d8fb57d1d84e3d86c5fd4c3c775251e3\""} Aug 12 01:32:43 box:tasks update 807: {"percent":26,"message":"Copying part 34 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=35433480192-36507222015"} Aug 12 01:32:50 box:tasks update 807: {"percent":26,"message":"Copying part 32 - Etag: \"c4eb47b97227479e2aa9001813c5f542\""} Aug 12 01:32:50 box:tasks update 807: {"percent":26,"message":"Copying part 35 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=36507222016-37580963839"} Aug 12 01:32:55 box:tasks update 807: {"percent":26,"message":"Copying part 33 - Etag: \"e4c504d31e5e6519defbb07add7b89c1\""} Aug 12 01:32:55 box:tasks update 807: {"percent":26,"message":"Copying part 36 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=37580963840-38654705663"} Aug 12 01:33:15 box:tasks update 807: {"percent":26,"message":"Copying part 35 - Etag: \"0d707006393e440d6c69211e7f593f1d\""} Aug 12 01:33:15 box:tasks update 807: {"percent":26,"message":"Copying part 37 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=38654705664-39728447487"} Aug 12 01:33:16 box:tasks update 807: {"percent":26,"message":"Copying part 34 - Etag: \"04e318aa113a24a367b3151332edba59\""} Aug 12 01:33:16 box:tasks update 807: {"percent":26,"message":"Copying part 38 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=39728447488-40802189311"} Aug 12 01:33:19 box:tasks update 807: {"percent":26,"message":"Copying part 36 - Etag: \"6496e604aaeba3f9a973d38ab1913f05\""} Aug 12 01:33:19 box:tasks update 807: {"percent":26,"message":"Copying part 39 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=40802189312-41875931135"} Aug 12 01:33:38 box:tasks update 807: {"percent":26,"message":"Copying part 37 - Etag: \"2827507425bad61157b050f1cb5b6d1e\""} Aug 12 01:33:38 box:tasks update 807: {"percent":26,"message":"Copying part 40 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=41875931136-42949672959"} Aug 12 01:33:45 box:tasks update 807: {"percent":26,"message":"Copying part 39 - Etag: \"3f27d29e3a49a7ac96857fd5d132845e\""} Aug 12 01:33:45 box:tasks update 807: {"percent":26,"message":"Copying part 41 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=42949672960-44023414783"} Aug 12 01:33:52 box:tasks update 807: {"percent":26,"message":"Copying part 38 - Etag: \"07619738fdaec13e80842c4cc1456dc7\""} Aug 12 01:33:52 box:tasks update 807: {"percent":26,"message":"Copying part 42 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=44023414784-45097156607"} Aug 12 01:34:02 box:tasks update 807: {"percent":26,"message":"Copying part 40 - Etag: \"ca57c24119eb33f45cd47308a7c701b4\""} Aug 12 01:34:02 box:tasks update 807: {"percent":26,"message":"Copying part 43 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=45097156608-46170898431"} Aug 12 01:34:06 box:tasks update 807: {"percent":26,"message":"Copying part 41 - Etag: \"3efe9e5894a1d3ef9e09721db850d7bb\""} Aug 12 01:34:06 box:tasks update 807: {"percent":26,"message":"Copying part 44 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=46170898432-47244640255"} Aug 12 01:34:16 box:tasks update 807: {"percent":26,"message":"Copying part 42 - Etag: \"8354e6225d1c89c456756ef233d9d644\""} Aug 12 01:34:16 box:tasks update 807: {"percent":26,"message":"Copying part 45 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=47244640256-48318382079"} Aug 12 01:34:27 box:tasks update 807: {"percent":26,"message":"Copying part 43 - Etag: \"e7ee0adeb8af45fe482f3a5ccec52bf5\""} Aug 12 01:34:27 box:tasks update 807: {"percent":26,"message":"Copying part 46 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=48318382080-49392123903"} Aug 12 01:34:32 box:tasks update 807: {"percent":26,"message":"Copying part 44 - Etag: \"1a55cf4da00bd008550222d9c966ecf2\""} Aug 12 01:34:32 box:tasks update 807: {"percent":26,"message":"Copying part 47 - /cloudron/immich.space/snapshot/app_8ce0be89-2f54-4681-b298-dfe6d05d2223.tar.gz.enc bytes=49392123904-49936582835"} Aug 12 01:34:39 box:tasks update 807: {"percent":26,"message":"Copying part 45 - Etag: \"e1d98f2d2e7a1ecf4ae4bc08db2d86fc\""} Aug 12 01:34:45 box:tasks update 807: {"percent":26,"message":"Copying part 47 - Etag: \"36520896cfda09f00ef7dd65f9a8d097\""} Aug 12 01:34:53 box:tasks update 807: {"percent":26,"message":"Copying part 46 - Etag: \"2a7ba94f8d693a352a3d4d12cb995c21\""} Aug 12 01:34:53 box:tasks update 807: {"percent":26,"message":"Finishing multipart copy - immich.space/2024-08-11-230000-469/app_immich.space_v1.68.1.tar.gz.enc"} Aug 12 01:34:53 box:tasks update 807: {"percent":26,"message":"Copied 1 files with error: null"} Aug 12 01:34:53 box:backuptask copy: copied successfully to 2024-08-11-230000-469/app_immich.space_v1.68.1. Took 416.167 seconds Aug 12 01:34:53 box:taskworker Task took 2093.503 seconds Aug 12 01:34:53 box:tasks setCompleted - 807: {"result":null,"error":{"stack":"BoxError: Backup not found\n at Object.setState (/home/yellowtent/box/src/backups.js:237:42)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async rotateAppBackup (/home/yellowtent/box/src/backuptask.js:304:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:384:12)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Not found","details":{},"message":"Backup not found"}} Aug 12 01:34:53 box:tasks update 807: {"percent":100,"result":null,"error":{"stack":"BoxError: Backup not found\n at Object.setState (/home/yellowtent/box/src/backups.js:237:42)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async rotateAppBackup (/home/yellowtent/box/src/backuptask.js:304:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:384:12)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Not found","details":{},"message":"Backup not found"}} [no timestamp] Backup not found [no timestamp] at Object.setState (/home/yellowtent/box/src/backups.js:237:42) [no timestamp] at process.processTicksAndRejections (node:internal/process/task_queues:95:5) [no timestamp] at async rotateAppBackup (/home/yellowtent/box/src/backuptask.js:304:5) [no timestamp] at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:384:12) [no timestamp] at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)
-
I noticed similar issues since the update to Cloudron 8, but in my case it was caused by the fact that the backup storage wasn't mounted correctly.
After clicking the "remount storage" button this was fixed and backups started working again.I'm using sshfs and had this issue on 2 Cloudrons.
Maybe there is a general regression in the update procedure that causes the backup storage location to be in an unmounted state, resulting in failed backups? -
@guyds the sshfs issue is somehow related to DNS, but some users have reported this issue. Once you remount, the problem should be fixed thereafter.
@kubasbimbas not sure what's going on. Are you able to contact us on support@cloudron.io so we acn debug further?
-
I am also getting errors daily on backup. I use Backblaze B2. Here is a partial log:
2024-08-10T06:15:12.080Z box:storage/s3 Upload progress: {"loaded":2915041280,"part":274,"key":"snapshot/app_7481caee-b863-4501-8a36-4fb08884505c.tar.gz"}
2024-08-10T06:15:12.663Z box:storage/s3 Upload progress: {"loaded":2925527040,"part":281,"key":"snapshot/app_7481caee-b863-4501-8a36-4fb08884505c.tar.gz"}
2024-08-10T06:15:13.201Z box:storage/s3 Upload progress: {"loaded":2936012800,"part":282,"key":"snapshot/app_7481caee-b863-4501-8a36-4fb08884505c.tar.gz"}
2024-08-10T06:15:13.835Z box:storage/s3 Upload progress: {"loaded":2946498560,"part":283,"key":"snapshot/app_7481caee-b863-4501-8a36-4fb08884505c.tar.gz"}
2024-08-10T06:15:14.397Z box:storage/s3 Upload progress: {"loaded":2956984320,"part":284,"key":"snapshot/app_7481caee-b863-4501-8a36-4fb08884505c.tar.gz"}
2024-08-10T06:15:15.154Z box:storage/s3 Upload progress: {"loaded":2967470080,"part":285,"key":"snapshot/app_7481caee-b863-4501-8a36-4fb08884505c.tar.gz"}
2024-08-10T06:15:16.550Z box:tasks update 2092: {"percent":34.333333333333336,"message":"Uploading backup 2876M@7MBps (nc.dellis.co)"}
2024-08-10T06:15:21.116Z box:storage/s3 Upload progress: {"loaded":2967470080,"part":274,"key":"snapshot/app_7481caee-b863-4501-8a36-4fb08884505c.tar.gz"}
node:events:496
throw er; // Unhandled 'error' event
^Error: write EPIPE
at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16)
Emitted 'error' event on TLSSocket instance at:
at emitErrorNT (node:internal/streams/destroy:169:8)
at emitErrorCloseNT (node:internal/streams/destroy:128:3)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
errno: -32,
code: 'EPIPE',
syscall: 'write'
}Node.js v20.12.2
2024-08-10T06:15:21.150Z box:shell backup-snapshot/app_7481caee-b863-4501-8a36-4fb08884505c: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_7481caee-b863-4501-8a36-4fb08884505c tgz {"localRoot":"/home/yellowtent/appsdata/7481caee-b863-4501-8a36-4fb08884505c","layout":[]} errored BoxError: backup-snapshot/app_7481caee-b863-4501-8a36-4fb08884505c exited with code 1 signal null
at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:122:19)
at ChildProcess.emit (node:events:518:28)
at ChildProcess.emit (node:domain:488:12)
at ChildProcess._handle.onexit (node:internal/child_process:294:12) {
reason: 'Shell Error',
details: {},
code: 1,
signal: null
}
2024-08-10T06:15:21.154Z box:taskworker Task took 919.866 seconds
2024-08-10T06:15:21.164Z box:tasks setCompleted - 2092: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
2024-08-10T06:15:21.165Z box:tasks update 2092: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
BoxError: Backuptask crashed
at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)
at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)
at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29) -
Again today. And i changed the S3 from AWS to Backblaze. Manual backup is working but automatic doesn't work. But since the change have been done, it is like automatic update doesn't trigger all the time.. (The successful backup has been triggered by me manualy).
Full logs of the last automatic failed backup (from today).
Aug 14 04:04:12 box:backupformat/tgz addToPack: added ./data/data/admin/files/EM/M1-S2/Rapport d'Activité/Rapport d'Activité n°2.docx file Aug 14 04:04:12 box:backupformat/tgz tarPack: processing /mnt/volumes/6e6d77d1ac41447da93a95677f20c922/Drive/data/admin/files/IA Aug 14 04:04:12 box:backupformat/tgz addToPack: added ./data/data/admin/files/IA/Ressources directory Aug 14 04:04:12 box:backupformat/tgz tarPack: processing /mnt/volumes/6e6d77d1ac41447da93a95677f20c922/Drive/data/admin/files/IA/Ressources Aug 14 04:04:12 box:backupformat/tgz addToPack: added ./data/data/admin/files/IA/Ressources/SOP pour Vérifier l'Activité d'une Application sur GitHub.docx file Aug 14 04:04:12 box:backupformat/tgz tarPack: processing /mnt/volumes/6e6d77d1ac41447da93a95677f20c922/Drive/data/admin/files/Perso Aug 14 04:04:12 box:backupformat/tgz addToPack: added ./data/data/admin/files/Perso/Articles A Acheter directory Aug 14 04:04:12 box:backupformat/tgz addToPack: added ./data/data/admin/files/Perso/Cadeaux Idées directory Aug 14 04:04:12 box:backupformat/tgz addToPack: added ./data/data/admin/files/Perso/Casquettes directory Aug 14 04:04:12 box:backupformat/tgz addToPack: added ./data/data/admin/files/Perso/Dress 15-06-24 directory Aug 14 04:04:12 box:backupformat/tgz addToPack: added ./data/data/admin/files/Perso/Elle.docx file Aug 14 04:04:12 box:backupformat/tgz addToPack: added ./data/data/admin/files/Perso/Jeux Vidéos directory Aug 14 04:04:12 box:backupformat/tgz addToPack: added ./data/data/admin/files/Perso/Logo Propal 2 A-One.png file Aug 14 04:04:13 box:storage/s3 Upload progress: {"loaded":3221225472,"part":2,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"} Aug 14 04:04:13 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 4539M@44MBps (drive.example.com)"} Aug 14 04:04:14 box:backupformat/tgz addToPack: added ./data/data/admin/files/Perso/Monologue Rayane Insta.mp4 file Aug 14 04:04:14 box:backupformat/tgz addToPack: added ./data/data/admin/files/Perso/Musiques directory Aug 14 04:04:23 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 5009M@47MBps (drive.example.com)"} Aug 14 04:04:33 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 5400M@39MBps (drive.example.com)"} Aug 14 04:04:43 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 5860M@46MBps (drive.example.com)"} Aug 14 04:04:53 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 6144M@28MBps (drive.example.com)"} Aug 14 04:04:53 box:storage/s3 Upload progress: {"loaded":4294967296,"part":5,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"} Aug 14 04:05:03 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 6144M@0MBps (drive.example.com)"} Aug 14 04:05:13 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 6144M@0MBps (drive.example.com)"} Aug 14 04:05:19 box:storage/s3 Upload progress: {"loaded":4294967296,"part":5,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"} Aug 14 04:05:23 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 6144M@0MBps (drive.example.com)"} Aug 14 04:05:33 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 6144M@0MBps (drive.example.com)"} Aug 14 04:05:34 box:storage/s3 Upload progress: {"loaded":4294967296,"part":1,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"} Aug 14 04:05:34 box:storage/s3 Upload progress: {"loaded":5368709120,"part":4,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"} Aug 14 04:05:44 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 6144M@0MBps (drive.example.com)"} Aug 14 04:05:52 box:storage/s3 Upload progress: {"loaded":5368709120,"part":5,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"} Aug 14 04:05:54 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 6583M@44MBps (drive.example.com)"} Aug 14 04:06:04 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 7038M@46MBps (drive.example.com)"} Aug 14 04:06:14 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 7420M@38MBps (drive.example.com)"} Aug 14 04:06:23 box:storage/s3 Upload progress: {"loaded":5368709120,"part":5,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"} Aug 14 04:06:24 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 7891M@47MBps (drive.example.com)"} Aug 14 04:06:34 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 8192M@30MBps (drive.example.com)"} Aug 14 04:06:44 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 8192M@0MBps (drive.example.com)"} Aug 14 04:06:54 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 8192M@0MBps (drive.example.com)"} Aug 14 04:07:04 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 8192M@0MBps (drive.example.com)"} Aug 14 04:07:14 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 8192M@0MBps (drive.example.com)"} Aug 14 04:07:16 box:storage/s3 Upload progress: {"loaded":6442450944,"part":6,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"} Aug 14 04:07:24 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 8435M@24MBps (drive.example.com)"} Aug 14 04:07:34 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 8894M@46MBps (drive.example.com)"} Aug 14 04:07:43 box:storage/s3 Upload progress: {"loaded":7516192768,"part":8,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"} Aug 14 04:07:44 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 9216M@32MBps (drive.example.com)"} Aug 14 04:07:54 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 9216M@0MBps (drive.example.com)"} Aug 14 04:08:01 box:storage/s3 Upload progress: {"loaded":7516192768,"part":5,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"} Aug 14 04:08:04 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 9217M@0MBps (drive.example.com)"} Aug 14 04:08:14 box:storage/s3 Upload progress: {"loaded":7516192768,"part":8,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"} Aug 14 04:08:14 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 9685M@47MBps (drive.example.com)"} Aug 14 04:08:24 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 10143M@46MBps (drive.example.com)"} Aug 14 04:08:34 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 10240M@10MBps (drive.example.com)"} Aug 14 04:08:44 box:storage/s3 Upload progress: {"loaded":7516192768,"part":8,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"} Aug 14 04:08:44 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 10240M@0MBps (drive.example.com)"} Aug 14 04:08:54 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 10240M@0MBps (drive.example.com)"} Aug 14 04:09:04 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 10240M@0MBps (drive.example.com)"} Aug 14 04:09:05 box:storage/s3 Upload progress: {"loaded":8589934592,"part":7,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"} Aug 14 04:09:14 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 10240M@0MBps (drive.example.com)"} Aug 14 04:09:24 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 10554M@31MBps (drive.example.com)"} Aug 14 04:09:32 box:storage/s3 Upload progress: {"loaded":9663676416,"part":9,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"} Aug 14 04:09:34 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 11023M@47MBps (drive.example.com)"} Aug 14 04:09:44 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 11416M@39MBps (drive.example.com)"} Aug 14 04:09:54 box:tasks update 3103: {"percent":28.450980392156865,"message":"Uploading backup 11864M@45MBps (drive.example.com)"} Aug 14 04:09:59 box:storage/s3 Upload progress: {"loaded":10737418240,"part":10,"key":"snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872.tar.gz.enc"} Jan 01 00:00:00 node:events:496 [no timestamp] throw er; // Unhandled 'error' event [no timestamp] ^ [no timestamp] [no timestamp] write EPIPE [no timestamp] at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:94:16) [no timestamp] 'error' event on TLSSocket instance at: [no timestamp] at emitErrorNT (node:internal/streams/destroy:169:8) [no timestamp] at emitErrorCloseNT (node:internal/streams/destroy:128:3) [no timestamp] at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { [no timestamp] errno: -32, [no timestamp] code: 'EPIPE', [no timestamp] syscall: 'write' [no timestamp] } [no timestamp] [no timestamp] v20.12.2 Aug 14 04:09:59 box:shell backup-snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872 tgz {"localRoot":"/home/yellowtent/appsdata/370be07e-c281-4d35-95a1-5c9984f72872","layout":[{"localDir":"/mnt/volumes/6e6d77d1ac41447da93a95677f20c922/Drive","remoteDir":"data"}]} errored BoxError: backup-snapshot/app_370be07e-c281-4d35-95a1-5c9984f72872 exited with code 1 signal null [no timestamp] at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:122:19) [no timestamp] at ChildProcess.emit (node:events:518:28) [no timestamp] at ChildProcess.emit (node:domain:488:12) [no timestamp] at ChildProcess._handle.onexit (node:internal/child_process:294:12) { [no timestamp] reason: 'Shell Error', [no timestamp] details: {}, [no timestamp] code: 1, [no timestamp] signal: null [no timestamp] } Aug 14 04:09:59 box:taskworker Task took 598.935 seconds Aug 14 04:09:59 box:tasks setCompleted - 3103: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}} Aug 14 04:09:59 box:tasks update 3103: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5)\n at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}} [no timestamp] Backuptask crashed [no timestamp] at runBackupUpload (/home/yellowtent/box/src/backuptask.js:164:15) [no timestamp] at process.processTicksAndRejections (node:internal/process/task_queues:95:5) [no timestamp] at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:361:5) [no timestamp] at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:383:5) [no timestamp] at async fullBackup (/home/yellowtent/box/src/backuptask.js:504:29)
-
I have noticed that the backup issue affects also the app updates, as the app is backed up before it is updated. Last night the Immich app started scheduled update 2 times from v1.68.1, but never updated to v1.69.0. I have then started the update manually this morning and it was completed.
-
@kubasbimbas that is expected, apps are always backed up before an update. if backups fails, the update will abort and go back to previous version .
-
-
@joseph I have verified that the app data were uploaded to the backup backend. Let's wait for tonight scheduled update to v1.69.1.
-
Just looked into this. I think there is a regression in Cloudron 8. We fixed the backup cleaner to work properly but in the process it cleans up the backup entry in the database that is being created. https://git.cloudron.io/cloudron/box/-/commit/9704eefc21af313a4239e74999186ae44a43ed46 is the fix for that. @kubasbimbas can you try that one line change? You have to add that line to the file
/home/yellowtent/box/src/backupcleaner.js
on the server . No need to restart anything after applying the patch. -
@Dont-Worry no worries. if you send us a mail to support@cloudron.io , we can apply the patch on your server . (We debugged the issue on somebody's server, but we cannot match forum username and support requests... If you had sent us a mail ealier, we already patched your server)
-
@girish You patched my server. I confirm that automatic Cloudron backup as well as app update worked correctly. Thank you for your support.
-
Same here on multiple servers backing up against S3. What's an ETA for the hotfix release?
-
-
If that's not too much, I would appreciate if you could ping this thread, once the new version released.
-
Thank you!