Updating FreeScout failed
-
I try to update FreeScout (+ 2 GB data) but during app backup the backup gets stuck at around 2GB and after some time the backup stops and the update seems to be failed.
I remember this same issue was a few weeks ago with Matomo also?
-
@girish ok, here we go again:
logs (redacted, removed all the /polycast/ lines and replaced URL):
Mar 15 22:18:33 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:18:33.289Z box:backupupload process: rss: 422084608 heapTotal: 76046336 heapUsed: 35838880 external: 449119645 Mar 15 22:18:33 box:backupupload v8 heap : used 35843208 total: 76046336 max: 8640266240 Mar 15 22:18:33 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 429M@14MBps (---app URL---)"} Mar 15 22:18:36 172.18.0.1 - - [15/Mar/2021:21:18:36 +0000] "POST /polycast/receive HTTP/1.1" 200 1051 "URL" Mar 15 22:18:53 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 761M@15MBps (---app URL---)"} Mar 15 22:19:03 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:19:03.289Z box:backupupload process: rss: 432582656 heapTotal: 82280448 heapUsed: 40514152 external: 941970845 Mar 15 22:19:03 box:backupupload v8 heap : used 40519584 total: 82280448 max: 8640266240 Mar 15 22:19:03 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 898M@15MBps (---app URL---)"} Mar 15 22:19:27 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1024M@15MBps (---app URL---)"} Mar 15 22:19:39 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:19:39.090Z box:backupupload process: rss: 435388416 heapTotal: 51912704 heapUsed: 43137064 external: 2153077299 Mar 15 22:19:39 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:19:39.097Z box:backupupload v8 heap : used 43141392 total: 51912704 max: 8640266240 Mar 15 22:19:39 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1026M@15MBps (---app URL---)"} Mar 15 22:19:55 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1027M@15MBps (---app URL---)"} Mar 15 22:20:13 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:20:13.009Z box:backupupload process: rss: 428376064 heapTotal: 46276608 heapUsed: 29052936 external: 1098173454 Mar 15 22:20:13 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:20:13.009Z box:backupupload v8 heap : used 29057200 total: 46276608 max: 8640266240 Mar 15 22:20:13 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1036M@14MBps ---app URL---)"} Mar 15 22:20:23 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1131M@14MBps (---app URL---)"} Mar 15 22:20:33 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1249M@14MBps (---app URL---)"} Mar 15 22:20:43 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:20:43.029Z box:backupupload process: rss: 429768704 heapTotal: 55463936 heapUsed: 31817008 external: 1421413365 Mar 15 22:20:43 box:backupupload v8 heap : used 31822456 total: 55463936 max: 8640266240 Mar 15 22:20:43 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1352M@13MBps (---app URL---)"} Mar 15 22:20:53 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1467M@13MBps (---app URL---)"} Mar 15 22:21:03 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1563M@13MBps (---app URL---)"} Mar 15 22:21:13 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:21:13.033Z box:backupupload process: rss: 422223872 heapTotal: 76251136 heapUsed: 35887104 external: 1739975157 Mar 15 22:21:13 box:backupupload v8 heap : used 35892552 total: 76251136 max: 8640266240 Mar 15 22:21:13 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1653M@13MBps ---app URL---)"} Mar 15 22:21:23 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1750M@12MBps (---app URL---)"} Mar 15 22:21:33 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1847M@12MBps (---app URL---)"} Mar 15 22:21:43 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:21:43.030Z box:backupupload process: rss: 401838080 heapTotal: 79659008 heapUsed: 38668648 external: 2036756469 Mar 15 22:21:43 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:21:43.031Z box:backupupload v8 heap : used 38674096 total: 79659008 max: 8640266240 Mar 15 22:21:43 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1932M@12MBps (---app URL---)"} Mar 15 22:21:53 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 2022M@12MBps (---app URL---)"} Mar 15 22:23:31 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:23:30.976Z box:backupupload process: rss: 405590016 heapTotal: 49029120 heapUsed: 38707320 external: 3223178229 Mar 15 22:23:30 box:backupupload v8 heap : used 38711648 total: 49029120 max: 8640266240 Mar 15 22:23:31 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 2048M@12MBps (---app URL---)"} Mar 15 22:23:37 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 code: null, signal: SIGKILL Mar 15 22:23:37 box:apptask ---app URL--- update aborted because backup failed BoxError: Backuptask crashed at /home/yellowtent/box/src/backups.js:895:29 at f (/home/yellowtent/box/node_modules/once/once.js:25:25) at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9) at ChildProcess.emit (events.js:315:20) at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12) { reason: 'Internal Error', details: {}, backupError: true Mar 15 22:23:37 box:apptask r---app URL--- updating app with values: {"installationState":"installed","error":null,"health":null} Mar 15 22:23:37 box:taskworker Task took 336.712 seconds Mar 15 22:23:37 box:tasks setCompleted - 3924: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at /home/yellowtent/box/src/backups.js:895:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:315:20)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed","backupError":true}} Mar 15 22:23:37 box:tasks 3924: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at /home/yellowtent/box/src/backups.js:895:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:315:20)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed","backupError":true}}
Zabbix (although the server has 64GB memory the swap space disappears during app-backup!!!!):
The app has 4GB memory.
-
@imc67 Does your backup configuration have enough memory? Or rather, I'm curious how much memory it's been given so far. If it doesn't have enough memory, I presume it may crash like that as it has for me on occasion when I was tweaking how many concurrency for the backups but didn't give it enough overall memory to accomodate the concurrency values, for example.
-
@imc67 there is a bug here . Can you apply this one liner and then set the memory limit https://git.cloudron.io/cloudron/box/-/commit/750f313c6a36d8d961892df4a41cb7c41faa1f1b ? You have
systemctl restart box
after you change that line in/home/yellowtent/box/src/apps.js
. -
@girish said in Updating FreeScout failed:
@imc67 there is a bug here . Can you apply this one liner and then set the memory limit https://git.cloudron.io/cloudron/box/-/commit/750f313c6a36d8d961892df4a41cb7c41faa1f1b ? You have
systemctl restart box
after you change that line in/home/yellowtent/box/src/apps.js
.Will try this in a few hours
-
@imc67 said in Updating FreeScout failed:
@girish said in Updating FreeScout failed:
@imc67 there is a bug here . Can you apply this one liner and then set the memory limit https://git.cloudron.io/cloudron/box/-/commit/750f313c6a36d8d961892df4a41cb7c41faa1f1b ? You have
systemctl restart box
after you change that line in/home/yellowtent/box/src/apps.js
.Will try this in a few hours
@girish this solved the bug, thanks!