Updating FreeScout failed

[imc67]

I try to update FreeScout (+ 2 GB data) but during app backup the backup gets stuck at around 2GB and after some time the backup stops and the update seems to be failed.

I remember this same issue was a few weeks ago with Matomo also?

[girish]

@imc67 This seems to me like the backup failed (and when backup fails, the code just aborts the update and keeps it at the old version). Do you see any backup error in the app logs? Also, where are you backing up to?

[imc67]

@girish ok, here we go again:

logs (redacted, removed all the /polycast/ lines and replaced URL):

Mar 15 22:18:33 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:18:33.289Z box:backupupload process: rss: 422084608 heapTotal: 76046336 heapUsed: 35838880 external: 449119645
Mar 15 22:18:33 box:backupupload v8 heap : used 35843208 total: 76046336 max: 8640266240
Mar 15 22:18:33 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 429M@14MBps (---app URL---)"}
Mar 15 22:18:36 172.18.0.1 - - [15/Mar/2021:21:18:36 +0000] "POST /polycast/receive HTTP/1.1" 200 1051 "URL"

Mar 15 22:18:53 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 761M@15MBps (---app URL---)"}


Mar 15 22:19:03 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:19:03.289Z box:backupupload process: rss: 432582656 heapTotal: 82280448 heapUsed: 40514152 external: 941970845
Mar 15 22:19:03 box:backupupload v8 heap : used 40519584 total: 82280448 max: 8640266240
Mar 15 22:19:03 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 898M@15MBps (---app URL---)"}



Mar 15 22:19:27 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1024M@15MBps (---app URL---)"}



Mar 15 22:19:39 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:19:39.090Z box:backupupload process: rss: 435388416 heapTotal: 51912704 heapUsed: 43137064 external: 2153077299
Mar 15 22:19:39 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:19:39.097Z box:backupupload v8 heap : used 43141392 total: 51912704 max: 8640266240
Mar 15 22:19:39 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1026M@15MBps (---app URL---)"}


Mar 15 22:19:55 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1027M@15MBps (---app URL---)"}


Mar 15 22:20:13 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:20:13.009Z box:backupupload process: rss: 428376064 heapTotal: 46276608 heapUsed: 29052936 external: 1098173454
Mar 15 22:20:13 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:20:13.009Z box:backupupload v8 heap : used 29057200 total: 46276608 max: 8640266240
Mar 15 22:20:13 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1036M@14MBps ---app URL---)"}


Mar 15 22:20:23 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1131M@14MBps (---app URL---)"}


Mar 15 22:20:33 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1249M@14MBps (---app URL---)"}


Mar 15 22:20:43 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:20:43.029Z box:backupupload process: rss: 429768704 heapTotal: 55463936 heapUsed: 31817008 external: 1421413365
Mar 15 22:20:43 box:backupupload v8 heap : used 31822456 total: 55463936 max: 8640266240
Mar 15 22:20:43 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1352M@13MBps (---app URL---)"}

Mar 15 22:20:53 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1467M@13MBps (---app URL---)"}


Mar 15 22:21:03 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1563M@13MBps (---app URL---)"}


Mar 15 22:21:13 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:21:13.033Z box:backupupload process: rss: 422223872 heapTotal: 76251136 heapUsed: 35887104 external: 1739975157
Mar 15 22:21:13 box:backupupload v8 heap : used 35892552 total: 76251136 max: 8640266240
Mar 15 22:21:13 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1653M@13MBps ---app URL---)"}


Mar 15 22:21:23 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1750M@12MBps (---app URL---)"}






Mar 15 22:21:33 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1847M@12MBps (---app URL---)"}




Mar 15 22:21:43 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:21:43.030Z box:backupupload process: rss: 401838080 heapTotal: 79659008 heapUsed: 38668648 external: 2036756469
Mar 15 22:21:43 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:21:43.031Z box:backupupload v8 heap : used 38674096 total: 79659008 max: 8640266240
Mar 15 22:21:43 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 1932M@12MBps (---app URL---)"}



Mar 15 22:21:53 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 2022M@12MBps (---app URL---)"}









Mar 15 22:23:31 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 (stdout): 2021-03-15T21:23:30.976Z box:backupupload process: rss: 405590016 heapTotal: 49029120 heapUsed: 38707320 external: 3223178229
Mar 15 22:23:30 box:backupupload v8 heap : used 38711648 total: 49029120 max: 8640266240

Mar 15 22:23:31 box:tasks 3924: {"percent":15,"message":"Backup - Uploading backup 2048M@12MBps (---app URL---)"}
Mar 15 22:23:37 box:shell backup-snapshot/app_299b63f7-cea7-458e-96fb-94db7f9d17b0 code: null, signal: SIGKILL
Mar 15 22:23:37 box:apptask ---app URL--- update aborted because backup failed BoxError: Backuptask crashed
at /home/yellowtent/box/src/backups.js:895:29
at f (/home/yellowtent/box/node_modules/once/once.js:25:25)
at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)
at ChildProcess.emit (events.js:315:20)
at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12) {
reason: 'Internal Error',
details: {},
backupError: true
Mar 15 22:23:37 box:apptask r---app URL--- updating app with values: {"installationState":"installed","error":null,"health":null}
Mar 15 22:23:37 box:taskworker Task took 336.712 seconds
Mar 15 22:23:37 box:tasks setCompleted - 3924: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at /home/yellowtent/box/src/backups.js:895:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:315:20)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed","backupError":true}}
Mar 15 22:23:37 box:tasks 3924: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at /home/yellowtent/box/src/backups.js:895:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:315:20)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed","backupError":true}}

Zabbix (although the server has 64GB memory the swap space disappears during app-backup!!!!):

The app has 4GB memory.

[d19dotca]

@imc67 Does your backup configuration have enough memory? Or rather, I'm curious how much memory it's been given so far. If it doesn't have enough memory, I presume it may crash like that as it has for me on occasion when I was tweaking how many concurrency for the backups but didn't give it enough overall memory to accomodate the concurrency values, for example.

[girish]

@imc67 as @d19dotca said , give the backup more memory in backup config.

[girish]

@imc67 there is a bug here . Can you apply this one liner and then set the memory limit https://git.cloudron.io/cloudron/box/-/commit/750f313c6a36d8d961892df4a41cb7c41faa1f1b ? You have systemctl restart box after you change that line in /home/yellowtent/box/src/apps.js.

[imc67]

@girish already from the migration to this server 3/4 year ago it has the full 64GB memory setting, which it never uses of course. Full (box) backups don’t fail.

[imc67]

@girish said in Updating FreeScout failed:

@imc67 there is a bug here . Can you apply this one liner and then set the memory limit https://git.cloudron.io/cloudron/box/-/commit/750f313c6a36d8d961892df4a41cb7c41faa1f1b ? You have systemctl restart box after you change that line in /home/yellowtent/box/src/apps.js.

Will try this in a few hours

[imc67]

@imc67 said in Updating FreeScout failed:

@girish said in Updating FreeScout failed:

@imc67 there is a bug here . Can you apply this one liner and then set the memory limit https://git.cloudron.io/cloudron/box/-/commit/750f313c6a36d8d961892df4a41cb7c41faa1f1b ? You have systemctl restart box after you change that line in /home/yellowtent/box/src/apps.js.

Will try this in a few hours

@girish this solved the bug, thanks!