Chat woot cannot uninstall after breaking the backup system.

[roofboard]

Starting 4 weeks ago my backups started failing. At the same time apparently.. Chatwoot stopped working. From what I can tell the backup fails when it tries to snapshot chatwoot which was in error state.

After disabling backup on chatwoot the backup started working Here is a copy of the error log for refrence.

Mar 30 20:19:30 box:tasks update 4224: {"percent":5.166666666666667,"message":"Backing up support.draglabs.com (2/21)"}
Mar 30 20:19:30 box:tasks update 4224: {"percent":9.333333333333334,"message":"Snapshotting app support.draglabs.com"}
Mar 30 20:19:30 box:services backupAddons
Mar 30 20:19:30 box:services backupAddons: backing up ["localstorage","postgresql","sendmail","redis"]
Mar 30 20:19:30 box:services Backing up postgresql
Mar 30 20:29:30 box:services pipeRequestToFile: timeout - connect or post-connect idle timeout
Mar 30 20:29:30 box:taskworker Task took 600.253 seconds
Mar 30 20:29:30 box:tasks setCompleted - 4224: {"result":null,"error":{"stack":"Error: Request timedout\n at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1244:20)\n at Object.onceWrapper (node:events:627:28)\n at ClientRequest.emit (node:events:513:28)\n at Socket.emitRequestTimeout (node:_http_client:839:9)\n at Object.onceWrapper (node:events:627:28)\n at Socket.emit (node:events:525:35)\n at Socket._onTimeout (node:net:550:8)\n at listOnTimeout (node:internal/timers:559:17)\n at processTimers (node:internal/timers:502:7)","message":"Request timedout"}}
Mar 30 20:29:30 box:tasks update 4224: {"percent":100,"result":null,"error":{"stack":"Error: Request timedout\n at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1244:20)\n at Object.onceWrapper (node:events:627:28)\n at ClientRequest.emit (node:events:513:28)\n at Socket.emitRequestTimeout (node:_http_client:839:9)\n at Object.onceWrapper (node:events:627:28)\n at Socket.emit (node:events:525:35)\n at Socket._onTimeout (node:net:550:8)\n at listOnTimeout (node:internal/timers:559:17)\n at processTimers (node:internal/timers:502:7)","message":"Request timedout"}}
Request timedout
at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1244:20)
at Object.onceWrapper (node:events:627:28)
at ClientRequest.emit (node:events:513:28)
at Socket.emitRequestTimeout (node:_http_client:839:9)
at Object.onceWrapper (node:events:627:28)
at Socket.emit (node:events:525:35)
at Socket._onTimeout (node:net:550:8)
at listOnTimeout (node:internal/timers:559:17)
at processTimers (node:internal/timers:502:7)

Now more importantly... I disabled Chatwoot and tried to uninstall it. When I go to uninstall I get this error.

Mar 30 07:10:02 box:taskworker Task took 600.802 seconds
Mar 30 07:10:02 box:tasks setCompleted - 4217: {"result":null,"error":{"stack":"Error: Request timedout\n at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1244:20)\n at Object.onceWrapper (node:events:627:28)\n at ClientRequest.emit (node:events:513:28)\n at Socket.emitRequestTimeout (node:_http_client:839:9)\n at Object.onceWrapper (node:events:627:28)\n at Socket.emit (node:events:525:35)\n at Socket._onTimeout (node:net:550:8)\n at listOnTimeout (node:internal/timers:559:17)\n at processTimers (node:internal/timers:502:7)","message":"Request timedout","backupError":true}}
Mar 30 07:10:02 box:tasks update 4217: {"percent":100,"result":null,"error":{"stack":"Error: Request timedout\n at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1244:20)\n at Object.onceWrapper (node:events:627:28)\n at ClientRequest.emit (node:events:513:28)\n at Socket.emitRequestTimeout (node:_http_client:839:9)\n at Object.onceWrapper (node:events:627:28)\n at Socket.emit (node:events:525:35)\n at Socket._onTimeout (node:net:550:8)\n at listOnTimeout (node:internal/timers:559:17)\n at processTimers (node:internal/timers:502:7)","message":"Request timedout","backupError":true}}
Request timedout
at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1244:20)
at Object.onceWrapper (node:events:627:28)
at ClientRequest.emit (node:events:513:28)
at Socket.emitRequestTimeout (node:_http_client:839:9)
at Object.onceWrapper (node:events:627:28)
at Socket.emit (node:events:525:35)
at Socket._onTimeout (node:net:550:8)
at listOnTimeout (node:internal/timers:559:17)
at processTimers (node:internal/timers:502:7)
Mar 30 20:38:54 box:settings initCache: pre-load settings
Mar 30 20:38:54 box:taskworker Starting task 4228. Logs are at /home/yellowtent/platformdata/logs/06c754d8-f01a-4fa2-b5c9-099e1916d0e8/apptask.log
Mar 30 20:38:54 box:apptask run: startTask installationState: pending_uninstall runState: running
Mar 30 20:38:54 box:tasks update 4228: {"percent":20,"message":"Deleting container"}
Mar 30 20:38:54 box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
Mar 30 20:38:55 box:apptask deleteContainer: deleting app containers (app, scheduler)
Mar 30 20:38:55 box:shell removeLogrotateConfig spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/configurelogrotate.sh remove 06c754d8-f01a-4fa2-b5c9-099e1916d0e8
Mar 30 20:39:11 box:tasks update 4228: {"percent":30,"message":"Teardown addons"}
Mar 30 20:39:11 box:services teardownAddons: Tearing down ["localstorage","postgresql","sendmail","redis"]
Mar 30 20:39:11 box:services teardownAddons: Tearing down addon localstorage with options {}
Mar 30 20:39:11 box:services teardownLocalStorage
Mar 30 20:39:11 box:shell clearVolume spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/clearvolume.sh rmdir /home/yellowtent/appsdata/06c754d8-f01a-4fa2-b5c9-099e1916d0e8/data
Mar 30 20:39:11 box:services teardownAddons: Tearing down addon postgresql with options {}
Mar 30 20:39:16 box:apptask run: app error for state pending_uninstall: BoxError: Error tearing down postgresql. Status code: 500 message: database "db06c754d8f01a4fa2b5c9099e1916d0e8" is being accessed by other users
at Object.teardownPostgreSql [as teardown] (/home/yellowtent/box/src/services.js:1437:40)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Object.teardownAddons (/home/yellowtent/box/src/services.js:623:9)
at async uninstall (/home/yellowtent/box/src/apptask.js:727:5) {
reason: 'Addons Error',
details: {}
Mar 30 20:39:16 box:taskworker Task took 21.51 seconds
Mar 30 20:39:16 box:tasks setCompleted - 4228: {"result":null,"error":{"stack":"BoxError: Error tearing down postgresql. Status code: 500 message: database \"db06c754d8f01a4fa2b5c9099e1916d0e8\" is being accessed by other users\n at Object.teardownPostgreSql [as teardown] (/home/yellowtent/box/src/services.js:1437:40)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at async Object.teardownAddons (/home/yellowtent/box/src/services.js:623:9)\n at async uninstall (/home/yellowtent/box/src/apptask.js:727:5)","name":"BoxError","reason":"Addons Error","details":{},"message":"Error tearing down postgresql. Status code: 500 message: database \"db06c754d8f01a4fa2b5c9099e1916d0e8\" is being accessed by other users"}}
Mar 30 20:39:16 box:tasks update 4228: {"percent":100,"result":null,"error":{"stack":"BoxError: Error tearing down postgresql. Status code: 500 message: database \"db06c754d8f01a4fa2b5c9099e1916d0e8\" is being accessed by other users\n at Object.teardownPostgreSql [as teardown] (/home/yellowtent/box/src/services.js:1437:40)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at async Object.teardownAddons (/home/yellowtent/box/src/services.js:623:9)\n at async uninstall (/home/yellowtent/box/src/apptask.js:727:5)","name":"BoxError","reason":"Addons Error","details":{},"message":"Error tearing down postgresql. Status code: 500 message: database \"db06c754d8f01a4fa2b5c9099e1916d0e8\" is being accessed by other users"}}
Error tearing down postgresql. Status code: 500 message: database "db06c754d8f01a4fa2b5c9099e1916d0e8" is being accessed by other users
at Object.teardownPostgreSql [as teardown] (/home/yellowtent/box/src/services.js:1437:40)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Object.teardownAddons (/home/yellowtent/box/src/services.js:623:9)
at async uninstall (/home/yellowtent/box/src/apptask.js:727:5)
Mar 30 20:44:20 box:settings initCache: pre-load settings
Mar 30 20:44:20 box:taskworker Starting task 4232. Logs are at /home/yellowtent/platformdata/logs/06c754d8-f01a-4fa2-b5c9-099e1916d0e8/apptask.log
Mar 30 20:44:20 box:apptask run: startTask installationState: pending_uninstall runState: running
May 31 19:00:00 box:tasks update 4232: {"percent":20,"message":"Deleting container"}
Mar 30 20:44:20 box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
Mar 30 20:44:20 box:apptask deleteContainer: deleting app containers (app, scheduler)
Mar 30 20:44:20 box:shell removeLogrotateConfig spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/configurelogrotate.sh remove 06c754d8-f01a-4fa2-b5c9-099e1916d0e8
Mar 30 20:44:20 box:tasks update 4232: {"percent":30,"message":"Teardown addons"}
Mar 30 20:44:20 box:services teardownAddons: Tearing down ["localstorage","postgresql","sendmail","redis"]
Mar 30 20:44:20 box:services teardownAddons: Tearing down addon localstorage with options {}
Mar 30 20:44:20 box:services teardownLocalStorage
Mar 30 20:44:20 box:shell clearVolume spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/clearvolume.sh rmdir /home/yellowtent/appsdata/06c754d8-f01a-4fa2-b5c9-099e1916d0e8/data
Mar 30 20:44:20 box:shell clearVolume (stdout): rmdir: failed to remove '/home/yellowtent/appsdata/06c754d8-f01a-4fa2-b5c9-099e1916d0e8/data': No such file or directory
Mar 30 20:44:20 box:services teardownAddons: Tearing down addon postgresql with options {}
Mar 30 20:44:25 box:apptask run: app error for state pending_uninstall: BoxError: Error tearing down postgresql. Status code: 500 message: database "db06c754d8f01a4fa2b5c9099e1916d0e8" is being accessed by other users
at Object.teardownPostgreSql [as teardown] (/home/yellowtent/box/src/services.js:1437:40)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Object.teardownAddons (/home/yellowtent/box/src/services.js:623:9)
at async uninstall (/home/yellowtent/box/src/apptask.js:727:5) {
reason: 'Addons Error',
details: {}
Mar 30 20:44:25 box:taskworker Task took 5.869 seconds
Mar 30 20:44:25 box:tasks setCompleted - 4232: {"result":null,"error":{"stack":"BoxError: Error tearing down postgresql. Status code: 500 message: database \"db06c754d8f01a4fa2b5c9099e1916d0e8\" is being accessed by other users\n at Object.teardownPostgreSql [as teardown] (/home/yellowtent/box/src/services.js:1437:40)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at async Object.teardownAddons (/home/yellowtent/box/src/services.js:623:9)\n at async uninstall (/home/yellowtent/box/src/apptask.js:727:5)","name":"BoxError","reason":"Addons Error","details":{},"message":"Error tearing down postgresql. Status code: 500 message: database \"db06c754d8f01a4fa2b5c9099e1916d0e8\" is being accessed by other users"}}
Mar 30 20:44:25 box:tasks update 4232: {"percent":100,"result":null,"error":{"stack":"BoxError: Error tearing down postgresql. Status code: 500 message: database \"db06c754d8f01a4fa2b5c9099e1916d0e8\" is being accessed by other users\n at Object.teardownPostgreSql [as teardown] (/home/yellowtent/box/src/services.js:1437:40)\n at processTicksAndRejections (node:internal/process/task_queues:96:5)\n at async Object.teardownAddons (/home/yellowtent/box/src/services.js:623:9)\n at async uninstall (/home/yellowtent/box/src/apptask.js:727:5)","name":"BoxError","reason":"Addons Error","details":{},"message":"Error tearing down postgresql. Status code: 500 message: database \"db06c754d8f01a4fa2b5c9099e1916d0e8\" is being accessed by other users"}}
Error tearing down postgresql. Status code: 500 message: database "db06c754d8f01a4fa2b5c9099e1916d0e8" is being accessed by other users
at Object.teardownPostgreSql [as teardown] (/home/yellowtent/box/src/services.js:1437:40)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async Object.teardownAddons (/home/yellowtent/box/src/services.js:623:9)
at async uninstall (/home/yellowtent/box/src/apptask.js:727:5)
ECONNREFUSED 172.18.19.45:3000=> Healtheck error: Error: connect ECONNREFUSED 172.18.19.45:3000=> Healtheck error: Error: connect ECONNREFUSED 172.18.19.45:3000=> !!!REPEATS 100X!!!! ECONNREFUSED 172.18.19.45:30002023-03-30T22:19:43.000Z [GET] /healthcheck
Mar 30 20:30:46 [GET] /healthcheck

Any ideas how to get rid of it?

[roofboard]

@roofboard in the tradition and answering my own questions... "Have you tried rebooting the computer?" Sometimes that fixes things.

Rebooting the computer worked, thanks 10 year old me.

[girish]

@roofboard said in Chat woot cannot uninstall after breaking the backup system.:

Mar 30 20:19:30 box:services Backing up postgresql
Mar 30 20:29:30 box:services pipeRequestToFile: timeout - connect or post-connect idle timeout

It seems somehow postgresql was not having a happy time. Maybe give it a bit more memory (Services -> PostgreSQL -> increase memory limit). The reboot probably just freed up some memory.