Uninstall of AdGuard Home stuck

[msbt]

Hi there! I justed wanted to remove an instance of Adguard Home and it's stuck at deleteImage:

Mar 12 13:41:33 box:taskworker Starting task 21661. Logs are at /home/yellowtent/platformdata/logs/{appid}/apptask.log
Mar 12 13:41:33 box:taskworker Running task of type app
Mar 12 13:41:33 box:apptask run: startTask installationState: pending_uninstall runState: running
Mar 12 13:41:33 box:tasks update 21661: {"percent":20,"message":"Deleting container"}
Mar 12 13:41:33 box:shell reverseproxy /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
Mar 12 13:41:35 box:apptask deleteContainer: deleting app containers (app, scheduler)
Mar 12 13:41:35 box:shell apptask /usr/bin/sudo -S /home/yellowtent/box/src/scripts/configurelogrotate.sh remove {appid}
Mar 12 13:41:35 2025/03/12 12:41:35.142253 [info] received signal signal=terminated
Mar 12 13:41:35 2025/03/12 12:41:35.142890 [info] stopping AdGuard Home
Mar 12 13:41:35 2025/03/12 12:41:35.143471 [info] webapi: stopping http server
Mar 12 13:41:35 2025/03/12 12:41:35.146373 [info] webapi: stopped http server
Mar 12 13:41:35 2025/03/12 12:41:35.148390 [info] dnsproxy: stopping server
Mar 12 13:41:35 2025/03/12 12:41:35.152002 [info] dnsproxy: stopped dns proxy server
Mar 12 13:41:35 2025/03/12 12:41:35.156505 [error] closing query log: flushing log buffer: nothing to write to a file
Mar 12 13:41:35 2025/03/12 12:41:35.177912 [info] stopped
Mar 12 13:41:36 box:tasks update 21661: {"percent":30,"message":"Teardown addons"}
Mar 12 13:41:36 box:services teardownAddons: Tearing down ["localstorage","tls"]
Mar 12 13:41:36 box:services teardownAddons: Tearing down addon localstorage with options {}
Mar 12 13:41:36 box:services teardownLocalStorage
Mar 12 13:41:36 box:shell services /usr/bin/sudo -S /home/yellowtent/box/src/scripts/clearvolume.sh rmdir /home/yellowtent/appsdata/{appid}/data
Mar 12 13:41:36 box:services teardownAddons: Tearing down addon tls with options {}
Mar 12 13:41:36 box:tasks update 21661: {"percent":40,"message":"Cleanup file manager"}
Mar 12 13:41:36 box:tasks update 21661: {"percent":50,"message":"Deleting app data directory"}
Mar 12 13:41:36 box:tasks update 21661: {"percent":60,"message":"Deleting image"}
Mar 12 13:41:36 box:docker deleteImage: removing cloudron/com.adguard.home.cloudronapp:202503102114000000

It's been like that for ~20 minutes, what should I do? I haven't canceled and retried yet.

[joseph]

It looks like something is off with the deamon (also the restart lines above with hasBeenManuallyStopped=true) . Have you tried restarting docker already? of course this causes some small downtime .

[joseph]

That looks like docker is stuck... Anything in journalctl -u docker -fa ?

[msbt]

Thanks for the quick reply @joseph!

Logs are full of those:

2025-03-12T12:41:16+0000 {servername} dockerd[1302]: time="2025-03-12T12:41:16.653471098Z" level=info msg="ignoring event" container=b17a50802e4b2d76246e4308b995b7dcef3262ee2d3fc20bb73525fc3d85800d module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
2025-03-12T12:41:32+0000 {servername} dockerd[1302]: time="2025-03-12T12:41:32.302981321Z" level=info msg="ignoring event" container=36a695ece77474b199fb7c3ed6519e97d3d0c9faaaf88472e6b7e90ba46df1b2 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
2025-03-12T12:41:35+0000 {servername} dockerd[1302]: time="2025-03-12T12:41:35.221464997Z" level=info msg="ignoring event" container=0bce6375cf3ebf98785f1023073b60366333a42707d256124b467fbaaac5b260 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
2025-03-12T12:41:35+0000 {servername} dockerd[1302]: time="2025-03-12T12:41:35.270212958Z" level=warning msg="ShouldRestart failed, container will not be restarted" container=0bce6375cf3ebf98785f1023073b60366333a42707d256124b467fbaaac5b260 daemonShuttingDown=false error="restart canceled" execDuration=9h41m23.72251829s exitStatus="{0 2025-03-12 12:41:35.200057946 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
2025-03-12T12:42:16+0000 {servername} dockerd[1302]: time="2025-03-12T12:42:16.667127768Z" level=info msg="ignoring event" container=b17a50802e4b2d76246e4308b995b7dcef3262ee2d3fc20bb73525fc3d85800d module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
2025-03-12T12:42:26+0000 {servername} dockerd[1302]: time="2025-03-12T12:42:26.431437134Z" level=info msg="ignoring event" container=0cca28fbb5c05dce821b5f016f4bc3ff451b1a9697e3e5b9b1eba58cde256464 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
:26.416601 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
2025-03-12T12:42:32+0000 {servername} dockerd[1302]: time="2025-03-12T12:42:32.427434862Z" level=info msg="ignoring event" container=36a695ece77474b199fb7c3ed6519e97d3d0c9faaaf88472e6b7e90ba46df1b2 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
2025-03-12T12:43:02+0000 {servername} dockerd[1302]: time="2025-03-12T12:43:02.099025754Z" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.18.30.4:53213" dns-server="udp:172.18.0.1:53" error="read udp 172.18.30.4:53213->172.18.0.1:53: i/o timeout" question=";146.162.128.27.in-addr.arpa.\tIN\t PTR" spanID=34e94d11649c75e5 traceID=04deac8d9a3f3f9ad125ad4d00ae50dd
2025-03-12T12:43:16+0000 {servername} dockerd[1302]: time="2025-03-12T12:43:16.195163179Z" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.18.30.4:36365" dns-server="udp:172.18.0.1:53" error="read udp 172.18.30.4:36365->172.18.0.1:53: i/o timeout" question=";155.35.37.68.in-addr.arpa.\tIN\t PTR" spanID=94a3cb6299f57204 traceID=0df6cdd89ed9fe9f10f49fcf084e03d7
2025-03-12T12:43:16+0000 {servername} dockerd[1302]: time="2025-03-12T12:43:16.662388469Z" level=info msg="ignoring event" container=b17a50802e4b2d76246e4308b995b7dcef3262ee2d3fc20bb73525fc3d85800d module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
2025-03-12T12:43:19+0000 {servername} dockerd[1302]: time="2025-03-12T12:43:19.196197058Z" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.18.30.4:35068" dns-server="udp:172.18.0.1:53" error="read udp 172.18.30.4:35068->172.18.0.1:53: i/o timeout" question=";155.35.37.68.in-addr.arpa.\tIN\t PTR" spanID=8602e0515b7381e3 traceID=31c05cf972d7f953b8d959cc0714cd4f
2025-03-12T12:43:20+0000 {servername} dockerd[1302]: time="2025-03-12T12:43:20.909740253Z" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.18.30.4:54879" dns-server="udp:172.18.0.1:53" error="read udp 172.18.30.4:54879->172.18.0.1:53: i/o timeout" question=";131.92.94.80.in-addr.arpa.\tIN\t PTR" spanID=37d7ae29cd3fc801 traceID=e4b6cfaa9fb8ce4d3bad3e6cf99a501c
2025-03-12T12:43:24+0000 {servername} dockerd[1302]: time="2025-03-12T12:43:24.234161659Z" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.18.30.4:45814" dns-server="udp:172.18.0.1:53" error="read udp 172.18.30.4:45814->172.18.0.1:53: i/o timeout" question=";158.238.215.85.in-addr.arpa.\tIN\t PTR" spanID=5b1bdaa1fc73cba1 traceID=fc1fdda81a51bd18fb0c513a2852f36c

I'm blocking most ports via Hetzner firewall and one other app (Nextcloud) is unresponsive because of their storagebox maintenance, other than that all seems fine on the server.

[joseph]

It looks like something is off with the deamon (also the restart lines above with hasBeenManuallyStopped=true) . Have you tried restarting docker already? of course this causes some small downtime .

[msbt]

I just tried to restart Nextcloud because in the same minute I got the email that the storagebox is back. This is what Nextcloud says:

Mar 12 14:22:55 box:tasks update 21663: {"percent":10,"message":"Starting app services"}
Mar 12 14:22:55 box:tasks update 21663: {"percent":20,"message":"Restarting container"}
Mar 12 14:22:57 => Healtheck error: Error: Timeout of 7000ms exceeded
Mar 12 14:23:19 box:apptask run: app error for state pending_restart: BoxError: (HTTP code 500) server error - Cannot restart container 0e650789-7a15-44cb-ad69-5a914b3dea26: tried to kill container, but did not receive an exit event at Object.restartContainer (/home/yellowtent/box/src/docker.js:504:50) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async restart (/home/yellowtent/box/src/apptask.js:745:9) { reason: 'Docker Error', details: {}, nestedError: Error: (HTTP code 500) server error - Cannot restart container 0e650789-7a15-44cb-ad69-5a914b3dea26: tried to kill container, but did not receive an exit event at /home/yellowtent/box/node_modules/docker-modem/lib/modem.js:382:17 at getCause (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:417:7) at Modem.buildPayload (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:378:5) at IncomingMessage.<anonymous> (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:346:16) at IncomingMessage.emit (node:events:531:35) at endReadableNT (node:internal/streams/readable:1696:12) at process.processTicksAndRejections (node:internal/process/task_queues:82:21) { reason: 'server error', statusCode: 500, json: { message: 'Cannot restart container 0e650789-7a15-44cb-ad69-5a914b3dea26: tried to kill container, but did not receive an exit event' } } }
Mar 12 14:23:19 box:taskworker Task took 24.202 seconds
Mar 12 14:23:19 box:tasks setCompleted - 21663: {"result":null,"error":{"stack":"BoxError: (HTTP code 500) server error - Cannot restart container 0e650789-7a15-44cb-ad69-5a914b3dea26: tried to kill container, but did not receive an exit event \n at Object.restartContainer (/home/yellowtent/box/src/docker.js:504:50)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async restart (/home/yellowtent/box/src/apptask.js:745:9)","name":"BoxError","reason":"Docker Error","details":{},"message":"(HTTP code 500) server error - Cannot restart container 0e650789-7a15-44cb-ad69-5a914b3dea26: tried to kill container, but did not receive an exit event ","nestedError":{"stack":"Error: (HTTP code 500) server error - Cannot restart container 0e650789-7a15-44cb-ad69-5a914b3dea26: tried to kill container, but did not receive an exit event \n at /home/yellowtent/box/node_modules/docker-modem/lib/modem.js:382:17\n at getCause (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:417:7)\n at Modem.buildPayload (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:378:5)\n at IncomingMessage.<anonymous> (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:346:16)\n at IncomingMessage.emit (node:events:531:35)\n at endReadableNT (node:internal/streams/readable:1696:12)\n at process.processTicksAndRejections (node:internal/process/task_queues:82:21)","name":"Error","reason":"server error","message":"(HTTP code 500) server error - Cannot restart container 0e650789-7a15-44cb-ad69-5a914b3dea26: tried to kill container, but did not receive an exit event "}}}
Mar 12 14:23:19 box:tasks update 21663: {"percent":100,"result":null,"error":{"stack":"BoxError: (HTTP code 500) server error - Cannot restart container 0e650789-7a15-44cb-ad69-5a914b3dea26: tried to kill container, but did not receive an exit event \n at Object.restartContainer (/home/yellowtent/box/src/docker.js:504:50)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async restart (/home/yellowtent/box/src/apptask.js:745:9)","name":"BoxError","reason":"Docker Error","details":{},"message":"(HTTP code 500) server error - Cannot restart container 0e650789-7a15-44cb-ad69-5a914b3dea26: tried to kill container, but did not receive an exit event ","nestedError":{"stack":"Error: (HTTP code 500) server error - Cannot restart container 0e650789-7a15-44cb-ad69-5a914b3dea26: tried to kill container, but did not receive an exit event \n at /home/yellowtent/box/node_modules/docker-modem/lib/modem.js:382:17\n at getCause (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:417:7)\n at Modem.buildPayload (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:378:5)\n at IncomingMessage.<anonymous> (/home/yellowtent/box/node_modules/docker-modem/lib/modem.js:346:16)\n at IncomingMessage.emit (node:events:531:35)\n at endReadableNT (node:internal/streams/readable:1696:12)\n at process.processTicksAndRejections (node:internal/process/task_queues:82:21)","name":"Error","reason":"server error","message":"(HTTP code 500) server error - Cannot restart container 0e650789-7a15-44cb-ad69-5a914b3dea26: tried to kill container, but did not receive an exit event "}}}
Mar 12 14:23:19 BoxError: (HTTP code 500) server error - Cannot restart container 0e650789-7a15-44cb-ad69-5a914b3dea26: tried to kill container, but did not receive an exit event
Mar 12 14:23:19 at Object.restartContainer (/home/yellowtent/box/src/docker.js:504:50)
Mar 12 14:23:19 at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
Mar 12 14:23:19 at async restart (/home/yellowtent/box/src/apptask.js:745:9)

Restarting docker now

[msbt]

Yeah that made it worse, running:

cloudron-support --troubleshoot
Vendor: Hetzner Product: vServer
Linux: 5.15.0-134-generic
Ubuntu: jammy 22.04
Processor: Intel Xeon Processor (Skylake, IBRS, no TSX) x 16
RAM: 32086440KB
Disk: /dev/sda1       140G
[OK]	node version is correct
[OK]	IPv6 is enabled in kernel. No public IPv6 address
	Docker is down. Trying to restart docker ...

And it's stuck there. Rebooting the whole thing now

[msbt]

Hah alright, reboot did the trick, sorry for the noise!