Unbound Service not working

[skeats]

I came in this morning and I had a message from my US Office stating that Scan to e-mail is not working, and I use Cloudron to act as the mail server. I tried rebooting and unbound service is still not loading. It is on version 8.0.6 on Ubuntu Server 24.04. An automatic update to 8.0.6 happened on the weekend and now this service is broken. Below are the logs I found:

Sep 30 12:29:29 cloudron systemd[1]: Starting unbound.service - Unbound DNS Resolver...
Sep 30 12:29:29 cloudron systemd[1]: unbound.service: Control process exited, code=exited, status=203/EXEC
Sep 30 12:29:29 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 30 12:29:29 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
Sep 30 12:29:29 cloudron systemd[1]: unbound.service: Scheduled restart job, restart counter is at 1.
Sep 30 12:29:29 cloudron systemd[1]: Starting unbound.service - Unbound DNS Resolver...
Sep 30 12:29:29 cloudron systemd[1]: unbound.service: Control process exited, code=exited, status=203/EXEC
Sep 30 12:29:29 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 30 12:29:29 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
Sep 30 12:29:29 cloudron systemd[1]: unbound.service: Scheduled restart job, restart counter is at 2.
Sep 30 12:29:29 cloudron systemd[1]: Starting unbound.service - Unbound DNS Resolver...
Sep 30 12:29:29 cloudron systemd[1]: unbound.service: Control process exited, code=exited, status=203/EXEC
Sep 30 12:29:29 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 30 12:29:29 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
Sep 30 12:29:29 cloudron systemd[1]: unbound.service: Scheduled restart job, restart counter is at 3.
Sep 30 12:29:29 cloudron systemd[1]: Starting unbound.service - Unbound DNS Resolver...
Sep 30 12:29:29 cloudron systemd[1]: unbound.service: Control process exited, code=exited, status=203/EXEC
Sep 30 12:29:29 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 30 12:29:29 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
Sep 30 12:29:30 cloudron systemd[1]: unbound.service: Scheduled restart job, restart counter is at 4.
Sep 30 12:29:30 cloudron systemd[1]: Starting unbound.service - Unbound DNS Resolver...
Sep 30 12:29:30 cloudron systemd[1]: unbound.service: Control process exited, code=exited, status=203/EXEC
Sep 30 12:29:30 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 30 12:29:30 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
Sep 30 12:29:30 cloudron systemd[1]: unbound.service: Scheduled restart job, restart counter is at 5.
Sep 30 12:29:30 cloudron systemd[1]: unbound.service: Start request repeated too quickly.
Sep 30 12:29:30 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 30 12:29:30 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.

What can I do to get this resolved.

[nebulon]

Does unbound itself report any errors? Check journalctl -u unbound

[skeats]

Sep 28 05:01:44 cloudron unbound[819]: [819:0] info: generate keytag query _ta-4f66. NULL IN
Sep 28 05:30:33 cloudron systemd[1]: Stopping unbound.service - Unbound DNS Resolver...
Sep 28 05:30:33 cloudron unbound[819]: [819:0] info: service stopped (unbound 1.19.2).
Sep 28 05:30:33 cloudron unbound[819]: [819:0] info: server stats for thread 0: 1353 queries, 288 answers from cache, 1065 recursions, 0 prefetch, 0 rejected by ip ra>
Sep 28 05:30:33 cloudron unbound[819]: [819:0] info: server stats for thread 0: requestlist max 15 avg 0.586854 exceeded 28 jostled 0
Sep 28 05:30:33 cloudron unbound[819]: [819:0] info: average recursion processing time 0.108782 sec
Sep 28 05:30:33 cloudron unbound[819]: [819:0] info: histogram of recursion processing times
Sep 28 05:30:33 cloudron unbound[819]: [819:0] info: [25%]=0.0432723 median[50%]=0.0667963 [75%]=0.117063
Sep 28 05:30:33 cloudron unbound[819]: [819:0] info: lower(secs) upper(secs) recursions
Sep 28 05:30:33 cloudron unbound[819]: [819:0] info: 0.008192 0.016384 45
Sep 28 05:30:33 cloudron unbound[819]: [819:0] info: 0.016384 0.032768 95
Sep 28 05:30:33 cloudron unbound[819]: [819:0] info: 0.032768 0.065536 372
Sep 28 05:30:33 cloudron unbound[819]: [819:0] info: 0.065536 0.131072 338
Sep 28 05:30:33 cloudron unbound[819]: [819:0] info: 0.131072 0.262144 112
Sep 28 05:30:33 cloudron unbound[819]: [819:0] info: 0.262144 0.524288 43
Sep 28 05:30:33 cloudron unbound[819]: [819:0] info: 0.524288 1.000000 28
Sep 28 05:30:33 cloudron unbound[819]: [819:0] info: 1.000000 2.000000 4
Sep 28 05:30:33 cloudron systemd[1]: unbound.service: Deactivated successfully.
Sep 28 05:30:33 cloudron systemd[1]: Stopped unbound.service - Unbound DNS Resolver.
Sep 28 05:30:33 cloudron systemd[1]: unbound.service: Consumed 2.639s CPU time, 15.4M memory peak, 0B memory swap peak.
Sep 28 05:30:33 cloudron systemd[1]: Starting unbound.service - Unbound DNS Resolver...
Sep 28 05:30:33 cloudron systemd[1]: unbound.service: Control process exited, code=exited, status=203/EXEC
Sep 28 05:30:33 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 28 05:30:33 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
Sep 28 05:30:34 cloudron systemd[1]: unbound.service: Scheduled restart job, restart counter is at 1.
Sep 28 05:30:34 cloudron systemd[1]: Starting unbound.service - Unbound DNS Resolver...
Sep 28 05:30:34 cloudron systemd[1]: unbound.service: Control process exited, code=exited, status=203/EXEC
Sep 28 05:30:34 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 28 05:30:34 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
Sep 28 05:30:34 cloudron systemd[1]: unbound.service: Scheduled restart job, restart counter is at 2.
Sep 28 05:30:34 cloudron systemd[1]: Starting unbound.service - Unbound DNS Resolver...
Sep 28 05:30:34 cloudron systemd[1]: unbound.service: Control process exited, code=exited, status=203/EXEC
Sep 28 05:30:34 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 28 05:30:34 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
Sep 28 05:30:34 cloudron systemd[1]: unbound.service: Scheduled restart job, restart counter is at 3.
Sep 28 05:30:34 cloudron systemd[1]: Starting unbound.service - Unbound DNS Resolver...
Sep 28 05:30:34 cloudron systemd[1]: unbound.service: Control process exited, code=exited, status=203/EXEC
Sep 28 05:30:34 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 28 05:30:34 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
Sep 28 05:30:34 cloudron systemd[1]: unbound.service: Scheduled restart job, restart counter is at 4.
Sep 28 05:30:34 cloudron systemd[1]: Starting unbound.service - Unbound DNS Resolver...
Sep 28 05:30:34 cloudron systemd[1]: unbound.service: Control process exited, code=exited, status=203/EXEC
Sep 28 05:30:34 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 28 05:30:34 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
Sep 28 05:30:35 cloudron systemd[1]: unbound.service: Scheduled restart job, restart counter is at 5.
Sep 28 05:30:35 cloudron systemd[1]: unbound.service: Start request repeated too quickly.
Sep 28 05:30:35 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 28 05:30:35 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
-- Boot 41862792df8b41d4b1f0d020a113a679 --
Sep 30 14:06:58 cloudron systemd[1]: Starting unbound.service - Unbound DNS Resolver...
Sep 30 14:06:58 cloudron systemd[1]: unbound.service: Control process exited, code=exited, status=203/EXEC
Sep 30 14:06:58 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 30 14:06:58 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
Sep 30 14:06:58 cloudron systemd[1]: unbound.service: Scheduled restart job, restart counter is at 1.
Sep 30 14:06:58 cloudron systemd[1]: Starting unbound.service - Unbound DNS Resolver...
Sep 30 14:06:58 cloudron systemd[1]: unbound.service: Control process exited, code=exited, status=203/EXEC
Sep 30 14:06:58 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 30 14:06:58 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
Sep 30 14:06:58 cloudron systemd[1]: unbound.service: Scheduled restart job, restart counter is at 2.
Sep 30 14:06:59 cloudron systemd[1]: Starting unbound.service - Unbound DNS Resolver...
Sep 30 14:06:59 cloudron systemd[1]: unbound.service: Control process exited, code=exited, status=203/EXEC
Sep 30 14:06:59 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 30 14:06:59 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
Sep 30 14:06:59 cloudron systemd[1]: unbound.service: Scheduled restart job, restart counter is at 3.
Sep 30 14:06:59 cloudron systemd[1]: Starting unbound.service - Unbound DNS Resolver...
Sep 30 14:06:59 cloudron systemd[1]: unbound.service: Control process exited, code=exited, status=203/EXEC
Sep 30 14:06:59 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 30 14:06:59 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
Sep 30 14:06:59 cloudron systemd[1]: unbound.service: Scheduled restart job, restart counter is at 4.
Sep 30 14:06:59 cloudron systemd[1]: Starting unbound.service - Unbound DNS Resolver...
Sep 30 14:06:59 cloudron systemd[1]: unbound.service: Control process exited, code=exited, status=203/EXEC
Sep 30 14:06:59 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 30 14:06:59 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
Sep 30 14:06:59 cloudron systemd[1]: unbound.service: Scheduled restart job, restart counter is at 5.
Sep 30 14:06:59 cloudron systemd[1]: unbound.service: Start request repeated too quickly.
Sep 30 14:06:59 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 30 14:06:59 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
-- Boot c896dce9e2024762ab95f16909c5b548 --
Sep 30 14:27:08 cloudron systemd[1]: Starting unbound.service - Unbound DNS Resolver...
Sep 30 14:27:08 cloudron systemd[1]: unbound.service: Control process exited, code=exited, status=203/EXEC
Sep 30 14:27:08 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 30 14:27:08 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
Sep 30 14:27:08 cloudron systemd[1]: unbound.service: Scheduled restart job, restart counter is at 1.
Sep 30 14:27:08 cloudron systemd[1]: Starting unbound.service - Unbound DNS Resolver...
Sep 30 14:27:08 cloudron systemd[1]: unbound.service: Control process exited, code=exited, status=203/EXEC
Sep 30 14:27:08 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 30 14:27:08 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
Sep 30 14:27:08 cloudron systemd[1]: unbound.service: Scheduled restart job, restart counter is at 2.
Sep 30 14:27:08 cloudron systemd[1]: Starting unbound.service - Unbound DNS Resolver...
Sep 30 14:27:08 cloudron systemd[1]: unbound.service: Control process exited, code=exited, status=203/EXEC
Sep 30 14:27:08 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
Sep 30 14:27:08 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
Sep 30 14:27:08 cloudron systemd[1]: unbound.service: Scheduled restart job, restart counter is at 3.
Sep 30 14:27:08 cloudron systemd[1]: Starting unbound.service - Unbound DNS Resolver...
Sep 30 14:27:08 cloudron systemd[1]: unbound.service: Control process exited, code=exited, status=203/EXEC
lines 1355-1404

[nebulon]

That is indeed not a lot of information here. Upon search on the web apparently exit code 203 often means that the binary/executable does not exist. Is unbound still properly installed?

[skeats]

Yes, Unbound is properly installed

[nebulon]

If you manually run unbound -d is it any more chatty about the issue?

[skeats]

It is just hanging and nothing is coming up, but I can Control C to stop it

[skeats]

I ran this and got the following:

cloudron-support --troubleshoot
Vendor: QEMU Product: Standard PC (i440FX + PIIX, 1996)
Linux: 6.8.0-45-generic
Ubuntu: noble 24.04
Processor: AMD Ryzen 7 3800X 8-Core Processor
BIOS pc-i440fx-9.0 CPU @ 2.0GHz x 8
RAM: 32866264KB
Disk: /dev/sda2 701G
[OK] node version is correct
[OK] IPv6 is enabled
[OK] docker is running
[OK] MySQL is running
[OK] nginx is running
[OK] dashboard cert is valid
[OK] dashboard is reachable via loopback
[OK] box v8.0.6 is running
[OK] netplan is good
[OK] DNS is resolving via systemd-resolved
[FAIL] Could not load dashboard domain.

[skeats]

I ran this and got the following:

sudo journalctl -xeu unbound.service
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit unbound.service has begun execution.
░░
░░ The job identifier is 2602.
Sep 30 19:09:37 cloudron systemd[1]: unbound.service: Control process exited, code=exited, status=203/EXEC
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ An ExecStartPre= process belonging to unit unbound.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 203.
Sep 30 19:09:37 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit unbound.service has entered the 'failed' state with result 'exit-code'.
Sep 30 19:09:37 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
░░ Subject: A start job for unit unbound.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit unbound.service has finished with a failure.
░░
░░ The job identifier is 2602 and the job result is failed.
Sep 30 19:09:38 cloudron systemd[1]: unbound.service: Scheduled restart job, restart counter is at 5.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ Automatic restarting of the unit unbound.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Sep 30 19:09:38 cloudron systemd[1]: unbound.service: Start request repeated too quickly.
Sep 30 19:09:38 cloudron systemd[1]: unbound.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit unbound.service has entered the 'failed' state with result 'exit-code'.
Sep 30 19:09:38 cloudron systemd[1]: Failed to start unbound.service - Unbound DNS Resolver.
░░ Subject: A start job for unit unbound.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit unbound.service has finished with a failure.
░░
░░ The job identifier is 2791 and the job result is failed.

[girish]

@skeats I think this is because unbound-anchor package is not installed. Can you apt install unbound-anchor ? After that, systemctl restart unbound should work.

This mistake happened because we missed he installation of that package in the upgrade guide. It's been since added at https://docs.cloudron.io/guides/upgrade-ubuntu-24/#post-upgrade

[skeats]

Hi @girish,

I did that step and did a server reboot as the systemctl restart unbound did not work and when it came back up, all was good again! Thank you!