Mail: failed Solr/Tika readiness check on boot silently disables all inbound mail, while healthcheck reports green

[LoudLemur]

---

Category: Support / Mail

---

@staff could you please take a look at this as mail is not working?

Cloudron 9.0.0, mail image cloudron/mail:4.3.5.

After a reboot of a reasonably busy server, inbound mail was dead for nearly three hours. The dashboard showed the mail service healthy for all of it. There are four separate issues tangled together here, and the last two are what turned a bad morning into a long one.

1. A slow Solr start takes Haraka and Spamd down with it

services.js starts things in this order:

async function start() {
    await dovecot.start(); // start this before solr in case the fts has to be re-indexed
    await fts.start();
    await haraka.start();
    await spamd.start();
    ...
}

On our boot, Solr exceeded the readiness timeout:

08:55:47  Error: Could not verify fts readiness: Timed out waiting for solr/tika readiness
            at Object.start (/app/code/fts.js:286:27)
            at async Object.start (/app/code/services.js:32:5)
08:57:26  solr core "dovecot" comes up normally (14,093 docs)

Solr was fine, just about a hundred seconds slower than the check allowed, because the whole box was contending after a reboot. But the throw propagated out of start(), so haraka.start() and spamd.start() never ran:

dovecot        RUNNING
haraka         STOPPED   Not started
spamd          STOPPED   Not started
mail-service   RUNNING
solr           RUNNING
tika           RUNNING

Full-text search is a search index. It seems wrong for it to be a hard prerequisite of SMTP. Could fts.start() be made non-fatal, logging and continuing with a background retry, or simply moved after haraka.start()? Either would have turned a three-hour mail outage into a temporarily degraded search box.

2. The healthcheck reports green while all mail is being rejected

Throughout the outage, and afterwards in the broken state described below:

{"status":true,"haraka":{"status":true},"dovecot":{"status":true},"spamd":{"status":true},
 "redis":{"status":true},"solr":{"status":true},"tika":{"status":true}}

This is because getHealth() is process liveness only:

const out = safe.child_process.execSync(`supervisorctl status ${program} | grep RUNNING`, ...);
health[program].status = out && out.includes('RUNNING');

A running Haraka that rejects every recipient is reported exactly like a working one. Would it be reasonable for the healthcheck to assert that /run/haraka/config/host_list is non-empty whenever at least one domain has inbound set? That single check would have caught both this and issue 3.

3. Starting Haraka by hand skips config generation, and makes things worse

Seeing haraka STOPPED, the obvious operator move is:

docker exec mail supervisorctl start haraka spamd

Both come up RUNNING, the healthcheck goes green, and SMTP answers on 25 and 587. It looks fixed. It is not. haraka.start() is not just supervisorctl start haraka:

async function start() {
    const [syncConfigError] = await safe(syncConfig());
    if (syncConfigError) throw new Error(...);
    safe.child_process.execSync('supervisorctl start haraka', ...);
}

syncConfig() is what writes /run/haraka/config/host_list and populates the DKIM runtime directory. Starting the daemon directly skips it, so Haraka runs with an empty host list and no DKIM keys. The result:

RCPT TO:<valid-mailbox@example.com>     550 I cannot deliver mail for <valid-mailbox@example.com>
RCPT TO:<nosuchuser@example.com>        550 No such address

Note the asymmetry, which is what misled us for a while: the invalid address gets the correct "No such address" from the cloudron plugin, proving the plugin is loaded and knows the domain. The valid address falls through to Haraka's core rejection in connection.js, because the cloudron plugin calls plain next() for a good mailbox and relies on rcpt_to.in_host_list to accept it, and host_list is empty.

This state is worse than the service simply being down. With Haraka stopped, port 25 refuses connections and sending servers queue and retry for days, so nothing is lost. With Haraka running against an empty host list, every sender gets a permanent 550 and gives up. Mail is destroyed rather than delayed.

Two suggestions:

• Have Haraka refuse to serve, or exit, when host_list is empty while inbound domains are configured. Failing closed at connect level is far safer than 550-ing real mail.
• Consider a comment or a guard around the supervisorctl entries, since supervisorctl start haraka is the natural thing for an operator to type and it is silently wrong.

4. POST /mail/<domain>/inbound returns 202 without applying anything

Having worked out that the runtime config needed regenerating, the natural way to trigger it looks like a no-op write of a domain's existing inbound setting. It answers a confident 202 and does nothing relevant:

async function setDomainInbound(domain, enabled, auditSource) {
    const result = await database.query('UPDATE mail SET inbound = ? WHERE domain = ?', );
    if (result.affectedRows === 0) throw new MailError(...);
    await boxEventlog.add(...);
}

There is no call to restartHarakaService(). It is defined at domains.js:11 but called only from lines 92, 128 and 143, and the route returns 202 unconditionally. host_list stayed empty and Haraka's uptime never reset. Would it be worth having domain setters that change delivery-relevant state trigger a reconfigure, or at least not report success when nothing was applied?

The recovery, for anyone who lands here

The underlying config is untouched in the database; only Haraka's runtime copy is missing. A no-op write of the existing max email size triggers haraka.reconfigure(), which regenerates host_list and the DKIM keys and restarts Haraka, without touching Dovecot and without re-running the fts check. It also returns 503 with a real error if the reconfigure fails, rather than an unconditional success:

GET  /max_email_size?access_token=$CLOUDRON_MAIL_TOKEN     -> {"size":25000000}
POST /max_email_size?access_token=$CLOUDRON_MAIL_TOKEN
Body: {"size": 25000000}                                    (write back the same value)

Verify all three of these, because the first two can look right while mail is still broken:

cat /run/haraka/config/host_list      # your inbound domains
ls  /run/haraka/config/dkim           # one directory per domain
supervisorctl status haraka           # uptime MUST reset to seconds

A Haraka uptime that has not reset means the reconfigure did not run, whatever the HTTP code said. Then test a real recipient over SMTP. A valid mailbox must return 250 where it previously returned 550 I cannot deliver mail for, while a genuinely unknown address should still return 550 No such address.

One last note for anyone probing SMTP by hand: Haraka applies a greeting delay to unknown clients, so a short-timeout probe reports failure against a perfectly healthy server. Allow at least 20 seconds before concluding the port is dead.

[girish]

@loudlemur I think some of the issues are valid, fixing.

[necrevistonnezr]

Sorry, but pushing a wall of unread and unrefined AI-output is just super-rude.

[girish]

Right, I don't read AI stuff myself But since this was a bug report I fed it into another AI and got something meaningful out of it.

I think the issue is that solr (fts) start up failure blocks all mail from starting. On top, there is a bug that gives incorrect notification that mail is running. Both these are fixed . The points 3,4 are not correct since the mail server API is not public and meant to be used via the box code . I guess the enthusiastic AI went straight into the mail container and started having some fun.

[marcusquinn]

We're pretty-much now in the age of: "I'll have my AI speak to your AI, and TLDR me the results."

Wild times.

[LoudLemur]

@marcusquinn said:

We're pretty-much now in the age of: "I'll have my AI speak to your AI, and TLDR me the results."

Wild times.

It is like Hollywood: "I'll have my people talk to your people and we can figure out a time to have lunch."

@necrevistonnezr has a valid point, too. What else can we do, though? "Understand it better?" This problem meant people were not receiving emails. Not sending a bug report, particularly when the ai had found a fix, didn't seem public spirited. It also is a bit like self-censorship.

On the other hand, Cloudron has a policy about no AI generated text in the forum. This is just one case where there is an AI bug report. @girish kindly found the time to investigate it. In the future, there may be many AI bug reports and human intervention might not be able to handle them all.

There are arguments for and against both sides. It does seem rude though to paste a load of AI and expect a human to look at it. I know what you mean.

[necrevistonnezr]

@LoudLemur said:

@necrevistonnezr has a valid point, too. What else can we do, though? "Understand it better?" This problem meant people were not receiving emails. Not sending a bug report, particularly when the ai had found a fix, didn't seem public spirited. It also is a bit like self-censorship.

It’s pretty simple: Read the output, work with it it, make it your own. Don‘t just forward it unfiltered. If that’s too much, leave it.

In case of a bug report, if you don’t want to put in the work, what’s the point? How does the team know your report was verified by you and has merits? It would probably more helpful to leave the team your result in 3-4 sentences and provide the prompt - then they can see for themselves.