Cloudron 10.0.4: nightly Bayes learning crashes mail-service (EPIPE) on messages > 500 KB

[nilp]

Since updating to Cloudron 10.0.4 (mail container cloudron/mail:4.3.5, Ubuntu 24.04), the nightly
Bayes training never completes. It stops after a few mailboxes, every night.

Mail logs at midnight:

learnFromUserMail: learning start
learnFromUserMail: learning from mailbox ...            (6 mailboxes)
learnFromUserMail: learning as ham: /app/data/vmail/<user>/mail/cur/...,S=21845327,...
node:events:487
      throw er; // Unhandled 'error' event
Error: write EPIPE
    at WriteWrap.onWriteComplete [as oncomplete] (node:internal/stream_base_commons:87:19)
Node.js v24.19.0
WARN exited: mail-service (exit status 1; not expected)
INFO reaped unknown pid 11009 (exit status 98)
INFO spawned: 'mail-service' with pid 11010

"learning done" is never logged, all mailboxes after that one are never trained, and no summary
event is written.

Cause, as far as I can tell:

• spamcLearn() in /app/code/spamd.js pipes the whole file to spamc through
  shell.execFileWithInput().
• spamc's default --max-size is 500 KB. For a bigger message it exits immediately with 98
  (EX_TOOBIG), before even connecting to spamd.
• The rest of the write to child.stdin then fails with EPIPE. execFileWithInput() in
  /app/code/shell.js has no 'error' listener on child.stdin, so this becomes an uncaught
  exception that kills mail-service.
• Candidates are the last 3 days of mail (LEARN_DAYS), so any message with an attachment

  500 KB is enough to trigger it, which in practice means every night.

Reproduction inside the mail container, against a closed port so nothing is actually learned:

spamc -p 1 -u test@invalid --learntype ham < small.eml ; echo $?   # 69 (tries to connect)
spamc -p 1 -u test@invalid --learntype ham < 2MB.eml   ; echo $?   # 98 (too big, exits at once)

A small Node script doing execFile('spamc', ) followed by child.stdin.end(<2 MB buffer>)
dies with the same "Unhandled 'error' event / write EPIPE".

Possible fixes, either would do:

• in execFileWithInput(), add child.stdin.on('error', () => {}) so the failure is reported by
  the execFile callback instead of crashing the process;
• in findMailFiles(), skip files above spamc's max size (e.g. find -size -500k), or treat
  exit code 98 as "skipped" rather than as a failure.

Thanks!

[timconsidine]

@nilp Cloudron 10.0.4 is not latest. Any idea if it’s the same on 10.1.5 ? Seems a lot of changes to Mail recently so that may fix it, but no certainty, just a thought.

[nilp]

Thanks for the suggestion! I've updated to 10.0.5. The dashboard now says there is no further update
available, so 10.1.5 isn't offered to me yet.

Unfortunately the issue is still there. The mail container is now cloudron/mail:4.3.9, but
execFileWithInput() in shell.js and spamcLearn() in spamd.js are unchanged: there is still no 'error'
listener on child.stdin, and no size filter on the candidate files.

I re-ran the reproduction inside the new container, this time using its own /app/code/shell.js: spamc
exits with 98 on a 2 MB message, and the process dies with the same "Unhandled 'error' event / write
EPIPE". So the nightly learning run should still stop at the first message larger than 500 KB.

[timconsidine]

@nilp That's a shame.
My update to 10.1.5 did not show as available for some time, but then appeared magically.

I think this is probably one for @james or another in the Cloudron team.

[nilp]

I also checked the 10.1.5 mail image (cloudron/mail:4.4.4) before updating:
shell.js and spamcLearn() are unchanged, so the issue is still there in 10.1.5.

[nilp]

I've now updated to 10.1.5, and unfortunately it doesn't fix it.

The running mail container is now cloudron/mail:4.4.4 (Node.js v24.21.0). I ran the same reproduction
inside it, using its own /app/code/shell.js, against a closed port so nothing was learned: spamc exits
with 98 on a 2 MB message, and the process dies with the same "Unhandled 'error' event / write EPIPE".

So the nightly learnFromUserMail() run on 10.1.5 will still crash mail-service at the first candidate
message larger than spamc's 500 KB default, and the mailboxes after it are never trained.

[imc67]

Confirming this on Cloudron 10.1.5 (mail 4.4.4, Dovecot 2.4.2). On our three servers the nightly learning run has crashed with the same write EPIPE / exited: mail-service on 23 of 27 nights since 11 October; two of the servers have not completed a single run in that period ("learning done" never logged). Every time it stops at the first candidate above 500 KB: last night at messages of 2.5 MB, 3.4 MB and 14.8 MB. Spam/ham training on IMAP moves (IMAPSieve) is not affected, only the nightly run.

[nilp]

@girish can you fix it ?

[girish]

Yup, on it... The issue is that we are feeding spamc files larger than max-size which ends up crashing the bayes learn.

[nilp]

Thanks @girish! Confirmed again last night on 10.1.5: the run stopped at a 5.6 MB message.

One small suggestion alongside the size filter: an 'error' listener on child.stdin in
execFileWithInput() would also stop any other early spamc exit from taking mail-service down.

Happy to test the fix when it's out.

[girish]

Fixed. @nilp you can test with https://git.cloudron.io/platform/box/-/commit/307b8f5a0d145e43cb2f805aec638163c8e4d9ad if you like. You have to edit /home/yellowtent/box/src/infra_version.js and update that one line . Then systemctl restart box .

[nilp]

Thanks @girish! I've applied the commit (mail 4.5.0) and restarted box: only the mail container
was recreated, everything else stayed up.

I replayed the exact message that crashed last night's run (5.6 MB). It is no longer picked by
findMailFiles(), and even when fed directly with the new spamcLearn() arguments, spamc exits with 98,
it's handled as "skipped" and the process stays up. Over the last 3 days, 242 messages above 500 KB
are now excluded.

The sieve bin_path and default_vsz_limit = 2G changes are working too. I'll confirm "learning done"
after tonight's run.