Mail app - FTS indexing fails on mailboxes with large attachments (HTTP timeout)

[paradoxbound]

Hello Cloudron,

Thank you for fixing the Solr OOM issue, sadly that was just hiding another issue with users with large mailboxes and large attachments. Some users like to send enormous CAD and spreadsheets to each other and documents the size of war and peace.

Issue:
Full-text search indexing fails on mailboxes containing emails with large attachments due to a hardcoded 60-second HTTP timeout in Dovecot's FTS Solr communication.

Evidence:

• Mailbox A: 120,545 emails → 100% indexed successfully (small emails, batches complete in 1-3 seconds)

• Mailbox B: 35,212 emails → Only 48% indexed (emails with large attachments cause slow batches)

Root Cause:
When Tika processes large email attachments during indexing, some batches of 200 emails take longer than 60 seconds to complete. Dovecot's HTTP client times out and aborts the transaction, even though Solr successfully completes the processing.
Example from logs:

doveadm(user): Error: fts_solr: Indexing failed: Request timed out (Request queued 61.852 secs ago, 60.144 in http ioloop)
doveadm(user): Error: Mailbox [name]: Transaction commit failed: FTS transaction commit failed

Solr logs show a batch completed in 74 seconds, but Dovecot gave up at 60 seconds.
Requested Fix:
Make the HTTP timeout dynamic based on mailbox size (similar to how SOLR_HEAP was made dynamic in 9.1.6).

Suggested implementation:

• Environment variable: DOVECOT_FTS_TIMEOUT (default: 60, recommended: 180-300 for large mailboxes)
• Or alternatively: DOVECOT_FTS_BATCH_SIZE to reduce batch size from 200 to 50 which avoids calculating mailbox sizes.

Workaround:
None available - /app/code/dovecot-config/dovecot-fts.conf is read-only and resets on container updates.
Related:
This issue was discovered after the SOLR_HEAP fix in version 9.1.6 resolved OOM crashes. The timeout issue was previously masked by the memory errors.

cloudron-support --troubleshoot
Vendor: QEMU Product: Standard PC (i440FX + PIIX, 1996)
Linux: 5.15.0-176-generic
Ubuntu: jammy 22.04
Execution environment: kvm
Processor: AMD EPYC Processor (with IBPB) x 10
RAM: 61714212KB
Disk: /dev/sda3       1.2T
[OK]	node version is correct
[OK]	IPv6 is enabled and public IPv6 address is working
[OK]	docker is running
[OK]	docker version is correct
[OK]	MySQL is running
[OK]	netplan is good
[OK]	DNS is resolving via systemd-resolved
[OK]	unbound is running
[OK]	nginx is running
[OK]	dashboard cert is valid
[OK]	dashboard is reachable via loopback
[OK]	No pending database migrations
[OK]	Service 'mysql' is running and healthy
[OK]	Service 'postgresql' is running and healthy
[OK]	Service 'mongodb' is running and healthy
[OK]	Service 'mail' is running and healthy
[OK]	Service 'graphite' is running and healthy
[OK]	Service 'sftp' is running and healthy
[OK]	box v9.1.6 is running
[OK]	Dashboard is reachable via domain name
[OK]	Domain xxxxxx is valid and has not expired