Email search not working properly

[avatar1024]

Hello,

I've noticed that email search has been weird over the past few days (might have been longer, not sure when it started), where the search in roundcube and snappymail returned result from email only from the last week but not for older emails.

I tried to deactivated and reactive full text search in Cloudron but since then I get no search result at all ...

I read the Cloudron doc but couldn't see anything obvious.

Running Cloudron 8.3.1.

[BrutalBirdie]

Maybe try this:

@girish said in Searching mail error "Server Error: UID SEARCH: Internal error" when Full Text Search (Solr) indexing is enabled, resolves when disabled.:

If you exec into the mail container, you can run doveadm index -u mail@domain.com '*' . That does the indexing . Maybe the solr index got corrupt and it's throwing some errors?

You can also see the logs in /run/solr/solr.log

To clear the index:

Exec into the mail container (docker exec -ti mail /bin/bash)
supervisorctl stop dovecot
supervisorctl stop solr
rm -rf /app/data/solr/dovecot
supervisorctl start solr
gosu cloudron:cloudron /app/code/solr/bin/solr create -c dovecot
supervisorctl start dovecot
doveadm index -u mail@domain.com '*'

[avatar1024]

Thank you, I followed those steps but I'm getting a lot of: Error: fts_solr: Indexing failed: 404 Not Found

I've looked at that thread in more details as it seems like @d19dotca was facing similar issues when following those steps, yet the link that @girish provided doesn't exist anymore so I couldn't really see how to solve it. I tried to add the -c argument to the index command but that on its own didn't work and I was still getting those errors.

I have disabled full text search for now and at least basic searches are working normally until this is resolved.

[avatar1024]

I've tried again the above steps but now I used doveadm -c /run/dovecot.conf index -u mail@domain.com '*' as the index command and I'm not getting any errors...but also not getting any search results (all searches in roundcube return Search found no matches). I tried on a very small mailbox where indexing shouldn't take anytime at all. Don't know if that's what's supposed to happen but that index command return to the prompt instantaneously, which seems weird.

Also for info, I gave the mail service 6GB ram so shouldn't be a resource issue.

[joseph]

@avatar1024 try https://docs.cloudron.io/email/#solr-index-corruption . The index is probably corrupt .

[avatar1024]

@joseph said in Email search not working properly:

@avatar1024 try https://docs.cloudron.io/email/#solr-index-corruption . The index is probably corrupt .

Thanks! I'll properly test tonight in downtime hours....but how the f**k have I missed that in the doc

Sorry for the bother if the solution is that simple and thanks for everyone's help.

[joseph]

@avatar1024 sadly, the docs search is not the best It really needs some a natural language component to it (we are building one, just very low priority)

[avatar1024]

Ok so I followed the steps indicated in the doc and there was a quirk. On a mailbox I got some errors:
Error: Mailbox INBOX/somefolder: Precache for UID=47 failed: Internal error occurred. Refer to server log for more information. [2025-03-30 07:54:41] doveadm(mail@domain.coop): Error: fts_tika: PUT http://127.0.0.1:9998/tika/ failed: connect(127.0.0.1:9998) failed: Connection refused

And then this stopped the indexing process altogether. So I had to manually run the command in the rebuil-index.sh script separately for all my mailboxes, avoiding the one that caused troubles, and on any mailbox where rebuilding the index worked, then search works again

Here is the detailed log related to the error above

2025-03-30 07:54:41.788 INFO  (searcherExecutor-16-thread-18-processing-dovecot localhost-1846) [c: s: r: x:dovecot t:localhost-1846] o.a.s.c.QuerySenderListener QuerySenderListener done.
2025-03-30 07:54:41.789 INFO  (searcherExecutor-16-thread-18-processing-dovecot localhost-1846) [c: s: r: x:dovecot t:localhost-1846] o.a.s.c.SolrCore Registered new searcher autowarm time: 3 ms
2025-03-30 07:54:41.802 INFO  (qtp2067925017-24-localhost-1846) [c: s: r: x:dovecot t:localhost-1846] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={}{commit=} 0 129
2025-03-30 07:54:41.818 INFO  (qtp2067925017-25-localhost-1847) [c: s: r: x:dovecot t:localhost-1847] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={}{add=[1/7824dd26093d43618202000011c04ec4/mail@domain.coop (1828004856394153984)]} 0 9
2025-03-30 07:54:41.831 INFO  (searcherExecutor-16-thread-18-processing-dovecot localhost-1848) [c: s: r: x:dovecot t:localhost-1848] o.a.s.c.QuerySenderListener QuerySenderListener done.
2025-03-30 07:54:41.832 INFO  (searcherExecutor-16-thread-18-processing-dovecot localhost-1848) [c: s: r: x:dovecot t:localhost-1848] o.a.s.c.SolrCore Registered new searcher autowarm time: 4 ms
2025-03-30 07:54:41.833 INFO  (qtp2067925017-24-localhost-1848) [c: s: r: x:dovecot t:localhost-1848] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={}{commit=} 0 13
2025-03-30 07:54:41.922 INFO  (qtp2067925017-25-localhost-1849) [c: s: r: x:dovecot t:localhost-1849] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={}{add=[1/288a8522203d43618202000011c04ec4/mail@domain.coop (1828004856499011584)]} 0 13
2025-03-30 07:54:41.934 INFO  (searcherExecutor-16-thread-18-processing-dovecot localhost-1850) [c: s: r: x:dovecot t:localhost-1850] o.a.s.c.QuerySenderListener QuerySenderListener done.
2025-03-30 07:54:41.936 INFO  (searcherExecutor-16-thread-18-processing-dovecot localhost-1850) [c: s: r: x:dovecot t:localhost-1850] o.a.s.c.SolrCore Registered new searcher autowarm time: 3 ms
2025-03-30 07:54:41.936 INFO  (qtp2067925017-24-localhost-1850) [c: s: r: x:dovecot t:localhost-1850] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={}{commit=} 0 12

These seem to relate to some kind of time out issues. I also got some of these errors on another mailbox which I have set-up on a desktop email client, and by opening the problematic folders with the mail client and then rerunning the rebuilding of the index then it worked.

So wonder if something could be improve to make building/rebuilding the index more robust?

[joseph]

The tika error seems to be because tika was down . I think a UI button to rebuild index will help a lot.

[avatar1024]

@joseph said in Email search not working properly:

The tika error seems to be because tika was down .

What's tika?

Btw I did all mailboxes manually, I had to retry a couple but they all ended up indexing without errors in the end.

@joseph said in Email search not working properly:

I think a UI button to rebuild index will help a lot.

Yes to that! Though whatever the solution chosen it's gotta be robust against the process failing / encountering errors. So either a button per mailbox, or if global, then it needs to make sure it carries on even if if there are errors on one mailbox.

[joseph]

@avatar1024 said in Email search not working properly:

What's tika?

tika is this - https://tika.apache.org/ . it converts the pdf/docs attachments in emails to text to make them searchable