Email search not working properly

[BrutalBirdie]

Maybe try this:

@girish said in Searching mail error "Server Error: UID SEARCH: Internal error" when Full Text Search (Solr) indexing is enabled, resolves when disabled.:

If you exec into the mail container, you can run doveadm index -u mail@domain.com '*' . That does the indexing . Maybe the solr index got corrupt and it's throwing some errors?

You can also see the logs in /run/solr/solr.log

To clear the index:

Exec into the mail container (docker exec -ti mail /bin/bash)
supervisorctl stop dovecot
supervisorctl stop solr
rm -rf /app/data/solr/dovecot
supervisorctl start solr
gosu cloudron:cloudron /app/code/solr/bin/solr create -c dovecot
supervisorctl start dovecot
doveadm index -u mail@domain.com '*'

[avatar1024]

Thank you, I followed those steps but I'm getting a lot of: Error: fts_solr: Indexing failed: 404 Not Found

I've looked at that thread in more details as it seems like @d19dotca was facing similar issues when following those steps, yet the link that @girish provided doesn't exist anymore so I couldn't really see how to solve it. I tried to add the -c argument to the index command but that on its own didn't work and I was still getting those errors.

I have disabled full text search for now and at least basic searches are working normally until this is resolved.

[avatar1024]

I've tried again the above steps but now I used doveadm -c /run/dovecot.conf index -u mail@domain.com '*' as the index command and I'm not getting any errors...but also not getting any search results (all searches in roundcube return Search found no matches). I tried on a very small mailbox where indexing shouldn't take anytime at all. Don't know if that's what's supposed to happen but that index command return to the prompt instantaneously, which seems weird.

Also for info, I gave the mail service 6GB ram so shouldn't be a resource issue.

[joseph]

@avatar1024 try https://docs.cloudron.io/email/#solr-index-corruption . The index is probably corrupt .

[avatar1024]

@joseph said in Email search not working properly:

@avatar1024 try https://docs.cloudron.io/email/#solr-index-corruption . The index is probably corrupt .

Thanks! I'll properly test tonight in downtime hours....but how the f**k have I missed that in the doc

Sorry for the bother if the solution is that simple and thanks for everyone's help.

[joseph]

@avatar1024 sadly, the docs search is not the best It really needs some a natural language component to it (we are building one, just very low priority)

[avatar1024]

Ok so I followed the steps indicated in the doc and there was a quirk. On a mailbox I got some errors:
Error: Mailbox INBOX/somefolder: Precache for UID=47 failed: Internal error occurred. Refer to server log for more information. [2025-03-30 07:54:41] doveadm(mail@domain.coop): Error: fts_tika: PUT http://127.0.0.1:9998/tika/ failed: connect(127.0.0.1:9998) failed: Connection refused

And then this stopped the indexing process altogether. So I had to manually run the command in the rebuil-index.sh script separately for all my mailboxes, avoiding the one that caused troubles, and on any mailbox where rebuilding the index worked, then search works again

Here is the detailed log related to the error above

2025-03-30 07:54:41.788 INFO  (searcherExecutor-16-thread-18-processing-dovecot localhost-1846) [c: s: r: x:dovecot t:localhost-1846] o.a.s.c.QuerySenderListener QuerySenderListener done.
2025-03-30 07:54:41.789 INFO  (searcherExecutor-16-thread-18-processing-dovecot localhost-1846) [c: s: r: x:dovecot t:localhost-1846] o.a.s.c.SolrCore Registered new searcher autowarm time: 3 ms
2025-03-30 07:54:41.802 INFO  (qtp2067925017-24-localhost-1846) [c: s: r: x:dovecot t:localhost-1846] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={}{commit=} 0 129
2025-03-30 07:54:41.818 INFO  (qtp2067925017-25-localhost-1847) [c: s: r: x:dovecot t:localhost-1847] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={}{add=[1/7824dd26093d43618202000011c04ec4/mail@domain.coop (1828004856394153984)]} 0 9
2025-03-30 07:54:41.831 INFO  (searcherExecutor-16-thread-18-processing-dovecot localhost-1848) [c: s: r: x:dovecot t:localhost-1848] o.a.s.c.QuerySenderListener QuerySenderListener done.
2025-03-30 07:54:41.832 INFO  (searcherExecutor-16-thread-18-processing-dovecot localhost-1848) [c: s: r: x:dovecot t:localhost-1848] o.a.s.c.SolrCore Registered new searcher autowarm time: 4 ms
2025-03-30 07:54:41.833 INFO  (qtp2067925017-24-localhost-1848) [c: s: r: x:dovecot t:localhost-1848] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={}{commit=} 0 13
2025-03-30 07:54:41.922 INFO  (qtp2067925017-25-localhost-1849) [c: s: r: x:dovecot t:localhost-1849] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={}{add=[1/288a8522203d43618202000011c04ec4/mail@domain.coop (1828004856499011584)]} 0 13
2025-03-30 07:54:41.934 INFO  (searcherExecutor-16-thread-18-processing-dovecot localhost-1850) [c: s: r: x:dovecot t:localhost-1850] o.a.s.c.QuerySenderListener QuerySenderListener done.
2025-03-30 07:54:41.936 INFO  (searcherExecutor-16-thread-18-processing-dovecot localhost-1850) [c: s: r: x:dovecot t:localhost-1850] o.a.s.c.SolrCore Registered new searcher autowarm time: 3 ms
2025-03-30 07:54:41.936 INFO  (qtp2067925017-24-localhost-1850) [c: s: r: x:dovecot t:localhost-1850] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={}{commit=} 0 12

These seem to relate to some kind of time out issues. I also got some of these errors on another mailbox which I have set-up on a desktop email client, and by opening the problematic folders with the mail client and then rerunning the rebuilding of the index then it worked.

So wonder if something could be improve to make building/rebuilding the index more robust?

[joseph]

The tika error seems to be because tika was down . I think a UI button to rebuild index will help a lot.

[avatar1024]

@joseph said in Email search not working properly:

The tika error seems to be because tika was down .

What's tika?

Btw I did all mailboxes manually, I had to retry a couple but they all ended up indexing without errors in the end.

@joseph said in Email search not working properly:

I think a UI button to rebuild index will help a lot.

Yes to that! Though whatever the solution chosen it's gotta be robust against the process failing / encountering errors. So either a button per mailbox, or if global, then it needs to make sure it carries on even if if there are errors on one mailbox.

[joseph]

@avatar1024 said in Email search not working properly:

What's tika?

tika is this - https://tika.apache.org/ . it converts the pdf/docs attachments in emails to text to make them searchable

[hakunamatata]

I am having issues with email search as well. I tried using the rebuild-index.sh script in the docs but am running into the following error with a particular mailbox:

local-admin@cloudron:/$ sudo docker exec -it mail /app/code/dovecot-config/rebuild-index.sh
==> Indexing mailboxes for user: mail1@domain.com
==> Successfully indexed mailboxes for user: mail1@domain.com
==> Indexing mailboxes for user: mail2@domain.com
==> Successfully indexed mailboxes for user: mail2@domain.com
==> Indexing mailboxes for user: mail3@domain.com
==> Successfully indexed mailboxes for user: mail3@domain.com
==> Indexing mailboxes for user: mail4@domain.com
doveadm(mail4@domain.com): Error: fts_solr: Indexing failed: 500 Server Error
doveadm(mail4@domain.com): Error: Mailbox INBOX: Transaction commit failed: FTS transaction commit failed: backend deinit

How do I troubleshoot what is causing mail4@domain.com to fail the reindexing? Tried bumping up the RAM allocation for the mail container from 4GB to 8GB to no avail.

[james]

Hello @hakunamatata

Do you have Full text search enabled?

[hakunamatata]

@James Yes it is.

[james]

Hello @hakunamatata

Please run the following command to get the solr.log:

docker exec mail cat /run/solr/solr.log

Do you see any error in there?

One last resort could be to fully clean the index and build it again.
See:
https://forum.cloudron.io/post/98372

[hakunamatata]

Hi James, thanks for the prompt response. There seems to be some Java timeout error, see below. Should I go ahead with the instructions in https://forum.cloudron.io/post/98372 ?

2026-02-09 10:56:08.132 ERROR (qtp858232531-25-localhost-141) [c: s: r: x:dovecot t:localhost-141] o.a.s.h.RequestHandlerBase Server exception => [com.ctc.wstx.exc.WstxLazyException] com.ctc.wstx.exc.WstxIOException: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms
	at com.ctc.wstx.exc.WstxLazyException.throwLazily(WstxLazyException.java:40)
com.ctc.wstx.exc.WstxLazyException: [com.ctc.wstx.exc.WstxLazyException] java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms
	at com.ctc.wstx.exc.WstxLazyException.throwLazily(WstxLazyException.java:40) ~[?:?]
	at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:737) ~[?:?]
	at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3761) ~[?:?]
	at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:894) ~[?:?]
	at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:363) ~[?:?]
	at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:319) ~[?:?]
	at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:188) ~[?:?]
	at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:120) ~[?:?]
	at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:102) ~[?:?]
	at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:98) ~[?:?]
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:231) ~[?:?]
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2878) ~[?:?]
	at org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:905) ~[?:?]
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578) ~[?:?]
	at org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:241) ~[?:?]
	at org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilterRetry$0(SolrDispatchFilter.java:198) ~[?:?]
	at org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:227) ~[?:?]
	at org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:197) ~[?:?]
	at org.apache.solr.servlet.SolrDispatchFilter.doFilterRetry(SolrDispatchFilter.java:192) ~[?:?]
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181) ~[?:?]
	at javax.servlet.http.HttpFilter.doFilter(HttpFilter.java:97) ~[jetty-servlet-api-4.0.6.jar:?]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) ~[jetty-servlet-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) ~[jetty-servlet-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) ~[jetty-security-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) ~[jetty-servlet-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) ~[jetty-rewrite-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.Server.handle(Server.java:563) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) ~[jetty-io-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) ~[jetty-io-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) ~[jetty-util-10.0.26.jar:10.0.26]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: com.ctc.wstx.exc.WstxIOException: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms
	at com.ctc.wstx.sr.StreamScanner.constructFromIOE(StreamScanner.java:653) ~[?:?]
	at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:1017) ~[?:?]
	at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4727) ~[?:?]
	at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3805) ~[?:?]
	at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3759) ~[?:?]
	... 59 more
Caused by: java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms
	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:272) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.apache.solr.servlet.ServletInputStreamWrapper.read(ServletInputStreamWrapper.java:82) ~[?:?]
	at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:346) ~[?:?]
	at com.ctc.wstx.io.BaseReader.readBytes(BaseReader.java:155) ~[?:?]
	at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:369) ~[?:?]
	at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:116) ~[?:?]
	at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:88) ~[?:?]
	at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57) ~[?:?]
	at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:1011) ~[?:?]
	at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4727) ~[?:?]
	at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3805) ~[?:?]
	at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3759) ~[?:?]
	... 59 more
Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms
	at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:170) ~[jetty-io-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:112) ~[jetty-io-10.0.26.jar:10.0.26]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	... 1 more
2026-02-09 10:56:08.139 ERROR (qtp858232531-25-localhost-141) [c: s: r: x:dovecot t:localhost-141] o.a.s.s.HttpSolrCall 500 Exception => [com.ctc.wstx.exc.WstxLazyException] com.ctc.wstx.exc.WstxIOException: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms
	at com.ctc.wstx.exc.WstxLazyException.throwLazily(WstxLazyException.java:40)
com.ctc.wstx.exc.WstxLazyException: [com.ctc.wstx.exc.WstxLazyException] java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms
	at com.ctc.wstx.exc.WstxLazyException.throwLazily(WstxLazyException.java:40) ~[?:?]
	at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:737) ~[?:?]
	at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3761) ~[?:?]
	at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:894) ~[?:?]
	at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:363) ~[?:?]
	at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:319) ~[?:?]
	at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:188) ~[?:?]
	at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:120) ~[?:?]
	at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:102) ~[?:?]
	at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:98) ~[?:?]
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:231) ~[?:?]
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:2878) ~[?:?]
	at org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:905) ~[?:?]
	at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578) ~[?:?]
	at org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:241) ~[?:?]
	at org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilterRetry$0(SolrDispatchFilter.java:198) ~[?:?]
	at org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:227) ~[?:?]
	at org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:197) ~[?:?]
	at org.apache.solr.servlet.SolrDispatchFilter.doFilterRetry(SolrDispatchFilter.java:192) ~[?:?]
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181) ~[?:?]
	at javax.servlet.http.HttpFilter.doFilter(HttpFilter.java:97) ~[jetty-servlet-api-4.0.6.jar:?]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) ~[jetty-servlet-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) ~[jetty-servlet-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) ~[jetty-security-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) ~[jetty-servlet-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) ~[jetty-rewrite-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.Server.handle(Server.java:563) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) ~[jetty-io-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) ~[jetty-io-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) ~[jetty-util-10.0.26.jar:10.0.26]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: com.ctc.wstx.exc.WstxIOException: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms
	at com.ctc.wstx.sr.StreamScanner.constructFromIOE(StreamScanner.java:653) ~[?:?]
	at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:1017) ~[?:?]
	at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4727) ~[?:?]
	at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3805) ~[?:?]
	at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3759) ~[?:?]
	... 59 more
Caused by: java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms
	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:272) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.apache.solr.servlet.ServletInputStreamWrapper.read(ServletInputStreamWrapper.java:82) ~[?:?]
	at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:346) ~[?:?]
	at com.ctc.wstx.io.BaseReader.readBytes(BaseReader.java:155) ~[?:?]
	at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:369) ~[?:?]
	at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:116) ~[?:?]
	at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:88) ~[?:?]
	at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57) ~[?:?]
	at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:1011) ~[?:?]
	at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4727) ~[?:?]
	at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3805) ~[?:?]
	at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3759) ~[?:?]
	... 59 more
Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms
	at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:170) ~[jetty-io-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:112) ~[jetty-io-10.0.26.jar:10.0.26]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	... 1 more
2026-02-09 10:56:08.141 INFO  (qtp858232531-25) [c: s: r: x: t:localhost-141] o.a.s.s.ServletUtils Could not consume full client request => java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms
	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307)
java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms
	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:272) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:262) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.apache.solr.servlet.ServletInputStreamWrapper.read(ServletInputStreamWrapper.java:72) ~[?:?]
	at org.apache.solr.servlet.ServletUtils.consumeInputFully(ServletUtils.java:280) ~[?:?]
	at org.apache.solr.servlet.SolrDispatchFilter.doFilterRetry(SolrDispatchFilter.java:204) ~[?:?]
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181) ~[?:?]
	at javax.servlet.http.HttpFilter.doFilter(HttpFilter.java:97) ~[jetty-servlet-api-4.0.6.jar:?]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) ~[jetty-servlet-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) ~[jetty-servlet-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) ~[jetty-security-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) ~[jetty-servlet-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) ~[jetty-rewrite-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.Server.handle(Server.java:563) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287) ~[jetty-server-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) ~[jetty-io-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) ~[jetty-io-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) ~[jetty-util-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) ~[jetty-util-10.0.26.jar:10.0.26]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms
	at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:170) ~[jetty-io-10.0.26.jar:10.0.26]
	at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:112) ~[jetty-io-10.0.26.jar:10.0.26]
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?]
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	... 1 more

[james]

Hello @hakunamatata

Yes, but even better, we have an official documentation for that:
https://docs.cloudron.io/email/#solr-index-corruption
Please follow that one.

[hakunamatata]

Hi @James, that is what I originally ran when I noticed that the email search was not working and the error messages I have shared are as a result of running the rebuild-index.sh script. Please advise.

[hakunamatata]

I think the issue here is trying to figure out why mail4@domain.com is failing during execution of the rebuild-index.sh script. mail1,2,3 indexed successfully and the search is working through Roundcube.

I don't know if it is possible to modify the script to give more details about why the indexing for a particular mailbox failed and in case of such an event, to continue the indexing of the next mailbox in the list. The per mailbox reindexing UI option suggested earlier by @avatar1024 would be perfect for this.

[james]

Hello @hakunamatata

Is mail4 in any case special?
Some examples: very large, special setup like folder sharring, special named folder with unique characters?

The timeout already is 120000 ms / 2 Minutes, so I am unsure if increasing the timeout would solve anything.

[avatar1024]

As the OP I must say that I had to finally give up in enabling full text search as it kept breaking searches. It works for a while but then breaks again and people get no results for very simple searches. Rerunning the script manually to reindex every time is not really a solution.