Email search not working properly
-
Ok so I followed the steps indicated in the doc and there was a quirk. On a mailbox I got some errors:
Error: Mailbox INBOX/somefolder: Precache for UID=47 failed: Internal error occurred. Refer to server log for more information. [2025-03-30 07:54:41] doveadm(mail@domain.coop): Error: fts_tika: PUT http://127.0.0.1:9998/tika/ failed: connect(127.0.0.1:9998) failed: Connection refusedAnd then this stopped the indexing process altogether. So I had to manually run the command in the rebuil-index.sh script separately for all my mailboxes, avoiding the one that caused troubles, and on any mailbox where rebuilding the index worked, then search works again
Here is the detailed log related to the error above
2025-03-30 07:54:41.788 INFO (searcherExecutor-16-thread-18-processing-dovecot localhost-1846) [c: s: r: x:dovecot t:localhost-1846] o.a.s.c.QuerySenderListener QuerySenderListener done. 2025-03-30 07:54:41.789 INFO (searcherExecutor-16-thread-18-processing-dovecot localhost-1846) [c: s: r: x:dovecot t:localhost-1846] o.a.s.c.SolrCore Registered new searcher autowarm time: 3 ms 2025-03-30 07:54:41.802 INFO (qtp2067925017-24-localhost-1846) [c: s: r: x:dovecot t:localhost-1846] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={}{commit=} 0 129 2025-03-30 07:54:41.818 INFO (qtp2067925017-25-localhost-1847) [c: s: r: x:dovecot t:localhost-1847] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={}{add=[1/7824dd26093d43618202000011c04ec4/mail@domain.coop (1828004856394153984)]} 0 9 2025-03-30 07:54:41.831 INFO (searcherExecutor-16-thread-18-processing-dovecot localhost-1848) [c: s: r: x:dovecot t:localhost-1848] o.a.s.c.QuerySenderListener QuerySenderListener done. 2025-03-30 07:54:41.832 INFO (searcherExecutor-16-thread-18-processing-dovecot localhost-1848) [c: s: r: x:dovecot t:localhost-1848] o.a.s.c.SolrCore Registered new searcher autowarm time: 4 ms 2025-03-30 07:54:41.833 INFO (qtp2067925017-24-localhost-1848) [c: s: r: x:dovecot t:localhost-1848] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={}{commit=} 0 13 2025-03-30 07:54:41.922 INFO (qtp2067925017-25-localhost-1849) [c: s: r: x:dovecot t:localhost-1849] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={}{add=[1/288a8522203d43618202000011c04ec4/mail@domain.coop (1828004856499011584)]} 0 13 2025-03-30 07:54:41.934 INFO (searcherExecutor-16-thread-18-processing-dovecot localhost-1850) [c: s: r: x:dovecot t:localhost-1850] o.a.s.c.QuerySenderListener QuerySenderListener done. 2025-03-30 07:54:41.936 INFO (searcherExecutor-16-thread-18-processing-dovecot localhost-1850) [c: s: r: x:dovecot t:localhost-1850] o.a.s.c.SolrCore Registered new searcher autowarm time: 3 ms 2025-03-30 07:54:41.936 INFO (qtp2067925017-24-localhost-1850) [c: s: r: x:dovecot t:localhost-1850] o.a.s.u.p.LogUpdateProcessorFactory webapp=/solr path=/update params={}{commit=} 0 12These seem to relate to some kind of time out issues. I also got some of these errors on another mailbox which I have set-up on a desktop email client, and by opening the problematic folders with the mail client and then rerunning the rebuilding of the index then it worked.
So wonder if something could be improve to make building/rebuilding the index more robust?
-
The tika error seems to be because tika was down . I think a UI button to rebuild index will help a lot.
@joseph said in Email search not working properly:
The tika error seems to be because tika was down .
What's tika?
Btw I did all mailboxes manually, I had to retry a couple but they all ended up indexing without errors in the end.
@joseph said in Email search not working properly:
I think a UI button to rebuild index will help a lot.
Yes to that! Though whatever the solution chosen it's gotta be robust against the process failing / encountering errors. So either a button per mailbox, or if global, then it needs to make sure it carries on even if if there are errors on one mailbox.
-
@joseph said in Email search not working properly:
The tika error seems to be because tika was down .
What's tika?
Btw I did all mailboxes manually, I had to retry a couple but they all ended up indexing without errors in the end.
@joseph said in Email search not working properly:
I think a UI button to rebuild index will help a lot.
Yes to that! Though whatever the solution chosen it's gotta be robust against the process failing / encountering errors. So either a button per mailbox, or if global, then it needs to make sure it carries on even if if there are errors on one mailbox.
@avatar1024 said in Email search not working properly:
What's tika?
tika is this - https://tika.apache.org/ . it converts the pdf/docs attachments in emails to text to make them searchable
-
A avatar1024 referenced this topic on
-
I am having issues with email search as well. I tried using the rebuild-index.sh script in the docs but am running into the following error with a particular mailbox:
local-admin@cloudron:/$ sudo docker exec -it mail /app/code/dovecot-config/rebuild-index.sh ==> Indexing mailboxes for user: mail1@domain.com ==> Successfully indexed mailboxes for user: mail1@domain.com ==> Indexing mailboxes for user: mail2@domain.com ==> Successfully indexed mailboxes for user: mail2@domain.com ==> Indexing mailboxes for user: mail3@domain.com ==> Successfully indexed mailboxes for user: mail3@domain.com ==> Indexing mailboxes for user: mail4@domain.com doveadm(mail4@domain.com): Error: fts_solr: Indexing failed: 500 Server Error doveadm(mail4@domain.com): Error: Mailbox INBOX: Transaction commit failed: FTS transaction commit failed: backend deinitHow do I troubleshoot what is causing mail4@domain.com to fail the reindexing? Tried bumping up the RAM allocation for the mail container from 4GB to 8GB to no avail.
-
Hello @hakunamatata
Do you have Full text search enabled?
-
Hello @hakunamatata
Do you have Full text search enabled?
@James Yes it is.
-
Hello @hakunamatata
Please run the following command to get the
solr.log:docker exec mail cat /run/solr/solr.logDo you see any error in there?
One last resort could be to fully clean the index and build it again.
See:
https://forum.cloudron.io/post/98372 -
Hi James, thanks for the prompt response. There seems to be some Java timeout error, see below. Should I go ahead with the instructions in https://forum.cloudron.io/post/98372 ?
2026-02-09 10:56:08.132 ERROR (qtp858232531-25-localhost-141) [c: s: r: x:dovecot t:localhost-141] o.a.s.h.RequestHandlerBase Server exception => [com.ctc.wstx.exc.WstxLazyException] com.ctc.wstx.exc.WstxIOException: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms at com.ctc.wstx.exc.WstxLazyException.throwLazily(WstxLazyException.java:40) com.ctc.wstx.exc.WstxLazyException: [com.ctc.wstx.exc.WstxLazyException] java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms at com.ctc.wstx.exc.WstxLazyException.throwLazily(WstxLazyException.java:40) ~[?:?] at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:737) ~[?:?] at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3761) ~[?:?] at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:894) ~[?:?] at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:363) ~[?:?] at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:319) ~[?:?] at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:188) ~[?:?] at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:120) ~[?:?] at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:102) ~[?:?] at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:98) ~[?:?] at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:231) ~[?:?] at org.apache.solr.core.SolrCore.execute(SolrCore.java:2878) ~[?:?] at org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:905) ~[?:?] at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:241) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilterRetry$0(SolrDispatchFilter.java:198) ~[?:?] at org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:227) ~[?:?] at org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:197) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.doFilterRetry(SolrDispatchFilter.java:192) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181) ~[?:?] at javax.servlet.http.HttpFilter.doFilter(HttpFilter.java:97) ~[jetty-servlet-api-4.0.6.jar:?] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) ~[jetty-servlet-10.0.26.jar:10.0.26] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.26.jar:10.0.26] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) ~[jetty-servlet-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) ~[jetty-security-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) ~[jetty-servlet-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) ~[jetty-rewrite-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.Server.handle(Server.java:563) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) ~[jetty-io-10.0.26.jar:10.0.26] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) ~[jetty-io-10.0.26.jar:10.0.26] at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) ~[jetty-util-10.0.26.jar:10.0.26] at java.base/java.lang.Thread.run(Thread.java:840) [?:?] Caused by: com.ctc.wstx.exc.WstxIOException: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms at com.ctc.wstx.sr.StreamScanner.constructFromIOE(StreamScanner.java:653) ~[?:?] at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:1017) ~[?:?] at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4727) ~[?:?] at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3805) ~[?:?] at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3759) ~[?:?] ... 59 more Caused by: java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:272) ~[jetty-server-10.0.26.jar:10.0.26] at org.apache.solr.servlet.ServletInputStreamWrapper.read(ServletInputStreamWrapper.java:82) ~[?:?] at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:346) ~[?:?] at com.ctc.wstx.io.BaseReader.readBytes(BaseReader.java:155) ~[?:?] at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:369) ~[?:?] at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:116) ~[?:?] at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:88) ~[?:?] at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57) ~[?:?] at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:1011) ~[?:?] at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4727) ~[?:?] at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3805) ~[?:?] at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3759) ~[?:?] ... 59 more Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:170) ~[jetty-io-10.0.26.jar:10.0.26] at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:112) ~[jetty-io-10.0.26.jar:10.0.26] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] ... 1 more 2026-02-09 10:56:08.139 ERROR (qtp858232531-25-localhost-141) [c: s: r: x:dovecot t:localhost-141] o.a.s.s.HttpSolrCall 500 Exception => [com.ctc.wstx.exc.WstxLazyException] com.ctc.wstx.exc.WstxIOException: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms at com.ctc.wstx.exc.WstxLazyException.throwLazily(WstxLazyException.java:40) com.ctc.wstx.exc.WstxLazyException: [com.ctc.wstx.exc.WstxLazyException] java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms at com.ctc.wstx.exc.WstxLazyException.throwLazily(WstxLazyException.java:40) ~[?:?] at com.ctc.wstx.sr.StreamScanner.throwLazyError(StreamScanner.java:737) ~[?:?] at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3761) ~[?:?] at com.ctc.wstx.sr.BasicStreamReader.getText(BasicStreamReader.java:894) ~[?:?] at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:363) ~[?:?] at org.apache.solr.handler.loader.XMLLoader.readDoc(XMLLoader.java:319) ~[?:?] at org.apache.solr.handler.loader.XMLLoader.processUpdate(XMLLoader.java:188) ~[?:?] at org.apache.solr.handler.loader.XMLLoader.load(XMLLoader.java:120) ~[?:?] at org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:102) ~[?:?] at org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:98) ~[?:?] at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:231) ~[?:?] at org.apache.solr.core.SolrCore.execute(SolrCore.java:2878) ~[?:?] at org.apache.solr.servlet.HttpSolrCall.executeCoreRequest(HttpSolrCall.java:905) ~[?:?] at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:578) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:241) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilterRetry$0(SolrDispatchFilter.java:198) ~[?:?] at org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:227) ~[?:?] at org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:197) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.doFilterRetry(SolrDispatchFilter.java:192) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181) ~[?:?] at javax.servlet.http.HttpFilter.doFilter(HttpFilter.java:97) ~[jetty-servlet-api-4.0.6.jar:?] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) ~[jetty-servlet-10.0.26.jar:10.0.26] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.26.jar:10.0.26] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) ~[jetty-servlet-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) ~[jetty-security-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) ~[jetty-servlet-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) ~[jetty-rewrite-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.Server.handle(Server.java:563) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) ~[jetty-io-10.0.26.jar:10.0.26] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) ~[jetty-io-10.0.26.jar:10.0.26] at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) ~[jetty-util-10.0.26.jar:10.0.26] at java.base/java.lang.Thread.run(Thread.java:840) [?:?] Caused by: com.ctc.wstx.exc.WstxIOException: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms at com.ctc.wstx.sr.StreamScanner.constructFromIOE(StreamScanner.java:653) ~[?:?] at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:1017) ~[?:?] at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4727) ~[?:?] at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3805) ~[?:?] at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3759) ~[?:?] ... 59 more Caused by: java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:272) ~[jetty-server-10.0.26.jar:10.0.26] at org.apache.solr.servlet.ServletInputStreamWrapper.read(ServletInputStreamWrapper.java:82) ~[?:?] at org.apache.commons.io.input.ProxyInputStream.read(ProxyInputStream.java:346) ~[?:?] at com.ctc.wstx.io.BaseReader.readBytes(BaseReader.java:155) ~[?:?] at com.ctc.wstx.io.UTF8Reader.loadMore(UTF8Reader.java:369) ~[?:?] at com.ctc.wstx.io.UTF8Reader.read(UTF8Reader.java:116) ~[?:?] at com.ctc.wstx.io.ReaderSource.readInto(ReaderSource.java:88) ~[?:?] at com.ctc.wstx.io.BranchingReaderSource.readInto(BranchingReaderSource.java:57) ~[?:?] at com.ctc.wstx.sr.StreamScanner.loadMore(StreamScanner.java:1011) ~[?:?] at com.ctc.wstx.sr.BasicStreamReader.readTextSecondary(BasicStreamReader.java:4727) ~[?:?] at com.ctc.wstx.sr.BasicStreamReader.finishToken(BasicStreamReader.java:3805) ~[?:?] at com.ctc.wstx.sr.BasicStreamReader.safeFinishToken(BasicStreamReader.java:3759) ~[?:?] ... 59 more Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:170) ~[jetty-io-10.0.26.jar:10.0.26] at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:112) ~[jetty-io-10.0.26.jar:10.0.26] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] ... 1 more 2026-02-09 10:56:08.141 INFO (qtp858232531-25) [c: s: r: x: t:localhost-141] o.a.s.s.ServletUtils Could not consume full client request => java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307) java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:307) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:272) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.HttpInput.read(HttpInput.java:262) ~[jetty-server-10.0.26.jar:10.0.26] at org.apache.solr.servlet.ServletInputStreamWrapper.read(ServletInputStreamWrapper.java:72) ~[?:?] at org.apache.solr.servlet.ServletUtils.consumeInputFully(ServletUtils.java:280) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.doFilterRetry(SolrDispatchFilter.java:204) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181) ~[?:?] at javax.servlet.http.HttpFilter.doFilter(HttpFilter.java:97) ~[jetty-servlet-api-4.0.6.jar:?] at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210) ~[jetty-servlet-10.0.26.jar:10.0.26] at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635) ~[jetty-servlet-10.0.26.jar:10.0.26] at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527) ~[jetty-servlet-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598) ~[jetty-security-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1580) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484) ~[jetty-servlet-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301) ~[jetty-rewrite-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.Server.handle(Server.java:563) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287) ~[jetty-server-10.0.26.jar:10.0.26] at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314) ~[jetty-io-10.0.26.jar:10.0.26] at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100) ~[jetty-io-10.0.26.jar:10.0.26] at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53) ~[jetty-io-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194) ~[jetty-util-10.0.26.jar:10.0.26] at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149) ~[jetty-util-10.0.26.jar:10.0.26] at java.base/java.lang.Thread.run(Thread.java:840) [?:?] Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 120001/120000 ms at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:170) ~[jetty-io-10.0.26.jar:10.0.26] at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:112) ~[jetty-io-10.0.26.jar:10.0.26] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539) ~[?:?] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] ... 1 more -
Hello @hakunamatata
Yes, but even better, we have an official documentation for that:
https://docs.cloudron.io/email/#solr-index-corruption
Please follow that one. -
Hi @James, that is what I originally ran when I noticed that the email search was not working and the error messages I have shared are as a result of running the rebuild-index.sh script. Please advise.
-
I think the issue here is trying to figure out why mail4@domain.com is failing during execution of the rebuild-index.sh script. mail1,2,3 indexed successfully and the search is working through Roundcube.
I don't know if it is possible to modify the script to give more details about why the indexing for a particular mailbox failed and in case of such an event, to continue the indexing of the next mailbox in the list. The per mailbox reindexing UI option suggested earlier by @avatar1024 would be perfect for this.
-
Hello @hakunamatata
Is
mail4in any case special?
Some examples: very large, special setup like folder sharring, special named folder with unique characters?The timeout already is 120000 ms / 2 Minutes, so I am unsure if increasing the timeout would solve anything.
-
As the OP I must say that I had to finally give up in enabling full text search as it kept breaking searches. It works for a while but then breaks again and people get no results for very simple searches. Rerunning the script manually to reindex every time is not really a solution.
-
@James nothing special about the mailbox other than it is the largest one at 1.12 GB. The total size of all mailboxes on the server is 2.72 GB and with 8GB RAM assigned to mail container, I don't think lack of memory is what is causing it to fail. This particular mailbox is the only one with an alias associated with it. I removed the alias, re-ran the rebuild-index.sh script but ended up with the same error. So that's not it either.
doveadm(mail4@domain.com): Error: fts_solr: Indexing failed: 500 Server Error doveadm(mail4@domain.com): Error: Mailbox INBOX: Transaction commit failed: FTS transaction commit failed: backend deinitI think I will follow OP's suit and disable the full text search until it is more stable on Cloudron. If you would like to have access to my server for some more troubleshooting, just let me know .
-
@James nothing special about the mailbox other than it is the largest one at 1.12 GB. The total size of all mailboxes on the server is 2.72 GB and with 8GB RAM assigned to mail container, I don't think lack of memory is what is causing it to fail. This particular mailbox is the only one with an alias associated with it. I removed the alias, re-ran the rebuild-index.sh script but ended up with the same error. So that's not it either.
doveadm(mail4@domain.com): Error: fts_solr: Indexing failed: 500 Server Error doveadm(mail4@domain.com): Error: Mailbox INBOX: Transaction commit failed: FTS transaction commit failed: backend deinitI think I will follow OP's suit and disable the full text search until it is more stable on Cloudron. If you would like to have access to my server for some more troubleshooting, just let me know .
Hello @hakunamatata
Yes, please write a mail to support@cloudron.io and reference this issue.
-
@James , I think I found the issue. After googling the time out error message, I came across some threads on other forums which seemed to indicate that it could be related to large email messages that are not being indexed within the 120000 ms / 2 Minutes timeout interval. So I went to mail4's INBOX , deleted the largest email (17.7MB) and then re-ran the script. This time it failed during indexing mail4's TRASH folder. Once I deleted the email from the TRASH folder, voila, indexing was successful.
So perhaps the timeout needs to be increased or be a user defined value.
In my case, I am running my Cloudron instance as a VM on a Synology DS1621+ which features an AMD Ryzen V1500B processor (4 cores, 8 threads, 2.2 Ghz base clock). It isn't a very fast processor so that could be a contributing factor.
-
@James , I think I found the issue. After googling the time out error message, I came across some threads on other forums which seemed to indicate that it could be related to large email messages that are not being indexed within the 120000 ms / 2 Minutes timeout interval. So I went to mail4's INBOX , deleted the largest email (17.7MB) and then re-ran the script. This time it failed during indexing mail4's TRASH folder. Once I deleted the email from the TRASH folder, voila, indexing was successful.
So perhaps the timeout needs to be increased or be a user defined value.
In my case, I am running my Cloudron instance as a VM on a Synology DS1621+ which features an AMD Ryzen V1500B processor (4 cores, 8 threads, 2.2 Ghz base clock). It isn't a very fast processor so that could be a contributing factor.
