Searching mail error "Server Error: UID SEARCH: Internal error" when Full Text Search (Solr) indexing is enabled, resolves when disabled.

[girish]

I have seen this happen when the indexing is busy the first time around. Do you see this happen with small mailbxoes as well? If so, you might just have to wait till the large mailboxes finish indexing before searching.

[d19dotca]

@girish I hadn’t tried with a small mailbox to be fair, it was a larger one who noted it to me and my own account is fairly large too and I was able to easily reproduce it. Assumed at the time it was impacting everyone but that’s just a guess.

I had enabled the indexing though about 1-2 days before this issue was seen… is it expected to take that long to index everything? Is there a way to mitigate the indexing impact at all? Perhaps pausing when a user issues a search query?

[girish]

@d19dotca 1-2 days seems quite a lot. I know that when you search the very first time, dovecot starts putting stuff into solr. There is a progress shown in the IMAP connection but most mail clients don't show it to the user .

As noted in https://docs.cloudron.io/email/#full-text-search , it might take even 10mins for 1GB of mail. Unfortunately, there is no way around this AFAIK. Also, only search is blocked and not checking emails . But of course, this depends on how the mail client deals with connections (it has to open a separate connection to check mails).

[d19dotca]

@girish there’s about 70 GB of mail on my server at the moment, so at that rate I guess it should be done in approximately 700 minutes (less than 12 hours) but even if we gave it 24 hours… 1-2 days seems like way too long then. The problem is my client uses webmail (Roundcube) exclusively and it’s dealing with medical stuff so I can’t really have the search failing like that as they rely on search heavily.

My intention of enabling it was to speed up the search even faster than it is, but it ended up blowing up in my face, lol.

Is there perhaps anything else I can try or is recommended for me to do or monitor if I were to try enabling it again? Anything in the logs I should be searching for? I didn’t find anything relevant in the logs even though the webmail client says to check the logs when it throws that error, so it was hard to know if it just wasn’t logged or if it was perhaps logged somewhere else.

[d19dotca]

@girish, now I’m starting to wonder if the thing was just indexing then that particular mailbox. The mailbox they have is around 15 GB, so 150 minutes approximately would be “down” for searching after they initiate the first search. If that is what was happening… I wonder… is there a way from the command line I can trigger it to index all mailboxes that haven’t been indexed already? My thinking is I could have it running overnight and then when they walk in to start the day it works as expected for them.

[girish]

@d19dotca let me test and report back the exact behavior . would be nice to have this documented as well...

[girish]

@d19dotca If you exec into the mail container, you can run doveadm index -u mail@domain.com '*' . That does the indexing . Maybe the solr index got corrupt and it's throwing some errors?

You can also see the logs in /run/solr/solr.log

To clear the index:

• Exec into the mail container (docker exec -ti mail /bin/bash)
• supervisorctl stop dovecot
• supervisorctl stop solr
• rm -rf /app/data/solr/dovecot
• supervisorctl start solr
• gosu cloudron:cloudron /app/code/solr/bin/solr create -c dovecot
• supervisorctl start dovecot
• doveadm index -u mail@domain.com '*'

[d19dotca]

Thank you so much @girish, I'll try that.

[d19dotca]

I just enabled Solr and then immediately went into the mail container and see the following errors (which seems to jive closely with what webmail returned earlier):

root@3cbcbbb04246:/app/code/haraka# doveadm index -u <redacted> '*'
doveadm(<redacted>): Error: fts_solr: Lookup failed: 500 Server Error
doveadm(<redacted>): Error: Mailbox Trash: Status lookup failed: Internal error occurred. Refer to server log for more information. [2024-12-19 07:11:18]
doveadm(<redacted>): Error: fts_solr: Lookup failed: 500 Server Error
doveadm(<redacted>): Error: Mailbox Junk: Status lookup failed: Internal error occurred. Refer to server log for more information. [2024-12-19 07:11:18]
doveadm(<redacted>): Error: fts_solr: Lookup failed: 500 Server Error
doveadm(<redacted>): Error: Mailbox Archive: Status lookup failed: Internal error occurred. Refer to server log for more information. [2024-12-19 07:11:18]
doveadm(<redacted>): Error: fts_solr: Lookup failed: 500 Server Error
doveadm(<redacted>): Error: Mailbox Deleted Messages: Status lookup failed: Internal error occurred. Refer to server log for more information. [2024-12-19 07:11:18]
doveadm(<redacted>): Error: fts_solr: Lookup failed: 500 Server Error
doveadm(<redacted>): Error: Mailbox Sent Messages: Status lookup failed: Internal error occurred. Refer to server log for more information. [2024-12-19 07:11:18]
doveadm(<redacted>): Error: fts_solr: Lookup failed: 500 Server Error
doveadm(<redacted>): Error: Mailbox Receipts: Status lookup failed: Internal error occurred. Refer to server log for more information. [2024-12-19 07:11:18]
doveadm(<redacted>): Error: fts_solr: Lookup failed: 500 Server Error
doveadm(<redacted>): Error: Mailbox Spam: Status lookup failed: Internal error occurred. Refer to server log for more information. [2024-12-19 07:11:18]
doveadm(<redacted>): Error: fts_solr: Lookup failed: 500 Server Error
doveadm(<redacted>): Error: Mailbox Drafts: Status lookup failed: Internal error occurred. Refer to server log for more information. [2024-12-19 07:11:18]
doveadm(<redacted>): Error: fts_solr: Lookup failed: 500 Server Error
doveadm(<redacted>): Error: Mailbox Sent: Status lookup failed: Internal error occurred. Refer to server log for more information. [2024-12-19 07:11:18]
doveadm(<redacted>): Error: fts_solr: Lookup failed: 500 Server Error
doveadm(<redacted>): Error: Mailbox INBOX: Status lookup failed: Internal error occurred. Refer to server log for more information. [2024-12-19 07:11:18]

[d19dotca]

I disabled Solr for now to avoid any issues with my clients. I'd love to make this work though, I'm not sure how to find out what's going wrong with it. The logs don't seem to show anything that I can find. Is there perhaps Solr logs on their own that I'm missing?

[d19dotca]

I also seem to get the same errors by the way if I use a command I found to immediately index all users by using: doveadm index -A '*' from the mail container, but it outputs the errors for each mailbox user and folder of course.

Any suggestions on next steps?

[d19dotca]

One last update, I re-enabled Solr and then followed your steps above for clearing the index, and then re-ran the doveadm index -A '*' and now I get a 400 Bad Request lookup failed message instead of the 500 server error earlier. I've disabled it again now.

[girish]

@d19dotca apologies, my instructions were not complete. I put the docs at https://docs.cloudron.io/email/#search sometime back and forgot to update the thread here. doveadm needs the config argument. Can you please try with those instructions?

[d19dotca]

That's great, thanks @girish! It seems to work now. I enabled it and followed the steps in the new documentation you linked to, and then ran doveadm -c /run/dovecot.conf index -A '*' from the mail container which started indexing everything. Just waiting for the indexing to finish to confirm it'll be good.