Email Event Log loading very slowly, seems tied to overall Email domain list health checks

[nebulon]

The email eventlog routes and view is independent of the health checks. Also the amount of domains as such should not matter. Essentially it is a a simple select query against a sqlite database in the mail container.
The db is at /home/yellowtent/boxdata/mail/events/log.db maybe you can check how large that one is in your instances?

[girish]

@d19dotca said in Email Event Log loading very slowly, seems tied to overall Email domain list health checks:

I think this coincides with the load time of the general Email domain lists which do the health-checks every time that page is loaded. Basically that takes a while,

Are you referring to the indicators taking time or the "Loading...." thing ? The latter can be a bit slow, it has to ask dovecot for the mailbox size for each domain. Dovecot in turn has to calculate this by checking the size of each mailbox. So, it probably all adds up since it's based not just on domain count but also mailbox count!

[imc67]

I just found out: when you visit the Email setting page and wait until all domains are fetched with stats, THEN the next click on Event Logs is very quick.
It seems like the “fetching” in the Email page is still continuing while you wait for the results in the Event Logs.

[d19dotca]

@imc67 said in Email Event Log loading very slowly, seems tied to overall Email domain list health checks:

I just found out: when you visit the Email setting page and wait until all domains are fetched with stats, THEN the next click on Event Logs is very quick.
It seems like the “fetching” in the Email page is still continuing while you wait for the results in the Event Logs.

Exactly what I’m seeing too.

@nebulon said in Email Event Log loading very slowly, seems tied to overall Email domain list health checks:

The email eventlog routes and view is independent of the health checks. Also the amount of domains as such should not matter. Essentially it is a a simple select query against a sqlite database in the mail container.
The db is at /home/yellowtent/boxdata/mail/events/log.db maybe you can check how large that one is in your instances?

@nebulon I know you said it’s unrelated to the metrics collected and displayed on the main email page but it sure feels related based on timing of when the event log actually ends up loading data. If it wasn’t related then I’d expect to see very different timings, but I’m not. Perhaps just a coincidence but very much seems to be related.

[girish]

This seems related to the redis issue . Think it gets fixed with https://git.cloudron.io/cloudron/box/-/commit/e64182d79134e8828c2fa953c676a8f6b08247b7

[d19dotca]

Sounds like this bug fix may be needed for a few different issues then. Any ETA on a pre-release we can try with? Between this fix and that OVH DNS API one, it'd be super helpful to get these sooner than later.

[d19dotca]

It seems the issue is still present on 7.6.2 @girish. Seems maybe it wasn't caused by the redis issue you fixed.

[girish]

@d19dotca I might have misunderstood the original issue. You are referring to /#/emails-eventlog correct ? Do you see the same 30 second delay if you click the refresh button (the one in the page and not the browser button) ?

[d19dotca]

@girish I'm sorry for the late reply. Yes, I'm referring to the loading of the Event Log from the Email page, loading this URL: https://my.{domain}/#/emails-eventlog

The refresh button works fine after the main logs are loaded, but does not improve the situation at all when it's still doing the first page load of it.

You should be able to simulate this with these steps:

1. Navigate to the Email Event Log page
2. Click on Back to Email button upper left corner
3. Quickly click back on the Email Event Log page and notice the slow delay

As I noted in my original post though, this seems to coincide with the RBL checks when loading up the initial Email page, as if showing the Event Logs can only occur after the RBL & DNS checks are done for each email domain. This means it likely isn't noticeable if only a few email domains are enabled. I'm not sure of the trigger point but in my situation I have 27 domains, it takes about 1-2 seconds for each domain to do the RBL & DNS checks, which relates to the ~30 second wait time it takes to load the Email Event Logs on the initial load. The steps above simply reproduce it, but even if I only went to the main Email page for the first time (new browser session) and quickly clicked the Event Log button, you'll notice this is delayed.

[d19dotca]

Any update on this one, @girish?

[girish]

@d19dotca I think I will have debug this on your server directly. Can you please send a mail to support@cloudron.io ? I think as you said it's party related to have a large number of domains, since with 2-3 domains it's not really reproducible. From a cursory look at the code, the domain status is entirely async. But maybe something is blocking and I can't make out what it is.

[d19dotca]

@girish Just sent you an email with access.

[girish]

I have fixed this now

[robi]

@girish excellent! What was it?

[girish]

@robi @d19dotca identified the issue at https://forum.cloudron.io/topic/10434/email-event-log-loading-very-slowly-seems-tied-to-overall-email-domain-list-health-checks/9 . When we switch views, pending http requests of the old view are not canceled .