box crashes with "Lock wait timeout exceeded" from eventlog.upsertLoginEvent()
-
Environment:
Cloudron 10.0.5, Ubuntu 26.04.1, MySQL 8.4.11 (also seen before
the OS upgrade on 22.04 / MySQL 8.0.46), ~100 apps, mail enabled, eventlog
table ~38k rows.Symptom
The box service exits through the uncaughtException handler and is restarted
by systemd. 3 times in 23 minutes today ( NRestarts=3 ), and 3 times on
2026-09-24:From uncaughtException handler. BoxError: Lock wait timeout exceeded; try restarting transaction at Object.transaction (box/src/database.js:58:15) at upsertLoginEvent (box/src/eventlog.js:57)While box restarts, the dashboard, LDAP/OIDC and mail authentication are
briefly unavailable.What I found
The stack points at the UPDATE in upsertLoginEvent() . Because of
DATE(creationTime) in the WHERE clause, MySQL doesn't use
creationTime_index and scans the whole table:EXPLAIN UPDATE eventlog SET dataJson='{}' WHERE action='user.login' AND sourceJson LIKE '%x%' AND DATE(creationTime)=CURDATE(); type: index | possible_keys: NULL | key: PRIMARY | rows: 32747 | Extra: Using whereOn a test MySQL I could reproduce the lock wait: with that UPDATE open in
one transaction, a write to an unrelated eventlog row from a second
session times out.Some callers (the mail auth paths in app-bridge.js and ldapserver.js )
call upsertLoginEvent() without await , which would explain why the error
ends up in the uncaughtException handler.Happy to provide more logs if useful.
-
I have fixed the crash itself. I think the 38k rows is causing the timeout. Maybe your disk or CPU is slow? But the default transaction lock is 50 seconds, and I can't think why reading 38k rows will take that long. I have changed the transaction's isolation level that should at least make it faster. Let's see how it is in the next release.
-
G girish has marked this topic as solved
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login