How would one monitor the database for connectivity issues? I think that's locked down to only be accessed locally from Docker, so I can't use an external monitoring solution in that case right? I looked at the /var/log/mysql/error.log file and it's empty, FWIW.

/var/log/mysql/error.log is the correct file. There are actually 2 MySQL instances - one that runs outside of docker (this is used by the box code) and one that runs inside docker (this is the 'addon' used by apps).

Usually, the MySQL logs do show something when it restarts. Maybe the connectivity issue was caused by something else in your instance. Is this error happening periodically or was this a one off?

can you check dmesg output and see if there is anything there? Usually the kernel dumps an oom dump (unless you or your VPS provider has disabled this) and it stands out in dmesg output.

I just checked the dmesg* and kern* and syslog* files under /var/log/ but don't see any OOM or "out of memory" errors nor any kind of kernel panics or anything that seems relatable (IMO). Nothing about MySQL either.

@girish said in "Temporary translation failure" error in Mail logs:

Currently, we only watch for OOM messages from the containers and also the box code itself. We don't track OOM messages from the database.

How would one monitor the database for connectivity issues? I think that's locked down to only be accessed locally from Docker, so I can't use an external monitoring solution in that case right? I looked at the /var/log/mysql/error.log file and it's empty, FWIW.

Apps won't go down when MySQL goes down. New users cannot login via LDAP. The dashboard might be out for a couple of seconds but all this will be very hard to notice since apps will be working with existing "sessions" and Cloudron's dashboard is a SPA.

Just how of interest, how have you got your monitoring set-up? Are you using Uptime Kuma or some other Cloudron app for this? Or something else entirely? Thanks!

I’m using Uptime Robot services for now with Instatus used for the status page, but I also have been testing Uptime Kuma locally. I don’t think I could switch over to Uptime Kuma for certain until it has a subscription service for my clients to receive auto-notifications on certain services they pay for.

my monitors would have picked that up if any of the apps weren't responding for more than 2 minutes

Just how of interest, how have you got your monitoring set-up? Are you using Uptime Kuma or some other Cloudron app for this? Or something else entirely? Thanks!

2022-05-19T04:11:52.731Z box:apphealthmonitor app health: 39 alive / 0 dead.
2022-05-19T04:12:07.705Z box:apphealthmonitor setHealth: 492b4264-fea8-4d52-bbe6-1f15ce72c53a (<appHostname>) waiting for 1184.295 to update health
ServerError [ServiceUnavailableError]: Response timeout
    at IncomingMessage.<anonymous> (/home/yellowtent/box/node_modules/connect-timeout/index.js:84:8)
    at IncomingMessage.emit (node:events:526:28)
    at Timeout._onTimeout (/home/yellowtent/box/node_modules/connect-timeout/index.js:49:11)
    at listOnTimeout (node:internal/timers:559:17)
    at processTimers (node:internal/timers:502:7) {
  code: 'ETIMEDOUT',
  timeout: 20000
}
Box GET /api/v1/apps 500 Internal Server Error Response timeout 21417.204 ms - 72

And when I did the translation of UTC time to local time, 04 UTC is equal to 9 PM Pacific Time, which is one of the two times I have the backups running each day. I wonder if the backup job froze the system or something for a short bit? Seems a little strange though if true, as there was never any OOM messages present, and no other apps had issues (from what I can see so far anyways). Any suggestions on how I could try to avoid this in the future?

FWIW, the backup didn't complete until about 27 minutes past the start of the hour, and there certainly wasn't any freezing going on for that length of time (my monitors would have picked that up if any of the apps weren't responding for more than 2 minutes).

2022-05-19T04:27:28.863Z box:shell startTask (stdout): Finished with result: success
Main processes terminated with: code=exited/status=0
Service runtime: 27min 27.287s
2022-05-19T04:27:28.905Z box:shell startTask (stdout): Service box-task-15237 finished with exit code 0
2022-05-19T04:27:28.923Z box:tasks startTask: 15237 completed with code 0 and signal 0
2022-05-19T04:27:28.924Z box:locker Released : full_backup
2022-05-19T04:27:28.924Z box:tasks startTask: 15237 done. error: null