How do you diagnose PHP-based SQL problems?

[Lonkle]

Doing some dev work for plugins on single site Wordpress installs I keep running into a lot of MYSQL SIGKILLS. It kills itself like 5 times and then restarts, happens once every few hours (usually while dev-ing since it's just a dev site) and is back up in 5-minutes-ish.

Has anyone else run into this? I feel like I run into this every time I'm developing on the backend of Wordpress. How does everyone else diagnose MySQL SIGKILL errors like this? This only happens with one installation so I know it's a plugin thing.

But...the thing is - I didn't even go to the WP site today (and it's not public) and checked the log and look what I just saw.

This leads me to believe that my issue is coming from a CRON process and Cloudron runs WP-CRON once per minute which is far too often IMO so maybe there's something there, but that's literally the only thing that could have run if I didn't touch Cloudron at all today.

[Lonkle]

Btw this is what the log looks like when I'm in there dev-ing. But, that's a plugin thing I have no choice but to figure out.

I'm just astounded that it happened while the VPS wasn't being used at all which is why I'm posting.

[nebulon]

Is this a custom app package? I wonder why there is a mysql instance running inside the app. On Cloudron it is recommended to use the mysql addon https://docs.cloudron.io/custom-apps/addons/#mysql instead.

[girish]

Could be getting killed because it's running out of memory. Maybe you can bump the increase the memory limit of mysql service?

[Lonkle]

@nebulon Nope, not custom, it's your Wordpress (Developer) app. You're pointing me in the right direction tho - it's gotta be a plugin that, for some reason, is running its own instance of mysql.

I personally don't get how or why it's 'spawning' the mysql process, waiting a second to say it succeeds and then dies in 5 minutes. 5 minutes must be some kind of timeout and there must be a conflict in the first place that killed mysql.

Also weird that it eventually succeeds too.

[Lonkle]

@girish said in How do you diagnose PHP-based SQL problems?:

Could be getting killed because it's running out of memory. Maybe you can bump the increase the memory limit of mysql service?

That's happened to me before but Cloudron always reports it and I adjust memory. Cloudron is reporting no memory limitations. But just in case, I set it really high IF that is the problem occurring.

[nebulon]

@lonkle just to be on the same page, since this sounds far out with some plugin spawning a mysql process. The logs you have pasted earlier here, are those from the app or from the mysql service running in your Cloudron?

[Lonkle]

@nebulon said in How do you diagnose PHP-based SQL problems?:

@lonkle just to be on the same page, since this sounds far out with some plugin spawning a mysql process. The logs you have pasted earlier here, are those from the app or from the mysql service running in your Cloudron?

That is from the my SQL logs. I found out that's why the entire server would stop functioning for 5 minutes at a time - MySQL unexpectedly dies, retries, dies, retries, dies. I know it's gotta be a plugin. When you said it was spawning an SQL process somehow, I just ran with the idea since you're the expert but I realized that you just thought I was showing you app logs.

I'v watched the app logs even during one of the random MySQL deaths, and there's nothing on the logs except normal PHP stuff. I cleaned up all the notices and it still happens. I have ONE lead:

PHP Notice: Undefined offset: 1 in phar:///app/pkg/wp/vendor/wp-cli/wp-cli/php/WP_CLI/Runner.php(1197) : eval()'d code on line 70

I don't use the WP-CLI, which means it can only be that the 1 minute cron job set up that sometimes runs into some huge task. Something is throwing an undefined offset in the Cron anyway. I'm hoping that's it. If I can find it.

I'll have to time the next time it happens and see if it aligns with a cron job.

[Lonkle]

Oh, I forgot the Error reconnecting to the database This means that we lost contact with the database server at mysql:3306. This could mean your host’s database server is down. Are you sure the database server is running? Are you sure the database server is not under particularly heavy load? If you’re unsure what these terms mean you should probably contact your host. If you still need help you can always visit the WordPress Support Forums error from Wordpress if you wait till timeout.

That's why it was so obvious that the SQL server was going down.

[Lonkle]

Further hunting. There's a plugin called WP Time Capsule that provides a "staging" server to test things out before making them live. It just copies the WP DB and files to a folder at the root called "staging" and if "staging is currently activated" and then I I try to backup Cloudron, I get this Cloudron crashed/stopped.

But if I delete the staging folder and DB tables (you just hit delete in the plugin). Then Cloudron backs up fine.

Theories?

[Lonkle]

Okay, looks like if I open 20 backend "edit post" pages in new tabs at once I can kill SQL at will. But...how...and why. I understand it running out of memory, but...wait, what if it runs out of memory and doesn't realize it because of swap and so it doesn't kill itself.

[Lonkle]

Since I can easily reproduce this I did a bunch of tests. Could never pinpoint it but after I updated to V6.3.4, I can't reproduce it at all. So, lucky release for me.

[girish]

Ok, I will mark this as solved but let's investigate further if this comes up again.

FWIW, if you do docker exec -ti mysql /bin/bash then the details mysql logs are located at /tmp/mysqld.err .