Website running on managed WP unreachable after latest automatic upgrade

guyds

This past night one of my wordpress websites, running the managed version, was updated automatically from v2.21.3 to v2.22.0 and now it isn't reachable anymore.

According to the cloudron dashboard it is running, but in the logs I noticed the following error which is being repeated continuously:

Mar 15 08:33:28 ==> Changing permissions
Mar 15 08:33:29 => Updating db settings
Mar 15 08:33:29 Success: Updated the constant 'DB_HOST' in the 'wp-config.php' file with the value 'mysql:3306'.
Mar 15 08:33:29 Success: Updated the constant 'DB_NAME' in the 'wp-config.php' file with the value '<redacted>'.
Mar 15 08:33:29 Success: Updated the constant 'DB_USER' in the 'wp-config.php' file with the value '<redacted>'.
Mar 15 08:33:30 Success: Updated the constant 'DB_PASSWORD' in the 'wp-config.php' file with the value '<redacted>'.
Mar 15 08:33:30 ==> Updating wordpress database
Mar 15 08:33:31 Success: WordPress database already at latest db version 51917.
Mar 15 08:33:31 ==> Updating domain related settings
Mar 15 08:33:31 Success: Updated the constant 'WP_HOME' in the 'wp-config.php' file with the value 'https://subdomain.domain.tld'.
Mar 15 08:33:31 Success: Updated the constant 'WP_SITEURL' in the 'wp-config.php' file with the value 'https://subdomain.domain.tld'.
Mar 15 08:33:32 Success: Value passed for 'siteurl' option is unchanged.
Mar 15 08:33:33 Success: Value passed for 'home' option is unchanged.
Mar 15 08:33:33 ==> Configuring smtp mail
Mar 15 08:33:33 Error: Could not get 'wp_mail_smtp' option. Does it exist?
Mar 15 08:33:35 Success: Value passed for 'smtp_mailer_options' option is unchanged.
Mar 15 08:33:35 ==> Configuring LDAP
Mar 15 08:33:37 Success: Value passed for 'authLDAPOptions' option is unchanged.
Mar 15 08:33:37 ==> Migrating database tables to have a prefix
Mar 15 08:33:37 ==> Renaming using RENAME TABLE `<redacted>`.`banhammer` TO `<redacted>`.`wp_banhammer`,`<redacted>`.`commentmeta` TO `<redacted>`.`wp_commentmeta`,`<redacted>`.`comments` TO `<redacted>`.`wp_comments`,`<redacted>`.`frmt_form_entry` TO `<redacted>`.`wp_frmt_form_entry`,`<redacted>`.`frmt_form_entry_meta` TO `<redacted>`.`wp_frmt_form_entry_meta`,`<redacted>`.`frmt_form_views` TO `<redacted>`.`wp_frmt_form_views`,`<redacted>`.`icwp_wpsf_audit_trail` TO `<redacted>`.`wp_icwp_wpsf_audit_trail`,`<redacted>`.`icwp_wpsf_events` TO `<redacted>`.`wp_icwp_wpsf_events`,`<redacted>`.`icwp_wpsf_geoip` TO `<redacted>`.`wp_icwp_wpsf_geoip`,`<redacted>`.`icwp_wpsf_ip_lists` TO `<redacted>`.`wp_icwp_wpsf_ip_lists`,`<redacted>`.`icwp_wpsf_notes` TO `<redacted>`.`wp_icwp_wpsf_notes`,`<redacted>`.`icwp_wpsf_reports` TO `<redacted>`.`wp_icwp_wpsf_reports`,`<redacted>`.`icwp_wpsf_sessions` TO `<redacted>`.`w
Mar 15 08:33:37 ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '`w' at line 1
Mar 15 08:34:38 ==> Changing permissions
...

So it seems that after the update an incomplete query is being executed for prefixing the database tables?

nebulon

Seems like there is a bug in the table migration then. Please restore to the previous version to get up and running again and then maybe clone the app to test the update in an isolated way without interfering with your production website. Afterwards you can enable remote SSH for us and send us a mail with your dashboard domain/IP to support@cloudron.io for investigation.

guyds

@nebulon Fortunately it's not a production website and therefore I didn't restore it yet to a previous version in order to be able to provide more details in case you need them.

I noticed @girish said in WordPress Managed - Package updates:

[2.22.0]

Change the table prefix to wp_. This makes it easier to migrate to the Developer package and is also better supported by plugins.

So it's probably not a coincidence that this is happening and maybe other (production) installations might be affected by this too.

girish

@guyds Version 2.22.0 is not pushed as stable yet. I think it's because the error is because RENAME query is hitting length limits Let me see how we can work around this.

girish

@guyds I have pushed a new package with a fix now. Can you please try the new one? i.e try updating from v2.21.3 to v2.22.0-1 (and not from 2.22.0). If that doesn't work too, can you please write to us at support@cloudron.io ? I would like to understand what's going wrong.

guyds

@girish I see, but if it's marked as unstable then why is it still installed via an automatic update? Shouldn't unstable updates be ignored by the automatic update process?

I'll try to roll back to an earlier backup and then upgrade to v2.22.0-1 and let you know the results.

guyds

@girish I successfully downgraded to v2.21.3 but the upgrade to v2.22.0-1 still fails with a similar, but slightly different error:

Mar 15 20:16:55 ==> Changing permissions
Mar 15 20:16:55 => Updating db settings
Mar 15 20:16:56 Success: Updated the constant 'DB_HOST' in the 'wp-config.php' file with the value 'mysql:3306'.
Mar 15 20:16:56 Success: Updated the constant 'DB_NAME' in the 'wp-config.php' file with the value '<redacted>'.
Mar 15 20:16:56 Success: Updated the constant 'DB_USER' in the 'wp-config.php' file with the value '<redacted>'.
Mar 15 20:16:56 Success: Updated the constant 'DB_PASSWORD' in the 'wp-config.php' file with the value '<redacted>'.
Mar 15 20:16:57 ==> Updating wordpress database
Mar 15 20:16:57 Success: WordPress database already at latest db version 51917.
Mar 15 20:16:57 ==> Updating domain related settings
Mar 15 20:16:58 Success: Updated the constant 'WP_HOME' in the 'wp-config.php' file with the value 'https://subdomain.domain.tld'.
Mar 15 20:16:58 Success: Updated the constant 'WP_SITEURL' in the 'wp-config.php' file with the value 'https://subdomain.domain.tld'.
Mar 15 20:16:59 Success: Value passed for 'siteurl' option is unchanged.
Mar 15 20:17:00 Success: Value passed for 'home' option is unchanged.
Mar 15 20:17:00 ==> Configuring smtp mail
Mar 15 20:17:01 Error: Could not get 'wp_mail_smtp' option. Does it exist?
Mar 15 20:17:03 Success: Value passed for 'smtp_mailer_options' option is unchanged.
Mar 15 20:17:03 ==> Configuring LDAP
Mar 15 20:17:04 Success: Value passed for 'authLDAPOptions' option is unchanged.
Mar 15 20:17:05 ==> Migrating database tables to have a prefix
Mar 15 20:17:05 ==> Renaming using RENAME TABLE `banhammer` TO `wp_banhammer`,`commentmeta` TO `wp_commentmeta`,`comments` TO `wp_comments`,`frmt_form_entry` TO `wp_frmt_form_entry`,`frmt_form_entry_meta` TO `wp_frmt_form_entry_meta`,`frmt_form_views` TO `wp_frmt_form_views`,`icwp_wpsf_audit_trail` TO `wp_icwp_wpsf_audit_trail`,`icwp_wpsf_events` TO `wp_icwp_wpsf_events`,`icwp_wpsf_geoip` TO `wp_icwp_wpsf_geoip`,`icwp_wpsf_ip_lists` TO `wp_icwp_wpsf_ip_lists`,`icwp_wpsf_notes` TO `wp_icwp_wpsf_notes`,`icwp_wpsf_reports` TO `wp_icwp_wpsf_reports`,`icwp_wpsf_sessions` TO `wp_icwp_wpsf_sessions`,`icwp_wpsf_traffic` TO `wp_icwp_wpsf_traffic`,`itsec_bans` TO `wp_itsec_bans`,`itsec_distributed_storage` TO `wp_itsec_distributed_storage`,`itsec_fingerprints` TO `wp_itsec_fingerprints`,`itsec_geolocation_cache` TO `wp_itsec_geolocation_cache`,`itsec_lockouts` TO `wp_itsec_lockouts`,`itsec_logs` TO `wp_itsec_logs`,`itsec_mutexes` TO `wp_itsec_mutexes`,`itsec_opaque_tokens` TO `wp_itsec_opaque_tokens`,`itsec_temp` TO `wp_itsec_temp`,`itsec_user_groups` TO `wp_itsec_u
Mar 15 20:17:05 ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '`wp_itsec_u' at line 1

Apparently it's still hitting a length limit...

girish

@guyds When we make a thing is unstable, our managed instances and internal instances are updated first. It then starts rolling out things slowly. I think your app was one of the super early adopters.

girish

@guyds thanks for testing! Do you think you can send me the log file to support@cloudron.io ? (I really just want the full RENAME line which is getting truncated here. Maybe you can just paste that fully here?) You can do this by Downloading the log file and getting that line.

guyds

@girish said in Website running on managed WP unreachable after latest automatic upgrade:

@guyds When we make a thing is unstable, our managed instances and internal instances are updated first. It then starts rolling out things slowly. I think your app was one of the super early adopters.

Ok, that's a plausible explanation.

@girish said in Website running on managed WP unreachable after latest automatic upgrade:

@guyds thanks for testing! Do you think you can send me the log file to support@cloudron.io ? (I really just want the full RENAME line which is getting truncated here. Maybe you can just paste that fully here?) You can do this by Downloading the log file and getting that line.

I downloaded the full logfile but even then the RENAME command is truncated:

2022-03-15T19:35:24.000Z ==> Migrating database tables to have a prefix
2022-03-15T19:35:24.000Z ==> Renaming using RENAME TABLE `banhammer` TO `wp_banhammer`,`commentmeta` TO `wp_commentmeta`,`comments` TO `wp_comments`,`frmt_form_entry` TO `wp_frmt_form_entry`,`frmt_form_entry_meta` TO `wp_frmt_form_entry_meta`,`frmt_form_views` TO `wp_frmt_form_views`,`icwp_wpsf_audit_trail` TO `wp_icwp_wpsf_audit_trail`,`icwp_wpsf_events` TO `wp_icwp_wpsf_events`,`icwp_wpsf_geoip` TO `wp_icwp_wpsf_geoip`,`icwp_wpsf_ip_lists` TO `wp_icwp_wpsf_ip_lists`,`icwp_wpsf_notes` TO `wp_icwp_wpsf_notes`,`icwp_wpsf_reports` TO `wp_icwp_wpsf_reports`,`icwp_wpsf_sessions` TO `wp_icwp_wpsf_sessions`,`icwp_wpsf_traffic` TO `wp_icwp_wpsf_traffic`,`itsec_bans` TO `wp_itsec_bans`,`itsec_distributed_storage` TO `wp_itsec_distributed_storage`,`itsec_fingerprints` TO `wp_itsec_fingerprints`,`itsec_geolocation_cache` TO `wp_itsec_geolocation_cache`,`itsec_lockouts` TO `wp_itsec_lockouts`,`itsec_logs` TO `wp_itsec_logs`,`itsec_mutexes` TO `wp_itsec_mutexes`,`itsec_opaque_tokens` TO `wp_itsec_opaque_tokens`,`itsec_temp` TO `wp_itsec_temp`,`itsec_user_groups` TO `wp_itsec_u
2022-03-15T19:35:24.000Z ERROR 1064 (42000) at line 1: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '`wp_itsec_u' at line 1

I created a support ticket.

girish

@guyds thanks! I sent you a reply from support. But from your mail it seems it is not related to SQL query length as I initially thought but some bash scripting issue.

girish

The issue here was the GROUP_CONCAT has a default length of 1024 or so. Setting session variable SET group_concat_max_len=10000 fixes the issue.

guyds

@girish I noticed it's running v2.22.0-2 now and indeed everything seems to be ok again.
Thanks for getting this fixed!

jeau

@girish I had the same problem today following a Worpress update. I think the problem comes from the Civicrm plugin which has many tables. I restored with the last backup and disabled the automatic update, but that's not a solution.

girish

@jeau Can you check if it's updating to v2.22.0-5 ? If you can clone from latest backup and update, I am happy to look into this if you can write to support@cloudron.io.

girish

@jeau mm, per https://docs.civicrm.org/installation/en/latest/wordpress/, CiviCRM needs to be told about the prefix as well.

It's not entirely clear but it looks like we have to set $db_prefix = 'wp_'; in settings.php

girish

It seems civicrm does not use WP's table_prefix at all. Well, I pushed a new package which simply ignore civicrm tables.

jeau

@girish Ignoring the civicrm tables works.

However, I have the same kind of problem with an integration of MRBS in Wordpress for authentication. As the tables were properly renamed during the update, I simply set the ad hoc variable ($db_tbl_prefix = "wp_mrbs_";) in the configuration file and it works.

But I guess there are other use cases where renaming tables causes problems.

girish

@jeau Good to know, thanks!

We have to do this migration since WordPress it says it doesn't support not having a table prefix. See:

jeau

@girish thanks

I will ask the teams of civicrm and MRBS to take care of this issue.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Solved Website running on managed WP unreachable after latest automatic upgrade