Non-persistent Redis still loads the backup's dump.rdb after a reboot — stale WordPress object cache put 9 apps in a restart loop

[imc67]

Follow-up to topic 15801, where you turned off RDB persistence for the WordPress object cache (thanks again, the disk I/O dropped right away). We ran into a side effect of that change this morning, and it took nine WordPress apps down for almost three hours.

What happened

Cloudron 10.1.5. The box update ran at 00:00–00:10 UTC and went fine; all apps were healthy afterwards. At 03:31 UTC the host rebooted for a new kernel. From that moment nine of our ~20 WordPress apps (org.wordpress.unmanaged) restarted every ~13 seconds, 791 times for one of them, until we intervened at 06:13 UTC. The other WordPress apps on the same server started normally.

Why

Each redis container logs this on start:

==> Disabling RDB persistence
* Loading RDB produced by version 8.10.2
* RDB age 12401 seconds

So persistence is off, but redis still loads /home/yellowtent/platformdata/redis/<appid>/dump.rdb when it starts. That file is only written when the box takes a backup (the one loaded at 03:32 was 12401 s old, i.e. from the pre-update backup at ~00:05). Everything the app wrote to its cache between that backup and the reboot is gone, and the cache goes back three and a half hours, out of sync with the database.

For WordPress that broke the start script. start.sh runs with set -eu and does wp option update smtp_mailer_options <json> on every start. The cached smtp_mailer_options was the old value (12 keys; the SMTP Mailer plugin adds force_from_name/force_from_email), while the database already held the new value with the 10 keys start.sh writes. wp-cli compares against the cached value, sees a difference and calls update_option(); the UPDATE changes 0 rows because the database already has that value, so update_option() returns false, wp-cli prints Error: Could not update option 'smtp_mailer_options'., the script exits and the container is restarted. Same result on every start. The sites where cache and database happened to match started fine.

I checked the database side to rule it out: no read-only, full grants, CHECK TABLE OK, and the same UPDATE inside a transaction (rolled back) affects 1 row without error.

Workaround

wp --path=/app/data/public cache flush in each affected app container (during the seconds it is up). On the next start wp-cli reads the database value, reports Value passed for 'smtp_mailer_options' option is unchanged. and Apache starts. Afterwards a BGSAVE on each app's redis, because the dump from this morning's 06:00 backup still held the stale value and the next reboot would have repeated the loop.

Suggestions

• Don't load the dump at start for non-persistent redis. It's a cache, so starting empty is correct; the dump is only needed for a restore. Removing or renaming dump.rdb before redis starts (or having the backup write it outside the data dir and load it explicitly on restore) would fix it. As it stands, every host reboot rolls every WordPress cache back to the last backup.
• Make the WordPress start.sh tolerant here. A wp option update that has nothing to change shouldn't stop the container. For example || true on that line, or a wp cache flush early in start.sh (the cache is not persistent anyway, so flushing it at start costs nothing).

Happy to test a fix on our side.

[girish]

Right, this is a bug. I have fixed it now.

• Redis in non-persistent mode deletes the dump on start up
• Box code will skip backup and restore of non-persistent redis
• Box code deletes the dump when the redis non-persistent flag is set