Security alerts due to outdated apps with automatic updates

[iamthefij]

I got a notice that my Ghost instance is out of date. I have automatic updates on, so I figured one additional update would get me up to speed with the latest, but I got the security notice again.

It appears that each update doesn’t apply the latest, but only the very next update. This is a problem because some apps have updates more frequently than they get applied.

First, is there a way to apply multiple updates at once or skip updates? It seems like that would be important for apps with frequent updates. As an interim action, it may be wise to avoid pushing updates more frequently than auto updates apply.

[girish]

@iamthefij the latest ghost package is v4.54.0 . Which one are you on? The last two updates were quickly spaced (2 days), maybe that's why?

Checking the ghost update timelines:

4.53.0     2023-06-24T04:06:32.000Z
4.53.1     2023-06-29T10:16:15.000Z
4.53.2     2023-06-30T16:42:22.000Z
4.53.3     2023-07-05T07:19:52.000Z
4.54.0     2023-07-07T16:42:59.000Z

[iamthefij]

I’ve just done about 8 updates in the last 24 hours and I’m now on 4.17.2. So still pretty far behind the latest.

[iamthefij]

I'm 25 updates in and still going. I think this is probably because I had disabled auto updates for a while and fell behind and it never caught up because it only updates once a week and updates are more frequent than that.

It seems to me that applying updates should allow jumping more than one minor patch version at a time. If there are migrations that are necessary to apply, either they could be handled within the image such that any image is capable of upgrading any previous one, by using semver and allowing skipping minor updates and jumping to the next major update, or by using some metadata to indicate that a particular update cannot be skipped.

I'm at 4.22.7 now, so I'm assuming that I have at least 40 or so more updates to apply.

Is there a way to skip these manually? Is it possible for me to docker pull the latest or something?

[jdaviescoates]

@iamthefij said in Security alerts due to outdated apps with automatic updates:

Is there a way to skip these manually?

No.

But the hack is to adjust your settings to apply updates every hour then wait a while. It'll get through them eventually.

[girish]

@iamthefij I think a trick is to enable updates for every hour in the schedule - https://docs.cloudron.io/updates/#update-schedule. Then it should get caught up in a day or two.

As for jumping updates, it's possible with the CLI but I have no idea what the end result would be. (For example, cloudron update --appstore-id org.ghost.cloudronapp2@xxx --app <subdomain>). If you are in a rush, you can get the tags in https://git.cloudron.io/cloudron/ghost-app and then update one by one with the CLI. There is also a --no-backup argument to update, if you are feeling brave...

This is hard problem to solve though given that both upstream and Cloudron packages are only tested against the previous release. It's all optimized for rolling releases... For example, if a version in the middle removes some migration logic because they think an app has upgraded to latest, it will break things. These subtle breakages only come up later.

[iamthefij]

Digging through the app update code, it appears that the version installed is determined by the client side request, so I could modify that, but it's probably better to use the CLI. It probably calls the same API.

This is hard problem to solve though given that both upstream and Cloudron packages are only tested against the previous release. It's all optimized for rolling releases... For example, if a version in the middle removes some migration logic because they think an app has upgraded to latest, it will break things. These subtle breakages only come up later.

Yea, I figured this was the case. This is where semver standards can help. Often, a x.x.1 bump won't introduce any major breaking changes. Using a major version step for introduction of a required update seems reasonable. In that case, you would avoid removing migration logic until after a major version update, at which point you can safely remove it. In real terms, 4.x.x updates should all include anything required to go from any 4.x.x initial state. 5.0.0 should include anything from 4.0.0 to 5.0.0. As of 5.0.1, any upgrade path from 4.x can be dropped. In this scenario, the Cloudron updater could skip from any 4.x.x to any other 4.x.x up to 5.0.0 but require 5.0.0 to be applied before upgrading to 5.0.1.

[robi]

Might be nice to develop a script for this that can generalize to any app from it's URL as parameter.

Another thought is to make a N8N workflow that's an easy copy/paste for folks who run into this -- added to the docs.

[iamthefij]

I got everything working now by skipping some updates, finding a problematic one and debugging it a bit. I'm leaving a message I was going to send mid debugging before I managed to get it working just in case anyone else comes across the same issue.

I got it working by skipping versions until I got to one that wouldn't apply. It was v4.41.0. It would run but the database migration would never connect to MySQL. This happened even after I got to the version just before and tried to apply it a few times.

While I was debugging I found that the MySQL connection wouldn't work from the Terminal. Then I put the app into recovery to debug further. It MySQL worked in the Terminal there, so I turned off recovery mode and it booted just fine! Not sure what the root issue was though. Unusual that I was able to reproduce it but then it kind of resolved itself. Maybe switching to or from recovery resets some value causing an issue. I'm not sure.

Original message:

So, this is now pretty specific to Ghost. I've made it up to v4.40.2 just fine, but when I apply the next version v4.41.0, Ghost fails to start.

The log shows

Jul 11 14:39:18 ==> Start ghost
Jul 11 14:39:18 ==> Clear potential migration lock
Jul 11 14:39:18 mysql: [Warning] Using a password on the command line interface can be insecure.
Jul 11 14:39:20 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:39:30 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:39:40 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:39:50 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:40:00 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:40:10 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:40:20 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:40:30 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:40:40 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:40:50 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:41:00 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:41:10 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:41:20 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:41:30 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:41:32 json: updated "/app/data/config.production.json" in-place
Jul 11 14:41:32 json: updated "/app/data/config.production.json" in-place
Jul 11 14:41:32 json: updated "/app/data/config.production.json" in-place
Jul 11 14:41:32 ===> Copy frotend/public folder for asset generation
Jul 11 14:41:32 ==> Loading /app/data/env for potential overrides
Jul 11 14:41:32 ==> Ensure permissions
Jul 11 14:41:32 ==> Migrating database
Jul 11 14:41:40 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:41:44 [2023-07-11 21:41:44] ERROR Invalid database host.
Jul 11 14:41:44
Jul 11 14:41:44 Invalid database host.
Jul 11 14:41:44
Jul 11 14:41:44 "Please double check your database config."
Jul 11 14:41:44
Jul 11 14:41:44 Error ID:
Jul 11 14:41:44 500
Jul 11 14:41:44
Jul 11 14:41:44 Error Code:
Jul 11 14:41:44 DATABASE_CREATION_FAILED
Jul 11 14:41:44
Jul 11 14:41:44 ----------------------------------------
Jul 11 14:41:44
Jul 11 14:41:44 Error: connect ETIMEDOUT
Jul 11 14:41:44 at /home/cloudron/ghost/versions/5.41.0/node_modules/knex-migrator/lib/database.js:134:19
Jul 11 14:41:44 at /home/cloudron/ghost/versions/5.41.0/node_modules/knex-migrator/lib/database.js:50:23
Jul 11 14:41:44 at Connection._handleTimeoutError (/home/cloudron/ghost/versions/5.41.0/node_modules/knex-migrator/node_modules/mysql2/lib/connection.js:202:17)
Jul 11 14:41:44 at listOnTimeout (node:internal/timers:564:17)
Jul 11 14:41:44 at process.processTimers (node:internal/timers:507:7)

The diff for this version bump is here: https://git.cloudron.io/cloudron/ghost-app/-/commit/93e180df6ad9216f8f04480b9b60212816f86c28

I've tried restoring my backup to v4.40.2 and re-applying multiple times but it continues to fail. It also fails if I use the CLI to skip this version and jump to the latest.