Cloudron Forum

@girish must be private. I can't view that with my account.

I do see some of my local rules present in the logs. Only older ones though. I added a pretty specific one targeting a bunch of spam I've been getting and that doesn't seem to be applied.

Just in case these messages are benign and my rule is breaking things, I'll post it here:

body LOCAL_TINA_EGAN_SPAM_PHISH /Tina Egan/
score LOCAL_TINA_EGAN_SPAM_PHISH 7.0

I've been getting a bunch of spam addressed to "Tina Egan" trying to phish me. Trying to mark them all as junk.

When I drop into the mail container and look at the logs for spamd, I see some more information. I can actually see where it's checking for spam and see that it's indeed not using my rules. It says:

Oct  7 23:09:31.891 [109] info: spamd: using default config for iamthefij@example.com: /app/data/spamd/iamthefij@example.com/user_prefs

That file does not exist. The actual rules are in /app/data/spamd/custom.cf.

I can see in /etc/supervisor/conf.d/spamd.conf that spamd is started with:

/usr/sbin/spamd --max-children 2 --port 7833 --nouser-config --virtual-config /app/data/spamd/%%u --syslog stderr -u cloudron --allow-tell

This appears to tell it not to try and access /root/.spamassassin/user_prefs. When I look at the spamd logs, I actually don't see the error that I see in the mail logs. So not sure why that would be yet.

The thing that does stand out is that it looks like it's telling spamd to look at /app/data/spamd/iamthefij@example.com/user_prefs, as I'm seeing in the logs. This file also does not exist. I don't see it looking for the custom file anywhere.

I'm still seeing this error and new rules I save don't appear to be applied. Anywhere I can follow along with the fix or help out? I can't find the repo on git.cloudron.io.

Yea, I agree. I'd rather see it attached to the button, however confusing is better than broken for me.

This should maybe be bundled into the Cloudron defaults, but if you add export OIDC_REDIRECTION_ENABLED=true to your env file (using File Manager) and reboot, Wekan will not use a popup but instead redirect you to the ODIC login page automatically. This works in Safari and the Progressive Web App on iOS.

So there are some logs on the Nextcloud side too. It shows returning a 403, not authorized. It seems like the token it's retrieving is invalid or something. I didn't find anything too promising when searching Nextcloud forum either. I'll keep poking.

This is curious. I'm seeing an authentication error in my logs for CODE showing the following:

Jul 12 13:34:53 172.18.0.1 - - [12/Jul/2023:20:34:53 +0000] "POST /browser/10deb70/cool.html?WOPISrc=https%3A%2F%2Fcloud.example.com%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Ffiles%2F66920_ocwnxxigt9qg&title=Test%20Doc.odt&lang=en&closebutton=1&revisionhistory=1 HTTP/1.1" 200 11028 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/114.0"
Jul 12 13:34:54 wsd-00013-00418 2023-07-12 20:34:54.998823 +0000 [ docbroker_029 ] ERR WOPI::GetFile [https://cloud.example.com/index.php/apps/richdocuments/wopi/files/66920_ocwnxxigt9qg/contents?access_token=&access_token_ttl=1689230090000] failed with Status Code: Forbidden| wsd/Storage.cpp:1161
Jul 12 13:34:55 wsd-00013-00418 2023-07-12 20:34:54.999183 +0000 [ docbroker_029 ] ERR Cannot download document from WOPI storage uri [https://cloud.example.com/index.php/apps/richdocuments/wopi/files/66920_ocwnxxigt9qg/contents?access_token=&access_token_ttl=1689230090000]. Error: WOPI::GetFile [https://cloud.example.com/index.php/apps/richdocuments/wopi/files/66920_ocwnxxigt9qg/contents?access_token=&access_token_ttl=1689230090000] failed: []| wsd/Storage.cpp:1100
Jul 12 13:34:55 wsd-00013-00418 2023-07-12 20:34:55.002027 +0000 [ docbroker_029 ] ERR loading document exception: WOPI::GetFile [https://cloud.example.com/index.php/apps/richdocuments/wopi/files/66920_ocwnxxigt9qg/contents?access_token=&access_token_ttl=1689230090000] failed: []| wsd/DocumentBroker.cpp:2559
Jul 12 13:34:55 wsd-00013-00418 2023-07-12 20:34:55.002229 +0000 [ docbroker_029 ] ERR Failed to add session to [%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Ffiles%2F66920_ocwnxxigt9qg] with URI [https://cloud.example.com/index.php/apps/richdocuments/wopi/files/66920_ocwnxxigt9qg?access_token=<token>&access_token_ttl=1689230090000]: WOPI::GetFile [https://cloud.example.com/index.php/apps/richdocuments/wopi/files/66920_ocwnxxigt9qg/contents?access_token=&access_token_ttl=1689230090000] failed: []| wsd/DocumentBroker.cpp:2521
Jul 12 13:34:55 wsd-00013-00418 2023-07-12 20:34:55.002568 +0000 [ docbroker_029 ] ERR Storage error while starting session on %2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Ffiles%2F66920_ocwnxxigt9qg for socket #26. Terminating connection. Error: WOPI::GetFile [https://cloud.example.com/index.php/apps/richdocuments/wopi/files/66920_ocwnxxigt9qg/contents?access_token=&access_token_ttl=1689230090000] failed: []| wsd/COOLWSD.cpp:5019
Jul 12 13:34:55 wsd-00013-00418 2023-07-12 20:34:55.003162 +0000 [ docbroker_029 ] ERR #26: Socket write returned -1 (EPIPE: Broken pipe)| net/Socket.hpp:1427
Jul 12 13:34:55 172.18.0.1 - - [12/Jul/2023:20:34:55 +0000] "GET /cool/https%3A%2F%2Fcloud.example.com%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Ffiles%2F66920_ocwnxxigt9qg%3Faccess_token%3D<token>%26access_token_ttl%3D1689230090000/ws?WOPISrc=https%3A%2F%2Fcloud.example.com%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Ffiles%2F66920_ocwnxxigt9qg&compat=/ws HTTP/1.1" 101 111 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:109.0) Gecko/20100101 Firefox/114.0"
Jul 12 13:34:55 wsd-00013-00418 2023-07-12 20:34:55.011816 +0000 [ docbroker_029 ] ERR #35: Read failed, have 0 buffered bytes (ECONNRESET: Connection reset by peer)| net/Socket.hpp:1121
Jul 12 13:34:55 wsd-00013-00418 2023-07-12 20:34:55.011948 +0000 [ docbroker_029 ] WRN #35: Unassociated Kit (411) disconnected unexpectedly| wsd/COOLWSD.cpp:3465

However when I authenticate as my admin user, everything works fine. I thought maybe it was realted to LDAP, so I created another non-LDAP user, but that user also did not work. I've checked my Nextcloud settings and I have no group restrictions for view or edit. I even tested by adding some and instead of seeing this error, it instead downloads the file.

I got everything working now by skipping some updates, finding a problematic one and debugging it a bit. I'm leaving a message I was going to send mid debugging before I managed to get it working just in case anyone else comes across the same issue.

I got it working by skipping versions until I got to one that wouldn't apply. It was v4.41.0. It would run but the database migration would never connect to MySQL. This happened even after I got to the version just before and tried to apply it a few times.

While I was debugging I found that the MySQL connection wouldn't work from the Terminal. Then I put the app into recovery to debug further. It MySQL worked in the Terminal there, so I turned off recovery mode and it booted just fine! Not sure what the root issue was though. Unusual that I was able to reproduce it but then it kind of resolved itself. Maybe switching to or from recovery resets some value causing an issue. I'm not sure.

Original message:

So, this is now pretty specific to Ghost. I've made it up to v4.40.2 just fine, but when I apply the next version v4.41.0, Ghost fails to start.

The log shows

Jul 11 14:39:18 ==> Start ghost
Jul 11 14:39:18 ==> Clear potential migration lock
Jul 11 14:39:18 mysql: [Warning] Using a password on the command line interface can be insecure.
Jul 11 14:39:20 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:39:30 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:39:40 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:39:50 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:40:00 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:40:10 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:40:20 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:40:30 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:40:40 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:40:50 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:41:00 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:41:10 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:41:20 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:41:30 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:41:32 json: updated "/app/data/config.production.json" in-place
Jul 11 14:41:32 json: updated "/app/data/config.production.json" in-place
Jul 11 14:41:32 json: updated "/app/data/config.production.json" in-place
Jul 11 14:41:32 ===> Copy frotend/public folder for asset generation
Jul 11 14:41:32 ==> Loading /app/data/env for potential overrides
Jul 11 14:41:32 ==> Ensure permissions
Jul 11 14:41:32 ==> Migrating database
Jul 11 14:41:40 => Healtheck error: Error: connect ECONNREFUSED 172.18.17.238:2368
Jul 11 14:41:44 [2023-07-11 21:41:44] ERROR Invalid database host.
Jul 11 14:41:44
Jul 11 14:41:44 Invalid database host.
Jul 11 14:41:44
Jul 11 14:41:44 "Please double check your database config."
Jul 11 14:41:44
Jul 11 14:41:44 Error ID:
Jul 11 14:41:44 500
Jul 11 14:41:44
Jul 11 14:41:44 Error Code:
Jul 11 14:41:44 DATABASE_CREATION_FAILED
Jul 11 14:41:44
Jul 11 14:41:44 ----------------------------------------
Jul 11 14:41:44
Jul 11 14:41:44 Error: connect ETIMEDOUT
Jul 11 14:41:44 at /home/cloudron/ghost/versions/5.41.0/node_modules/knex-migrator/lib/database.js:134:19
Jul 11 14:41:44 at /home/cloudron/ghost/versions/5.41.0/node_modules/knex-migrator/lib/database.js:50:23
Jul 11 14:41:44 at Connection._handleTimeoutError (/home/cloudron/ghost/versions/5.41.0/node_modules/knex-migrator/node_modules/mysql2/lib/connection.js:202:17)
Jul 11 14:41:44 at listOnTimeout (node:internal/timers:564:17)
Jul 11 14:41:44 at process.processTimers (node:internal/timers:507:7)

The diff for this version bump is here: https://git.cloudron.io/cloudron/ghost-app/-/commit/93e180df6ad9216f8f04480b9b60212816f86c28

I've tried restoring my backup to v4.40.2 and re-applying multiple times but it continues to fail. It also fails if I use the CLI to skip this version and jump to the latest.

Digging through the app update code, it appears that the version installed is determined by the client side request, so I could modify that, but it's probably better to use the CLI. It probably calls the same API.

This is hard problem to solve though given that both upstream and Cloudron packages are only tested against the previous release. It's all optimized for rolling releases... For example, if a version in the middle removes some migration logic because they think an app has upgraded to latest, it will break things. These subtle breakages only come up later.

Yea, I figured this was the case. This is where semver standards can help. Often, a x.x.1 bump won't introduce any major breaking changes. Using a major version step for introduction of a required update seems reasonable. In that case, you would avoid removing migration logic until after a major version update, at which point you can safely remove it. In real terms, 4.x.x updates should all include anything required to go from any 4.x.x initial state. 5.0.0 should include anything from 4.0.0 to 5.0.0. As of 5.0.1, any upgrade path from 4.x can be dropped. In this scenario, the Cloudron updater could skip from any 4.x.x to any other 4.x.x up to 5.0.0 but require 5.0.0 to be applied before upgrading to 5.0.1.

I'm 25 updates in and still going. I think this is probably because I had disabled auto updates for a while and fell behind and it never caught up because it only updates once a week and updates are more frequent than that.

It seems to me that applying updates should allow jumping more than one minor patch version at a time. If there are migrations that are necessary to apply, either they could be handled within the image such that any image is capable of upgrading any previous one, by using semver and allowing skipping minor updates and jumping to the next major update, or by using some metadata to indicate that a particular update cannot be skipped.

I'm at 4.22.7 now, so I'm assuming that I have at least 40 or so more updates to apply.

Is there a way to skip these manually? Is it possible for me to docker pull the latest or something?

I’ve just done about 8 updates in the last 24 hours and I’m now on 4.17.2. So still pretty far behind the latest.

I got a notice that my Ghost instance is out of date. I have automatic updates on, so I figured one additional update would get me up to speed with the latest, but I got the security notice again.

It appears that each update doesn’t apply the latest, but only the very next update. This is a problem because some apps have updates more frequently than they get applied.

First, is there a way to apply multiple updates at once or skip updates? It seems like that would be important for apps with frequent updates. As an interim action, it may be wise to avoid pushing updates more frequently than auto updates apply.

@iamthefij I should say... it worked to increase the limit. My contacts still don't load.

@nebulon said in Increase Nextcloud PHP memory limit:

/app/data/php.ini

I undid everything I set, but then I grepped through /app/data and found php_value memory_limit 512M in /app/data/htaccess. It's not present in /app/data/.htaccess though. I'm confused as to why both of these exist in the first place.

But anyway, I commented out the lines in /app/data/htaccess, restarted and it looks like it worked!

Thanks for the help.

I know very little about PHP, so I'm running low on things to try. I've tried grepping through /app/code for other places where memory_limit is referenced, and I don't see anything else limiting it.

@girish deleted and restarted. Nextcloud still says 512MB but php -i shows memory_limit => 1024M => 1024M.

@girish I had updated the .user.ini because it had previously said memory_limit=512M, so that's why I changed it. Should I remove it or change it back? I've already removed it from everywhere else.

@girish @nebulon I've restarted the app with all those values set and Settings > System still shows Memory Limit: 512 MB

I'm having this issue but it is with contacts: https://forum.cloudron.io/topic/6813/increase-nextcloud-php-memory-limit