Cloudron Forum

no ...

@d19dotca said in Sharing custom SpamAssassin Rules:

@imc67 said in Sharing custom SpamAssassin Rules:

@msbt said in Sharing custom SpamAssassin Rules:

Thanks a bunch for the list @d19dotca! Quick question about the rest of the setup though: Do you still have entries in the Email ACL DNSBL Zones or is that empty because everything is handled in the custom rules? Like those:

zen.spamhaus.org
bl.mailspike.net
noptr.spamrats.com
dnsbl.sorbs.net

Or is that empty on your side?

I think this is still a relevant question, @d19dotca your spam-rules are amazing, however you are "calling" ACL DSNBL's that are not default in a Cloudron install (https://docs.cloudron.io/email/#dnsbl) so I guess that they are not working until you add them?

I asked ChatGPT to analyse your latest rules and it advised to add the below ones to the DNSLBL Zones ACL (https://my.domain.com/#/email-settings). Is that in your opinion correct to make them all work?

zen.spamhaus.org
bl.mailspike.net
noptr.spamrats.com
all.spamrats.com
backscatter.spameatingmonkey.net
bl.spameatingmonkey.net
netbl.spameatingmonkey.net

So just to clarify… if you add those to the DNSBL list in Cloudron mail settings, it will completely reject mail that has a hit on any of those services. That mail setting in Cloudron is used by Dovecot/Haraka, not SpamAssassin. The reason you don’t want all those DNSBLs there is because not all of them are super accurate (some are too aggressive), which is why they’re in the SpamAssassin rules instead.

Basically the DNSBL list for Cloudron should only be if you want anything that has a hit to be outright rejected and never arrive in your mailbox (not even the junk folder). I prefer to keep that to just Abusix and SpamHaus myself because they have proven to be very accurate in the sense that they return no false positives, so they’re “safe” in rejecting only the most obvious of spam.

Then everything else that passes through that part will simply be scanned by SpamAssassin against the other DNSBLs in the custom rules and are therefore not rejected but just categorized as either spam or ham. It’s safer that way.

But also totally up to you. If you trust the other DNSBLs, then certainly feel free to add them to the Cloudron DNSBL list, but just know that doing so will most likely result in rejected/dropped messages that you’ll never know about until you look at the mail sever logs.

Ultimately… the DNSBLs in the custom SpamAssassin rule set doesn’t really have anything to do with the DNSBL setting used in Cloudron, as they are different levels of filtering and unrelated to each other.

Hopefully that makes sense. I’m just waking up while writing this so let me know if I can clarify further as I may not be explaining myself perfectly, lol.

WOW thank you very very much for this extraordinary clarification! I expected a necessary connection between the two but it isn’t. Thanks for your great work and explanation!

@msbt said in Sharing custom SpamAssassin Rules:

Thanks a bunch for the list @d19dotca! Quick question about the rest of the setup though: Do you still have entries in the Email ACL DNSBL Zones or is that empty because everything is handled in the custom rules? Like those:

zen.spamhaus.org
bl.mailspike.net
noptr.spamrats.com
dnsbl.sorbs.net

Or is that empty on your side?

I think this is still a relevant question, @d19dotca your spam-rules are amazing, however you are "calling" ACL DSNBL's that are not default in a Cloudron install (https://docs.cloudron.io/email/#dnsbl) so I guess that they are not working until you add them?

I asked ChatGPT to analyse your latest rules and it advised to add the below ones to the DNSLBL Zones ACL (https://my.domain.com/#/email-settings). Is that in your opinion correct to make them all work?

zen.spamhaus.org
bl.mailspike.net
noptr.spamrats.com
all.spamrats.com
backscatter.spameatingmonkey.net
bl.spameatingmonkey.net
netbl.spameatingmonkey.net

Ok, thanks for your hints!!

The result was PID 19974

However:

● mysql.service - MySQL Community Server
     Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2025-12-13 05:57:30 UTC; 1 day 5h ago
    Process: 874 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
   Main PID: 910 (mysqld)
     Status: "Server is operational"
      Tasks: 47 (limit: 77023)
     Memory: 601.7M
        CPU: 59min 14.538s
     CGroup: /system.slice/mysql.service
             └─910 /usr/sbin/mysqld

And docker top mysql

UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                9842                8908                0                   Dec13               ?                   00:00:17            /usr/bin/python3 /usr/bin/supervisord --configuration /etc/supervisor/supervisord.conf --nodaemon -i Mysql
message+            19974               9842                6                   Dec13               ?                   01:56:43            /usr/sbin/mysqld
message+            19976               9842                0                   Dec13               ?                   00:01:31            node /app/code/service.js

So ps uax | grep -i 19974 gives:

message+   19974  6.6  1.8 4249604 1229136 ?     Sl   Dec13 116:48 /usr/sbin/mysqld

So at least we now know that it's the Docker MySQL

I did a manual plugin update of openid-connect-generic to 3.10.1 within a Wordpress (developer) website and the result was that the Login with Cloudron button was gone!

Do we have to wait for the app update to fix something?

For now I restored the latest backup and wait with manual updates.

It’s a production server, isn’t it ridiculous to stop these apps to watch resource behavior? There must be tools or ways to find the root cause don’t you think?

Beside that it’s the host MySQL does it has anything to do with apps?

Thanks @nebulon for your time, together with ChatGPT I did deeper analysis but I also read this: https://docs.cloudron.io/troubleshooting/#mysql

Two instances of MySQL
There are two instances of MySQL on Cloudron. One instance runs on the host and is used by the platform. Another instance is the MySQL addon which runs in a container named mysql and is shared by apps. This test is related to the host MySQL.

Doesn't this mean that the mysql service in iotop is the "host version" that has nothing to do with the apps?

For now "we" (I) have seen this:

Summary of Disk Write I/O Observation on Cloudron Host

• Using iotop, the host shows consistently high disk write I/O (4–5 MB/s).
• Analysis of MySQL processes (mysqld) indicates these are responsible for the majority of the write load.
• The high write I/O is primarily due to InnoDB internal activity: buffer pool flushes, redo log writes, and metadata updates, mostly from the box database (eventlog, tasks, backups).

In about 10 minutes this is the Disk Write I/O (so 1.5GB in 10 minutes)

Total DISK READ:         0.00 B/s | Total DISK WRITE:         2.73 M/s
Current DISK READ:       0.00 B/s | Current DISK WRITE:       4.25 M/s
    TID  PRIO  USER     DISK READ DISK WRITE>  SWAPIN      IO    COMMAND                                                                                                                  
  21250 be/4 messageb      0.00 B   1038.50 M  ?unavailable?  mysqld
    936 be/4 mysql         0.00 B    465.28 M  ?unavailable?  mysqld

I stopped about 25% of the apps at a certain moment with no significant result, this is the current situation (IMHO not really intensive application and they have low traffic):

App 	Status 
Yourls	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Stopped 
WordPress (Developer)	Running 
WordPress (Developer)	Stopped 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Stopped 
WordPress (Developer)	Running 
Taiga	Stopped 
Surfer	Running 
Surfer	Stopped 
Roundcube	Running 
Roundcube	Running 
Omeka S	Stopped 
Moodle	Stopped 
LAMP	Running 
Roundcube	Running 
Roundcube	Running 
Roundcube	Running 
Pretix	Stopped 
MiroTalk SFU	Running 
Matomo	Running 
FreeScout	Running 
FreeScout	Running 
Espo CRM	Running 

What to do next to find the root cause?

Because an auto-update to 9.0.14 caused serious backup errors on one of my 3 Cloudrons, I wanted to prevent Cloudron from auto-update box on the other 2.

However auto-update functionality is (still) coupled! So the only one setting auto-update is for apps AND box.

It would be very useful to decouple this and make it two separate settings.

Last night one of my 3 Cloudron servers auto-updated to 9.0.14.

The first auto-backup after it errored (see below).
I tried to edit/safe config and remount and manual started a backup an again the same error!

Server info:

Cloudron version 9.0.14
Ubuntu version Ubuntu 22.04.5 LTS Linux 5.15.0-163-generic
Vendor netcup
Product KVM Server
CPU 10 Core "AMD EPYC 7702P 64-Core Processor"
Memory 67.42 GB RAM & 4.29 GB Swap
Uptime 4 days
Cloudron creation time unknown

Backup type:

DefaultEverything
Storage: sshfs (rsync) at u*******.your-storagebox.de:/home
Schedule: Every day @ 00:00, 08:00, 12:00, 16:00, 20:00
Retention Policy: 7 daily, 4 weekly, 12 monthly
Last run: 12 dec 2025, 08:01
SSH connection error: ssh exited with code 255 signal null 

Error:

Dec 12 08:01:53 box:shell filesystem: ssh -o "StrictHostKeyChecking no" -i /tmp/identity_file-mnt-managedbackups-6de7dc51-23ad-45e7-b6e4-137d769995ab -p 23 *****@******.your-storagebox.de rm -rf snapshot/app_1ceb591b-97ad-4793-8942-cf5853d739fb/data/public/wp-content/cache/****** errored BoxError: ssh exited with code 255 signal null
Dec 12 08:01:53 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:82:23)
Dec 12 08:01:53 at ChildProcess.emit (node:events:519:28)
Dec 12 08:01:53 at maybeClose (node:internal/child_process:1101:16)
Dec 12 08:01:53 at ChildProcess._handle.onexit (node:internal/child_process:304:5) {
Dec 12 08:01:53 reason: 'Shell Error',
Dec 12 08:01:53 details: {},
Dec 12 08:01:53 stdout: <Buffer >,
Dec 12 08:01:53 stdoutLineCount: 0,
Dec 12 08:01:53 stderr: <Buffer 57 61 72 6e 69 6e 67 3a 20 49 64 65 6e 74 69 74 79 20 66 69 6c 65 20 2f 74 6d 70 2f 69 64 65 6e 74 69 74 79 5f 66 69 6c 65 2d 6d 6e 74 2d 6d 61 6e 61 ... 244 more bytes>,
Dec 12 08:01:53 stderrLineCount: 4,
Dec 12 08:01:53 code: 255,
Dec 12 08:01:53 signal: null,
Dec 12 08:01:53 timedOut: false,
Dec 12 08:01:53 terminated: false
Dec 12 08:01:53 }
Dec 12 08:01:53 box:backupformat/rsync sync: done processing deletes. error: BoxError: SSH connection error: ssh exited with code 255 signal null at Object.removeDir (/home/yellowtent/box/src/storage/filesystem.js:286:47) at process.processTicksAndRejections (node:internal/process/task_queues:105:5) at async processSyncerChange (/home/yellowtent/box/src/backupformat/rsync.js:110:13) at async /home/yellowtent/box/src/backupformat/rsync.js:128:92 { reason: 'External Error', details: {} }
Dec 12 08:01:53 box:backupupload upload completed. error: BoxError: SSH connection error: ssh exited with code 255 signal null at Object.removeDir (/home/yellowtent/box/src/storage/filesystem.js:286:47) at process.processTicksAndRejections (node:internal/process/task_queues:105:5) at async processSyncerChange (/home/yellowtent/box/src/backupformat/rsync.js:110:13) at async /home/yellowtent/box/src/backupformat/rsync.js:128:92 { reason: 'External Error', details: {} }
Dec 12 08:01:53 box:backuptask runBackupUpload: result - {"errorMessage":"SSH connection error: ssh exited with code 255 signal null"}
Dec 12 08:01:54 box:shell backuptask: /usr/bin/sudo --non-interactive -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_1ceb591b-97ad-4793-8942-cf5853d739fb 6de7dc51-23ad-45e7-b6e4-137d769995ab {"localRoot":"/home/yellowtent/appsdata/1ceb591b-97ad-4793-8942-cf5853d739fb","layout":[]} errored BoxError: /usr/bin/sudo exited with code 50 signal null
Dec 12 08:01:54 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:82:23)
Dec 12 08:01:54 at ChildProcess.emit (node:events:519:28)
Dec 12 08:01:54 at maybeClose (node:internal/child_process:1101:16)
Dec 12 08:01:54 at ChildProcess._handle.onexit (node:internal/child_process:304:5) {
Dec 12 08:01:54 reason: 'Shell Error',
Dec 12 08:01:54 details: {},
Dec 12 08:01:54 stdout: '',
Dec 12 08:01:54 stdoutLineCount: 0,
Dec 12 08:01:54 stderr: '',
Dec 12 08:01:54 stderrLineCount: 0,
Dec 12 08:01:54 code: 50,
Dec 12 08:01:54 signal: null,
Dec 12 08:01:54 timedOut: false,
Dec 12 08:01:54 terminated: false
Dec 12 08:01:54 }
Dec 12 08:01:54 box:backuptask fullBackup: app *******.nl backup finished. Took 4.634 seconds
Dec 12 08:01:54 box:locks write: current locks: {"full_backup_task_6de7dc51-23ad-45e7-b6e4-137d769995ab":null}
Dec 12 08:01:54 box:locks release: app_backup_1ceb591b-97ad-4793-8942-cf5853d739fb
Dec 12 08:01:54 box:tasks setCompleted - 18576: {"result":null,"error":{"message":"SSH connection error: ssh exited with code 255 signal null","reason":"External Error"},"percent":100}
Dec 12 08:01:54 box:tasks updating task 18576 with: {"completed":true,"result":null,"error":{"message":"SSH connection error: ssh exited with code 255 signal null","reason":"External Error"},"percent":100}
Dec 12 08:01:54 box:taskworker Task took 24.755 seconds
Dec 12 08:01:54 BoxError: SSH connection error: ssh exited with code 255 signal null
Dec 12 08:01:54 at runBackupUpload (/home/yellowtent/box/src/backuptask.js:204:15)
Dec 12 08:01:54 at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
Dec 12 08:01:54 at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:370:34)
Dec 12 08:01:54 at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:393:26)
Dec 12 08:01:54 Exiting with code 0

@girish said in Suggestion: Change WordPress app health check endpoint to /wp-json/ or alternatives:

Have to investigate, but I think WP JSON API can be disabled (in fact, many sites recommend this) and one of these WP firewall plugins disables it.

Agree, the first thing I do after installing a new Wordpress site is deactivation of the rest-api:
https://nl.wordpress.org/plugins/disable-wp-rest-api/

ok thanks, below the result after just a few minutes, I'm not a technician but as far as I can see it's mainly mysql which is writing (I sorted Write):

and now?

its not default installed:

Command 'iotop' not found, but can be installed with:
apt install iotop    # version 0.6-24-g733f3f8-1.1ubuntu0.1, or
apt install iotop-c  # version 1.21-1

@girish right but where to look?

@girish said in Cloudron v9: huge disk I/O is this normal/safe/needed?:

'docker stats'

it's impossible to have a view with this, every second tens of docker containers are created (cron?) so it keeps listing and growing.

Is there a proper way to do some inspections with disk I/O in mind? Or shall I give you access to have a view?

Also in the Netcup SCP it shows, so it seems there is seriously something wrong?

(The last part of the graph is not representative, zooming in to 6 hours is straight line)

@girish said in Cloudron v9: huge disk I/O is this normal/safe/needed?:

We have even more I doubt our server wrote or read so much. Must be something wrong with the reporting, investigating...

But at least your Writ I/O graph/speed is almost zero?!

Description

I noticed since v9 on my 3 Cloudron-Pro instances with the new graphs that the Disk I/O is non-stop average at Write ~ 2.5 MB/s and one even at ~4MB/s. This seems to me extremely high as the 3 servers are IMHO not really highly productive. The total Write in 24 hours is about 4-7TB!!!

Steps to reproduce

Have a look in your graph

Cloudron Version

9.0.12 and since last night 9.0.13

Ubuntu Version

Ubuntu 22.04.5 LTS

In this tutorial it shows how to set reverse geocoding, but how to do this in the Cloudron app?:

https://dawarich.app/docs/tutorials/reverse-geocoding

just tried again but without the very last comma .... and now it works.
poeh, what a comma can do ... or not