Cloudron v9: huge disk I/O is this normal/safe/needed?

[imc67]

ok thanks, below the result after just a few minutes, I'm not a technician but as far as I can see it's mainly mysql which is writing (I sorted Write):

[nebulon]

Since we debugged some other issue on that server, was also taking a look at the disk I/O. So basically the mysql service is doing a lot of disk I/O (also as see in the screenshot).

It does seem the mysql addon is just queried and written to a lot. So likely one of the many installed apps using it might commit a lot to the database. I didn't want to stop apps, but maybe you can try to stop individual apps which use mysql one-by-one to hopefully find the one which causes the constant writes.

[imc67]

Thanks @nebulon for your time, together with ChatGPT I did deeper analysis but I also read this: https://docs.cloudron.io/troubleshooting/#mysql

Two instances of MySQL
There are two instances of MySQL on Cloudron. One instance runs on the host and is used by the platform. Another instance is the MySQL addon which runs in a container named mysql and is shared by apps. This test is related to the host MySQL.

Doesn't this mean that the mysql service in iotop is the "host version" that has nothing to do with the apps?

For now "we" (I) have seen this:

Summary of Disk Write I/O Observation on Cloudron Host

• Using iotop, the host shows consistently high disk write I/O (4–5 MB/s).
• Analysis of MySQL processes (mysqld) indicates these are responsible for the majority of the write load.
• The high write I/O is primarily due to InnoDB internal activity: buffer pool flushes, redo log writes, and metadata updates, mostly from the box database (eventlog, tasks, backups).

In about 10 minutes this is the Disk Write I/O (so 1.5GB in 10 minutes)

Total DISK READ:         0.00 B/s | Total DISK WRITE:         2.73 M/s
Current DISK READ:       0.00 B/s | Current DISK WRITE:       4.25 M/s
    TID  PRIO  USER     DISK READ DISK WRITE>  SWAPIN      IO    COMMAND                                                                                                                  
  21250 be/4 messageb      0.00 B   1038.50 M  ?unavailable?  mysqld
    936 be/4 mysql         0.00 B    465.28 M  ?unavailable?  mysqld

I stopped about 25% of the apps at a certain moment with no significant result, this is the current situation (IMHO not really intensive application and they have low traffic):

App 	Status 
Yourls	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Stopped 
WordPress (Developer)	Running 
WordPress (Developer)	Stopped 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Stopped 
WordPress (Developer)	Running 
Taiga	Stopped 
Surfer	Running 
Surfer	Stopped 
Roundcube	Running 
Roundcube	Running 
Omeka S	Stopped 
Moodle	Stopped 
LAMP	Running 
Roundcube	Running 
Roundcube	Running 
Roundcube	Running 
Pretix	Stopped 
MiroTalk SFU	Running 
Matomo	Running 
FreeScout	Running 
FreeScout	Running 
Espo CRM	Running 

What to do next to find the root cause?

[avatar1024]

@imc67 said in Cloudron v9: huge disk I/O is this normal/safe/needed?:

I stopped about 25% of the apps at a certain moment with no significant result

I think @nebulon was suggesting to stop apps one by one to see if one particular app is causing the problem.

[robi]

Generally such system behavior is accompanied by higher CPU and Memory usage, so you can start with stopping those, and see which one causes a dip MySQL usage.

[imc67]

It’s a production server, isn’t it ridiculous to stop these apps to watch resource behavior? There must be tools or ways to find the root cause don’t you think?

Beside that it’s the host MySQL does it has anything to do with apps?

[robi]

@imc67 Holding that limiting belief is keeping your problem unresolved, no?

Sure, then trace it from the MySQL side, find which user, which container and so on..

Yes, it has everything to do with the Apps that are using that DB instance.

[james]

Hello @imc67
You can use the PID from the process to figure out what mysql service it is.

e.g. your iotop shows for mysqld the pid 1994756.
You can run systemctl status mysql.service and there is the pid displayed:

● mysql.service - MySQL Community Server
     Loaded: loaded (/usr/lib/systemd/system/mysql.service; enabled; preset: enabled)
     Active: active (running) since Mon 2025-12-01 09:17:59 UTC; 1 week 5 days ago
   Main PID: 1994756 (mysqld)
     Status: "Server is operational"
      Tasks: 48 (limit: 4603)
     Memory: 178.7M (peak: 298.0M swap: 95.4M swap peak: 108.7M)
        CPU: 1h 41min 31.520s
     CGroup: /system.slice/mysql.service
             └─1994756 /usr/sbin/mysqld

Notice: journal has been rotated since unit was started, output may be incomplete.

So from iotop I can confirm that the system mysqld service is pid 1994756 so I'd know to inspect the system mysqld service and not the docker mysql service.

You can also get the pid from the mysqld inside the docker container with docker top mysql:

docker top mysql
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                1889                1512                0                   Nov07               ?                   00:06:17            /usr/bin/python3 /usr/bin/supervisord --configuration /etc/supervisor/supervisord.conf --nodaemon -i Mysql
usbmux              3079                1889                0                   Nov07               ?                   03:49:38            /usr/sbin/mysqld
usbmux              3099                1889                0                   Nov07               ?                   00:00:11            node /app/code/service.js

Then I know the mysqld pid of the docker service is 3079 which I can check again with the system:

ps uax | grep -i 3079
usbmux      3079  0.4  1.0 1587720 43692 ?       Sl   Nov07 229:38 /usr/sbin/mysqld

Now we can differentiate between the two.

---

Okay.
Now that we can differentiate between the two, you can observe iotop and see which one has a high I/O.
After you narrow it down to either one, then we can do some analysis what database / table get accesses the most even further narrow it down.

[imc67]

Ok, thanks for your hints!!

The result was PID 19974

However:

● mysql.service - MySQL Community Server
     Loaded: loaded (/lib/systemd/system/mysql.service; enabled; vendor preset: enabled)
     Active: active (running) since Sat 2025-12-13 05:57:30 UTC; 1 day 5h ago
    Process: 874 ExecStartPre=/usr/share/mysql/mysql-systemd-start pre (code=exited, status=0/SUCCESS)
   Main PID: 910 (mysqld)
     Status: "Server is operational"
      Tasks: 47 (limit: 77023)
     Memory: 601.7M
        CPU: 59min 14.538s
     CGroup: /system.slice/mysql.service
             └─910 /usr/sbin/mysqld

And docker top mysql

UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                9842                8908                0                   Dec13               ?                   00:00:17            /usr/bin/python3 /usr/bin/supervisord --configuration /etc/supervisor/supervisord.conf --nodaemon -i Mysql
message+            19974               9842                6                   Dec13               ?                   01:56:43            /usr/sbin/mysqld
message+            19976               9842                0                   Dec13               ?                   00:01:31            node /app/code/service.js

So ps uax | grep -i 19974 gives:

message+   19974  6.6  1.8 4249604 1229136 ?     Sl   Dec13 116:48 /usr/sbin/mysqld

So at least we now know that it's the Docker MySQL

[james]

Hello @imc67
Now we can start analysing.
Edit the file /home/yellowtent/platformdata/mysql/custom.cnf and add the following lines:

[mysqld]
general_log = 1
slow_query_log = 1

Restart the MySQL service in the Cloudron Dashboard.
The log files are stored at /home/yellowtent/platformdata/mysql/mysql.log and /home/yellowtent/platformdata/mysql/mysql-slow.log.

Let it run for a day or more.
Then you can download the log files and see what queries run very often causing disk I/O.

[imc67]

I enabled this en within seconds the log file was enormous, I asked ChatGPT to analyse it and here is it's observations: (too technical for me):

---

Some observations after briefly enabling the MySQL general log (Cloudron v9)

I enabled the MySQL general log only for a short time because of disk I/O concerns, but even within a few minutes a clear pattern showed up.

What I’m seeing:

• A very high number of
  INSERT INTO session (...) and
  INSERT ... ON DUPLICATE KEY UPDATE
• These happen continuously and come from 172.18.0.1
• As far as I understand, this IP is the Docker bridge gateway in Cloudron, so it likely represents multiple apps

I temporarily disabled Matomo to rule that out, but disk I/O and session-related writes did not noticeably decrease, so it does not seem to be the main contributor.

From the log it looks like:

• Multiple applications are storing sessions in MySQL
• Session rows are updated on almost every request
• This can generate a lot of InnoDB redo log and disk I/O, even with low traffic

Nothing looks obviously broken, but I’m trying to understand whether this level of session write activity is:

• expected behavior in Cloudron v9
• something that can be tuned or configured
• or if there are recommended best practices (e.g. Redis for sessions)

Any guidance on how Cloudron expects apps to handle sessions, or how to reduce unnecessary MySQL write I/O, would be much appreciated.

Thanks for looking into this.

[joseph]

Do you have happen to use nextcloud on the server? I think nextcloud+ldap keeps doing a login request when syncing for each file (which might trigger a login eventlog in mysql)

[imc67]

@joseph said in Cloudron v9: huge disk I/O is this normal/safe/needed?:

Do you have happen to use nextcloud on the server? I think nextcloud+ldap keeps doing a login request when syncing for each file (which might trigger a login eventlog in mysql)

No there is no Nextcloud on this server

[joseph]

@imc67 not sure I remember why Does this mean that if you disable matomo temporarily, the disk usage goes down a lot?

Seems easy to fix now that we know the root cause

[imc67]

@joseph I’m pretty sure that more apps suffer from this issue since the introduction of OIDC, I see EspoCRM and FreeScout, also has a Healthcheck to root/ (where the OIDC login is), didn’t check the sessions.

[joseph]

I have to test, but it seems like a matomo bug here (if this is all true). There is no need to create an OIDC session when visiting '/' . You have to only create OIDC session when OIDC login button is clicked.

[luckow]

My two cents: as soon as #28 is correct, this should happen with every Cloudron instance that has Matomo (and OIDC enabled). I looked at one of my instances that met the criteria. One of the Matomo instances had about 300 sessions stored in MySQL. The oldest entry is from Feb 26.
So maybe #28 isn't correct, or it's something that only happens on this instance.

[imc67]

Maybe because the three installs are 5-6 years old and had many many updates/upgrades etc?

can you check how many sessions per hour are being created? Run this query:
sql

SELECT HOUR(FROM_UNIXTIME(modified)) AS hour, COUNT(*) AS sessions
FROM `<your_matomo_db>`.session
WHERE DATE(FROM_UNIXTIME(modified)) = CURDATE() - INTERVAL 1 DAY
GROUP BY hour ORDER BY hour;

On my instances this shows exactly 360 per hour = 1 per 10 seconds = health check interval. If yours shows much less, the health checker behaves differently on your setup.

[luckow]

@imc67 one app instance (4y old)

+------+----------+
| hour | sessions |
+------+----------+
|    0 |        2 |
|    2 |        1 |
|    7 |        2 |
|    8 |        1 |
|    9 |        1 |
|   13 |        3 |
|   15 |        1 |
|   17 |        3 |
|   19 |        1 |
|   20 |        3 |
|   21 |        4 |
|   22 |        1 |
+------+----------+

different app instance (7y old)

+------+----------+
| hour | sessions |
+------+----------+
|    3 |        1 |
|    5 |        2 |
|   15 |        4 |
|   18 |        2 |
|   19 |        2 |
|   20 |        2 |
|   21 |        4 |
|   22 |        2 |
+------+----------+

health check is every 10 sec.

Mar 07 18:00:50 - - - [07/Mar/2026:17:00:50 +0000] "GET / HTTP/1.1" 302 - "-" "Mozilla (CloudronHealth)"
Mar 07 18:00:50 172.18.0.1 - - [07/Mar/2026:17:00:50 +0000] "GET / HTTP/1.1" 302 299 "-" "Mozilla (CloudronHealth)"
Mar 07 18:01:00 - - - [07/Mar/2026:17:01:00 +0000] "GET / HTTP/1.1" 302 - "-" "Mozilla (CloudronHealth)"
Mar 07 18:01:00 172.18.0.1 - - [07/Mar/2026:17:01:00 +0000] "GET / HTTP/1.1" 302 299 "-" "Mozilla (CloudronHealth)"
Mar 07 18:01:10 - - - [07/Mar/2026:17:01:10 +0000] "GET / HTTP/1.1" 302 - "-" "Mozilla (CloudronHealth)"
Mar 07 18:01:10 172.18.0.1 - - [07/Mar/2026:17:01:10 +0000] "GET / HTTP/1.1" 302 299 "-" "Mozilla (CloudronHealth)"
Mar 07 18:01:20 - - - [07/Mar/2026:17:01:20 +0000] "GET / HTTP/1.1" 302 - "-" "Mozilla (CloudronHealth)"
Mar 07 18:01:20 172.18.0.1 - - [07/Mar/2026:17:01:20 +0000] "GET / HTTP/1.1" 302 299 "-" "Mozilla (CloudronHealth)"
Mar 07 18:01:30 - - - [07/Mar/2026:17:01:30 +0000] "GET / HTTP/1.1" 302 - "-" "Mozilla (CloudronHealth)"
Mar 07 18:01:30 172.18.0.1 - - [07/Mar/2026:17:01:30 +0000] "GET / HTTP/1.1" 302 299 "-" "Mozilla (CloudronHealth)"

[imc67]

We also found some huge MySQL tables from a Wordpress-app with dedicated MainWP due to incorrect retention settings, after correction and deletion the 1 minute iotop -aoP -d 5 is still:

• Docker MySQL: 70 MB
• Host MySQL: 33 MB
• go-carbon: 6.7 MB
• jbd2: 9.9 MB
• Total: ~103 MB per minute

To put this in perspective:

• 103 MB/min = 6.2 GB/hour
• 6.2 GB/hour = 148 GB/day
• 148 GB/day = 4.4 TB/month

This is on a server with relatively low visitors across 10 sites. The vast majority of this write activity is caused by the issues identified above (Matomo health checker sessions, box.tasks accumulation, and app-level retention misconfigurations) — not by actual user traffic.

Note: these are cumulative iotop counters, not sustained rates. The actual average write speed shown by Cloudron's dashboard is ~2.5-4 MB/s, which still translates to 216-345 GB/day of unnecessary disk writes on a lightly loaded server.