Cloudron v9: huge disk I/O is this normal/safe/needed?

[imc67]

Description

I noticed since v9 on my 3 Cloudron-Pro instances with the new graphs that the Disk I/O is non-stop average at Write ~ 2.5 MB/s and one even at ~4MB/s. This seems to me extremely high as the 3 servers are IMHO not really highly productive. The total Write in 24 hours is about 4-7TB!!!

Steps to reproduce

Have a look in your graph

Cloudron Version

9.0.12 and since last night 9.0.13

Ubuntu Version

Ubuntu 22.04.5 LTS

[girish]

We have even more I doubt our server wrote or read so much. Must be something wrong with the reporting, investigating...

[imc67]

@girish said in Cloudron v9: huge disk I/O is this normal/safe/needed?:

We have even more I doubt our server wrote or read so much. Must be something wrong with the reporting, investigating...

But at least your Writ I/O graph/speed is almost zero?!

[imc67]

Also in the Netcup SCP it shows, so it seems there is seriously something wrong?

(The last part of the graph is not representative, zooming in to 6 hours is straight line)

[girish]

I checked many servers and it's mostly under 1MBps all the time . Does 'docker stats' show anything interesting (if it's a container that is hogging cpu)? Not sure what else you have installed on your server?

[girish]

Also, the total read/write counters are cumulative counters maintained by the kernel since boot time.

[imc67]

@girish said in Cloudron v9: huge disk I/O is this normal/safe/needed?:

'docker stats'

it's impossible to have a view with this, every second tens of docker containers are created (cron?) so it keeps listing and growing.

Is there a proper way to do some inspections with disk I/O in mind? Or shall I give you access to have a view?

[james]

Hello @imc67
Could it be that you simply need to zoom out of your terminal or make the window larger?
Had the same first thought when running this command but when resizing the window or zooming out:

[imc67]

@girish right but where to look?

[james]

Hello @imc67
Can you run the following command:

iotop -aoP

This gives a live view of what is currently writing IO to the disk.
Maybe this output can give some more indications where this is coming from.

[imc67]

its not default installed:

Command 'iotop' not found, but can be installed with:
apt install iotop    # version 0.6-24-g733f3f8-1.1ubuntu0.1, or
apt install iotop-c  # version 1.21-1

[nebulon]

yeah it is not installed by default, but you can safely install iotop via apt on your system.

[imc67]

and now?

[james]

Hello @imc67
You can either deselect it or press OK, restarting services should cause no issues.

[imc67]

ok thanks, below the result after just a few minutes, I'm not a technician but as far as I can see it's mainly mysql which is writing (I sorted Write):

[nebulon]

Since we debugged some other issue on that server, was also taking a look at the disk I/O. So basically the mysql service is doing a lot of disk I/O (also as see in the screenshot).

It does seem the mysql addon is just queried and written to a lot. So likely one of the many installed apps using it might commit a lot to the database. I didn't want to stop apps, but maybe you can try to stop individual apps which use mysql one-by-one to hopefully find the one which causes the constant writes.

[imc67]

Thanks @nebulon for your time, together with ChatGPT I did deeper analysis but I also read this: https://docs.cloudron.io/troubleshooting/#mysql

Two instances of MySQL
There are two instances of MySQL on Cloudron. One instance runs on the host and is used by the platform. Another instance is the MySQL addon which runs in a container named mysql and is shared by apps. This test is related to the host MySQL.

Doesn't this mean that the mysql service in iotop is the "host version" that has nothing to do with the apps?

For now "we" (I) have seen this:

Summary of Disk Write I/O Observation on Cloudron Host

• Using iotop, the host shows consistently high disk write I/O (4–5 MB/s).
• Analysis of MySQL processes (mysqld) indicates these are responsible for the majority of the write load.
• The high write I/O is primarily due to InnoDB internal activity: buffer pool flushes, redo log writes, and metadata updates, mostly from the box database (eventlog, tasks, backups).

In about 10 minutes this is the Disk Write I/O (so 1.5GB in 10 minutes)

Total DISK READ:         0.00 B/s | Total DISK WRITE:         2.73 M/s
Current DISK READ:       0.00 B/s | Current DISK WRITE:       4.25 M/s
    TID  PRIO  USER     DISK READ DISK WRITE>  SWAPIN      IO    COMMAND                                                                                                                  
  21250 be/4 messageb      0.00 B   1038.50 M  ?unavailable?  mysqld
    936 be/4 mysql         0.00 B    465.28 M  ?unavailable?  mysqld

I stopped about 25% of the apps at a certain moment with no significant result, this is the current situation (IMHO not really intensive application and they have low traffic):

App 	Status 
Yourls	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Stopped 
WordPress (Developer)	Running 
WordPress (Developer)	Stopped 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Running 
WordPress (Developer)	Stopped 
WordPress (Developer)	Running 
Taiga	Stopped 
Surfer	Running 
Surfer	Stopped 
Roundcube	Running 
Roundcube	Running 
Omeka S	Stopped 
Moodle	Stopped 
LAMP	Running 
Roundcube	Running 
Roundcube	Running 
Roundcube	Running 
Pretix	Stopped 
MiroTalk SFU	Running 
Matomo	Running 
FreeScout	Running 
FreeScout	Running 
Espo CRM	Running 

What to do next to find the root cause?

[avatar1024]

@imc67 said in Cloudron v9: huge disk I/O is this normal/safe/needed?:

I stopped about 25% of the apps at a certain moment with no significant result

I think @nebulon was suggesting to stop apps one by one to see if one particular app is causing the problem.

[robi]

Generally such system behavior is accompanied by higher CPU and Memory usage, so you can start with stopping those, and see which one causes a dip MySQL usage.

[imc67]

It’s a production server, isn’t it ridiculous to stop these apps to watch resource behavior? There must be tools or ways to find the root cause don’t you think?

Beside that it’s the host MySQL does it has anything to do with apps?