Cloudron services are slow

[nebulon]

Maybe you can get some better immediate overview when running htop or top via SSH to see if systemload is maybe high, in which case the system may just be underpowered for the apps you are running. Also sometimes disk I/O is an important factor.

[robi]

Also look at the disk IO on the VPS next to the CPU graphs. If it's under 300, it may be overloaded.

[timconsidine]

Errr it’s your server @KhalilZammeli !
Not being awkward but you can’t really post a question like this one without information on the server, RAM, CPU, disk space total and free, location (not sure VPS on the end of starlink in Antarctica will be great)

[KhalilZammeli]

Sorry for the lack of info.

Here are some additional info gathered by the team:

Platform Versionv8.2.3 (Ubuntu 24.04.1 LTS)
VendorQEMU
ProductStandard PC (i440FX + PIIX, 1996)
CPU6 Core "AMD EPYC 7282 16-Core Processor"
Memory16.77 GB RAM & 4.29 GB Swap

We noticed something is related to MySQL?

[robi]

Please post the Cloudron top right Menu > System Info view

Your DB is super busy for some reason as well. So include the > Services view

[nebulon]

@KhalilZammeli looking at your screenshot, the user of that very busy mysql instance seems to be some pollina+ which I have no clue what that would be. At least this is not coming from any Cloudron bits. Have you installed anything on the side on this server, which may be causing that?

[timconsidine]

https://github.com/pollina-s/mysql_project ??

But that doesn’t give much info so I’m not sure

[KhalilZammeli]

Hello Cloudron Team,

After further investigation into the server performance issues, we've identified some suspicious activity that may not be related to Cloudron itself. Here's a summary of our findings:

1. System and Security Checks Performed:
   • We confirmed that no software other than Cloudron was intentionally installed before today.
   • We reviewed recent command history to ensure no unintended actions were performed manually.
2. Unrecognized User Activity:
   • Initially, we noticed a MySQL instance running under the user pollinate, which is not part of Cloudron’s standard services.
   • Upon further analysis, we found that after removing the pollinate user, a new user with UID 106 took over running similar services, including MySQL, PostgreSQL, and MongoDB.
   • Most recently, these processes have transitioned to running under the netdata user, which we installed today for monitoring purposes.
     • is it possible it’s a malicious entity changing user each time ?
3. System Observations:
   • We have gathered the following screenshots for further analysis:

     • Service View Screenshot: (Attach the image)
       

     • System Info Screenshot: (Attach the image)
       

     • Top Command Output Screenshot: (Attach the image)
       

       update of the screenshot for our wordpress service, very high CPU usage, how can we investigate these ?
       

Next Steps:

• We're investigating if the unauthorized services were installed via a privilege escalation method.
• We're considering reinstallation of Cloudron on a fresh system if no other resolution is found.

Any insights or suggestions from the Cloudron team would be greatly appreciated.

Thank you for your continued support!

[jdaviescoates]

I did wonder earlier if a hacked instance of WordPress (or even just WordPress malicious or badly designed plugin) could potentially be the culprit. Seems worth investigating.

[joseph]

You can also use docker stats to check which container is taking CPU.

For the confusing user names: top command has no idea about containers. A uid in container will just appear as uid in host and top will blindly translate the uid not realizing that the uid is for the container. This is why you see mysqld running as netdtaa. Most likely mysqld is running in the mysql container and the mysqld user of container is mapped as netdata in host (i.e their uids are the same). See something like pstree to make out the pids

[nebulon]

Sorry for the mysql user confusion. I had not realized that Ubuntu does come with a pollinate user through the pollinate package by default!

So then the question comes back to finding out which app seems to utilize the mysql service that heavily. Maybe you can also stop one app after the other to find out which one is the one causing that.

[jdaviescoates]

@nebulon

@KhalilZammeli said in Cloudron services are slow:

update of the screenshot for our wordpress service, very high CPU usage, how can we investigate these ?

Given the above and the fact WordPress uses mysql (and is very often attacked as it's so widely used, and malicious plugins are also quite common) that seem to me to be the obvious first app to investigate.

[micmc]

@jdaviescoates said in Cloudron services are slow:

@nebulon

@KhalilZammeli said in Cloudron services are slow:

update of the screenshot for our wordpress service, very high CPU usage, how can we investigate these ?

Given the above and the fact WordPress uses mysql (and is very often attacked as it's so widely used, and malicious plugins are also quite common) that seem to me to be the obvious first app to investigate.

Absolutely! Most of the time a WP plugin is the culprit. A WP plugin that's been acquired through a 'friend' which is not the original developer. Extremely, dangerous these are compromised in 99% of the times.