High CPU Usage by www-data php -f app/code/cron.php after Cloudron Migration

[girish]

@archos do you know which app this is? I don't recall any specific app having a cron.php . You can find out using systemd-cgls and systemd-cgtop commands.

[archos]

@girish
Thank you for the response.

I used the commands and found that the container docker/50a7ac5a21a7a92e0d7e8e846625f2285167801b6e30c66ad45fcc827af8d7d1 has the highest CPU usage. I used docker inspect and discovered that the container is using the image cloudron/com.nextcloud.cloudronapp, which suggests it is a Nextcloud application.

Additionally, on the new server, I have an HDD connected via the filesystem /mnt/data, where I have mounted the Nextcloud data. Could the issue be related to this?

[girish]

@archos I wonder if it's busy thumbnailing or something post migration Is there any hint as to what the cron job is doing?

[girish]

The log file of nextcloud is at /run/nextcloud/nextcloud.log inside the container . Maybe there is also some errors in the Admin view of nextcloud itself (I think it's under system or something)

[archos]

@girish
In the /run/nextcloud/nextcloud.log file, I found the following entries:

"reqId":"gOoHxX7NY8BTxEzZ5VyM","level":3,"time":"2024-05-30T18:34:31+00:00","remoteAddr":"172.18.0.1","user":"--","app":"PHP","method":"GET","url":"/index.php/login","message":"Optional parameter $userId dec>
{"reqId":"N8nYMVeRlzkdKH7QYp4a","level":3,"time":"2024-05-30T18:59:34+00:00","remoteAddr":"31.220.75.8","user":"--","app":"no app in context","method":"GET","url":"/index.php/apps/files/api/v1/thumbnail/256/2>
{"reqId":"aMRHN0Pc346zRht23UE2","level":3,"time":"2024-05-30T18:59:34+00:00","remoteAddr":"31.220.75.8","user":"--","app":"no app in context","method":"GET","url":"/index.php/apps/files/api/v1/thumbnail/256/2>
{"reqId":"XPD5Qv09LbyZDHo54rSj","level":3,"time":"2024-05-30T18:59:35+00:00","remoteAddr":"31.220.75.8","user":"--","app":"core","method":"GET","url":"/index.php/apps/files/api/v1/thumbnail/256/256/Nextcloud%>
{"reqId":"KETPR0CXBY6eDIdQVGLA","level":3,"time":"2024-05-30T18:59:35+00:00","remoteAddr":"31.220.75.8","user":"--","app":"core","method":"GET","url":"/index.php/apps/files/api/v1/thumbnail/256/256/Readme.md">
{"reqId":"xnJGRyXLgykGvlEUSePr","level":3,"time":"2024-05-30T19:06:43+00:00","remoteAddr":"31.220.75.8","user":"--","app":"no app in context","method":"GET","url":"/index.php/apps/files/api/v1/thumbnail/256/2>
{"reqId":"oZvewwEmJALYOMXtMamf","level":3,"time":"2024-05-30T19:06:51+00:00","remoteAddr":"31.220.75.8","user":"--","app":"no app in context","method":"GET","url":"/index.php/apps/files/api/v1/thumbnail/256/2>
{"reqId":"F4WyT2QAbT36igHAnyLN","level":3,"time":"2024-05-30T19:07:04+00:00","remoteAddr":"31.220.75.8","user":"--","app":"no app in context","method":"PROPFIND","url":"/remote.php/dav/files/zde/Insta>
{"reqId":"4tzlclGhXjdZoeGR0GXI","level":3,"time":"2024-05-30T19:07:06+00:00","remoteAddr":"31.220.75.8","user":"--","app":"no app in context","method":"GET","url":"/index.php/apps/files/api/v1/thumbnail/256/2>
{"reqId":"bS32DtpCm5SALP4Y3h8m","level":3,"time":"2024-05-30T19:07:16+00:00","remoteAddr":"31.220.75.8","user":"--","app":"no app in context","method":"GET","url":"/index.php/apps/files/api/v1/thumbnail/256/2>
{"reqId":"WFNbvM7iKNtSnHyzSmyY","level":3,"time":"2024-05-30T19:07:26+00:00","remoteAddr":"31.220.75.8","user":"--","app":"no app in context","method":"PROPFIND","url":"/remote.php/dav/files/zdenekkreml/Insta>
{"reqId":"KPWPjkSen6S7zdDLMptF","level":3,"time":"2024-05-30T19:07:28+00:00","remoteAddr":"31.220.75.8","user":"--","app":"no app in context","method":"GET","url":"/index.php/apps/files/api/v1/thumbnail/256/2>
{"reqId":"IvKkTVH8cBrrlrAJLWM7","level":3,"time":"2024-05-30T19:07:35+00:00","remoteAddr":"31.220.75.8","user":"--","app":"no app in context","method":"PROPFIND","url":"/remote.php/dav/files/zdenekkreml/Insta>
{"reqId":"457bZ1rwsPlbOiQ924LS","level":3,"time":"2024-05-30T19:07:37+00:00","remoteAddr":"31.220.75.8","user":"--","app":"no app in context","method":"GET","url":"/index.php/apps/files/api/v1/thumbnail/256/2>
{"reqId":"sLUTrOfVwp6Kb4qHNaFc","level":3,"time":"2024-05-30T19:07:45+00:00","remoteAddr":"31.220.75.8","user":"--","app":"no app in context","method":"GET","url":"/status.php","message":"Could not decrypt or>
{"reqId":"jGL241edGUMd5RIWufU1","level":3,"time":"2024-05-30T19:09:00+00:00","remoteAddr":"31.220.75.8","user":"--","app":"no app in context","method":"GET","url":"/index.php/apps/files/api/v1/thumbnail/256/2>
{"reqId":"DfwchaHbbQaANW1nVxjo","level":3,"time":"2024-05-30T19:09:13+00:00","remoteAddr":"31.220.75.8","user":"--","app":"no app in context","method":"GET","url":"/index.php/apps/files/api/v1/thumbnail/256/2>
{"reqId":"hv0MtsGWldKlWlZ32JNj","level":3,"time":"2024-05-30T19:13:59+00:00","remoteAddr":"31.220.75.8","user":"--","app":"no app in context","method":"GET","url":"/index.php/apps/files/api/v1/thumbnail/256/2>
{"reqId":"MdJX3M7UFfjthQWN5GPz","level":3,"time":"2024-05-30T19:14:00+00:00","remoteAddr":"31.220.75.8","user":"--","app":"no app in context","method":"GET","url":"/status.php","message":"Could not decrypt or>
{"reqId":"EVGReqYQcc4wKsK4Z8HR","level":3,"time":"2024-05-31T01:00:33+00:00","remoteAddr":"","user":"--","app":"suspicious_login","method":"","url":"--","message":"Caught unknown error during IPv4 background >
{"reqId":"FAijjY3308XoKUhQcvTR","level":3,"time":"2024-05-31T05:58:31+00:00","remoteAddr":"172.18.0.1","user":"--","app":"PHP","method":"GET","url":"/index.php/login","message":"Optional parameter $userId dec>
{"reqId":"pY0CXxAiqyXn6nIh9lVH","level":3,"time":"2024-05-31T06:05:12+00:00","remoteAddr":"31.220.75.8","user":"zdenekkreml","app":"jsresourceloader","method":"GET","url":"/settings/user/sync-clients","messag>
{"reqId":"OG6CHHlWo1Rb2JbidYn2","level":3,"time":"2024-05-31T06:06:46+00:00","remoteAddr":"31.220.75.8","user":"zdenekkreml","app":"PHP","method":"GET","url":"/settings/user","message":"Undefined array key 1 >
{"reqId":"YNnUOn7bP0m0u9mNDS2R","level":3,"time":"2024-05-31T06:14:21+00:00","remoteAddr":"","user":"--","app":"core","method":"","url":"--","message":"Error while running background job OC\\FilesMetadata\\Jo>
{"reqId":"YNnUOn7bP0m0u9mNDS2R","level":3,"time":"2024-05-31T06:14:21+00:00","remoteAddr":"","user":"--","app":"cron","method":"","url":"--","message":"An exception occurred while executing a query: SQLSTATE[>
{"reqId":"oOyNuORofriqhF4tbgVW","level":3,"time":"2024-05-31T06:25:41+00:00","remoteAddr":"172.18.0.1","user":"--","app":"PHP","method":"GET","url":"/index.php/login","message":"Optional parameter $userId dec>

[archos]

Here are some additional errors from the Nextcloud administration:

[archos]

I don't know if this is normal, but in the system info, the connected HDD changes color, sometimes it's red and sometimes it's light blue.

[girish]

I can't make out from the logs atleast what the cron job is doing. Maybe kill that process and see if it always hangs. Otherwise, we might have to ask upstream to figure what that process is doing.

[archos]

@girish May I contact you, please? I'm not sure how to kill the process and what exactly to monitor. Thank you very much.

[girish]

@archos you can just ssh and kill -9 pid (pid is 398333 in your screenshot)

Of course, you are welcome to contact on support@cloudron.io too.

[archos]

I tried to terminate the process using the command kill -9 389333, but it seems it has already been terminated, as the error bash: kill: (389333) - No such process appeared. Now the process /usr/sbin/Apache2- DFOREGROUND is using around 50% of the CPU, but it’s not as bad as it was in the morning. I’ll monitor it for a while and will provide further updates.

[archos]

Hello,

I have been monitoring the situation for a few days, but the process /usr/sbin/Apache2 -DFOREGROUND is still consuming too much CPU, around 50%. Additionally, there are now 11 errors appearing in the Nextcloud administration settings.

[girish]

For the "suspicious login" plugin atleast, maybe try reporting it upstream . Just make sure it's up to date before reporting upstream . There are some issues upstream already - https://github.com/search?q=repo%3Anextcloud%2Fsuspicious_login+caught+unknown+error+ipv4+background+training&type=issues

The other one comes from some fediverse thing in nextcloud ...