Sporadic server freeze

[hpz24]

Hi, I have the problem that my Cloudron server freezes again and again, but not regularly, and only works again with an externally triggered reboot (even ping to no longer respond). Always at night, around 3 a.m. (UTC)

Server runs at the provider, I have already asked there and the answer is:
During a detailed investigation, I could see that the CPU usage was triggered sometimes, which might have caused the server to go down for a moment.
Could you please optimize the process you are running from your end?

Sure

CPU-Usage in my dashboard:

My first suspicion was the backup, but that is created every Monday at 11pm, so no change to be responsible in that case.

I then looked through the journal logs and these are the last entries in the log. The next ones only after the reboot.

Dec 05 02:04:24 hostname syslog.js[867]: <30>1 2024-12-05T02:04:24Z hostname 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 1060 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 - [2024-12-05 02:04:24,547] [WARNING] [celery.redirected]
Dec 05 02:04:24 hostname syslog.js[867]: <30>1 2024-12-05T02:04:24Z hostname 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 1060 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 - [2024-12-05 02:04:24,547] [WARNING] [celery.redirected] self.dict[key.encode(self.keyencoding)] = f.getvalue()
Dec 05 02:04:24 hostname syslog.js[867]: <30>1 2024-12-05T02:04:24Z hostname 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 1060 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 - [2024-12-05 02:04:24,547] [WARNING] [celery.redirected] _dbm
Dec 05 02:04:24 hostname syslog.js[867]: <30>1 2024-12-05T02:04:24Z hostname 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 1060 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 - [2024-12-05 02:04:24,547] [WARNING] [celery.redirected] .
Dec 05 02:04:24 hostname syslog.js[867]: <30>1 2024-12-05T02:04:24Z hostname 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 1060 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 - [2024-12-05 02:04:24,547] [WARNING] [celery.redirected] error
Dec 05 02:04:24 hostname syslog.js[867]: <30>1 2024-12-05T02:04:24Z hostname 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 1060 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 - [2024-12-05 02:04:24,548] [WARNING] [celery.redirected] :
Dec 05 02:04:24 hostname syslog.js[867]: <30>1 2024-12-05T02:04:24Z hostname 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 1060 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 - [2024-12-05 02:04:24,548] [WARNING] [celery.redirected] cannot add item to database
Dec 05 02:04:24 hostname syslog.js[867]: 2024-12-05T02:04:24.966Z syslog:server Unable to parse: <30>1 2024-12-05T02:04:24Z hostname 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 1060 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 - celery beat v5.4.0 (opalescent) is starting.
Dec 05 02:04:24 hostname syslog.js[867]: <30>1 2024-12-05T02:04:24Z hostname 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 1060 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 - __    -    ... __   -        _
Dec 05 02:04:24 hostname syslog.js[867]: <30>1 2024-12-05T02:04:24Z hostname 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 1060 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 - LocalTime -> 2024-12-05 02:04:24
Dec 05 02:04:24 hostname syslog.js[867]: <30>1 2024-12-05T02:04:24Z hostname 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 1060 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 - Configuration ->
Dec 05 02:04:24 hostname syslog.js[867]: <30>1 2024-12-05T02:04:24Z hostname 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 1060 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 -     . broker -> redis://redis-9f58c89e-f08c-429a-8092-2cf7d2b6fe75:6379//
Dec 05 02:04:24 hostname syslog.js[867]: <30>1 2024-12-05T02:04:24Z hostname 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 1060 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 -     . loader -> celery.loaders.app.AppLoader
Dec 05 02:04:24 hostname syslog.js[867]: <30>1 2024-12-05T02:04:24Z hostname 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 1060 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 -     . scheduler -> celery.beat.PersistentScheduler
Dec 05 02:04:24 hostname syslog.js[867]: <30>1 2024-12-05T02:04:24Z hostname 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 1060 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 -     . db -> /app/data/data/celerybeat-schedule.db
Dec 05 02:04:24 hostname syslog.js[867]: <30>1 2024-12-05T02:04:24Z hostname 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 1060 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 -     . logfile -> [stderr]@%INFO
Dec 05 02:04:24 hostname syslog.js[867]: <30>1 2024-12-05T02:04:24Z hostname 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 1060 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 -     . maxinterval -> 5.00 minutes (300s)
Dec 05 02:04:25 hostname syslog.js[867]: 2024-12-05T02:04:25.502Z syslog:server Unable to parse: <30>1 2024-12-05T02:04:25Z hostname 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 1060 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 - 2024-12-05 02:04:25,500 INFO exited: paperless-scheduler.service (exit status 1; not expected)
Dec 05 02:04:26 hostname syslog.js[867]: 2024-12-05T02:04:26.510Z syslog:server Unable to parse: <30>1 2024-12-05T02:04:26Z hostname 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 1060 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 - 2024-12-05 02:04:26,508 INFO spawned: 'paperless-scheduler.service' with pid 3032591
Dec 05 02:04:27 hostname syslog.js[867]: 2024-12-05T02:04:27.512Z syslog:server Unable to parse: <30>1 2024-12-05T02:04:27Z hostname 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 1060 9f58c89e-f08c-429a-8092-2cf7d2b6fe75 - 2024-12-05 02:04:27,511 INFO success: paperless-scheduler.service entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

I'm at a bit of a loss: The server is otherwise running without any problems.
Where else can I troubleshoot?
Thank you!

[nebulon]

Since paperless shows up here, maybe you hit the same issue as https://forum.cloudron.io/topic/12195/high-cpu-utilization-since-update-1-25-1/29 ?

[hpz24]

Thx, I´ve look into that, it looks like the same!