Onlyoffice high CPU usage

[im-fabian]

I had to split the logs into two posts because the post have been too long otherwise:

Feb 26 13:34:06## ## RabbitMQ 3.9.13
Feb 26 13:34:06## ##
Feb 26 13:34:06########## Copyright (c) 2007-2022 VMware, Inc. or its affiliates.
Feb 26 13:34:06###### ##
Feb 26 13:34:06########## Licensed under the MPL 2.0. Website: https://rabbitmq.com
Feb 26 13:34:06
Feb 26 13:34:06Erlang: 24.2.1 [jit]
Feb 26 13:34:06TLS Library: OpenSSL - OpenSSL 3.0.2 15 Mar 2022
Feb 26 13:34:06
Feb 26 13:34:06Doc guides: https://rabbitmq.com/documentation.html
Feb 26 13:34:06Support: https://rabbitmq.com/contact.html
Feb 26 13:34:06Tutorials: https://rabbitmq.com/getstarted.html
Feb 26 13:34:06Monitoring: https://rabbitmq.com/monitoring.html
Feb 26 13:34:06
Feb 26 13:34:06Logs: /var/log/rabbitmq/rabbit@c03a8f57-4218-49ed-9462-69e17e7cf0ad.log
Feb 26 13:34:06/var/log/rabbitmq/rabbit@c03a8f57-4218-49ed-9462-69e17e7cf0ad_upgrade.log
Feb 26 13:34:06<stdout>
Feb 26 13:34:06
Feb 26 13:34:06Config file(s): (none)
Feb 26 13:34:06
Feb 26 13:34:06Starting broker...[2024-02-26T12:34:06.343] [WARN] [localhost] [docId] [userId] nodeJS - worker 481 started.
Feb 26 13:34:062024-02-26 12:34:06,344 INFO reaped unknown pid 346 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,344 INFO reaped unknown pid 346 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,344 INFO reaped unknown pid 363 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,344 INFO reaped unknown pid 363 (exit status 0)
Feb 26 13:34:06[2024-02-26T12:34:06.347] [WARN] [localhost] [docId] [userId] nodeJS - worker 482 started.
Feb 26 13:34:06[2024-02-26T12:34:06.351] [WARN] [localhost] [docId] [userId] nodeJS - worker 495 started.
Feb 26 13:34:06[2024-02-26T12:34:06.357] [WARN] [localhost] [docId] [userId] nodeJS - worker 515 started.
Feb 26 13:34:06[2024-02-26T12:34:06.363] [WARN] [localhost] [docId] [userId] nodeJS - worker 521 started.
Feb 26 13:34:06[2024-02-26T12:34:06.371] [WARN] [localhost] [docId] [userId] nodeJS - worker 527 started.
Feb 26 13:34:062024-02-26 12:34:06,372 INFO reaped unknown pid 381 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,372 INFO reaped unknown pid 381 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,372 INFO reaped unknown pid 387 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,372 INFO reaped unknown pid 387 (exit status 0)
Feb 26 13:34:06[2024-02-26T12:34:06.380] [WARN] [localhost] [docId] [userId] nodeJS - worker 533 started.
Feb 26 13:34:062024-02-26 12:34:06,380 INFO reaped unknown pid 340 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,380 INFO reaped unknown pid 340 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,380 INFO reaped unknown pid 375 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,380 INFO reaped unknown pid 375 (exit status 0)
Feb 26 13:34:06[2024-02-26T12:34:06.386] [WARN] [localhost] [docId] [userId] nodeJS - worker 534 started.
Feb 26 13:34:06[2024-02-26T12:34:06.393] [WARN] [localhost] [docId] [userId] nodeJS - worker 545 started.
Feb 26 13:34:06[2024-02-26T12:34:06.400] [WARN] [localhost] [docId] [userId] nodeJS - worker 552 started.
Feb 26 13:34:06[2024-02-26T12:34:06.408] [WARN] [localhost] [docId] [userId] nodeJS - worker 561 started.
Feb 26 13:34:06[2024-02-26T12:34:06.416] [WARN] [localhost] [docId] [userId] nodeJS - worker 569 started.
Feb 26 13:34:06[2024-02-26T12:34:06.427] [WARN] [localhost] [docId] [userId] nodeJS - worker 575 started.
Feb 26 13:34:06[2024-02-26T12:34:06.440] [WARN] [localhost] [docId] [userId] nodeJS - worker 581 started.
Feb 26 13:34:06[2024-02-26T12:34:06.455] [WARN] [localhost] [docId] [userId] nodeJS - worker 591 started.
Feb 26 13:34:06[2024-02-26T12:34:06.465] [WARN] [localhost] [docId] [userId] nodeJS - worker 597 started.
Feb 26 13:34:06[2024-02-26T12:34:06.474] [WARN] [localhost] [docId] [userId] nodeJS - worker 603 started.
Feb 26 13:34:062024-02-26 12:34:06,475 INFO reaped unknown pid 390 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,475 INFO reaped unknown pid 390 (exit status 0)
Feb 26 13:34:06[2024-02-26T12:34:06.487] [WARN] [localhost] [docId] [userId] nodeJS - worker 609 started.
Feb 26 13:34:06[2024-02-26T12:34:06.503] [WARN] [localhost] [docId] [userId] nodeJS - worker 615 started.
Feb 26 13:34:06[2024-02-26T12:34:06.519] [WARN] [localhost] [docId] [userId] nodeJS - worker 621 started.
Feb 26 13:34:06[2024-02-26T12:34:06.530] [WARN] [localhost] [docId] [userId] nodeJS - worker 627 started.
Feb 26 13:34:06[2024-02-26T12:34:06.541] [WARN] [localhost] [docId] [userId] nodeJS - worker 631 started.
Feb 26 13:34:06[2024-02-26T12:34:06.559] [WARN] [localhost] [docId] [userId] nodeJS - worker 634 started.
Feb 26 13:34:06[2024-02-26T12:34:06.584] [WARN] [localhost] [docId] [userId] nodeJS - worker 645 started.
Feb 26 13:34:06[2024-02-26T12:34:06.601] [WARN] [localhost] [docId] [userId] nodeJS - worker 651 started.
Feb 26 13:34:06[2024-02-26T12:34:06.615] [WARN] [localhost] [docId] [userId] nodeJS - worker 652 started.
Feb 26 13:34:06[2024-02-26T12:34:06.629] [WARN] [localhost] [docId] [userId] nodeJS - worker 663 started.
Feb 26 13:34:06[2024-02-26T12:34:06.644] [WARN] [localhost] [docId] [userId] nodeJS - worker 669 started.
Feb 26 13:34:07Killed
Feb 26 13:34:072024-02-26 12:34:07,626 INFO reaped unknown pid 22 (exit status 0)
Feb 26 13:34:072024-02-26 12:34:07,626 INFO reaped unknown pid 22 (exit status 0)
Feb 26 13:34:072024-02-26 12:34:07,635 INFO exited: converter (terminated by SIGKILL; not expected)
Feb 26 13:34:072024-02-26 12:34:07,635 INFO exited: converter (terminated by SIGKILL; not expected)
Feb 26 13:34:072024-02-26 12:34:07,640 INFO exited: docservice (terminated by SIGKILL; not expected)
Feb 26 13:34:072024-02-26 12:34:07,640 INFO exited: docservice (terminated by SIGKILL; not expected)
Feb 26 13:34:072024-02-26 12:34:07,640 INFO reaped unknown pid 44 (exit status 0)
Feb 26 13:34:072024-02-26 12:34:07,640 INFO reaped unknown pid 44 (exit status 0)
Feb 26 13:34:072024-02-26 12:34:07,640 INFO reaped unknown pid 152 (exit status 0)
Feb 26 13:34:072024-02-26 12:34:07,640 INFO reaped unknown pid 152 (exit status 0)
Feb 26 13:34:072024-02-26 12:34:07,773 INFO spawned: 'converter' with pid 675
Feb 26 13:34:072024-02-26 12:34:07,773 INFO spawned: 'converter' with pid 675
Feb 26 13:34:072024-02-26 12:34:07,893 INFO spawned: 'docservice' with pid 676
Feb 26 13:34:072024-02-26 12:34:07,893 INFO spawned: 'docservice' with pid 676
Feb 26 13:34:082024-02-26 12:34:08,994 INFO success: converter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:34:082024-02-26 12:34:08,994 INFO success: converter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:34:082024-02-26 12:34:08,996 INFO success: docservice entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:34:082024-02-26 12:34:08,996 INFO success: docservice entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:34:082024-02-26 12:34:08,996 INFO reaped unknown pid 521 (terminated by SIGKILL)
Feb 26 13:34:082024-02-26 12:34:08,996 INFO reaped unknown pid 521 (terminated by SIGKILL)
Feb 26 13:34:102024-02-26 12:34:10,009 INFO reaped unknown pid 482 (terminated by SIGKILL)
Feb 26 13:34:102024-02-26 12:34:10,009 INFO reaped unknown pid 482 (terminated by SIGKILL)
Feb 26 13:34:102024/02/26 12:34:10 [error] 173#173: *4 connect() failed (111: Unknown error) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /healthcheck HTTP/1.1", upstream: "http://127.0.0.1:8000/healthcheck", host: "onlyoffice.DOMAINNAME.TLD"
Feb 26 13:34:102024/02/26 12:34:10 [error] 173#173: *4 connect() failed (111: Unknown error) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /healthcheck HTTP/1.1", upstream: "http://[::1]:8000/healthcheck", host: "onlyoffice.DOMAINNAME.TLD"
Feb 26 13:34:10172.18.0.1 - - [26/Feb/2024:12:34:10 +0000] "GET /healthcheck HTTP/1.1" 502 150 "-" "Mozilla (CloudronHealth)"
Feb 26 13:34:10=> Healtheck error got response status 502
Feb 26 13:34:102024-02-26 12:34:10,462 INFO reaped unknown pid 527 (terminated by SIGKILL)
Feb 26 13:34:102024-02-26 12:34:10,462 INFO reaped unknown pid 527 (terminated by SIGKILL)
Feb 26 13:34:192024-02-26 12:34:19,472 INFO reaped unknown pid 515 (terminated by SIGKILL)
Feb 26 13:34:192024-02-26 12:34:19,472 INFO reaped unknown pid 515 (terminated by SIGKILL)
Feb 26 13:34:202024/02/26 12:34:20 [error] 173#173: *7 connect() failed (111: Unknown error) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /healthcheck HTTP/1.1", upstream: "http://[::1]:8000/healthcheck", host: "onlyoffice.DOMAINNAME.TLD"
Feb 26 13:34:202024/02/26 12:34:20 [error] 173#173: *7 connect() failed (111: Unknown error) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /healthcheck HTTP/1.1", upstream: "http://127.0.0.1:8000/healthcheck", host: "onlyoffice.DOMAINNAME.TLD"
Feb 26 13:34:20172.18.0.1 - - [26/Feb/2024:12:34:20 +0000] "GET /healthcheck HTTP/1.1" 502 150 "-" "Mozilla (CloudronHealth)"
Feb 26 13:34:20=> Healtheck error got response status 502
Feb 26 13:34:212024-02-26 12:34:21,220 INFO reaped unknown pid 469 (terminated by SIGKILL)
Feb 26 13:34:212024-02-26 12:34:21,220 INFO reaped unknown pid 469 (terminated by SIGKILL)
Feb 26 13:34:212024-02-26 12:34:21,220 INFO reaped unknown pid 475 (terminated by SIGKILL)
Feb 26 13:34:212024-02-26 12:34:21,220 INFO reaped unknown pid 475 (terminated by SIGKILL)
Feb 26 13:34:232024-02-26 12:34:23,224 INFO reaped unknown pid 441 (terminated by SIGKILL)
Feb 26 13:34:232024-02-26 12:34:23,224 INFO reaped unknown pid 441 (terminated by SIGKILL)
Feb 26 13:34:232024-02-26 12:34:23,225 INFO reaped unknown pid 481 (terminated by SIGKILL)
Feb 26 13:34:232024-02-26 12:34:23,225 INFO reaped unknown pid 481 (terminated by SIGKILL)
Feb 26 13:34:232024-02-26 12:34:23,225 INFO reaped unknown pid 534 (terminated by SIGKILL)
Feb 26 13:34:232024-02-26 12:34:23,225 INFO reaped unknown pid 534 (terminated by SIGKILL)
Feb 26 13:34:242024-02-26 12:34:24,264 INFO reaped unknown pid 609 (terminated by SIGKILL)
Feb 26 13:34:242024-02-26 12:34:24,264 INFO reaped unknown pid 609 (terminated by SIGKILL)
Feb 26 13:34:252024-02-26 12:34:25,312 INFO reaped unknown pid 561 (terminated by SIGKILL)
Feb 26 13:34:252024-02-26 12:34:25,312 INFO reaped unknown pid 561 (terminated by SIGKILL)
Feb 26 13:34:252024-02-26 12:34:25,312 INFO reaped unknown pid 627 (terminated by SIGKILL)
Feb 26 13:34:252024-02-26 12:34:25,312 INFO reaped unknown pid 627 (terminated by SIGKILL)
Feb 26 13:34:262024-02-26 12:34:26,330 INFO reaped unknown pid 495 (terminated by SIGKILL)
Feb 26 13:34:262024-02-26 12:34:26,330 INFO reaped unknown pid 495 (terminated by SIGKILL)
Feb 26 13:34:262024-02-26 12:34:26,330 INFO reaped unknown pid 552 (terminated by SIGKILL)
Feb 26 13:34:262024-02-26 12:34:26,330 INFO reaped unknown pid 552 (terminated by SIGKILL)
Feb 26 13:34:262024-02-26 12:34:26,330 INFO reaped unknown pid 603 (terminated by SIGKILL)
Feb 26 13:34:262024-02-26 12:34:26,330 INFO reaped unknown pid 603 (terminated by SIGKILL)
Feb 26 13:34:272024-02-26 12:34:27,349 INFO reaped unknown pid 597 (terminated by SIGKILL)
Feb 26 13:34:272024-02-26 12:34:27,349 INFO reaped unknown pid 597 (terminated by SIGKILL)
Feb 26 13:34:302024/02/26 12:34:30 [error] 173#173: *10 connect() failed (111: Unknown error) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /healthcheck HTTP/1.1", upstream: "http://127.0.0.1:8000/healthcheck", host: "onlyoffice.DOMAINNAME.TLD"
Feb 26 13:34:302024/02/26 12:34:30 [error] 173#173: *10 connect() failed (111: Unknown error) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /healthcheck HTTP/1.1", upstream: "http://[::1]:8000/healthcheck", host: "onlyoffice.DOMAINNAME.TLD"
Feb 26 13:34:30=> Healtheck error got response status 502
Feb 26 13:34:30172.18.0.1 - - [26/Feb/2024:12:34:30 +0000] "GET /healthcheck HTTP/1.1" 502 150 "-" "Mozilla (CloudronHealth)"
Feb 26 13:34:422024/02/26 12:34:42 [error] 173#173: *13 connect() failed (111: Unknown error) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /healthcheck HTTP/1.1", upstream: "http://[::1]:8000/healthcheck", host: "onlyoffice.DOMAINNAME.TLD"
Feb 26 13:34:422024/02/26 12:34:42 [error] 173#173: *13 connect() failed (111: Unknown error) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /healthcheck HTTP/1.1", upstream: "http://127.0.0.1:8000/healthcheck", host: "onlyoffice.DOMAINNAME.TLD"
Feb 26 13:34:42172.18.0.1 - - [26/Feb/2024:12:34:42 +0000] "GET /healthcheck HTTP/1.1" 502 150 "-" "Mozilla (CloudronHealth)"
Feb 26 13:34:42=> Healtheck error got response status 502
Feb 26 13:34:42[2024-02-26T12:34:42.446] [WARN] [localhost] [docId] [userId] nodeJS - num of CPUs: 32; availableParallelism: undefined
Feb 26 13:34:422024-02-26 12:34:42,449 INFO reaped unknown pid 533 (terminated by SIGKILL)
Feb 26 13:34:422024-02-26 12:34:42,449 INFO reaped unknown pid 533 (terminated by SIGKILL)
Feb 26 13:34:422024-02-26 12:34:42,449 INFO reaped unknown pid 575 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,449 INFO reaped unknown pid 575 (exit status 0)
Feb 26 13:34:42[2024-02-26T12:34:42.464] [WARN] [localhost] [docId] [userId] nodeJS - update cluster with 32 workers
Feb 26 13:34:422024-02-26 12:34:42,464 INFO reaped unknown pid 569 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,464 INFO reaped unknown pid 569 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,464 INFO reaped unknown pid 615 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,464 INFO reaped unknown pid 615 (exit status 0)
Feb 26 13:34:42[2024-02-26T12:34:42.478] [WARN] [localhost] [docId] [userId] nodeJS - worker 707 started.
Feb 26 13:34:42[2024-02-26T12:34:42.485] [WARN] [localhost] [docId] [userId] nodeJS - worker 708 started.
Feb 26 13:34:422024-02-26 12:34:42,485 INFO reaped unknown pid 581 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,485 INFO reaped unknown pid 581 (exit status 0)
Feb 26 13:34:42[2024-02-26T12:34:42.491] [WARN] [localhost] [docId] [userId] nodeJS - worker 719 started.
Feb 26 13:34:42[2024-02-26T12:34:42.495] [WARN] [localhost] [docId] [userId] nodeJS - worker 720 started.
Feb 26 13:34:42[2024-02-26T12:34:42.501] [WARN] [localhost] [docId] [userId] nodeJS - worker 726 started.
Feb 26 13:34:42[2024-02-26T12:34:42.511] [WARN] [localhost] [docId] [userId] nodeJS - worker 737 started.
Feb 26 13:34:42[2024-02-26T12:34:42.519] [WARN] [localhost] [docId] [userId] nodeJS - worker 751 started.
Feb 26 13:34:42[2024-02-26T12:34:42.529] [WARN] [localhost] [docId] [userId] nodeJS - worker 757 started.
Feb 26 13:34:42[2024-02-26T12:34:42.536] [WARN] [localhost] [docId] [userId] nodeJS - worker 762 started.
Feb 26 13:34:422024-02-26 12:34:42,536 INFO reaped unknown pid 545 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,536 INFO reaped unknown pid 545 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,537 INFO reaped unknown pid 631 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,537 INFO reaped unknown pid 631 (exit status 0)
Feb 26 13:34:42[2024-02-26T12:34:42.547] [WARN] [localhost] [docId] [userId] nodeJS - worker 773 started.
Feb 26 13:34:42[2024-02-26T12:34:42.558] [WARN] [localhost] [docId] [userId] nodeJS - worker 783 started.
Feb 26 13:34:42[2024-02-26T12:34:42.566] [WARN] [localhost] [docId] [userId] nodeJS - worker 793 started.
Feb 26 13:34:422024-02-26 12:34:42,567 INFO reaped unknown pid 651 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,567 INFO reaped unknown pid 651 (exit status 0)
Feb 26 13:34:42[2024-02-26T12:34:42.578] [WARN] [localhost] [docId] [userId] nodeJS - worker 803 started.
Feb 26 13:34:42[2024-02-26T12:34:42.590] [WARN] [localhost] [docId] [userId] nodeJS - worker 813 started.
Feb 26 13:34:422024-02-26 12:34:42,590 INFO reaped unknown pid 591 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,590 INFO reaped unknown pid 591 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,591 INFO reaped unknown pid 634 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,591 INFO reaped unknown pid 634 (exit status 0)
Feb 26 13:34:42[2024-02-26T12:34:42.603] [WARN] [localhost] [docId] [userId] nodeJS - worker 827 started.
Feb 26 13:34:422024-02-26 12:34:42,603 INFO reaped unknown pid 663 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,603 INFO reaped unknown pid 663 (exit status 0)
Feb 26 13:34:42[2024-02-26T12:34:42.613] [WARN] [localhost] [docId] [userId] nodeJS - worker 833 started.
Feb 26 13:34:42[2024-02-26T12:34:42.624] [WARN] [localhost] [docId] [userId] nodeJS - worker 843 started.
Feb 26 13:34:422024-02-26 12:34:42,625 INFO reaped unknown pid 458 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,625 INFO reaped unknown pid 458 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,625 INFO reaped unknown pid 645 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,625 INFO reaped unknown pid 645 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,625 INFO reaped unknown pid 669 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,625 INFO reaped unknown pid 669 (exit status 0)
Feb 26 13:34:42[2024-02-26T12:34:42.634] [WARN] [localhost] [docId] [userId] nodeJS - worker 849 started.
Feb 26 13:34:42[2024-02-26T12:34:42.640] [WARN] [localhost] [docId] [userId] nodeJS - worker 855 started.
Feb 26 13:34:422024-02-26 12:34:42,641 INFO reaped unknown pid 621 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,641 INFO reaped unknown pid 621 (exit status 0)
Feb 26 13:34:42[2024-02-26T12:34:42.645] [WARN] [localhost] [docId] [userId] nodeJS - worker 856 started.
Feb 26 13:34:42[2024-02-26T12:34:42.650] [WARN] [localhost] [docId] [userId] nodeJS - worker 862 started.
Feb 26 13:34:42[2024-02-26T12:34:42.659] [WARN] [localhost] [docId] [userId] nodeJS - worker 868 started.
Feb 26 13:34:42[2024-02-26T12:34:42.674] [WARN] [localhost] [docId] [userId] nodeJS - worker 879 started.
Feb 26 13:34:42[2024-02-26T12:34:42.685] [WARN] [localhost] [docId] [userId] nodeJS - worker 885 started.
Feb 26 13:34:42[2024-02-26T12:34:42.697] [WARN] [localhost] [docId] [userId] nodeJS - worker 891 started.
Feb 26 13:34:42[2024-02-26T12:34:42.714] [WARN] [localhost] [docId] [userId] nodeJS - worker 901 started.
Feb 26 13:34:42[2024-02-26T12:34:42.726] [WARN] [localhost] [docId] [userId] nodeJS - worker 907 started.
Feb 26 13:34:42[2024-02-26T12:34:42.754] [WARN] [localhost] [docId] [userId] nodeJS - worker 908 started.
Feb 26 13:34:422024-02-26 12:34:42,758 INFO reaped unknown pid 652 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,758 INFO reaped unknown pid 652 (exit status 0)
Feb 26 13:34:42[2024-02-26T12:34:42.770] [WARN] [localhost] [docId] [userId] nodeJS - worker 914 started.
Feb 26 13:34:42[2024-02-26T12:34:42.792] [WARN] [localhost] [docId] [userId] nodeJS - worker 925 started.
Feb 26 13:34:42[2024-02-26T12:34:42.811] [WARN] [localhost] [docId] [userId] nodeJS - worker 931 started.
Feb 26 13:34:42[2024-02-26T12:34:42.837] [WARN] [localhost] [docId] [userId] nodeJS - worker 932 started.
Feb 26 13:34:43box:taskworker Starting task 850. Logs are at /home/yellowtent/platformdata/logs/c03a8f57-4218-49ed-9462-69e17e7cf0ad/apptask.log
Feb 26 13:34:43box:apptask run: startTask installationState: pending_stop runState: stopped
Feb 26 13:34:43box:tasks update 850: {"percent":20,"message":"Stopping container"}
Feb 26 13:34:43box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
Feb 26 13:34:432024-02-26 12:34:43,335 INFO exited: docservice (terminated by SIGKILL; not expected)
Feb 26 13:34:432024-02-26 12:34:43,335 INFO exited: docservice (terminated by SIGKILL; not expected)
Feb 26 13:34:43[2024-02-26T12:34:43.448] [WARN] [localhost] [docId] [userId] nodeJS - worker 719 died (code = null; signal = SIGKILL).
Feb 26 13:34:432024-02-26 12:34:43,458 INFO spawned: 'docservice' with pid 943
Feb 26 13:34:432024-02-26 12:34:43,458 INFO spawned: 'docservice' with pid 943
Feb 26 13:34:432024-02-26 12:34:43,459 WARN received SIGTERM indicating exit request
Feb 26 13:34:432024-02-26 12:34:43,459 WARN received SIGTERM indicating exit request
Feb 26 13:34:432024-02-26 12:34:43,461 INFO waiting for converter, docservice, metrics, nginx to die
Feb 26 13:34:432024-02-26 12:34:43,461 INFO waiting for converter, docservice, metrics, nginx to die
Feb 26 13:34:43[2024-02-26T12:34:43.465] [WARN] [localhost] [docId] [userId] nodeJS - worker 944 started.
Feb 26 13:34:43[2024-02-26T12:34:43.467] [WARN] [localhost] [docId] [userId] nodeJS - worker 726 died (code = null; signal = SIGKILL).
Feb 26 13:34:43[2024-02-26T12:34:43.487] [WARN] [localhost] [docId] [userId] nodeJS - worker 945 started.
Feb 26 13:34:43[2024-02-26T12:34:43.937] [WARN] [localhost] [docId] [userId] nodeJS - worker 720 died (code = null; signal = SIGKILL).
Feb 26 13:34:44[2024-02-26T12:34:44.040] [WARN] [localhost] [docId] [userId] nodeJS - worker 956 started.
Feb 26 13:34:44[2024-02-26T12:34:44.041] [WARN] [localhost] [docId] [userId] nodeJS - worker 762 died (code = null; signal = SIGKILL).
Feb 26 13:34:44[2024-02-26T12:34:44.052] [WARN] [localhost] [docId] [userId] nodeJS - worker 957 started.
Feb 26 13:34:442024-02-26 12:34:44,076 INFO stopped: nginx (exit status 0)
Feb 26 13:34:442024-02-26 12:34:44,076 INFO stopped: nginx (exit status 0)
Feb 26 13:34:44[2024-02-26T12:34:44.286] [WARN] [localhost] [docId] [userId] nodeJS - worker 737 died (code = null; signal = SIGKILL).
Feb 26 13:34:44[2024-02-26T12:34:44.309] [WARN] [localhost] [docId] [userId] nodeJS - worker 973 started.
Feb 26 13:34:44[2024-02-26T12:34:44.401] [WARN] [localhost] [docId] [userId] nodeJS - worker 708 died (code = null; signal = SIGKILL).
Feb 26 13:34:44[2024-02-26T12:34:44.413] [WARN] [localhost] [docId] [userId] nodeJS - worker 974 started.
Feb 26 13:34:452024-02-26 12:34:45,425 INFO success: docservice entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:34:452024-02-26 12:34:45,425 INFO success: docservice entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:34:46[2024-02-26T12:34:46.290] [WARN] [localhost] [docId] [userId] nodeJS - worker 707 died (code = null; signal = SIGKILL).
Feb 26 13:34:46[2024-02-26T12:34:46.315] [WARN] [localhost] [docId] [userId] nodeJS - worker 980 started.
Feb 26 13:34:472024-02-26 12:34:47,333 INFO waiting for converter, docservice, metrics to die
Feb 26 13:34:472024-02-26 12:34:47,333 INFO waiting for converter, docservice, metrics to die
Feb 26 13:34:48[2024-02-26T12:34:48.004] [WARN] [localhost] [docId] [userId] nodeJS - worker 803 died (code = null; signal = SIGKILL).
Feb 26 13:34:48[2024-02-26T12:34:48.011] [WARN] [localhost] [docId] [userId] nodeJS - worker 991 started.
Feb 26 13:34:48Flushing stats at Mon Feb 26 2024 12:34:44 GMT+0000 (Coordinated Universal Time)
Feb 26 13:34:48[2024-02-26T12:34:48.242] [WARN] [localhost] [docId] [userId] nodeJS - worker 751 died (code = null; signal = SIGKILL).
Feb 26 13:34:48[2024-02-26T12:34:48.251] [WARN] [localhost] [docId] [userId] nodeJS - worker 997 started.
Feb 26 13:34:48[2024-02-26T12:34:48.252] [WARN] [localhost] [docId] [userId] nodeJS - worker 757 died (code = null; signal = SIGKILL).
Feb 26 13:34:48[2024-02-26T12:34:48.270] [WARN] [localhost] [docId] [userId] nodeJS - worker 998 started.
Feb 26 13:34:48{
Feb 26 13:34:48counters: {
Feb 26 13:34:48'statsd.bad_lines_seen': 0,
Feb 26 13:34:48'statsd.packets_received': 0,
Feb 26 13:34:48'statsd.metrics_received': 0
Feb 26 13:34:48},
Feb 26 13:34:48timers: {},
Feb 26 13:34:48gauges: {},
Feb 26 13:34:48timer_data: {},
Feb 26 13:34:48counter_rates: {
Feb 26 13:34:48'statsd.bad_lines_seen': 0,
Feb 26 13:34:48'statsd.packets_received': 0,
Feb 26 13:34:48'statsd.metrics_received': 0
Feb 26 13:34:48},
Feb 26 13:34:48sets: {},
Feb 26 13:34:48pctThreshold: [ 90 ]
Feb 26 13:34:48}
Feb 26 13:34:48[2024-02-26T12:34:48.462] [WARN] [localhost] [docId] [userId] nodeJS - worker 773 died (code = null; signal = SIGKILL).
Feb 26 13:34:482024-02-26 12:34:48,491 INFO stopped: metrics (exit status 0)
Feb 26 13:34:482024-02-26 12:34:48,491 INFO stopped: metrics (exit status 0)
Feb 26 13:34:482024-02-26 12:34:48,504 INFO stopped: docservice (terminated by SIGTERM)
Feb 26 13:34:482024-02-26 12:34:48,504 INFO stopped: docservice (terminated by SIGTERM)
Feb 26 13:34:492024-02-26 12:34:49,609 INFO stopped: converter (terminated by SIGTERM)
Feb 26 13:34:492024-02-26 12:34:49,609 INFO stopped: converter (terminated by SIGTERM)
Feb 26 13:34:53box:tasks update 850: {"percent":50,"message":"Stopping app services"}
Feb 26 13:34:542024-02-26 12:34:54,108 WARN received SIGTERM indicating exit request
Feb 26 13:34:542024-02-26 12:34:54,108 INFO waiting for redis, redis-service to die
Feb 26 13:34:542024-02-26 12:34:54,111 INFO stopped: redis-service (terminated by SIGTERM)
Feb 26 13:34:5412:signal-handler (1708950894) Received SIGTERM scheduling shutdown...
Feb 26 13:34:5412:M 26 Feb 2024 12:34:54.174 * User requested shutdown...
Feb 26 13:34:5412:M 26 Feb 2024 12:34:54.174 * Saving the final RDB snapshot before exiting.
Feb 26 13:34:5412:M 26 Feb 2024 12:34:54.178 * DB saved on disk
Feb 26 13:34:5412:M 26 Feb 2024 12:34:54.178 * Removing the pid file.
Feb 26 13:34:5412:M 26 Feb 2024 12:34:54.178 # Redis is now ready to exit, bye bye...
Feb 26 13:34:542024-02-26 12:34:54,179 INFO stopped: redis (exit status 0)
Feb 26 13:34:54box:tasks update 850: {"percent":100,"message":"Done"}
Feb 26 13:34:54box:taskworker Task took 11.177 seconds
Feb 26 13:34:54box:tasks setCompleted - 850: {"result":null,"error":null}
Feb 26 13:34:54box:tasks update 850: {"percent":100,"result":null,"error":null}

[nebulon]

Just since there was a new update today, do you see the same behavior with version 8.0.1 now?
Also it appears from your logs that the worker keeps on exiting, do you see any memory shortage or so? Can you try to increase the memory limit, just so we can rule this out?

[im-fabian]

This update does not fix it but it changes the behavior. Since the update I see a memory shortage. However it is impossible to increase the memory limit. When I try to do this I get stuck with this error message:

Inactive: Error getting IP of redis-da6b0965-386f-467c-a9b1-53f0484c81d8 service

I already tired deleting the App and creating a brand-new one. Same issue when trying to resize.

[im-fabian]

Okay, I got it fixed now; rebooting did the trick with redis.
For those who come along with this thread:

I guess when used on machines with many CPU-Cores, especially with hyperhreading, OnlyOffice starts some kind of worker per Core, which then eats up some memory. In my case, with 16 cores (32 HT-Cores), this eats about 3.25 Gbytes. If the memory limit is too low, this results in a restart loop. Since I resized the memory to 6 GB, all is fine.

[girish]

@im-fabian Investigating this a bit more. I found that there is a config variable FileConverter.converter.maxprocesscount . This defaults to 1. In new instances, I can see that one 1 instance of that converter runs by default.

However, I do see the behavior that you mention of many converter processes in the Enterprise version. Are you using Onlyoffice EE ? That config variable seemingly has no effect in the enterprise version.

[girish]

The code is from https://github.com/ONLYOFFICE/server/blob/7d2104f16e0cda9cdb11002c581a8792e9e915d7/FileConverter/sources/convertermaster.js#L47 but the EE version uses some binary looks like.

[im-fabian]

@girish I can confirm, we are using the EE Version. So this might be a bug in the OnlyOffice EE-Version not Cloudron?

[girish]

@im-fabian in many GitHub issues (but this only applies to the community version), they mention that they launch as many processes as CPUs . Maybe this is the default in the EE. If you can reach out to their support and ask if there is a config variable for this, I think we can fix the package accordingly.

[im-fabian]

@girish I opened an ticket with OnlyOffice on that cause. Will keep you posted.

[im-fabian]

@girish the support replied. I did not have the time to look into it, but wanted to share the feedback with you asap:

You are correct, but you can decrease the amount of converter processes by changing maxprocesscount parameter in default.json file:
/etc/onlyoffice/documentserver/default.json
 
"FileConverter": {
  "converter": {
   
   "maxprocesscount": 1,
The default value is "1" but you can change it (to 0.5 or 0.25 for instance) to decrease the number of processes, since maxprocesscount is a multiplier, the number of converter processes will be equal to maxprocesscount * number of CPUs.
 
Please, do not forget to restart Document Server's services after applying the changes:
supervisorctl restart all