Onlyoffice high CPU usage

[im-fabian]

The problem just reappered. We moved from a small vm to a bigger server and now onlyoffice is totally out of control in regard to cpu usage.

I already tried uninstalling the onlyoffice app and reinstalling it.

[nebulon]

Do you have any more logs of the app which may indicate an issue? Also given that it seems busy with some document converter process, maybe one document triggered this and its worth looking at upstream issues?

[im-fabian]

Thanks for your reply and support.

Sure, I can provide you with the Cloudron App logs. Those are recorded after starting and waiting about a minute at full CPU usage. If I leave the App running for longer, this will continue.

I also have to note that the problem occurs even with a changed secret – so the servers connecting to OnlyOffice are not causing the issue because they can't trigger a conversion, as with a changed secret, they do not have the right to do such a thing. And as noted before, it even occurs with a brand-new OnlyOffice instance.

For privacy reasons, I replaced the actual domain with DOMAINNAME.TLD.

Feb 26 13:33:49box:taskworker Starting task 849. Logs are at /home/yellowtent/platformdata/logs/c03a8f57-4218-49ed-9462-69e17e7cf0ad/apptask.log
Feb 26 13:33:49box:apptask run: startTask installationState: pending_start runState: running
Feb 26 13:33:49box:tasks update 849: {"percent":10,"message":"Starting app services"}
Feb 26 13:33:49box:tasks update 849: {"percent":35,"message":"Starting container"}
Feb 26 13:33:49Starting supervisor
Feb 26 13:33:492024-02-26 12:33:49,952 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
Feb 26 13:33:492024-02-26 12:33:49,953 INFO Included extra file "/etc/supervisor/conf.d/redis-service.conf" during parsing
Feb 26 13:33:492024-02-26 12:33:49,953 INFO Included extra file "/etc/supervisor/conf.d/redis.conf" during parsing
Feb 26 13:33:492024-02-26 12:33:49,955 INFO RPC interface 'supervisor' initialized
Feb 26 13:33:492024-02-26 12:33:49,955 CRIT Server 'inet_http_server' running without any HTTP authentication checking
Feb 26 13:33:492024-02-26 12:33:49,955 INFO RPC interface 'supervisor' initialized
Feb 26 13:33:492024-02-26 12:33:49,955 CRIT Server 'unix_http_server' running without any HTTP authentication checking
Feb 26 13:33:492024-02-26 12:33:49,955 INFO supervisord started with pid 1
Feb 26 13:33:50=> Setup directories
Feb 26 13:33:50box:tasks update 849: {"percent":80,"message":"Configuring reverse proxy"}
Feb 26 13:33:50=> Patch configuration files
Feb 26 13:33:50box:reverseproxy providerMatchesSync: subject=CN = onlyoffice.DOMAINNAME.TLD domain=onlyoffice.DOMAINNAME.TLD issuer=C = US, O = Let's Encrypt, CN = R3 wildcard=false/false prod=true/true issuerMismatch=false wildcardMismatch=false match=true
Feb 26 13:33:50box:reverseproxy expiryDate: subject=CN = onlyoffice.DOMAINNAME.TLD notBefore=Feb 18 14:11:00 2024 GMT notAfter=May 18 14:10:59 2024 GMT daysLeft=82.0674635300926
Feb 26 13:33:50box:reverseproxy needsRenewal: false. force: false
Feb 26 13:33:50box:reverseproxy ensureCertificate: onlyoffice.DOMAINNAME.TLD acme cert exists and is up to date
Feb 26 13:33:50box:reverseproxy writeAppLocationNginxConfig: writing config for "onlyoffice.DOMAINNAME.TLD" to /home/yellowtent/platformdata/nginx/applications/c03a8f57-4218-49ed-9462-69e17e7cf0ad/onlyoffice.DOMAINNAME.TLD.conf with options {"sourceDir":"/home/yellowtent/box","vhost":"onlyoffice.DOMAINNAME.TLD","hasIPv6":true,"ip":"172.18.19.242","port":80,"endpoint":"app","redirectTo":null,"certFilePath":"/home/yellowtent/platformdata/nginx/cert/onlyoffice.DOMAINNAME.TLD.cert","keyFilePath":"/home/yellowtent/platformdata/nginx/cert/onlyoffice.DOMAINNAME.TLD.key","robotsTxtQuoted":null,"cspQuoted":null,"hideHeaders":[],"proxyAuth":{"enabled":false,"id":"c03a8f57-4218-49ed-9462-69e17e7cf0ad","location":"/"},"upstreamUri":"","ocsp":true,"hstsPreload":false}
Feb 26 13:33:50box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
Feb 26 13:33:50Reading config from /app/data/config/production-linux.json ...
Feb 26 13:33:50{
Feb 26 13:33:50log: { filePath: '/etc/onlyoffice/documentserver/log4js/production.json' },
Feb 26 13:33:50storage: {
Feb 26 13:33:50fs: {
Feb 26 13:33:50folderPath: '/var/lib/onlyoffice/documentserver/App_Data/cache/files'
Feb 26 13:33:50}
Feb 26 13:33:50},
Feb 26 13:33:50services: {
Feb 26 13:33:50CoAuthoring: {
Feb 26 13:33:50server: [Object],
Feb 26 13:33:50utils: [Object],
Feb 26 13:33:50sockjs: [Object],
Feb 26 13:33:50sql: [Object],
Feb 26 13:33:50redis: [Object],
Feb 26 13:33:50secret: [Object],
Feb 26 13:33:50token: [Object]
Feb 26 13:33:50}
Feb 26 13:33:50},
Feb 26 13:33:50license: {
Feb 26 13:33:50license_file: '/app/data/license.lic',
Feb 26 13:33:50warning_limit_percents: '70'
Feb 26 13:33:50},
Feb 26 13:33:50FileConverter: {
Feb 26 13:33:50converter: {
Feb 26 13:33:50fontDir: '/usr/share/fonts',
Feb 26 13:33:50presentationThemesDir: '/var/www/onlyoffice/documentserver/sdkjs/slide/themes',
Feb 26 13:33:50x2tPath: '/var/www/onlyoffice/documentserver/server/FileConverter/bin/x2t',
Feb 26 13:33:50docbuilderPath: '/var/www/onlyoffice/documentserver/server/FileConverter/bin/docbuilder',
Feb 26 13:33:50docbuilderAllFontsPath: '/var/lib/onlyoffice/documentserver/App_Data/docbuilder/AllFonts.js'
Feb 26 13:33:50}
Feb 26 13:33:50},
Feb 26 13:33:50FileStorage: { directory: '/app/data/documentserver/App_Data' },
Feb 26 13:33:50SpellChecker: {
Feb 26 13:33:50server: { dictDir: '/var/www/onlyoffice/documentserver/dictionaries' }
Feb 26 13:33:50}
Feb 26 13:33:50}
Feb 26 13:33:50Patching Postgres credentials ...
Feb 26 13:33:50Patching Redis credentials ...
Feb 26 13:33:50Ensuring spellcheck dictionaries ...
Feb 26 13:33:50Writing config to /app/data/config/production-linux.json ...
Feb 26 13:33:50Done
Feb 26 13:33:50=> Collect fonts
Feb 26 13:33:50box:tasks update 849: {"percent":100,"message":"Done"}
Feb 26 13:33:50box:taskworker Task took 1.086 seconds
Feb 26 13:33:50box:tasks setCompleted - 849: {"result":null,"error":null}
Feb 26 13:33:50box:tasks update 849: {"percent":100,"result":null,"error":null}
Feb 26 13:33:50=> Initialize and migrate database
Feb 26 13:33:50psql:/var/www/onlyoffice/documentserver/server/schema/postgresql/createdb.sql:21: NOTICE: relation "doc_changes" already exists, skipping
Feb 26 13:33:50CREATE TABLE
Feb 26 13:33:50psql:/var/www/onlyoffice/documentserver/server/schema/postgresql/createdb.sql:41: NOTICE: relation "task_result" already exists, skipping
Feb 26 13:33:50CREATE TABLE
Feb 26 13:33:50CREATE FUNCTION
Feb 26 13:33:50=> Start rabbitmq server
Feb 26 13:33:50=> Fixup permissions
Feb 26 13:33:50=> Start ONLYOFFICE daemons
Feb 26 13:33:502024-02-26 12:33:50,815 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
Feb 26 13:33:502024-02-26 12:33:50,815 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
Feb 26 13:33:502024-02-26 12:33:50,815 INFO Included extra file "/etc/supervisor/conf.d/ds-converter.conf" during parsing
Feb 26 13:33:502024-02-26 12:33:50,815 INFO Included extra file "/etc/supervisor/conf.d/ds-converter.conf" during parsing
Feb 26 13:33:502024-02-26 12:33:50,815 INFO Included extra file "/etc/supervisor/conf.d/ds-docservice.conf" during parsing
Feb 26 13:33:502024-02-26 12:33:50,815 INFO Included extra file "/etc/supervisor/conf.d/ds-docservice.conf" during parsing
Feb 26 13:33:502024-02-26 12:33:50,815 INFO Included extra file "/etc/supervisor/conf.d/ds-metrics.conf" during parsing
Feb 26 13:33:502024-02-26 12:33:50,815 INFO Included extra file "/etc/supervisor/conf.d/ds-metrics.conf" during parsing
Feb 26 13:33:502024-02-26 12:33:50,815 INFO Included extra file "/etc/supervisor/conf.d/nginx.conf" during parsing
Feb 26 13:33:502024-02-26 12:33:50,815 INFO Included extra file "/etc/supervisor/conf.d/nginx.conf" during parsing
Feb 26 13:33:50Unlinking stale socket /var/run/supervisor.sock
Feb 26 13:33:502024-02-26 12:33:50,957 INFO spawned: 'redis' with pid 12
Feb 26 13:33:502024-02-26 12:33:50,958 INFO spawned: 'redis-service' with pid 13
Feb 26 13:33:5012:C 26 Feb 2024 12:33:50.963 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
Feb 26 13:33:5012:C 26 Feb 2024 12:33:50.963 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
Feb 26 13:33:5012:C 26 Feb 2024 12:33:50.963 * Redis version=7.2.1, bits=64, commit=00000000, modified=0, pid=12, just started
Feb 26 13:33:5012:C 26 Feb 2024 12:33:50.963 * Configuration loaded
Feb 26 13:33:5012:M 26 Feb 2024 12:33:50.964 * monotonic clock: POSIX clock_gettime
Feb 26 13:33:5012:M 26 Feb 2024 12:33:50.964 # Failed to write PID file: Permission denied
Feb 26 13:33:5012:M 26 Feb 2024 12:33:50.964 * Running mode=standalone, port=6379.
Feb 26 13:33:5012:M 26 Feb 2024 12:33:50.964 * Server initialized
Feb 26 13:33:5012:M 26 Feb 2024 12:33:50.964 * Loading RDB produced by version 7.2.1
Feb 26 13:33:5012:M 26 Feb 2024 12:33:50.964 * RDB age 47152 seconds
Feb 26 13:33:5012:M 26 Feb 2024 12:33:50.964 * RDB memory usage when created 1.64 Mb
Feb 26 13:33:5012:M 26 Feb 2024 12:33:50.966 * Done loading RDB, keys loaded: 9, keys expired: 0.
Feb 26 13:33:5012:M 26 Feb 2024 12:33:50.966 * DB loaded from disk: 0.001 seconds
Feb 26 13:33:5012:M 26 Feb 2024 12:33:50.966 * Ready to accept connections tcp
Feb 26 13:33:51Redis service endpoint listening on http://:::3000
Feb 26 13:33:512024-02-26 12:33:51,118 INFO RPC interface 'supervisor' initialized
Feb 26 13:33:512024-02-26 12:33:51,118 INFO RPC interface 'supervisor' initialized
Feb 26 13:33:512024-02-26 12:33:51,118 CRIT Server 'unix_http_server' running without any HTTP authentication checking
Feb 26 13:33:512024-02-26 12:33:51,118 CRIT Server 'unix_http_server' running without any HTTP authentication checking
Feb 26 13:33:512024-02-26 12:33:51,118 INFO supervisord started with pid 1
Feb 26 13:33:512024-02-26 12:33:51,118 INFO supervisord started with pid 1
Feb 26 13:33:512024-02-26 12:33:51.684201+00:00 [info] <0.228.0> Feature flags: list of feature flags found:
Feb 26 13:33:512024-02-26 12:33:51.688647+00:00 [info] <0.228.0> Feature flags: [x] implicit_default_bindings
Feb 26 13:33:512024-02-26 12:33:51.688665+00:00 [info] <0.228.0> Feature flags: [x] maintenance_mode_status
Feb 26 13:33:512024-02-26 12:33:51.688673+00:00 [info] <0.228.0> Feature flags: [x] quorum_queue
Feb 26 13:33:512024-02-26 12:33:51.688683+00:00 [info] <0.228.0> Feature flags: [x] stream_queue
Feb 26 13:33:512024-02-26 12:33:51.688711+00:00 [info] <0.228.0> Feature flags: [x] user_limits
Feb 26 13:33:512024-02-26 12:33:51.688718+00:00 [info] <0.228.0> Feature flags: [x] virtual_host_metadata
Feb 26 13:33:512024-02-26 12:33:51.688725+00:00 [info] <0.228.0> Feature flags: feature flag states written to disk: yes
Feb 26 13:33:512024-02-26 12:33:51.814743+00:00 [noti] <0.44.0> Application syslog exited with reason: stopped
Feb 26 13:33:512024-02-26 12:33:51.814774+00:00 [noti] <0.228.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
Feb 26 13:33:522024-02-26 12:33:52,047 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:33:522024-02-26 12:33:52,047 INFO success: redis-service entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:33:522024-02-26 12:33:52,121 INFO spawned: 'converter' with pid 154
Feb 26 13:33:522024-02-26 12:33:52,121 INFO spawned: 'converter' with pid 154
Feb 26 13:33:522024-02-26 12:33:52,122 INFO spawned: 'docservice' with pid 155
Feb 26 13:33:522024-02-26 12:33:52,122 INFO spawned: 'docservice' with pid 155
Feb 26 13:33:522024-02-26 12:33:52,123 INFO spawned: 'metrics' with pid 156
Feb 26 13:33:522024-02-26 12:33:52,123 INFO spawned: 'metrics' with pid 156
Feb 26 13:33:522024-02-26 12:33:52,125 INFO spawned: 'nginx' with pid 157
Feb 26 13:33:522024-02-26 12:33:52,125 INFO spawned: 'nginx' with pid 157
Feb 26 13:33:522024-02-26 12:33:52,125 INFO reaped unknown pid 60 (terminated by SIGUSR1)
Feb 26 13:33:522024-02-26 12:33:52,125 INFO reaped unknown pid 60 (terminated by SIGUSR1)
Feb 26 13:33:522024-02-26 12:33:52,125 INFO reaped unknown pid 68 (exit status 0)
Feb 26 13:33:522024-02-26 12:33:52,125 INFO reaped unknown pid 68 (exit status 0)
Feb 26 13:33:522024-02-26 12:33:52,125 INFO reaped unknown pid 151 (terminated by SIGUSR1)
Feb 26 13:33:522024-02-26 12:33:52,125 INFO reaped unknown pid 151 (terminated by SIGUSR1)
Feb 26 13:33:522024-02-26 12:33:52,125 INFO reaped unknown pid 73 (exit status 0)
Feb 26 13:33:522024-02-26 12:33:52,125 INFO reaped unknown pid 73 (exit status 0)
Feb 26 13:33:5226 Feb 12:33:52 - [156] reading config file: ./config/config.js
Feb 26 13:33:5226 Feb 12:33:52 - server is up INFO
Feb 26 13:33:52[2024-02-26T12:33:52.529] [WARN] [localhost] [docId] [userId] nodeJS - num of CPUs: 32; availableParallelism: undefined
Feb 26 13:33:52[2024-02-26T12:33:52.537] [WARN] [localhost] [docId] [userId] nodeJS - update cluster with 32 workers
Feb 26 13:33:52[2024-02-26T12:33:52.541] [WARN] [localhost] [docId] [userId] nodeJS - worker 213 started.
Feb 26 13:33:52[2024-02-26T12:33:52.544] [WARN] [localhost] [docId] [userId] nodeJS - worker 214 started.
Feb 26 13:33:52[2024-02-26T12:33:52.547] [WARN] [localhost] [docId] [userId] nodeJS - worker 220 started.
Feb 26 13:33:52[2024-02-26T12:33:52.551] [WARN] [localhost] [docId] [userId] nodeJS - worker 226 started.
Feb 26 13:33:52[2024-02-26T12:33:52.553] [WARN] [localhost] [docId] [userId] nodeJS - worker 232 started.
Feb 26 13:33:52[2024-02-26T12:33:52.556] [WARN] [localhost] [docId] [userId] nodeJS - worker 238 started.
Feb 26 13:33:52[2024-02-26T12:33:52.559] [WARN] [localhost] [docId] [userId] nodeJS - worker 244 started.
Feb 26 13:33:52[2024-02-26T12:33:52.562] [WARN] [localhost] [docId] [userId] nodeJS - worker 250 started.
Feb 26 13:33:52[2024-02-26T12:33:52.565] [WARN] [localhost] [docId] [userId] nodeJS - worker 256 started.
Feb 26 13:33:52[2024-02-26T12:33:52.569] [WARN] [localhost] [docId] [userId] nodeJS - worker 262 started.
Feb 26 13:33:52[2024-02-26T12:33:52.572] [WARN] [localhost] [docId] [userId] nodeJS - worker 268 started.
Feb 26 13:33:52[2024-02-26T12:33:52.575] [WARN] [localhost] [docId] [userId] nodeJS - worker 274 started.
Feb 26 13:33:52[2024-02-26T12:33:52.579] [WARN] [localhost] [docId] [userId] nodeJS - worker 280 started.
Feb 26 13:33:52[2024-02-26T12:33:52.582] [WARN] [localhost] [docId] [userId] nodeJS - worker 286 started.
Feb 26 13:33:52[2024-02-26T12:33:52.586] [WARN] [localhost] [docId] [userId] nodeJS - worker 292 started.
Feb 26 13:33:52[2024-02-26T12:33:52.590] [WARN] [localhost] [docId] [userId] nodeJS - worker 298 started.
Feb 26 13:33:52[2024-02-26T12:33:52.595] [WARN] [localhost] [docId] [userId] nodeJS - worker 304 started.
Feb 26 13:33:52[2024-02-26T12:33:52.601] [WARN] [localhost] [docId] [userId] nodeJS - worker 310 started.
Feb 26 13:33:52[2024-02-26T12:33:52.608] [WARN] [localhost] [docId] [userId] nodeJS - worker 316 started.
Feb 26 13:33:52[2024-02-26T12:33:52.614] [WARN] [localhost] [docId] [userId] nodeJS - worker 322 started.
Feb 26 13:33:52[2024-02-26T12:33:52.620] [WARN] [localhost] [docId] [userId] nodeJS - worker 328 started.
Feb 26 13:33:52[2024-02-26T12:33:52.626] [WARN] [localhost] [docId] [userId] nodeJS - worker 334 started.
Feb 26 13:33:52[2024-02-26T12:33:52.632] [WARN] [localhost] [docId] [userId] nodeJS - worker 340 started.
Feb 26 13:33:52[2024-02-26T12:33:52.639] [WARN] [localhost] [docId] [userId] nodeJS - worker 346 started.
Feb 26 13:33:52[2024-02-26T12:33:52.646] [WARN] [localhost] [docId] [userId] nodeJS - worker 352 started.
Feb 26 13:33:52[2024-02-26T12:33:52.658] [WARN] [localhost] [docId] [userId] nodeJS - worker 363 started.
Feb 26 13:33:52[2024-02-26T12:33:52.673] [WARN] [localhost] [docId] [userId] nodeJS - worker 369 started.
Feb 26 13:33:52[2024-02-26T12:33:52.687] [WARN] [localhost] [docId] [userId] nodeJS - worker 375 started.
Feb 26 13:33:52[2024-02-26T12:33:52.700] [WARN] [localhost] [docId] [userId] nodeJS - worker 381 started.
Feb 26 13:33:52[2024-02-26T12:33:52.712] [WARN] [localhost] [docId] [userId] nodeJS - worker 387 started.
Feb 26 13:33:52[2024-02-26T12:33:52.730] [WARN] [localhost] [docId] [userId] nodeJS - worker 388 started.
Feb 26 13:33:52[2024-02-26T12:33:52.748] [WARN] [localhost] [docId] [userId] nodeJS - worker 390 started.
Feb 26 13:33:532024-02-26 12:33:53,867 INFO success: converter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:33:532024-02-26 12:33:53,867 INFO success: converter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:33:532024-02-26 12:33:53,901 INFO success: docservice entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:33:532024-02-26 12:33:53,901 INFO success: docservice entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:33:532024-02-26 12:33:53,901 INFO success: metrics entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:33:532024-02-26 12:33:53,901 INFO success: metrics entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:33:532024-02-26 12:33:53,901 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:33:532024-02-26 12:33:53,901 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:33:532024-02-26 12:33:53,992 INFO exited: converter (terminated by SIGKILL; not expected)
Feb 26 13:33:532024-02-26 12:33:53,992 INFO exited: converter (terminated by SIGKILL; not expected)
Feb 26 13:33:542024-02-26 12:33:54,107 INFO spawned: 'converter' with pid 405
Feb 26 13:33:542024-02-26 12:33:54,107 INFO spawned: 'converter' with pid 405
Feb 26 13:33:542024-02-26 12:33:54,526 INFO exited: docservice (terminated by SIGKILL; not expected)
Feb 26 13:33:542024-02-26 12:33:54,526 INFO exited: docservice (terminated by SIGKILL; not expected)
Feb 26 13:33:542024-02-26 12:33:54,531 INFO spawned: 'docservice' with pid 406
Feb 26 13:33:542024-02-26 12:33:54,531 INFO spawned: 'docservice' with pid 406
Feb 26 13:33:552024-02-26 12:33:55,535 INFO success: converter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:33:552024-02-26 12:33:55,535 INFO success: converter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:33:552024-02-26 12:33:55,535 INFO success: docservice entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:33:552024-02-26 12:33:55,535 INFO success: docservice entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:33:552024-02-26 12:33:55,535 INFO reaped unknown pid 232 (terminated by SIGKILL)
Feb 26 13:33:552024-02-26 12:33:55,535 INFO reaped unknown pid 232 (terminated by SIGKILL)
Feb 26 13:33:562024-02-26 12:33:56,592 INFO reaped unknown pid 226 (terminated by SIGKILL)
Feb 26 13:33:562024-02-26 12:33:56,592 INFO reaped unknown pid 226 (terminated by SIGKILL)
Feb 26 13:33:562024-02-26 12:33:56,592 INFO reaped unknown pid 238 (terminated by SIGKILL)
Feb 26 13:33:562024-02-26 12:33:56,592 INFO reaped unknown pid 238 (terminated by SIGKILL)
Feb 26 13:33:562024-02-26 12:33:56,592 INFO reaped unknown pid 244 (terminated by SIGKILL)
Feb 26 13:33:562024-02-26 12:33:56,592 INFO reaped unknown pid 244 (terminated by SIGKILL)
Feb 26 13:33:562024-02-26 12:33:56,592 INFO reaped unknown pid 250 (terminated by SIGKILL)
Feb 26 13:33:562024-02-26 12:33:56,592 INFO reaped unknown pid 250 (terminated by SIGKILL)
Feb 26 13:33:562024-02-26 12:33:56,592 INFO reaped unknown pid 292 (terminated by SIGKILL)
Feb 26 13:33:562024-02-26 12:33:56,592 INFO reaped unknown pid 292 (terminated by SIGKILL)
Feb 26 13:33:562024-02-26 12:33:56,592 INFO reaped unknown pid 310 (terminated by SIGKILL)
Feb 26 13:33:562024-02-26 12:33:56,592 INFO reaped unknown pid 310 (terminated by SIGKILL)
Feb 26 13:33:572024-02-26 12:33:57,594 INFO reaped unknown pid 268 (terminated by SIGKILL)
Feb 26 13:33:572024-02-26 12:33:57,594 INFO reaped unknown pid 268 (terminated by SIGKILL)
Feb 26 13:33:572024-02-26 12:33:57,595 INFO reaped unknown pid 274 (terminated by SIGKILL)
Feb 26 13:33:572024-02-26 12:33:57,595 INFO reaped unknown pid 274 (terminated by SIGKILL)
Feb 26 13:33:572024-02-26 12:33:57,595 INFO reaped unknown pid 280 (terminated by SIGKILL)
Feb 26 13:33:572024-02-26 12:33:57,595 INFO reaped unknown pid 280 (terminated by SIGKILL)
Feb 26 13:33:572024-02-26 12:33:57,595 INFO reaped unknown pid 304 (terminated by SIGKILL)
Feb 26 13:33:572024-02-26 12:33:57,595 INFO reaped unknown pid 304 (terminated by SIGKILL)
Feb 26 13:33:582024-02-26 12:33:58,616 INFO reaped unknown pid 328 (terminated by SIGKILL)
Feb 26 13:33:582024-02-26 12:33:58,616 INFO reaped unknown pid 328 (terminated by SIGKILL)
Feb 26 13:34:002024/02/26 12:34:00 [error] 173#173: *1 connect() failed (111: Unknown error) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /healthcheck HTTP/1.1", upstream: "http://[::1]:8000/healthcheck", host: "onlyoffice.DOMAINNAME.TLD"
Feb 26 13:34:002024/02/26 12:34:00 [error] 173#173: *1 connect() failed (111: Unknown error) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /healthcheck HTTP/1.1", upstream: "http://127.0.0.1:8000/healthcheck", host: "onlyoffice.DOMAINNAME.TLD"
Feb 26 13:34:00172.18.0.1 - - [26/Feb/2024:12:34:00 +0000] "GET /healthcheck HTTP/1.1" 502 150 "-" "Mozilla (CloudronHealth)"
Feb 26 13:34:00=> Healtheck error got response status 502
Feb 26 13:34:012024-02-26 12:34:01,729 INFO reaped unknown pid 213 (terminated by SIGKILL)
Feb 26 13:34:012024-02-26 12:34:01,729 INFO reaped unknown pid 213 (terminated by SIGKILL)
Feb 26 13:34:012024-02-26 12:34:01,730 INFO reaped unknown pid 256 (terminated by SIGKILL)
Feb 26 13:34:012024-02-26 12:34:01,730 INFO reaped unknown pid 256 (terminated by SIGKILL)
Feb 26 13:34:012024-02-26 12:34:01,730 INFO reaped unknown pid 298 (terminated by SIGKILL)
Feb 26 13:34:012024-02-26 12:34:01,730 INFO reaped unknown pid 298 (terminated by SIGKILL)
Feb 26 13:34:012024-02-26 12:34:01,730 INFO reaped unknown pid 369 (terminated by SIGKILL)
Feb 26 13:34:012024-02-26 12:34:01,730 INFO reaped unknown pid 369 (terminated by SIGKILL)
Feb 26 13:34:012024-02-26 12:34:01,730 INFO reaped unknown pid 388 (terminated by SIGKILL)
Feb 26 13:34:012024-02-26 12:34:01,730 INFO reaped unknown pid 388 (terminated by SIGKILL)
Feb 26 13:34:06[2024-02-26T12:34:01.728] [WARN] [localhost] [docId] [userId] nodeJS - num of CPUs: 32; availableParallelism: undefined
Feb 26 13:34:06[2024-02-26T12:34:06.301] [WARN] [localhost] [docId] [userId] nodeJS - update cluster with 32 workers
Feb 26 13:34:062024-02-26 12:34:06,302 INFO reaped unknown pid 334 (terminated by SIGKILL)
Feb 26 13:34:062024-02-26 12:34:06,302 INFO reaped unknown pid 334 (terminated by SIGKILL)
Feb 26 13:34:06[2024-02-26T12:34:06.313] [WARN] [localhost] [docId] [userId] nodeJS - worker 441 started.
Feb 26 13:34:062024-02-26 12:34:06,313 INFO reaped unknown pid 286 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,313 INFO reaped unknown pid 286 (exit status 0)
Feb 26 13:34:06[2024-02-26T12:34:06.320] [WARN] [localhost] [docId] [userId] nodeJS - worker 458 started.
Feb 26 13:34:062024-02-26 12:34:06,321 INFO reaped unknown pid 220 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,321 INFO reaped unknown pid 220 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,321 INFO reaped unknown pid 352 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,321 INFO reaped unknown pid 352 (exit status 0)
Feb 26 13:34:06[2024-02-26T12:34:06.328] [WARN] [localhost] [docId] [userId] nodeJS - worker 469 started.
Feb 26 13:34:062024-02-26 12:34:06,329 INFO reaped unknown pid 262 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,329 INFO reaped unknown pid 262 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,329 INFO reaped unknown pid 322 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,329 INFO reaped unknown pid 322 (exit status 0)
Feb 26 13:34:06[2024-02-26T12:34:06.339] [WARN] [localhost] [docId] [userId] nodeJS - worker 475 started.
Feb 26 13:34:062024-02-26 12:34:06,340 INFO reaped unknown pid 214 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,340 INFO reaped unknown pid 214 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,340 INFO reaped unknown pid 316 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,340 INFO reaped unknown pid 316 (exit status 0)
Feb 26 13:34:06

[im-fabian]

I had to split the logs into two posts because the post have been too long otherwise:

Feb 26 13:34:06## ## RabbitMQ 3.9.13
Feb 26 13:34:06## ##
Feb 26 13:34:06########## Copyright (c) 2007-2022 VMware, Inc. or its affiliates.
Feb 26 13:34:06###### ##
Feb 26 13:34:06########## Licensed under the MPL 2.0. Website: https://rabbitmq.com
Feb 26 13:34:06
Feb 26 13:34:06Erlang: 24.2.1 [jit]
Feb 26 13:34:06TLS Library: OpenSSL - OpenSSL 3.0.2 15 Mar 2022
Feb 26 13:34:06
Feb 26 13:34:06Doc guides: https://rabbitmq.com/documentation.html
Feb 26 13:34:06Support: https://rabbitmq.com/contact.html
Feb 26 13:34:06Tutorials: https://rabbitmq.com/getstarted.html
Feb 26 13:34:06Monitoring: https://rabbitmq.com/monitoring.html
Feb 26 13:34:06
Feb 26 13:34:06Logs: /var/log/rabbitmq/rabbit@c03a8f57-4218-49ed-9462-69e17e7cf0ad.log
Feb 26 13:34:06/var/log/rabbitmq/rabbit@c03a8f57-4218-49ed-9462-69e17e7cf0ad_upgrade.log
Feb 26 13:34:06<stdout>
Feb 26 13:34:06
Feb 26 13:34:06Config file(s): (none)
Feb 26 13:34:06
Feb 26 13:34:06Starting broker...[2024-02-26T12:34:06.343] [WARN] [localhost] [docId] [userId] nodeJS - worker 481 started.
Feb 26 13:34:062024-02-26 12:34:06,344 INFO reaped unknown pid 346 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,344 INFO reaped unknown pid 346 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,344 INFO reaped unknown pid 363 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,344 INFO reaped unknown pid 363 (exit status 0)
Feb 26 13:34:06[2024-02-26T12:34:06.347] [WARN] [localhost] [docId] [userId] nodeJS - worker 482 started.
Feb 26 13:34:06[2024-02-26T12:34:06.351] [WARN] [localhost] [docId] [userId] nodeJS - worker 495 started.
Feb 26 13:34:06[2024-02-26T12:34:06.357] [WARN] [localhost] [docId] [userId] nodeJS - worker 515 started.
Feb 26 13:34:06[2024-02-26T12:34:06.363] [WARN] [localhost] [docId] [userId] nodeJS - worker 521 started.
Feb 26 13:34:06[2024-02-26T12:34:06.371] [WARN] [localhost] [docId] [userId] nodeJS - worker 527 started.
Feb 26 13:34:062024-02-26 12:34:06,372 INFO reaped unknown pid 381 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,372 INFO reaped unknown pid 381 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,372 INFO reaped unknown pid 387 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,372 INFO reaped unknown pid 387 (exit status 0)
Feb 26 13:34:06[2024-02-26T12:34:06.380] [WARN] [localhost] [docId] [userId] nodeJS - worker 533 started.
Feb 26 13:34:062024-02-26 12:34:06,380 INFO reaped unknown pid 340 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,380 INFO reaped unknown pid 340 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,380 INFO reaped unknown pid 375 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,380 INFO reaped unknown pid 375 (exit status 0)
Feb 26 13:34:06[2024-02-26T12:34:06.386] [WARN] [localhost] [docId] [userId] nodeJS - worker 534 started.
Feb 26 13:34:06[2024-02-26T12:34:06.393] [WARN] [localhost] [docId] [userId] nodeJS - worker 545 started.
Feb 26 13:34:06[2024-02-26T12:34:06.400] [WARN] [localhost] [docId] [userId] nodeJS - worker 552 started.
Feb 26 13:34:06[2024-02-26T12:34:06.408] [WARN] [localhost] [docId] [userId] nodeJS - worker 561 started.
Feb 26 13:34:06[2024-02-26T12:34:06.416] [WARN] [localhost] [docId] [userId] nodeJS - worker 569 started.
Feb 26 13:34:06[2024-02-26T12:34:06.427] [WARN] [localhost] [docId] [userId] nodeJS - worker 575 started.
Feb 26 13:34:06[2024-02-26T12:34:06.440] [WARN] [localhost] [docId] [userId] nodeJS - worker 581 started.
Feb 26 13:34:06[2024-02-26T12:34:06.455] [WARN] [localhost] [docId] [userId] nodeJS - worker 591 started.
Feb 26 13:34:06[2024-02-26T12:34:06.465] [WARN] [localhost] [docId] [userId] nodeJS - worker 597 started.
Feb 26 13:34:06[2024-02-26T12:34:06.474] [WARN] [localhost] [docId] [userId] nodeJS - worker 603 started.
Feb 26 13:34:062024-02-26 12:34:06,475 INFO reaped unknown pid 390 (exit status 0)
Feb 26 13:34:062024-02-26 12:34:06,475 INFO reaped unknown pid 390 (exit status 0)
Feb 26 13:34:06[2024-02-26T12:34:06.487] [WARN] [localhost] [docId] [userId] nodeJS - worker 609 started.
Feb 26 13:34:06[2024-02-26T12:34:06.503] [WARN] [localhost] [docId] [userId] nodeJS - worker 615 started.
Feb 26 13:34:06[2024-02-26T12:34:06.519] [WARN] [localhost] [docId] [userId] nodeJS - worker 621 started.
Feb 26 13:34:06[2024-02-26T12:34:06.530] [WARN] [localhost] [docId] [userId] nodeJS - worker 627 started.
Feb 26 13:34:06[2024-02-26T12:34:06.541] [WARN] [localhost] [docId] [userId] nodeJS - worker 631 started.
Feb 26 13:34:06[2024-02-26T12:34:06.559] [WARN] [localhost] [docId] [userId] nodeJS - worker 634 started.
Feb 26 13:34:06[2024-02-26T12:34:06.584] [WARN] [localhost] [docId] [userId] nodeJS - worker 645 started.
Feb 26 13:34:06[2024-02-26T12:34:06.601] [WARN] [localhost] [docId] [userId] nodeJS - worker 651 started.
Feb 26 13:34:06[2024-02-26T12:34:06.615] [WARN] [localhost] [docId] [userId] nodeJS - worker 652 started.
Feb 26 13:34:06[2024-02-26T12:34:06.629] [WARN] [localhost] [docId] [userId] nodeJS - worker 663 started.
Feb 26 13:34:06[2024-02-26T12:34:06.644] [WARN] [localhost] [docId] [userId] nodeJS - worker 669 started.
Feb 26 13:34:07Killed
Feb 26 13:34:072024-02-26 12:34:07,626 INFO reaped unknown pid 22 (exit status 0)
Feb 26 13:34:072024-02-26 12:34:07,626 INFO reaped unknown pid 22 (exit status 0)
Feb 26 13:34:072024-02-26 12:34:07,635 INFO exited: converter (terminated by SIGKILL; not expected)
Feb 26 13:34:072024-02-26 12:34:07,635 INFO exited: converter (terminated by SIGKILL; not expected)
Feb 26 13:34:072024-02-26 12:34:07,640 INFO exited: docservice (terminated by SIGKILL; not expected)
Feb 26 13:34:072024-02-26 12:34:07,640 INFO exited: docservice (terminated by SIGKILL; not expected)
Feb 26 13:34:072024-02-26 12:34:07,640 INFO reaped unknown pid 44 (exit status 0)
Feb 26 13:34:072024-02-26 12:34:07,640 INFO reaped unknown pid 44 (exit status 0)
Feb 26 13:34:072024-02-26 12:34:07,640 INFO reaped unknown pid 152 (exit status 0)
Feb 26 13:34:072024-02-26 12:34:07,640 INFO reaped unknown pid 152 (exit status 0)
Feb 26 13:34:072024-02-26 12:34:07,773 INFO spawned: 'converter' with pid 675
Feb 26 13:34:072024-02-26 12:34:07,773 INFO spawned: 'converter' with pid 675
Feb 26 13:34:072024-02-26 12:34:07,893 INFO spawned: 'docservice' with pid 676
Feb 26 13:34:072024-02-26 12:34:07,893 INFO spawned: 'docservice' with pid 676
Feb 26 13:34:082024-02-26 12:34:08,994 INFO success: converter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:34:082024-02-26 12:34:08,994 INFO success: converter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:34:082024-02-26 12:34:08,996 INFO success: docservice entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:34:082024-02-26 12:34:08,996 INFO success: docservice entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:34:082024-02-26 12:34:08,996 INFO reaped unknown pid 521 (terminated by SIGKILL)
Feb 26 13:34:082024-02-26 12:34:08,996 INFO reaped unknown pid 521 (terminated by SIGKILL)
Feb 26 13:34:102024-02-26 12:34:10,009 INFO reaped unknown pid 482 (terminated by SIGKILL)
Feb 26 13:34:102024-02-26 12:34:10,009 INFO reaped unknown pid 482 (terminated by SIGKILL)
Feb 26 13:34:102024/02/26 12:34:10 [error] 173#173: *4 connect() failed (111: Unknown error) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /healthcheck HTTP/1.1", upstream: "http://127.0.0.1:8000/healthcheck", host: "onlyoffice.DOMAINNAME.TLD"
Feb 26 13:34:102024/02/26 12:34:10 [error] 173#173: *4 connect() failed (111: Unknown error) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /healthcheck HTTP/1.1", upstream: "http://[::1]:8000/healthcheck", host: "onlyoffice.DOMAINNAME.TLD"
Feb 26 13:34:10172.18.0.1 - - [26/Feb/2024:12:34:10 +0000] "GET /healthcheck HTTP/1.1" 502 150 "-" "Mozilla (CloudronHealth)"
Feb 26 13:34:10=> Healtheck error got response status 502
Feb 26 13:34:102024-02-26 12:34:10,462 INFO reaped unknown pid 527 (terminated by SIGKILL)
Feb 26 13:34:102024-02-26 12:34:10,462 INFO reaped unknown pid 527 (terminated by SIGKILL)
Feb 26 13:34:192024-02-26 12:34:19,472 INFO reaped unknown pid 515 (terminated by SIGKILL)
Feb 26 13:34:192024-02-26 12:34:19,472 INFO reaped unknown pid 515 (terminated by SIGKILL)
Feb 26 13:34:202024/02/26 12:34:20 [error] 173#173: *7 connect() failed (111: Unknown error) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /healthcheck HTTP/1.1", upstream: "http://[::1]:8000/healthcheck", host: "onlyoffice.DOMAINNAME.TLD"
Feb 26 13:34:202024/02/26 12:34:20 [error] 173#173: *7 connect() failed (111: Unknown error) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /healthcheck HTTP/1.1", upstream: "http://127.0.0.1:8000/healthcheck", host: "onlyoffice.DOMAINNAME.TLD"
Feb 26 13:34:20172.18.0.1 - - [26/Feb/2024:12:34:20 +0000] "GET /healthcheck HTTP/1.1" 502 150 "-" "Mozilla (CloudronHealth)"
Feb 26 13:34:20=> Healtheck error got response status 502
Feb 26 13:34:212024-02-26 12:34:21,220 INFO reaped unknown pid 469 (terminated by SIGKILL)
Feb 26 13:34:212024-02-26 12:34:21,220 INFO reaped unknown pid 469 (terminated by SIGKILL)
Feb 26 13:34:212024-02-26 12:34:21,220 INFO reaped unknown pid 475 (terminated by SIGKILL)
Feb 26 13:34:212024-02-26 12:34:21,220 INFO reaped unknown pid 475 (terminated by SIGKILL)
Feb 26 13:34:232024-02-26 12:34:23,224 INFO reaped unknown pid 441 (terminated by SIGKILL)
Feb 26 13:34:232024-02-26 12:34:23,224 INFO reaped unknown pid 441 (terminated by SIGKILL)
Feb 26 13:34:232024-02-26 12:34:23,225 INFO reaped unknown pid 481 (terminated by SIGKILL)
Feb 26 13:34:232024-02-26 12:34:23,225 INFO reaped unknown pid 481 (terminated by SIGKILL)
Feb 26 13:34:232024-02-26 12:34:23,225 INFO reaped unknown pid 534 (terminated by SIGKILL)
Feb 26 13:34:232024-02-26 12:34:23,225 INFO reaped unknown pid 534 (terminated by SIGKILL)
Feb 26 13:34:242024-02-26 12:34:24,264 INFO reaped unknown pid 609 (terminated by SIGKILL)
Feb 26 13:34:242024-02-26 12:34:24,264 INFO reaped unknown pid 609 (terminated by SIGKILL)
Feb 26 13:34:252024-02-26 12:34:25,312 INFO reaped unknown pid 561 (terminated by SIGKILL)
Feb 26 13:34:252024-02-26 12:34:25,312 INFO reaped unknown pid 561 (terminated by SIGKILL)
Feb 26 13:34:252024-02-26 12:34:25,312 INFO reaped unknown pid 627 (terminated by SIGKILL)
Feb 26 13:34:252024-02-26 12:34:25,312 INFO reaped unknown pid 627 (terminated by SIGKILL)
Feb 26 13:34:262024-02-26 12:34:26,330 INFO reaped unknown pid 495 (terminated by SIGKILL)
Feb 26 13:34:262024-02-26 12:34:26,330 INFO reaped unknown pid 495 (terminated by SIGKILL)
Feb 26 13:34:262024-02-26 12:34:26,330 INFO reaped unknown pid 552 (terminated by SIGKILL)
Feb 26 13:34:262024-02-26 12:34:26,330 INFO reaped unknown pid 552 (terminated by SIGKILL)
Feb 26 13:34:262024-02-26 12:34:26,330 INFO reaped unknown pid 603 (terminated by SIGKILL)
Feb 26 13:34:262024-02-26 12:34:26,330 INFO reaped unknown pid 603 (terminated by SIGKILL)
Feb 26 13:34:272024-02-26 12:34:27,349 INFO reaped unknown pid 597 (terminated by SIGKILL)
Feb 26 13:34:272024-02-26 12:34:27,349 INFO reaped unknown pid 597 (terminated by SIGKILL)
Feb 26 13:34:302024/02/26 12:34:30 [error] 173#173: *10 connect() failed (111: Unknown error) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /healthcheck HTTP/1.1", upstream: "http://127.0.0.1:8000/healthcheck", host: "onlyoffice.DOMAINNAME.TLD"
Feb 26 13:34:302024/02/26 12:34:30 [error] 173#173: *10 connect() failed (111: Unknown error) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /healthcheck HTTP/1.1", upstream: "http://[::1]:8000/healthcheck", host: "onlyoffice.DOMAINNAME.TLD"
Feb 26 13:34:30=> Healtheck error got response status 502
Feb 26 13:34:30172.18.0.1 - - [26/Feb/2024:12:34:30 +0000] "GET /healthcheck HTTP/1.1" 502 150 "-" "Mozilla (CloudronHealth)"
Feb 26 13:34:422024/02/26 12:34:42 [error] 173#173: *13 connect() failed (111: Unknown error) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /healthcheck HTTP/1.1", upstream: "http://[::1]:8000/healthcheck", host: "onlyoffice.DOMAINNAME.TLD"
Feb 26 13:34:422024/02/26 12:34:42 [error] 173#173: *13 connect() failed (111: Unknown error) while connecting to upstream, client: 172.18.0.1, server: , request: "GET /healthcheck HTTP/1.1", upstream: "http://127.0.0.1:8000/healthcheck", host: "onlyoffice.DOMAINNAME.TLD"
Feb 26 13:34:42172.18.0.1 - - [26/Feb/2024:12:34:42 +0000] "GET /healthcheck HTTP/1.1" 502 150 "-" "Mozilla (CloudronHealth)"
Feb 26 13:34:42=> Healtheck error got response status 502
Feb 26 13:34:42[2024-02-26T12:34:42.446] [WARN] [localhost] [docId] [userId] nodeJS - num of CPUs: 32; availableParallelism: undefined
Feb 26 13:34:422024-02-26 12:34:42,449 INFO reaped unknown pid 533 (terminated by SIGKILL)
Feb 26 13:34:422024-02-26 12:34:42,449 INFO reaped unknown pid 533 (terminated by SIGKILL)
Feb 26 13:34:422024-02-26 12:34:42,449 INFO reaped unknown pid 575 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,449 INFO reaped unknown pid 575 (exit status 0)
Feb 26 13:34:42[2024-02-26T12:34:42.464] [WARN] [localhost] [docId] [userId] nodeJS - update cluster with 32 workers
Feb 26 13:34:422024-02-26 12:34:42,464 INFO reaped unknown pid 569 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,464 INFO reaped unknown pid 569 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,464 INFO reaped unknown pid 615 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,464 INFO reaped unknown pid 615 (exit status 0)
Feb 26 13:34:42[2024-02-26T12:34:42.478] [WARN] [localhost] [docId] [userId] nodeJS - worker 707 started.
Feb 26 13:34:42[2024-02-26T12:34:42.485] [WARN] [localhost] [docId] [userId] nodeJS - worker 708 started.
Feb 26 13:34:422024-02-26 12:34:42,485 INFO reaped unknown pid 581 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,485 INFO reaped unknown pid 581 (exit status 0)
Feb 26 13:34:42[2024-02-26T12:34:42.491] [WARN] [localhost] [docId] [userId] nodeJS - worker 719 started.
Feb 26 13:34:42[2024-02-26T12:34:42.495] [WARN] [localhost] [docId] [userId] nodeJS - worker 720 started.
Feb 26 13:34:42[2024-02-26T12:34:42.501] [WARN] [localhost] [docId] [userId] nodeJS - worker 726 started.
Feb 26 13:34:42[2024-02-26T12:34:42.511] [WARN] [localhost] [docId] [userId] nodeJS - worker 737 started.
Feb 26 13:34:42[2024-02-26T12:34:42.519] [WARN] [localhost] [docId] [userId] nodeJS - worker 751 started.
Feb 26 13:34:42[2024-02-26T12:34:42.529] [WARN] [localhost] [docId] [userId] nodeJS - worker 757 started.
Feb 26 13:34:42[2024-02-26T12:34:42.536] [WARN] [localhost] [docId] [userId] nodeJS - worker 762 started.
Feb 26 13:34:422024-02-26 12:34:42,536 INFO reaped unknown pid 545 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,536 INFO reaped unknown pid 545 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,537 INFO reaped unknown pid 631 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,537 INFO reaped unknown pid 631 (exit status 0)
Feb 26 13:34:42[2024-02-26T12:34:42.547] [WARN] [localhost] [docId] [userId] nodeJS - worker 773 started.
Feb 26 13:34:42[2024-02-26T12:34:42.558] [WARN] [localhost] [docId] [userId] nodeJS - worker 783 started.
Feb 26 13:34:42[2024-02-26T12:34:42.566] [WARN] [localhost] [docId] [userId] nodeJS - worker 793 started.
Feb 26 13:34:422024-02-26 12:34:42,567 INFO reaped unknown pid 651 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,567 INFO reaped unknown pid 651 (exit status 0)
Feb 26 13:34:42[2024-02-26T12:34:42.578] [WARN] [localhost] [docId] [userId] nodeJS - worker 803 started.
Feb 26 13:34:42[2024-02-26T12:34:42.590] [WARN] [localhost] [docId] [userId] nodeJS - worker 813 started.
Feb 26 13:34:422024-02-26 12:34:42,590 INFO reaped unknown pid 591 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,590 INFO reaped unknown pid 591 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,591 INFO reaped unknown pid 634 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,591 INFO reaped unknown pid 634 (exit status 0)
Feb 26 13:34:42[2024-02-26T12:34:42.603] [WARN] [localhost] [docId] [userId] nodeJS - worker 827 started.
Feb 26 13:34:422024-02-26 12:34:42,603 INFO reaped unknown pid 663 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,603 INFO reaped unknown pid 663 (exit status 0)
Feb 26 13:34:42[2024-02-26T12:34:42.613] [WARN] [localhost] [docId] [userId] nodeJS - worker 833 started.
Feb 26 13:34:42[2024-02-26T12:34:42.624] [WARN] [localhost] [docId] [userId] nodeJS - worker 843 started.
Feb 26 13:34:422024-02-26 12:34:42,625 INFO reaped unknown pid 458 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,625 INFO reaped unknown pid 458 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,625 INFO reaped unknown pid 645 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,625 INFO reaped unknown pid 645 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,625 INFO reaped unknown pid 669 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,625 INFO reaped unknown pid 669 (exit status 0)
Feb 26 13:34:42[2024-02-26T12:34:42.634] [WARN] [localhost] [docId] [userId] nodeJS - worker 849 started.
Feb 26 13:34:42[2024-02-26T12:34:42.640] [WARN] [localhost] [docId] [userId] nodeJS - worker 855 started.
Feb 26 13:34:422024-02-26 12:34:42,641 INFO reaped unknown pid 621 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,641 INFO reaped unknown pid 621 (exit status 0)
Feb 26 13:34:42[2024-02-26T12:34:42.645] [WARN] [localhost] [docId] [userId] nodeJS - worker 856 started.
Feb 26 13:34:42[2024-02-26T12:34:42.650] [WARN] [localhost] [docId] [userId] nodeJS - worker 862 started.
Feb 26 13:34:42[2024-02-26T12:34:42.659] [WARN] [localhost] [docId] [userId] nodeJS - worker 868 started.
Feb 26 13:34:42[2024-02-26T12:34:42.674] [WARN] [localhost] [docId] [userId] nodeJS - worker 879 started.
Feb 26 13:34:42[2024-02-26T12:34:42.685] [WARN] [localhost] [docId] [userId] nodeJS - worker 885 started.
Feb 26 13:34:42[2024-02-26T12:34:42.697] [WARN] [localhost] [docId] [userId] nodeJS - worker 891 started.
Feb 26 13:34:42[2024-02-26T12:34:42.714] [WARN] [localhost] [docId] [userId] nodeJS - worker 901 started.
Feb 26 13:34:42[2024-02-26T12:34:42.726] [WARN] [localhost] [docId] [userId] nodeJS - worker 907 started.
Feb 26 13:34:42[2024-02-26T12:34:42.754] [WARN] [localhost] [docId] [userId] nodeJS - worker 908 started.
Feb 26 13:34:422024-02-26 12:34:42,758 INFO reaped unknown pid 652 (exit status 0)
Feb 26 13:34:422024-02-26 12:34:42,758 INFO reaped unknown pid 652 (exit status 0)
Feb 26 13:34:42[2024-02-26T12:34:42.770] [WARN] [localhost] [docId] [userId] nodeJS - worker 914 started.
Feb 26 13:34:42[2024-02-26T12:34:42.792] [WARN] [localhost] [docId] [userId] nodeJS - worker 925 started.
Feb 26 13:34:42[2024-02-26T12:34:42.811] [WARN] [localhost] [docId] [userId] nodeJS - worker 931 started.
Feb 26 13:34:42[2024-02-26T12:34:42.837] [WARN] [localhost] [docId] [userId] nodeJS - worker 932 started.
Feb 26 13:34:43box:taskworker Starting task 850. Logs are at /home/yellowtent/platformdata/logs/c03a8f57-4218-49ed-9462-69e17e7cf0ad/apptask.log
Feb 26 13:34:43box:apptask run: startTask installationState: pending_stop runState: stopped
Feb 26 13:34:43box:tasks update 850: {"percent":20,"message":"Stopping container"}
Feb 26 13:34:43box:shell reload spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
Feb 26 13:34:432024-02-26 12:34:43,335 INFO exited: docservice (terminated by SIGKILL; not expected)
Feb 26 13:34:432024-02-26 12:34:43,335 INFO exited: docservice (terminated by SIGKILL; not expected)
Feb 26 13:34:43[2024-02-26T12:34:43.448] [WARN] [localhost] [docId] [userId] nodeJS - worker 719 died (code = null; signal = SIGKILL).
Feb 26 13:34:432024-02-26 12:34:43,458 INFO spawned: 'docservice' with pid 943
Feb 26 13:34:432024-02-26 12:34:43,458 INFO spawned: 'docservice' with pid 943
Feb 26 13:34:432024-02-26 12:34:43,459 WARN received SIGTERM indicating exit request
Feb 26 13:34:432024-02-26 12:34:43,459 WARN received SIGTERM indicating exit request
Feb 26 13:34:432024-02-26 12:34:43,461 INFO waiting for converter, docservice, metrics, nginx to die
Feb 26 13:34:432024-02-26 12:34:43,461 INFO waiting for converter, docservice, metrics, nginx to die
Feb 26 13:34:43[2024-02-26T12:34:43.465] [WARN] [localhost] [docId] [userId] nodeJS - worker 944 started.
Feb 26 13:34:43[2024-02-26T12:34:43.467] [WARN] [localhost] [docId] [userId] nodeJS - worker 726 died (code = null; signal = SIGKILL).
Feb 26 13:34:43[2024-02-26T12:34:43.487] [WARN] [localhost] [docId] [userId] nodeJS - worker 945 started.
Feb 26 13:34:43[2024-02-26T12:34:43.937] [WARN] [localhost] [docId] [userId] nodeJS - worker 720 died (code = null; signal = SIGKILL).
Feb 26 13:34:44[2024-02-26T12:34:44.040] [WARN] [localhost] [docId] [userId] nodeJS - worker 956 started.
Feb 26 13:34:44[2024-02-26T12:34:44.041] [WARN] [localhost] [docId] [userId] nodeJS - worker 762 died (code = null; signal = SIGKILL).
Feb 26 13:34:44[2024-02-26T12:34:44.052] [WARN] [localhost] [docId] [userId] nodeJS - worker 957 started.
Feb 26 13:34:442024-02-26 12:34:44,076 INFO stopped: nginx (exit status 0)
Feb 26 13:34:442024-02-26 12:34:44,076 INFO stopped: nginx (exit status 0)
Feb 26 13:34:44[2024-02-26T12:34:44.286] [WARN] [localhost] [docId] [userId] nodeJS - worker 737 died (code = null; signal = SIGKILL).
Feb 26 13:34:44[2024-02-26T12:34:44.309] [WARN] [localhost] [docId] [userId] nodeJS - worker 973 started.
Feb 26 13:34:44[2024-02-26T12:34:44.401] [WARN] [localhost] [docId] [userId] nodeJS - worker 708 died (code = null; signal = SIGKILL).
Feb 26 13:34:44[2024-02-26T12:34:44.413] [WARN] [localhost] [docId] [userId] nodeJS - worker 974 started.
Feb 26 13:34:452024-02-26 12:34:45,425 INFO success: docservice entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:34:452024-02-26 12:34:45,425 INFO success: docservice entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Feb 26 13:34:46[2024-02-26T12:34:46.290] [WARN] [localhost] [docId] [userId] nodeJS - worker 707 died (code = null; signal = SIGKILL).
Feb 26 13:34:46[2024-02-26T12:34:46.315] [WARN] [localhost] [docId] [userId] nodeJS - worker 980 started.
Feb 26 13:34:472024-02-26 12:34:47,333 INFO waiting for converter, docservice, metrics to die
Feb 26 13:34:472024-02-26 12:34:47,333 INFO waiting for converter, docservice, metrics to die
Feb 26 13:34:48[2024-02-26T12:34:48.004] [WARN] [localhost] [docId] [userId] nodeJS - worker 803 died (code = null; signal = SIGKILL).
Feb 26 13:34:48[2024-02-26T12:34:48.011] [WARN] [localhost] [docId] [userId] nodeJS - worker 991 started.
Feb 26 13:34:48Flushing stats at Mon Feb 26 2024 12:34:44 GMT+0000 (Coordinated Universal Time)
Feb 26 13:34:48[2024-02-26T12:34:48.242] [WARN] [localhost] [docId] [userId] nodeJS - worker 751 died (code = null; signal = SIGKILL).
Feb 26 13:34:48[2024-02-26T12:34:48.251] [WARN] [localhost] [docId] [userId] nodeJS - worker 997 started.
Feb 26 13:34:48[2024-02-26T12:34:48.252] [WARN] [localhost] [docId] [userId] nodeJS - worker 757 died (code = null; signal = SIGKILL).
Feb 26 13:34:48[2024-02-26T12:34:48.270] [WARN] [localhost] [docId] [userId] nodeJS - worker 998 started.
Feb 26 13:34:48{
Feb 26 13:34:48counters: {
Feb 26 13:34:48'statsd.bad_lines_seen': 0,
Feb 26 13:34:48'statsd.packets_received': 0,
Feb 26 13:34:48'statsd.metrics_received': 0
Feb 26 13:34:48},
Feb 26 13:34:48timers: {},
Feb 26 13:34:48gauges: {},
Feb 26 13:34:48timer_data: {},
Feb 26 13:34:48counter_rates: {
Feb 26 13:34:48'statsd.bad_lines_seen': 0,
Feb 26 13:34:48'statsd.packets_received': 0,
Feb 26 13:34:48'statsd.metrics_received': 0
Feb 26 13:34:48},
Feb 26 13:34:48sets: {},
Feb 26 13:34:48pctThreshold: [ 90 ]
Feb 26 13:34:48}
Feb 26 13:34:48[2024-02-26T12:34:48.462] [WARN] [localhost] [docId] [userId] nodeJS - worker 773 died (code = null; signal = SIGKILL).
Feb 26 13:34:482024-02-26 12:34:48,491 INFO stopped: metrics (exit status 0)
Feb 26 13:34:482024-02-26 12:34:48,491 INFO stopped: metrics (exit status 0)
Feb 26 13:34:482024-02-26 12:34:48,504 INFO stopped: docservice (terminated by SIGTERM)
Feb 26 13:34:482024-02-26 12:34:48,504 INFO stopped: docservice (terminated by SIGTERM)
Feb 26 13:34:492024-02-26 12:34:49,609 INFO stopped: converter (terminated by SIGTERM)
Feb 26 13:34:492024-02-26 12:34:49,609 INFO stopped: converter (terminated by SIGTERM)
Feb 26 13:34:53box:tasks update 850: {"percent":50,"message":"Stopping app services"}
Feb 26 13:34:542024-02-26 12:34:54,108 WARN received SIGTERM indicating exit request
Feb 26 13:34:542024-02-26 12:34:54,108 INFO waiting for redis, redis-service to die
Feb 26 13:34:542024-02-26 12:34:54,111 INFO stopped: redis-service (terminated by SIGTERM)
Feb 26 13:34:5412:signal-handler (1708950894) Received SIGTERM scheduling shutdown...
Feb 26 13:34:5412:M 26 Feb 2024 12:34:54.174 * User requested shutdown...
Feb 26 13:34:5412:M 26 Feb 2024 12:34:54.174 * Saving the final RDB snapshot before exiting.
Feb 26 13:34:5412:M 26 Feb 2024 12:34:54.178 * DB saved on disk
Feb 26 13:34:5412:M 26 Feb 2024 12:34:54.178 * Removing the pid file.
Feb 26 13:34:5412:M 26 Feb 2024 12:34:54.178 # Redis is now ready to exit, bye bye...
Feb 26 13:34:542024-02-26 12:34:54,179 INFO stopped: redis (exit status 0)
Feb 26 13:34:54box:tasks update 850: {"percent":100,"message":"Done"}
Feb 26 13:34:54box:taskworker Task took 11.177 seconds
Feb 26 13:34:54box:tasks setCompleted - 850: {"result":null,"error":null}
Feb 26 13:34:54box:tasks update 850: {"percent":100,"result":null,"error":null}

[nebulon]

Just since there was a new update today, do you see the same behavior with version 8.0.1 now?
Also it appears from your logs that the worker keeps on exiting, do you see any memory shortage or so? Can you try to increase the memory limit, just so we can rule this out?

[im-fabian]

This update does not fix it but it changes the behavior. Since the update I see a memory shortage. However it is impossible to increase the memory limit. When I try to do this I get stuck with this error message:

Inactive: Error getting IP of redis-da6b0965-386f-467c-a9b1-53f0484c81d8 service

I already tired deleting the App and creating a brand-new one. Same issue when trying to resize.

[im-fabian]

Okay, I got it fixed now; rebooting did the trick with redis.
For those who come along with this thread:

I guess when used on machines with many CPU-Cores, especially with hyperhreading, OnlyOffice starts some kind of worker per Core, which then eats up some memory. In my case, with 16 cores (32 HT-Cores), this eats about 3.25 Gbytes. If the memory limit is too low, this results in a restart loop. Since I resized the memory to 6 GB, all is fine.

[girish]

@im-fabian Investigating this a bit more. I found that there is a config variable FileConverter.converter.maxprocesscount . This defaults to 1. In new instances, I can see that one 1 instance of that converter runs by default.

However, I do see the behavior that you mention of many converter processes in the Enterprise version. Are you using Onlyoffice EE ? That config variable seemingly has no effect in the enterprise version.

[girish]

The code is from https://github.com/ONLYOFFICE/server/blob/7d2104f16e0cda9cdb11002c581a8792e9e915d7/FileConverter/sources/convertermaster.js#L47 but the EE version uses some binary looks like.

[im-fabian]

@girish I can confirm, we are using the EE Version. So this might be a bug in the OnlyOffice EE-Version not Cloudron?

[girish]

@im-fabian in many GitHub issues (but this only applies to the community version), they mention that they launch as many processes as CPUs . Maybe this is the default in the EE. If you can reach out to their support and ask if there is a config variable for this, I think we can fix the package accordingly.

[im-fabian]

@girish I opened an ticket with OnlyOffice on that cause. Will keep you posted.

[im-fabian]

@girish the support replied. I did not have the time to look into it, but wanted to share the feedback with you asap:

You are correct, but you can decrease the amount of converter processes by changing maxprocesscount parameter in default.json file:
/etc/onlyoffice/documentserver/default.json
 
"FileConverter": {
  "converter": {
   
   "maxprocesscount": 1,
The default value is "1" but you can change it (to 0.5 or 0.25 for instance) to decrease the number of processes, since maxprocesscount is a multiplier, the number of converter processes will be equal to maxprocesscount * number of CPUs.
 
Please, do not forget to restart Document Server's services after applying the changes:
supervisorctl restart all