OnlyOffice is unresponsive after update to 1.18

[jagan]

Earlier Logs

2024-10-22T22:30:07.746Z box:backuptask copy: copied successfully to 2024-10-22-223000-579/app_docs.cloudintegral.com_v1.18.0. Took 0.309 seconds
2024-10-22T22:30:07.749Z box:tasks update 20566: {"percent":20,"message":"Updating checklist"}
2024-10-22T22:30:07.755Z box:tasks update 20566: {"percent":25,"message":"Downloading image"}
2024-10-22T22:30:07.767Z box:shell file execArgs: df ["-B1","--output=source,fstype,size,used,avail,pcent,target","/var/lib/docker"]
2024-10-22T22:30:07.771Z box:docker downloadImage cloudron/com.onlyoffice.coudronapp:20241022-130707-971fcd61e
2024-10-22T22:30:07.773Z box:docker pullImage: will pull cloudron/com.onlyoffice.coudronapp:20241022-130707-971fcd61e. auth: no
2024-10-22T22:30:09.100Z box:docker pullImage: {"status":"Pulling from cloudron/com.onlyoffice.coudronapp","id":"20241022-130707-971fcd61e"}
2024-10-22T22:30:09.101Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"445a6a12be2b"}
2024-10-22T22:30:09.104Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"4cfe0cdc770e"}
2024-10-22T22:30:09.108Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"e6a0eb1fa9b7"}
2024-10-22T22:30:09.112Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"e995e5b957f9"}
2024-10-22T22:30:09.115Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"e6d226089461"}
2024-10-22T22:30:09.118Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"b3243df2776e"}
2024-10-22T22:30:09.122Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"debd247c1af3"}
2024-10-22T22:30:09.125Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"ea1f575bfbef"}
2024-10-22T22:30:09.128Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"566e1eaf48e1"}
2024-10-22T22:30:09.132Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"68da526a8544"}
2024-10-22T22:30:09.135Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"2f3677647d18"}
2024-10-22T22:30:09.138Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"90984d402264"}
2024-10-22T22:30:09.144Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"802deede2955"}
2024-10-22T22:30:09.147Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"1861003a8fe7"}
2024-10-22T22:30:09.150Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"524cf22ec2b3"}
2024-10-22T22:30:09.153Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"758195db2cd9"}
2024-10-22T22:30:09.156Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"7af86d1134c7"}
2024-10-22T22:30:09.161Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"a8755a249dc6"}
2024-10-22T22:30:09.165Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"8baacb295220"}
2024-10-22T22:30:09.168Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"fa01604db307"}
2024-10-22T22:30:09.172Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"3c23621e8d0f"}
2024-10-22T22:30:09.175Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"5853e682f0d7"}
2024-10-22T22:30:09.179Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"c46e254b8a9a"}
2024-10-22T22:30:09.183Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"c1e719d63179"}
2024-10-22T22:30:09.187Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"9b9d10c3e182"}
2024-10-22T22:30:09.190Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"a2f2b0925732"}
2024-10-22T22:30:09.194Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"90ddba323643"}
2024-10-22T22:30:09.199Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"04f95a19ff53"}
2024-10-22T22:30:09.203Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"2391cdb18ac1"}
2024-10-22T22:30:09.207Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"9c076dec7d14"}
2024-10-22T22:30:09.210Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"f640656c91ed"}
2024-10-22T22:30:09.214Z box:docker pullImage: {"status":"Already exists","progressDetail":{},"id":"ee1ae70bb368"}
2024-10-22T22:30:09.218Z box:docker pullImage: {"status":"Pulling fs layer","progressDetail":{},"id":"06b64bf0aa3b"}
2024-10-22T22:30:09.543Z box:docker pullImage: {"status":"Downloading","progressDetail":{"current":2266,"total":2266},"progress":"[==================================================>]  2.266kB/2.266kB","id":"06b64bf0aa3b"}
2024-10-22T22:30:09.543Z box:docker pullImage: {"status":"Downloading","progressDetail":{"current":689,"total":2266},"progress":"[===============>                                   ]     689B/2.266kB","id":"06b64bf0aa3b"}
2024-10-22T22:30:09.543Z box:docker pullImage: {"status":"Verifying Checksum","progressDetail":{},"id":"06b64bf0aa3b"}
2024-10-22T22:30:09.544Z box:docker pullImage: {"status":"Download complete","progressDetail":{},"id":"06b64bf0aa3b"}
2024-10-22T22:30:09.544Z box:docker pullImage: {"status":"Extracting","progressDetail":{"current":2266,"total":2266},"progress":"[==================================================>]  2.266kB/2.266kB","id":"06b64bf0aa3b"}
2024-10-22T22:30:09.560Z box:docker pullImage: {"status":"Extracting","progressDetail":{"current":2266,"total":2266},"progress":"[==================================================>]  2.266kB/2.266kB","id":"06b64bf0aa3b"}
2024-10-22T22:30:09.569Z box:docker pullImage: {"status":"Pull complete","progressDetail":{},"id":"06b64bf0aa3b"}
2024-10-22T22:30:09.577Z box:docker pullImage: {"status":"Digest: sha256:87c419b30ba005fc05130f70cec4e3c76c49c442617befbaf9232b927ab6cc14"}
2024-10-22T22:30:09.580Z box:docker pullImage: {"status":"Status: Downloaded newer image for cloudron/com.onlyoffice.coudronapp:20241022-130707-971fcd61e"}
2024-10-22T22:30:09.582Z box:docker downloaded image cloudron/com.onlyoffice.coudronapp:20241022-130707-971fcd61e . error: false
2024-10-22T22:30:09.582Z box:tasks update 20566: {"percent":35,"message":"Deleting old containers"}
2024-10-22T22:30:09.585Z box:apptask deleteContainer: deleting app containers (app, scheduler)
2024-10-22T22:30:09.585Z box:shell removeLogrotateConfig /usr/bin/sudo -S /home/yellowtent/box/src/scripts/configurelogrotate.sh remove fddc9476-71ad-4285-b986-7ea32b383fa2
2024-10-22T22:30:10.783Z box:services teardownAddons: Tearing down []
2024-10-22T22:30:10.788Z box:tasks update 20566: {"percent":45,"message":"Downloading icon"}
2024-10-22T22:30:10.789Z box:apptask downloadIcon: Downloading icon of com.onlyoffice.coudronapp@1.18.1
2024-10-22T22:30:11.254Z box:tasks update 20566: {"percent":60,"message":"Updating addons"}
2024-10-22T22:30:11.257Z box:services setupAddons: Setting up ["localstorage","redis","postgresql"]
2024-10-22T22:30:11.257Z box:services setupAddons: setting up addon localstorage with options {}
2024-10-22T22:30:11.257Z box:services setupLocalStorage
2024-10-22T22:30:11.257Z box:shell createVolume /usr/bin/sudo -S /home/yellowtent/box/src/scripts/mkdirvolume.sh /home/yellowtent/appsdata/fddc9476-71ad-4285-b986-7ea32b383fa2/data
2024-10-22T22:30:11.267Z box:services setupAddons: setting up addon redis with options {"noPassword":true}
2024-10-22T22:30:11.269Z box:services Re-using existing redis container with state: {"Status":"running","Running":true,"Paused":false,"Restarting":false,"OOMKilled":false,"Dead":false,"Pid":13346,"ExitCode":0,"Error":"","StartedAt":"2024-10-22T18:24:56.719658355Z","FinishedAt":"2024-10-22T18:23:59.045408638Z"}
2024-10-22T22:30:11.273Z box:services Waiting for redis-fddc9476-71ad-4285-b986-7ea32b383fa2
2024-10-22T22:30:11.481Z box:services Setting up postgresql
2024-10-22T22:30:11.481Z box:services setupAddons: setting up addon postgresql with options {}
2024-10-22T22:30:11.511Z box:services Setting postgresql addon config to [{"name":"CLOUDRON_POSTGRESQL_URL","value":"postgres://userfddc--------------redacted------------@postgresql/dbfddc--------------redacted------------"},{"name":"CLOUDRON_POSTGRESQL_USERNAME","value":"userfddc947671ad4285b9867ea32b383fa2"},{"name":"CLOUDRON_POSTGRESQL_PASSWORD","value":"d--------------redacted------------"},{"name":"CLOUDRON_POSTGRESQL_HOST","value":"postgresql"},{"name":"CLOUDRON_POSTGRESQL_PORT","value":"5432"},{"name":"CLOUDRON_POSTGRESQL_DATABASE","value":"dbfddc--------------redacted------------"}]
2024-10-22T22:30:11.520Z box:tasks update 20566: {"percent":70,"message":"Creating container"}
2024-10-22T22:30:11.521Z box:apptask createContainer: creating container
2024-10-22T22:30:11.643Z box:shell addLogrotateConfig /usr/bin/sudo -S /home/yellowtent/box/src/scripts/configurelogrotate.sh add fddc9476-71ad-4285-b986-7ea32b383fa2 /tmp/fddc9476-71ad-4285-b986-7ea32b383fa2.logrotate
2024-10-22T22:30:11.654Z box:apptask startApp: starting container
2024-10-22T22:30:11.920Z box:tasks update 20566: {"percent":90,"message":"Configuring reverse proxy"}
2024-10-22T22:30:11.923Z box:tasks update 20566: {"percent":100,"message":"Done"}
2024-10-22T22:30:11.927Z box:tasks setCompleted - 20566: {"result":null,"error":null}
2024-10-22T22:30:11.927Z box:tasks update 20566: {"percent":100,"result":null,"error":null}
2024-10-22T22:30:11.927Z box:taskworker Task took 11.39 seconds
2024-10-22T22:30:11Z - [GET] /healthcheck
2024-10-23T00:00:10.000Z => Healtheck error got response status 502

[nebulon]

The "Already exists" status is referring to layers of the app package image. That is expected and good.

"=> Healtheck error got response status 502" is showing some issue, although very limited insight.

Have you restarted the app? Does it maybe run out of memory? Instances here work fine, so shouldn't be a generic issue.

[imc67]

I had the same yesterday, didn’t have time to debug so restored backup.

[jagan]

@nebulon Yep, I did the usual: restarted, increased memory, and even restarted the redis. A reboot was due so I did that too for good measure.

This is the limit of what I know to do

I kicked the tires around a bit and peeked under the logs. Couldn't find anything. This is supposed to be a simple application, with no plugins or such to complicate stuff.

Maybe I will have to restore the backup and wait for a newer version to test.

[jdaviescoates]

@imc67 said in OnlyOffice is unresponsive after update to 1.18:

I had the same yesterday, didn’t have time to debug so restored backup.

Me too. But restoring previous backups didn't work. I had to go back to 1.17.2 which still didn't work until I restarted post restore. Now have 1.17.2 running again.

Here are some logs:
https://paste.uniteddiversity.coop/?9e0d71d64bec7812#5oM2ZdBDZLSqMm9oBGLpjUxYQWjzdeeuWTzZXW5KW3XF

[jdaviescoates]

@nebulon said in OnlyOffice is unresponsive after update to 1.18:

Instances here work fine, so shouldn't be a generic issue.

At least 3 of us are here who are having this issue so I'd guess it's actually pretty widespread.

[jdaviescoates]

@jdaviescoates said in OnlyOffice is unresponsive after update to 1.18:

@nebulon said in OnlyOffice is unresponsive after update to 1.18:

Instances here work fine, so shouldn't be a generic issue.

At least 3 of us are here who are having this issue so I'd guess it's actually pretty widespread.

Although I just checked and fresh install works fine. So it's seemingly just updates from 1.17 - 1.18 that aren't working.

[imc67]

@girish @nebulon any update on this topic after 5 days of waiting?

[nebulon]

at least so far I wasn't able to reproduce this to be able to debug that and without any further errors or logs to work with, not sure what can be done.

[tobiasb]

Depending how old your onlyoffice is:

PGPASSWORD=${CLOUDRON_POSTGRESQL_PASSWORD} psql -h ${CLOUDRON_POSTGRESQL_HOST} -p ${CLOUDRON_POSTGRESQL_PORT} -U ${CLOUDRON_POSTGRESQL_USERNAME} -d ${CLOUDRON_POSTGRESQL_DATABASE} -f /var/www/onlyoffice/documentserver/server/schema/postgresql/upgrade/upgradev630.sql

PGPASSWORD=${CLOUDRON_POSTGRESQL_PASSWORD} psql -h ${CLOUDRON_POSTGRESQL_HOST} -p ${CLOUDRON_POSTGRESQL_PORT} -U ${CLOUDRON_POSTGRESQL_USERNAME} -d ${CLOUDRON_POSTGRESQL_DATABASE} -f /var/www/onlyoffice/documentserver/server/schema/postgresql/upgrade/upgradev710.sql

In case the db schema upgrade is already up2date, the output looks like:

root@78d36e9e-8352-4cd1-a356-42fa5e22702f:/app/code# PGPASSWORD=${CLOUDRON_POSTGRESQL_PASSWORD} psql -h ${CLOUDRON_POSTGRESQL_HOST} -p ${CLOUDRON_POSTGRESQL_PORT} -U ${CLOUDRON_POSTGRESQL_USERNAME} -d ${CLOUDRON_POSTGRESQL_DATABASE} -f /var/www/onlyoffice/documentserver/server/schema/postgresql/upgrade/upgradev630.sql 
psql:/var/www/onlyoffice/documentserver/server/schema/postgresql/upgrade/upgradev630.sql:15: NOTICE:  column created_at already exists.
psql:/var/www/onlyoffice/documentserver/server/schema/postgresql/upgrade/upgradev630.sql:15: NOTICE:  column password already exists.
DO

Based on the git history, cloudron only did the schema migration for 720.sql.

The Dockerfile should fail on docker build in case there new sql files and apply the sql files. atm rerunning the sql files does not exit with exit code > 0. But I believe it is better to store somewhere which sql-files was already applied, just for case sql file would break something on a rerun.

[nebulon]

Ah that is a good hint then. So we have to be more careful here with the db migrations going forward. For the time I guess a re-installation would work fine, especially since the app is mostly stateless anyways.

[imc67]

Re-install and config key solved it. Thanks!

[jdaviescoates]

@imc67 said in OnlyOffice is unresponsive after update to 1.18:

config key

config key?

[jagan]

@tobiasb This Greek and Latin is exactly why I am such a fan of Cloudron - users range from utterly computer challenged to gods of code and linux.

Thank you for solving the mystery for all of us.

We must reinstall with the same config key / secret then!

[imc67]

@jdaviescoates said in OnlyOffice is unresponsive after update to 1.18:

@imc67 said in OnlyOffice is unresponsive after update to 1.18:

config key

config key?

Secret (key) in the Config folder