Posts made by vladimir.d

@girish we are using drone app packaged by @fbartels.

@girish said in Drone:

@marcusquinn Which package are you using ? Or are you running this outside Cloudron ?

We are running Docker runner and SSH runner as docker containers and Exec runner as a service on the Cloudron host, basically exactly as described in the Drone documentation.
https://docs.drone.io/runner/overview/

@msbt said in Drone:

@marcusquinn same here, but the Drone runners are still not part of the equation, would be nice if they could work together with the docker addon to have it 100% "portable"

Maybe we'll package runners as cloudron apps later that could build & deploy an app on Cloudron.

@girish sorry, a school boy mistake, the Content-Type header did the trick, but I made a typo in the backupId, that's why I got the error.

still have no luck.

If I add -H 'Content-Type: application/json' to the backup restoring request I get:

{ "status": "Internal Server Error", "message": "Cannot read properties of null (reading 'manifest')" }

I tried to send the request using curl and postman, the response is the same. Cloudron is v7.3.2

ps. the CLI tool works well though.

@subven said in cloudron-cli in cron:

You did something like: '{"backupId": "2022-11-11-030000-618/app_archiv.domain.com_v2.2.1"}'?

No, I put Id value from the backup list response, e.g. a backup list response from REST API documentation:

{
  "backups": [
    {
      "id": "string",
      "remotePath": "2022-10-24-060000-679/app_cubby.nebulon.space_v1.0.1",
      "label": "My important backup",
      "identifier": "e7d4cc36-9ac0-4f93-8534-62514abe5d9c",
      "creationTime": 0,
      "type": "app",
      "state": "normal",
      "format": "rsync",
      "preserveSecs": 604800,
      "encryptionVersion": "string",
      "dependsOn": [
        null
      ],
      "manifest": null
    }
  ]
}

Do you mean remotePath value is being used as backupId in the backup restore request?

@girish said in cloudron-cli in cron:

Is the use case here to have some sort of a demo app instance that keeps resetting state ?

Instead of using the CLI tool, maybe you can just use the REST API for this. It's just a POST request. Something like:

curl -X POST -H 'Authorization: Bearer <token>' -d '{"backupId": "id"}' https://my.domain.com/api/v1/apps/<appid>/restore

I created a token in the profile and got backupId using the REST API call:

curl -X GET -H 'Authorization: Bearer <token>' https://my.domain/api/v1/apps/<appid>/backups

I get the following error on the backup restore REST API call:

{
  "status": "Bad Request",
  "message": "backupId must be non-empty string"
}

So I'm sure the token and backupId are correct and valid.
The CLI tool successfully restores the backup copy by the same token and backupId.
What am I missing?

@subven said in cloudron-cli in cron:

You can create a personal token and pass it to the cron cli command via --token Dunno how it works with 2FA enabled but there should be no need to manually log in.

EDIT: Just noticed that you want to run the cli command on the Cloudron host. The CLI tool is not intended to run on the Cloudron host itself!

Token seems to work very well.
What are drawbacks of using CLI tool on the Cloudron host?

@girish said in cloudron-cli in cron:

Is the use case here to have some sort of a demo app instance that keeps resetting state ?

Instead of using the CLI tool, maybe you can just use the REST API for this. It's just a POST request. Something like:

curl -X POST -H 'Authorization: Bearer <token>' -d '{"backupId": "id"}' https://my.domain.com/api/v1/apps/<appid>/restore

Yes, it's for a demo app instance and yes, REST API is a graceful solution.
I wonder if there is also a REST API call to get appid by CLOUDRON_APP_DOMAIN?

Hi, need a suggestion how to run cloudron-cli as a cronjob on the cloudron host. Namely we'd like to restore an app backup by cron.
Considering that the command requires cloudron login as admin first, I doubt it's secure to create and use an admin user without 2FA enabled so I'm thinking to use oathtool for 2FA in console.
I haven't tried it yet, so not sure it's doable or not.

Or maybe there is another easier way to perform such admin cloudron tasks without cloudron-cli on the cloudron server?

Any suggestions very welcomed!

We sorted this issue by amending /app/data/htaccess (/app/code/.htaccess) as follows:

<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteCond %{HTTP_USER_AGENT} DavClnt
  RewriteRule ^$ /remote.php/webdav/ [L,R=302]
  RewriteRule .* - [env=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
  RewriteRule ^\.well-known/host-meta /public.php?service=host-meta [QSA,L]
  RewriteRule ^\.well-known/host-meta\.json /public.php?service=host-meta-json [QSA,L]
-  RewriteRule ^\.well-known/webfinger /public.php?service=webfinger [QSA,L]
-  RewriteRule ^\.well-known/nodeinfo /public.php?service=nodeinfo [QSA,L]
+ RewriteRule ^/\.well-known/webfinger /index.php/.well-known/webfinger [R=301,L]
+ RewriteRule ^/\.well-known/nodeinfo /index.php/.well-known/nodeinfo [R=301,L]
  RewriteRule ^\.well-known/carddav https://%{HTTP_HOST}/remote.php/dav/ [R=301,L]
  RewriteRule ^\.well-known/caldav https://%{HTTP_HOST}/remote.php/dav/ [R=301,L]
  RewriteRule ^remote/(.*) remote.php [QSA,L]
  RewriteRule ^(?:build|tests|config|lib|3rdparty|templates)/.* - [R=404,L]
  RewriteCond %{REQUEST_URI} !^/\.well-known/(acme-challenge|pki-validation)/.*
  RewriteRule ^(?:\.|autotest|occ|issue|indie|db_|console).* - [R=404,L]
</IfModule>

@jdaviescoates said in How to cancel an update?:

@vladimir-d said in How to cancel an update?:

FTR, "Retry update task" and then "Restart App" have sorted the issue.

And then revert to previous backup?

@jdaviescoates It would be yes if there was a recent backup copy.

FTR, "Retry update task" and then "Restart App" have sorted the issue.

@girish the License has been added to the repository.
Please feel free to change anything if needed.

@nebulon for the beginning we'd like to enable slow queries log to identify what causes postgresql taking almost 100% of CPU time all the time.

@girish could you amend the postgresql image so supervisord runs postgresql using the config /var/lib/postgresql/12/main/postgresql.conf (/home/yellowtent/platformdata/postgresql/12/main/postgresql.conf)?

Namely I suggest creating a symlink to the file:

$ ln -s /var/lib/postgresql/12/main/postgresql.conf /etc/postgresql/12/main/postgresql.conf

OR changing supervisord config file /etc/supervisor/conf.d/postgresql.conf:

- command=/usr/lib/postgresql/12/bin/postmaster --config-file=/etc/postgresql/12/main/postgresql.conf
+ command=/usr/lib/postgresql/12/bin/postmaster --config-file=/var/lib/postgresql/12/main/postgresql.conf

We need to enable postgresql slow query logging and we have done the following:

$ docker exec -ti postgresql bash
$ psql -d MY_DATABASE_NAME

psql (12.10 (Ubuntu 12.10-1.pgdg20.04+1+b1))
Type "help" for help.

MY_DATABASE_NAME =# ALTER DATABASE MY_DATABASE_NAME SET log_min_duration_statement = 5000;

MY_DATABASE_NAME =# SELECT pg_sleep(15);

But the slow log file didn't appear anywhere (/run/postgresql, /var/log/postgresql, /var/lib/postgresql/12/main/).

We need to set logging_collector to on in the postgresql config /etc/postgresql/12/main/postgresql.conf which is read-only.

Also please advise how to amend postgresql configuration in reboot/upgrade safe way.

By default, Nextcloud generates photo thumbnails on-demand that leads to slow Photos thumbnails load. To speed it up we’re using the Preview Generator app for Nextcloud to pre-generate previews regularly.

First we set the default thumbnail sizes by using the "occ config" commands:

sudo -u www-data /usr/bin/php /app/code/occ config:app:set --value="32 256" previewgenerator squareSizes
sudo -u www-data /usr/bin/php /app/code/occ config:app:set --value="256 384" previewgenerator widthSizes
sudo -u www-data /usr/bin/php /app/code/occ config:app:set --value="256" previewgenerator heightSizes

Then we generated previews for all existent images:

sudo -u www-data /usr/bin/php /app/code/occ preview:generate-all -vvv

It took about 3 weeks on our host to generate preview images for 4Tb of data which takes additional 500Gb.

Once it's finished we setup a cron job to generate previews:

0 */6 * * * sudo -u www-data /usr/bin/php -f /app/code/occ preview:pre-generate

TBH, it helped us a little but our Photos are still loading for 50-60 seconds and we are seeking any solution to speed it up.

I've packaged netbox application available at

https://git.cloudron.io/vladimir.d/netbox

Just a few notes:

1. For now I haven't managed to run it properly as WSGI app with gunicorn. It's taking lots of server resources.
   It's running using the Dev server.
   https://docs.netbox.dev/en/stable/installation/3-netbox/#test-the-application

2. Super Admin username is specified in start.sh, the Postgress password is taken for Super Admin.

@girish it looks good.
What would involve to move existent data to the new data directory (/app/data/data)?

@girish It has never ended to backup Nextcloud app using the native cloudron backup routine in our case so we disabled them for this app at all. We use rclone to backup the data to S3 for this application. Restoring would be also done using rclone running under www-data user so there will be no issues with the ownership in our case.

Yes, agree the workaround could be updating ownership only of certain files and folders on start.
Maybe a system wide background service/job (maybe using AQMP) to fix permissions/ownership files when necessary in applications?

@girish well, I've updated the app so it triggered chowning again.

I created the flag /run/nextcloud/chowned manually, killed the chown process and restarted the app. The upgrade downtime was just a few minutes this time.

Ideally to have a workaround for upgrades as well (a flag in /app/data?).

@girish it looks promising, but a few points I need to mention about:

1. There is Preview Generator app which creates a cache of thumbnails in /app/data/appdata_XXXXXXXXX/preview/. It also contains lots of small files - it took 30 minutes to process 300Gb files in the folder on SSD drive (chown or even du -hs). Better to exclude it from chowning as well.

2. A timeout occurs on time consuming file operations (i.e. chowning) so Cloudron shows 'Not responding' app status for a quite long time (3 hours in our case). It's a little misleading. So ideally to move all such these operations to another process in the background (supervisord or even cron) and make the app running whilst it's updating permissions.

3. A new app release is very welcomed even if it's a dirty hotfix with start.sh on writable FS (I'd comment 'chowning' out for now).

@girish could you also review the code on lines 10-11 of start.sh.
and can you move start.sh on writable fs?

Looks good, looking forward for a new Nextcloud release.
Thank you.

Can we have an option to disable fixing file permissions on startup or fix permissions later?
It takes 2-3 hours to start Nextcloud app (3Tb, ~2.5 million files).

@nebulon it seems 7.2.2 is still affected. We faced with the same issue and resolved it by restating the mail service.

@girish Nextcloud-annotate needs the following dependencies installed in Nextcloud app.

DEPENDENCIES:
svg2pdf
pdftk
gs

@girish here is some initial work.
https://git.cloudron.io/vladimir.d/mailpiler

TBH I didn't have a chance to complete it yet, I think the issue described here still needs to be addressed.

@girish I used 'Syncronize' button nothing else.
Did you have a chance to reproduce the group assignment issue on our server?

@luckow we have created Cloudron users group in UCS and put some users to the group.

On the Connect an External Directory section on Cloudron we have configured to import users and groups from UCS.

Users and Groups are imported in Cloudron.

But it fails to assign Test.Test1 user to groups. If we rename Test.Test1 username to lowercase as test.test1, it starts working properly - see test.test user on the screenshots.
I suspect there is a bug or misconfiguration somewhere in Cloudron.

We cannot rename real usernames to lower case due to our company policy.

We are trying to link Univention LDAP as External Directory and I noticed that it doesn't import group memberships properly.

It looks like it's case sensitive issue, when I import test.test user, it puts the user into a group properly.

Mar 30 12:50:12 box:externalldap syncUsers: [adding user] username=test.test email=test.test@dev.mynet displayName=test test
Mar 30 12:50:12 box:externalldap syncUsers: done
...

Mar 30 12:50:12 box:tasks update 4074: {"percent":68,"message":"Syncing... uni-all-cloudron-users"}
Mar 30 12:50:12 box:externalldap syncGroups: [up-to-date group] groupname=uni-all-cloudron-users
Mar 30 12:50:12 box:externalldap syncGroups: sync done
...
Mar 30 12:50:12 box:externalldap syncGroupUsers: Sync users for group uni-all-cloudron-users
Mar 30 12:50:12 box:externalldap syncGroupUsers: Group uni-all-cloudron-users has 2 members.
Mar 30 12:50:12 box:externalldap ldapGetByDN: Get object at uid=test.test,ou=Users,dc=dev,dc=mynet
Mar 30 12:50:12 box:externalldap syncGroupUsers: Found member object at uid=test.test,ou=Users,dc=dev,dc=mynet adding to group uni-all-cloudron-users

If I rename it to Test.Test it fails to import.

Mar 30 12:12:37 box:tasks update 4068: {"percent":30,"message":"Syncing... Test.Test"}
Mar 30 12:12:37 box:externalldap syncUsers: [adding user] username=Test.Test email=test.test@dev.mynet displayName=test test
Mar 30 12:12:37 box:externalldap syncUsers: done
...
Mar 30 12:12:37 box:externalldap syncGroupUsers: Group uni-all-cloudron-users has 2 members.
Mar 30 12:12:37 box:externalldap ldapGetByDN: Get object at uid=Test.Test,ou=Users,dc=dev,dc=mynet
Mar 30 12:12:37 box:externalldap syncGroupUsers: Found member object at uid=Test.Test,ou=Users,dc=dev,dc=mynet adding to group uni-all-cloudron-users
Mar 30 12:12:37 box:externalldap syncGroupUsers: Failed to get user by username Test.Test User not found

I can get the same data using ldapsearch using both test.test and Test.Test from Univention LDAP.

We need to make External Directory synchronisation working with usernames in Test.Test format.

@girish said in LDAP not exposing outside:

@vladimir-d said in LDAP not exposing outside:

Yes, this works for ldaps://<cloudron-server-domain>:636 but not for local IP ,e.g.
ldaps://192.168.10.10:636

As @nebulon said, IP addresses cannot have a valid certificate. But (for whatever reason), you really want to use an IP, you can export LDAPTLS_REQCERT=never which disables the cert check for ldapsearch and friends.

Yes, sorted that with passing a flag to switch the cert validation off to ldapsearch.

LDAPTLS_REQCERT=never ldapsearch  -v -c -x -b "ou=users,dc=cloudron" -D "cn=admin,ou=system,dc=cloudron" -W -H ldaps://192.168.10.10:636

In fact, we need to connect various services to ldap not just ldapsearch, so apparently there is only one way out - we have to use a domain name.

Thank you @girish and @nebulon

@girish said in LDAP not exposing outside:

@vladimir-d said in LDAP not exposing outside:

ldap://<cloudron-server-domain>:636

the above has to be ldaps:// . Do you get the same error with that?

Yes, this works for ldaps://<cloudron-server-domain>:636 but not for local IP ,e.g.
ldaps://192.168.10.10:636

By <cloudron-server-domain>:

# ldapsearch  -v -c -x -b "ou=users,dc=cloudron" -D "cn=admin,ou=system,dc=cloudron" -W -H ldaps://<cloudron-server-domain>:636
ldap_initialize( ldaps://<cloudron-server-domain>:636/??base )
Enter LDAP Password: 

# extended LDIF
#
# LDAPv3
# base <ou=users,dc=cloudron> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# uid-0430c072-331b-4280-8a95-e92029fd16af, users, cloudron
dn: cn=uid-0430c072-331b-4280-8a95-e92029fd16af,ou=users,dc=cloudron
objectclass: user
objectclass: inetorgperson
objectclass: person
objectcategory: person
...

By local IP:

# ldapsearch  -v -c -x -b "ou=users,dc=cloudron" -D "cn=admin,ou=system,dc=cloudron" -W -H ldaps://192.168.10.10:636 "cn=uid-13f9d18f-afd2-4c41-b78d-f2a32c0a3e18,ou=users,dc=cloudron"
ldap_initialize( ldaps://192.168.10.10:636/??base )
Enter LDAP Password: 
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)

Wrong cert for the local IP?

Also I noticed that if I save Directory Server configuration several times, multiple duplicate rules added to iptables:

-A PREROUTING -p tcp -m tcp --dport 636 -j REDIRECT --to-ports 3004
-A PREROUTING -p tcp -m tcp --dport 636 -j REDIRECT --to-ports 3004
-A PREROUTING -p tcp -m tcp --dport 636 -j REDIRECT --to-ports 3004
-A PREROUTING -p tcp -m tcp --dport 636 -j REDIRECT --to-ports 3004

@girish well, telnet <cloudron-server-ip> 636 started to work today, probably another reboot helped.

Now I cannot get anything via ldapsearch neither by a local ip nor by <cloudron-server-domain>.

# ldapsearch  -x -b "ou=users,dc=cloudron" -D "cn=admin,ou=system,dc=cloudron" -W -H ldap://<cloudron-server-domain>:636
ldap_initialize( ldap://<cloudron-server-domain>:636/??base )
Enter LDAP Password: 
ldap_result: Can't contact LDAP server (-1)

Hi,

It doesn't seem that a long waited feature - the ability to expose LDAP outside is working in our environment after the recent upgrade to 7.1.3.

For some reason 636 port doesn't get available even locally.
netstat -tulpn | grep 636 shows nothing, but 3002 port is still there though.

Also wondering if there is an ability to expose LDAP to a local network only on a specific network interface, e.g. 192.168.10.0/24.

@murgero We'd like to give it a try, could you share the app?

Is it possible to expose 10K or 20K UDP ports by one directive in Cloudronmanifest?

I think I was unclear about what folder being made writable, sorry.

What we need is to deploy some files and directories in /app/code/current/public folder which is readonly now. This /app/code/current/public folder contains just directories/files available for public visitors. We don't need to update or patch release files.

There are some EspoCRM extensions require a public content being uploaded to /app/code/7.0.8/public folder, which is on read-only filesystem in the cloudron app now.

Could you amend the app so the public folder was linked to /app/data/public?

@girish nothing in /home/yellowtent/platformdata/logs/box.log, will email to the mailbox now.

@girish I'm not able to reproduce the issue on the demo, it works on the demo as expected.
Ok, will debug further what happens on our instance.

I mean as admin I get the invite link on the invite user popup, copy it and open in another browser to set a password. It submits the data without errors (in the web console we see they successfully posted to cloudron), we enter exactly the same password to login, but it fails to login. The user password and salt fields are left unchanged in the cloudron database.
I tried it in Safari and Chrome on Monterey.
I have no issues to logout and login with my user. Also if I update the password and salt directly in the database, the user is able to login.

I believe you still have SSH access to our host if you'd like to investigate it.

After upgrade to 7.0.3 the invite page doesn't seem to set password for new users.
It successfully submits data, but the respective DB record isn't updated.

@nebulon Yes, it's the instance you have SSH access.
We understand it's a temporary workaround and there could be upgrade issues (re-applying the patch), just need this done as soon as possible.
Later we can look at packaging of some LDAP proxy app or using external LDAP or wait when you implement a solution in the mainstream.

Thanks a lot!
ps. just sent details by email.

@nebulon could you suggest a workaround of App not found to connect to LDAP?

Could you amend the code so it reads the allowed IPs from a config file or Cloudron console?

Managed to expose LDAP to VPN by amending iptables on the cloudron host:

$ iptables -I CLOUDRON 4 -s 10.100.10.0/16 -i tun0 -p tcp -m tcp --dport 3002 -j ACCEPT

But when I try to connect to ldap using ldapsearch, it returns an error.

ldapsearch -x -b "ou=users,dc=cloudron" -D "cn=0111a111b-111-1111-1111-1111111,ou=apps,dc=cloudron" -W${CLOUDRON_LDAP_BIND_PASSWORD} -H ldap://10.100.10.1:3002

Error:

# extended LDIF
#
# LDAPv3
# base <ou=users,dc=cloudron> with scope subtree
# filter: (objectclass=*)
# requesting: ALL
#

# search result
search: 2
result: 1 Operations error
matchedDN: ou=users, dc=cloudron
text: App not found

# numResponses: 1

The same DN and password work well in the application.

Hi,

Could anyone suggest a tool for real-time MySQL performance monitoring?
I'm thinking of Grafana and Prometheus apps.
Any hints are very appreciated.

I've packaged openHab Cloud app, it can be built from

https://git.cloudron.io/vladimir.d/openhab-cloud

Comments are very welcome.

@girish We created these postgresql databases manually so it works on our instances.

@girish could it be related to the number of files in the archive?
It fails to unpack custom 67.zip containing 8400 files.
It unpacks when we randomly delete files from the archive down to 7830 files.

Also it doesn't seem related to the archive size, as I tried to unpack archives of different sizes (but containing just a few files) and it worked just fine.

@girish It still needs multiple postgresql database support (two databases for the app).
https://forum.cloudron.io/topic/4989/postgresql-multiple-databases-support

@girish You are right, it requires an external elastic search server so we have built ElasticSearch app for cloudron.
https://forum.cloudron.io/topic/1339/nextcloud-fulltext-search/15

Yes, fulltextsearch:live should be always run and being restarted if it gets stuck.
https://github.com/nextcloud/fulltextsearch/wiki/Basic-Installation#live-index-service

@girish It needs to expose 25 port for piler-smtp daemon. it says 25 (and other smtp ports) are reserved.
For now I've added tcpPorts field to Cloudronmanifest.json and exposed it as 12525 but I guess it won't properly work in most cases.
Is there a workaround to expose 25 port for the app?

@timconsidine I've updated Dockerfile to fix all building issues you faced with. After that everything went just fine on a fresh cloned copy.

@timconsidine said in Mailpiler - self hosted email archive:

Blundering around in the dark, but I notice that start.sh has on line 53

local SSL_CERT_DATA="/C=US/ST=Denial/L=Springfield/O=Dis/CN=www.example.com"

Should this not reference the Cloudron variable for location ?

When tls add-on is enabled in the manifest, the script doesn't generate a self signed certificate and uses the Cloudron certificate of the primary domain. I've amended it to use $CLOUDRON_APP_DOMAIN variable.

@timconsidine said in Mailpiler - self hosted email archive:

Successfully built and pushed to repository (docker in my case).
App installed on Cloudron w/o hassle .... but hangs in 'starting' mode.
So I uninstalled and tried cloudron install --image <repo>/<image>:<tag> without the --no-wait option.
This installs but similarly gets stuck

 => Wait for health check ............................^C

Out of my depth now. What to check / amend ?

It doesn't get stuck for me - I've tried to build&install the app with the recent changes.

I guess there is an issue with piler or nginx directories in /etc/, probably a wrong path or an invalid symlink or so.
You need to look at the app logs to see the issue.

Mailpiler app for cloudron is here, built based on the original docker configuration.
https://git.cloudron.io/vladimir.d/mailpiler

Comments are very welcome.

@mehdi not sure why we need to copy config files from /app/code to /app/data at runtime. They are intended to be specific for every instance.

What I'm trying to achieve is
https://github.com/nextcloud/fulltextsearch/wiki/Basic-Installation#live-index-service

Is there a better approach?

We've created an ElasticSearch app for Cloudron.

https://git.cloudron.io/vladimir.d/elasticsearch

Comments are welcome!

Well, we figured out that there were not enough system resources so apache was dying unexpectedly. BTW, the Graphs tab did show nothing. We increased CPU app limits significantly up to 75% so we get much less freezes now.

Also we implemented a Node-Red workflow to monitor actual healthstatus of the application using EspoCRM API and submit notifications to a messenger app in case of failure.

All of that doesn't resolve the issue but allows us to keep the system running somehow by restating the app services manually.

Hi,

We need to run Live index service for full-text search so I suggest adding the ability to configure supervisord in the application.

Here is the patch I propose to include into the mainstream:

https://git.cloudron.io/vladimir.d/nextcloud-app/-/merge_requests/1/diffs

In this case we will be able to create our own config to run indexing service using supervisor, e.g.

/app/data/supervisor/conf.d/fulltextsearch_index.conf

[program:fulltextsearch_index]
directory=/app/code
user=www-data
command=/usr/bin/php /app/code/occ fulltextsearch:live -q
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

ftr @staff and @appdev

@nebulon just checked we have the same issue on another cloudron host.
There are some different applications, some others the same.
I suspect it could be EspoCRM app.

@nebulon said in Errors logs to identify issues:

inspector tools?

Nothing in logs but get JS error in the console

Behavior is that when it gets frozen, the page refreshes fine but no data loaded, the Loading... message doesn't disappear.