Best posts made by vladimir.d

I've packaged openHab Cloud app, it can be built from

https://git.cloudron.io/vladimir.d/openhab-cloud

Comments are very welcome.

@girish the License has been added to the repository.
Please feel free to change anything if needed.

I've packaged netbox application available at

https://git.cloudron.io/vladimir.d/netbox

Just a few notes:

1. For now I haven't managed to run it properly as WSGI app with gunicorn. It's taking lots of server resources.
   It's running using the Dev server.
   https://docs.netbox.dev/en/stable/installation/3-netbox/#test-the-application

2. Super Admin username is specified in start.sh, the Postgress password is taken for Super Admin.

Mailpiler app for cloudron is here, built based on the original docker configuration.
https://git.cloudron.io/vladimir.d/mailpiler

Comments are very welcome.

By default, Nextcloud generates photo thumbnails on-demand that leads to slow Photos thumbnails load. To speed it up we’re using the Preview Generator app for Nextcloud to pre-generate previews regularly.

First we set the default thumbnail sizes by using the "occ config" commands:

sudo -u www-data /usr/bin/php /app/code/occ config:app:set --value="32 256" previewgenerator squareSizes
sudo -u www-data /usr/bin/php /app/code/occ config:app:set --value="256 384" previewgenerator widthSizes
sudo -u www-data /usr/bin/php /app/code/occ config:app:set --value="256" previewgenerator heightSizes

Then we generated previews for all existent images:

sudo -u www-data /usr/bin/php /app/code/occ preview:generate-all -vvv

It took about 3 weeks on our host to generate preview images for 4Tb of data which takes additional 500Gb.

Once it's finished we setup a cron job to generate previews:

0 */6 * * * sudo -u www-data /usr/bin/php -f /app/code/occ preview:pre-generate

TBH, it helped us a little but our Photos are still loading for 50-60 seconds and we are seeking any solution to speed it up.

We've created an ElasticSearch app for Cloudron.

https://git.cloudron.io/vladimir.d/elasticsearch

Comments are welcome!

Can we have an option to disable fixing file permissions on startup or fix permissions later?
It takes 2-3 hours to start Nextcloud app (3Tb, ~2.5 million files).

Looks good, looking forward for a new Nextcloud release.
Thank you.

We sorted this issue by amending /app/data/htaccess (/app/code/.htaccess) as follows:

<IfModule mod_rewrite.c>
  RewriteEngine on
  RewriteCond %{HTTP_USER_AGENT} DavClnt
  RewriteRule ^$ /remote.php/webdav/ [L,R=302]
  RewriteRule .* - [env=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
  RewriteRule ^\.well-known/host-meta /public.php?service=host-meta [QSA,L]
  RewriteRule ^\.well-known/host-meta\.json /public.php?service=host-meta-json [QSA,L]
-  RewriteRule ^\.well-known/webfinger /public.php?service=webfinger [QSA,L]
-  RewriteRule ^\.well-known/nodeinfo /public.php?service=nodeinfo [QSA,L]
+ RewriteRule ^/\.well-known/webfinger /index.php/.well-known/webfinger [R=301,L]
+ RewriteRule ^/\.well-known/nodeinfo /index.php/.well-known/nodeinfo [R=301,L]
  RewriteRule ^\.well-known/carddav https://%{HTTP_HOST}/remote.php/dav/ [R=301,L]
  RewriteRule ^\.well-known/caldav https://%{HTTP_HOST}/remote.php/dav/ [R=301,L]
  RewriteRule ^remote/(.*) remote.php [QSA,L]
  RewriteRule ^(?:build|tests|config|lib|3rdparty|templates)/.* - [R=404,L]
  RewriteCond %{REQUEST_URI} !^/\.well-known/(acme-challenge|pki-validation)/.*
  RewriteRule ^(?:\.|autotest|occ|issue|indie|db_|console).* - [R=404,L]
</IfModule>

@nebulon Yes, it's the instance you have SSH access.
We understand it's a temporary workaround and there could be upgrade issues (re-applying the patch), just need this done as soon as possible.
Later we can look at packaging of some LDAP proxy app or using external LDAP or wait when you implement a solution in the mainstream.

Thanks a lot!
ps. just sent details by email.

FTR, "Retry update task" and then "Restart App" have sorted the issue.

@girish works now, thank you!

The app has been pushed to git.cloudron.io.

@timconsidine I've updated Dockerfile to fix all building issues you faced with. After that everything went just fine on a fresh cloned copy.

@timconsidine said in Mailpiler - self hosted email archive:

Blundering around in the dark, but I notice that start.sh has on line 53

local SSL_CERT_DATA="/C=US/ST=Denial/L=Springfield/O=Dis/CN=www.example.com"

Should this not reference the Cloudron variable for location ?

When tls add-on is enabled in the manifest, the script doesn't generate a self signed certificate and uses the Cloudron certificate of the primary domain. I've amended it to use $CLOUDRON_APP_DOMAIN variable.

@timconsidine said in Mailpiler - self hosted email archive:

Successfully built and pushed to repository (docker in my case).
App installed on Cloudron w/o hassle .... but hangs in 'starting' mode.
So I uninstalled and tried cloudron install --image <repo>/<image>:<tag> without the --no-wait option.
This installs but similarly gets stuck

 => Wait for health check ............................^C

Out of my depth now. What to check / amend ?

It doesn't get stuck for me - I've tried to build&install the app with the recent changes.

I guess there is an issue with piler or nginx directories in /etc/, probably a wrong path or an invalid symlink or so.
You need to look at the app logs to see the issue.

@girish It still needs multiple postgresql database support (two databases for the app).
https://forum.cloudron.io/topic/4989/postgresql-multiple-databases-support

@girish It needs to expose 25 port for piler-smtp daemon. it says 25 (and other smtp ports) are reserved.
For now I've added tcpPorts field to Cloudronmanifest.json and exposed it as 12525 but I guess it won't properly work in most cases.
Is there a workaround to expose 25 port for the app?

@girish said in Drone:

@marcusquinn Which package are you using ? Or are you running this outside Cloudron ?

We are running Docker runner and SSH runner as docker containers and Exec runner as a service on the Cloudron host, basically exactly as described in the Drone documentation.
https://docs.drone.io/runner/overview/

@msbt said in Drone:

@marcusquinn same here, but the Drone runners are still not part of the equation, would be nice if they could work together with the docker addon to have it 100% "portable"

Maybe we'll package runners as cloudron apps later that could build & deploy an app on Cloudron.

Hi,

We need to run Live index service for full-text search so I suggest adding the ability to configure supervisord in the application.

Here is the patch I propose to include into the mainstream:

https://git.cloudron.io/vladimir.d/nextcloud-app/-/merge_requests/1/diffs

In this case we will be able to create our own config to run indexing service using supervisor, e.g.

/app/data/supervisor/conf.d/fulltextsearch_index.conf

[program:fulltextsearch_index]
directory=/app/code
user=www-data
command=/usr/bin/php /app/code/occ fulltextsearch:live -q
stdout_logfile=/dev/stdout
stdout_logfile_maxbytes=0
stderr_logfile=/dev/stderr
stderr_logfile_maxbytes=0

ftr @staff and @appdev

@girish could it be related to the number of files in the archive?
It fails to unpack custom 67.zip containing 8400 files.
It unpacks when we randomly delete files from the archive down to 7830 files.

Also it doesn't seem related to the archive size, as I tried to unpack archives of different sizes (but containing just a few files) and it worked just fine.

Hi everyone,

I've faced with an issue when I get EspoCRM app frozen but there is nothing in logs.
I inspected:

1. /app/data/data/logs/espo-2021-06-10.log

2. Full App logs downloaded from the container logs console

3. mysql logs in the mysql docker container on the cloudron host

What did I miss? Are Apache Error logs directed to stdout and to Full App logs?

@girish it looks like I don't have permissions to create new projects on git.cloudron.io.
I get Namespace is not valid error when I try to create a project for Chirpstack App.

@girish said in LDAP not exposing outside:

@vladimir-d said in LDAP not exposing outside:

Yes, this works for ldaps://<cloudron-server-domain>:636 but not for local IP ,e.g.
ldaps://192.168.10.10:636

As @nebulon said, IP addresses cannot have a valid certificate. But (for whatever reason), you really want to use an IP, you can export LDAPTLS_REQCERT=never which disables the cert check for ldapsearch and friends.

Yes, sorted that with passing a flag to switch the cert validation off to ldapsearch.

LDAPTLS_REQCERT=never ldapsearch  -v -c -x -b "ou=users,dc=cloudron" -D "cn=admin,ou=system,dc=cloudron" -W -H ldaps://192.168.10.10:636

In fact, we need to connect various services to ldap not just ldapsearch, so apparently there is only one way out - we have to use a domain name.

Thank you @girish and @nebulon

Well, we figured out that there were not enough system resources so apache was dying unexpectedly. BTW, the Graphs tab did show nothing. We increased CPU app limits significantly up to 75% so we get much less freezes now.

Also we implemented a Node-Red workflow to monitor actual healthstatus of the application using EspoCRM API and submit notifications to a messenger app in case of failure.

All of that doesn't resolve the issue but allows us to keep the system running somehow by restating the app services manually.

@girish here is some initial work.
https://git.cloudron.io/vladimir.d/mailpiler

TBH I didn't have a chance to complete it yet, I think the issue described here still needs to be addressed.

Hi,

Any plans to make multipleDatabases option available for postgresql add-on in the same was as it's for mysql add-on?

Thanks!

We need to enable postgresql slow query logging and we have done the following:

$ docker exec -ti postgresql bash
$ psql -d MY_DATABASE_NAME

psql (12.10 (Ubuntu 12.10-1.pgdg20.04+1+b1))
Type "help" for help.

MY_DATABASE_NAME =# ALTER DATABASE MY_DATABASE_NAME SET log_min_duration_statement = 5000;

MY_DATABASE_NAME =# SELECT pg_sleep(15);

But the slow log file didn't appear anywhere (/run/postgresql, /var/log/postgresql, /var/lib/postgresql/12/main/).

We need to set logging_collector to on in the postgresql config /etc/postgresql/12/main/postgresql.conf which is read-only.

Also please advise how to amend postgresql configuration in reboot/upgrade safe way.

@girish Nextcloud-annotate needs the following dependencies installed in Nextcloud app.

DEPENDENCIES:
svg2pdf
pdftk
gs

I'm building a custom app for Chirpstack as All-in-One application container.
It requires two postgresql databases: one for the application server and another for the network server (or maybe even more for other services). I've managed to create two databases for the app user manually and set appropriate access privileges, but not sure if it would be persistent after the cloudron host reboot.
I hoped multipleDatabases option was available for postgresql add-on but it's not.

@girish could you amend the postgresql image so supervisord runs postgresql using the config /var/lib/postgresql/12/main/postgresql.conf (/home/yellowtent/platformdata/postgresql/12/main/postgresql.conf)?

Namely I suggest creating a symlink to the file:

$ ln -s /var/lib/postgresql/12/main/postgresql.conf /etc/postgresql/12/main/postgresql.conf

OR changing supervisord config file /etc/supervisor/conf.d/postgresql.conf:

- command=/usr/lib/postgresql/12/bin/postmaster --config-file=/etc/postgresql/12/main/postgresql.conf
+ command=/usr/lib/postgresql/12/bin/postmaster --config-file=/var/lib/postgresql/12/main/postgresql.conf

@girish we are using drone app packaged by @fbartels.

@jdaviescoates said in How to cancel an update?:

@vladimir-d said in How to cancel an update?:

FTR, "Retry update task" and then "Restart App" have sorted the issue.

And then revert to previous backup?

@jdaviescoates It would be yes if there was a recent backup copy.

@girish sorry, a school boy mistake, the Content-Type header did the trick, but I made a typo in the backupId, that's why I got the error.

@mehdi not sure why we need to copy config files from /app/code to /app/data at runtime. They are intended to be specific for every instance.

What I'm trying to achieve is
https://github.com/nextcloud/fulltextsearch/wiki/Basic-Installation#live-index-service

Is there a better approach?

@girish You are right, it requires an external elastic search server so we have built ElasticSearch app for cloudron.
https://forum.cloudron.io/topic/1339/nextcloud-fulltext-search/15

Yes, fulltextsearch:live should be always run and being restarted if it gets stuck.
https://github.com/nextcloud/fulltextsearch/wiki/Basic-Installation#live-index-service

@girish could you also review the code on lines 10-11 of start.sh.
and can you move start.sh on writable fs?

@girish it looks promising, but a few points I need to mention about:

1. There is Preview Generator app which creates a cache of thumbnails in /app/data/appdata_XXXXXXXXX/preview/. It also contains lots of small files - it took 30 minutes to process 300Gb files in the folder on SSD drive (chown or even du -hs). Better to exclude it from chowning as well.

2. A timeout occurs on time consuming file operations (i.e. chowning) so Cloudron shows 'Not responding' app status for a quite long time (3 hours in our case). It's a little misleading. So ideally to move all such these operations to another process in the background (supervisord or even cron) and make the app running whilst it's updating permissions.

3. A new app release is very welcomed even if it's a dirty hotfix with start.sh on writable FS (I'd comment 'chowning' out for now).

@girish well, I've updated the app so it triggered chowning again.

I created the flag /run/nextcloud/chowned manually, killed the chown process and restarted the app. The upgrade downtime was just a few minutes this time.

Ideally to have a workaround for upgrades as well (a flag in /app/data?).

@girish it looks good.
What would involve to move existent data to the new data directory (/app/data/data)?