Baserow 1.9 and it's Dockerfile and packaging changes

alex-a-soto

@girish said in Baserow 1.9 and it's Dockerfile and packaging changes:

@nigel_baserow sorry for the late reply!

As @robi mentioned, all our apps are based on the same base image. This is mostly to maintain consistency and saves us a lot of developer time in terms of maintaining these images. The base image also needs to have many packages that are expected by the Cloudron tooling - the web terminal, upload, extract features all depend on the base image having various packages pre-installed.

In general, we are comfortable with the cloudron packaging and the upstream docker image diverging. From what we have seen, the goals of these images are quite different. Upstream images want to support a wide variety of configurations and have different optimizations in mind. The Cloudron image on the other hand is pre-configured (database, email, storage, auth etc) and has various security restrictions (read only file system, non-root user etc).

We are happy to make the changes needed to the cloudron packaging with your help (this is just a matter of adapting what you have written to the cloudron image, I think). So, I think 1) is the way to go. Another thing is that removing mjml service is not a problem at all. In fact, that is the advantage of cloudron packaging, the docker images can be thrown away and replaced with another as long as the docker image can handle the "data" from the previous version.

Hello, thank you for building Cloudron, and all of the work that has gone into it.

Baserow is one of the applications that I use a lot on Cloudron and I intend to use it further in combination with n8n and other applications.

What's the status of updating Baserow to 1.9 in Cloudron?

Thank you for your time and support.

robi

@alex-a-soto said in Baserow 1.9 and it's Dockerfile and packaging changes:

Baserow is one of the applications that I use a lot on Cloudron

How do you use it?

girish

@alex-a-soto we will continue to update/maintain the package. Expect an update next week or so.

tobru

@girish do you have any plans to provide 1.9 soon?

nebulon

I have taken another run at this and for a start, lets try to get 1.9 packaged, then see how we can sync up with @nigel_baserow suggestions on how to streamline the package/images.

At the moment, just building it for 1.9 results in a yarn nuxt error from RUN (cd web-frontend && yarn install && yarn build):

Entrypoint app = server.js server.js.map

ERROR in ./modules/database/store/view/bufferedRows.js 31:153
Module parse failed: Unexpected token (31:153)
File was processed with these loaders:
 * ./node_modules/babel-loader/lib/index.js
You may need an additional loader to handle the result of these loaders.
|  * ]
|  * ```
>  */export default(({service,customPopulateRow})=>{let lastRequestSource=null;const populateRow=row=>{if(customPopulateRow){customPopulateRow(row);}row._??={};// Matching rows for front-end only search is not yet properly
| // supported and tested in this store mixin. Only server-side search
| // implementation is finished.
 @ ./modules/database/store/view/gallery.js 1:0-76 1:322-334
 @ ./.nuxt/database.plugin.6555c3fe.js
 @ ./.nuxt/index.js
 @ ./.nuxt/server.js
 @ multi ./.nuxt/server.js

I am not sure at all what this means and what could be missing, does anyone have an idea here?

nigel_baserow

Apologizes for the delayed response above. We're happy to continue on with different ways of packaging Baserow upstream/downstream.

@nebulon Baserow 1.9 and prior was only built and tested against node 12. It looks like 1.8 and earlier versions happened to also work on node 14/16 present in the cloudron:3.2.0 image. However in 1.9 we introduced a change which accidentally breaks the build when not using node 12.

For Baserow 1.10 (not yet released) we've upgraded to using node 16 and fixed the issue above.

For now I think you will have to manually downgrade the node version in the Baserow cloudron image to get 1.9 working. Below is a patch for your https://git.cloudron.io/cloudron/baserow-app.git repo where I have:

Bumped the Baserow version to 1.9.1
Downgraded the image's node version to 12 to fix the build and ensure Baserow is running against its supported version of node.
Removed the MJML CLI install as it is no longer needed at runtime in version 1.9 for sending emails.

Index: Dockerfile
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/Dockerfile b/Dockerfile
--- a/Dockerfile	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/Dockerfile	(date 1650559344287)
@@ -16,13 +16,18 @@
 RUN ln -sf /dev/stdout /var/log/supervisor/supervisord.log
 RUN ln -sf /dev/stdout /app/code/supervisord.log
 
-RUN npm install -g mjml
-
-ARG BASEROW_VERSION=1.8.3
+ARG BASEROW_VERSION=1.9.1
 
 RUN curl -LSs "https://gitlab.com/bramw/baserow/-/archive/${BASEROW_VERSION}/baserow-${BASEROW_VERSION}.tar.gz" | tar -xz -C /app/code/ --strip-components 1 -f -
 RUN virtualenv -p python3 env
 RUN env/bin/pip install --no-cache -r backend/requirements/base.txt
+
+# Baserow 1.9 requires node 12. When Baserow 1.10 is released, this can be removed as it will support node 16 already present in cloudron/base:3.2.0.
+ARG NODEVERSION=12.22.2
+RUN mkdir -p /usr/local/node-${NODEVERSION} && \
+    curl -L https://nodejs.org/dist/v${NODEVERSION}/node-v${NODEVERSION}-linux-x64.tar.xz | tar Jxf - --strip-components 1 -C /usr/local/node-${NODEVERSION}
+ENV PATH /usr/local/node-${NODEVERSION}/bin:$PATH
+
 RUN (cd web-frontend && yarn install && yarn build)
 
 RUN (mkdir -p /app/code/cloudron/cloudron && touch /app/code/cloudron/cloudron/__init__.py)
Index: settings.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/settings.py b/settings.py
--- a/settings.py	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/settings.py	(date 1650557484604)
@@ -3,9 +3,6 @@
 
 MEDIA_ROOT = '/app/data/media'
 
-MJML_BACKEND_MODE = 'cmd'
-MJML_EXEC_CMD = 'mjml'
-
 FROM_EMAIL = os.environ['CLOUDRON_MAIL_FROM']
 EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend"
 EMAIL_USE_TLS = False

I won't have a chance today to manually test this image, however it now builds for me and should work.

nebulon

@nigel_baserow thanks for the quick response, I have adjusted the Dockerfile and it does build now with node 12.

For some reason it struggles to connect to redis upon installation, I will debug this tomorrow.

nigel_baserow

@nebulon

This is how we setup our redis connection with the Baserow django/celery backend:

REDIS_HOST = os.getenv("REDIS_HOST", "redis")
REDIS_PORT = os.getenv("REDIS_PORT", "6379")
REDIS_USERNAME = os.getenv("REDIS_USER", "")
REDIS_PASSWORD = os.getenv("REDIS_PASSWORD", "")
REDIS_PROTOCOL = os.getenv("REDIS_PROTOCOL", "redis")
REDIS_URL = os.getenv(
    "REDIS_URL",
    f"{REDIS_PROTOCOL}://{REDIS_USERNAME}:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}/0",
)

I'm guessing if there are redis problems then its due to the setup in https://git.cloudron.io/cloudron/baserow-app/-/blob/master/supervisor.conf

    REDIS_HOST='%(ENV_CLOUDRON_REDIS_HOST)s',
    REDIS_PORT='%(ENV_CLOUDRON_REDIS_PORT)s',
    REDIS_PASSWORD='%(ENV_CLOUDRON_REDIS_PASSWORD)s',

It looks like you instead provide a CLOUDRON_REDIS_URL with your redis addon, maybe using this instead will fix things?

So changing supervisor.conf to only have one env variable for redis being:

    REDIS_URL='%(ENV_CLOUDRON_REDIS_URL)s',

nebulon

@nigel_baserow thanks for the hints. I have tried the url as env variable but also even hardcoded the env vars to match yours without success.
The error remains:

Clearing Baserow's internal generated model cache...
Traceback (most recent call last):
  File "/app/code/env/lib/python3.8/site-packages/django_redis/cache.py", line 31, in _decorator
    return method(self, *args, **kwargs)
  File "/app/code/env/lib/python3.8/site-packages/django_redis/cache.py", line 117, in clear
    return self.client.clear()
  File "/app/code/env/lib/python3.8/site-packages/django_redis/client/default.py", line 434, in clear
    raise ConnectionInterrupted(connection=client) from e
django_redis.exceptions.ConnectionInterrupted: Redis ConnectionError: Error -2 connecting to redis:6379. Name or service not known.

During handling of the above exception, another exception occurred:

Which kinda looks like it doesn't even pick those up, as it tries to connect to redis:6379 which is not correct (the port is but the host would be redis-<appid>)

using the redis-cli works fine with the cloudron provided URL, so I guess it just didn't pick up the env vars by those names. Also I have seen the following warning:

WARNING: Baserow is configured to use a PUBLIC_BACKEND_URL of http://localhost:8000. If you attempt to access Baserow on any other hostname requests to the backend will fail as they will be from an unknown host.Please ensure you set PUBLIC_BACKEND_URL if you will be accessing Baserow from any other URL then http://localhost.
WARNING: Baserow is configured to use a default PUBLIC_WEB_FRONTEND_URL of http://localhost:3000. Emails sent by Baserow will use links pointing to http://localhost:3000 when telling users how to access your server. If this is incorrect please ensure you have set PUBLIC_WEB_FRONTEND_URL to the URL where users can access your Baserow server.

which might indicate none of the env vars are correctly picked up? Is there a chance they are differently named and somewhat translated by your Dockerfile start script?

nebulon

Hm clearly the env vars are not somehow translated looking at https://github.com/bram2w/baserow/blob/e1dd374010de5d7870f5e02503a5ed4dfa6bed19/backend/src/baserow/config/settings/base.py

Something off with forwarding the environment it seems.

nigel_baserow

@nebulon Figured out the problem!

In start.sh we are calling migrate and sync_templates before we have setup the REDIS_HOST etc env variables from the CLOUDRON_ ones. These commands are the ones you are seeing failing to connect to redis and not the actual backend processes of Baserow. Previously in version 1.8 and prior these commands did not connect to redis. Now they do (they clear some cached data from redis which potentially becomes invalid on Baserow upgrade/database changes).

Below is a patch where i've entirely removed the supervisor.conf environment section and just moved those variables setup into start.sh prior to any command usage. This way we can be sure the management commands run with the same and correct environment as the actual supervisord run processes.

Index: supervisor.conf
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/supervisor.conf b/supervisor.conf
--- a/supervisor.conf	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/supervisor.conf	(date 1650878179553)
@@ -2,14 +2,6 @@
 nodaemon = true
 logfile=/dev/null
 logfile_maxbytes=0
-environment =
-    DJANGO_SETTINGS_MODULE='cloudron.settings',
-    REDIS_HOST='%(ENV_CLOUDRON_REDIS_HOST)s',
-    REDIS_PORT='%(ENV_CLOUDRON_REDIS_PORT)s',
-    REDIS_PASSWORD='%(ENV_CLOUDRON_REDIS_PASSWORD)s',
-    PRIVATE_BACKEND_URL='http://localhost:8000',
-    PUBLIC_WEB_FRONTEND_URL='https://%(ENV_CLOUDRON_APP_DOMAIN)s',
-    PUBLIC_BACKEND_URL='https://%(ENV_CLOUDRON_APP_DOMAIN)s'
 
 [program:gunicorn]
 user=cloudron
Index: start.sh
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/start.sh b/start.sh
--- a/start.sh	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/start.sh	(date 1650878843914)
@@ -7,11 +7,19 @@
 fi
 source /app/data/.secret
 
+export REDIS_HOST="$CLOUDRON_REDIS_HOST"
+export REDIS_PORT="$CLOUDRON_REDIS_PORT"
+export REDIS_PASSWORD="$CLOUDRON_REDIS_PASSWORD"
+export DJANGO_SETTINGS_MODULE="cloudron.settings"
+export PRIVATE_BACKEND_URL="http://localhost:8000"
+export PUBLIC_WEB_FRONTEND_URL="https://$CLOUDRON_APP_DOMAIN"
+export PUBLIC_BACKEND_URL="https://$CLOUDRON_APP_DOMAIN"
+
 echo "==> Executing database migrations"
-/app/code/env/bin/python /app/code/backend/src/baserow/manage.py migrate --settings=cloudron.settings
+/app/code/env/bin/python /app/code/backend/src/baserow/manage.py migrate
 
 echo "==> Syncing templates"
-/app/code/env/bin/python /app/code/backend/src/baserow/manage.py sync_templates --settings=cloudron.settings
+/app/code/env/bin/python /app/code/backend/src/baserow/manage.py sync_templates
 
 chown -R cloudron:cloudron /app/data

With these changes I can now startup this image and it all works for me locally.

BrutalBirdie

@nigel_baserow I tried applying these patches but a normal git apply this.patch resulted in some errors.
I cleanup up the diffs and split them up. (It seems the intellij version does not work nicely with git)

diff --git a/supervisor.conf b/supervisor.conf
--- a/supervisor.conf	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/supervisor.conf	(date 1650878179553)
@@ -2,14 +2,6 @@
 nodaemon = true
 logfile=/dev/null
 logfile_maxbytes=0
-environment =
-    DJANGO_SETTINGS_MODULE='cloudron.settings',
-    REDIS_HOST='%(ENV_CLOUDRON_REDIS_HOST)s',
-    REDIS_PORT='%(ENV_CLOUDRON_REDIS_PORT)s',
-    REDIS_PASSWORD='%(ENV_CLOUDRON_REDIS_PASSWORD)s',
-    PRIVATE_BACKEND_URL='http://localhost:8000',
-    PUBLIC_WEB_FRONTEND_URL='https://%(ENV_CLOUDRON_APP_DOMAIN)s',
-    PUBLIC_BACKEND_URL='https://%(ENV_CLOUDRON_APP_DOMAIN)s'
 
 [program:gunicorn]
 user=cloudron

diff --git a/start.sh b/start.sh
--- a/start.sh	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/start.sh	(date 1650878843914)
@@ -7,11 +7,19 @@
 fi
 source /app/data/.secret
 
+export REDIS_HOST="$CLOUDRON_REDIS_HOST"
+export REDIS_PORT="$CLOUDRON_REDIS_PORT"
+export REDIS_PASSWORD="$CLOUDRON_REDIS_PASSWORD"
+export DJANGO_SETTINGS_MODULE="cloudron.settings"
+export PRIVATE_BACKEND_URL="http://localhost:8000"
+export PUBLIC_WEB_FRONTEND_URL="https://$CLOUDRON_APP_DOMAIN"
+export PUBLIC_BACKEND_URL="https://$CLOUDRON_APP_DOMAIN"
+
 echo "==> Executing database migrations"
-/app/code/env/bin/python /app/code/backend/src/baserow/manage.py migrate --settings=cloudron.settings
+/app/code/env/bin/python /app/code/backend/src/baserow/manage.py migrate
 
 echo "==> Syncing templates"
-/app/code/env/bin/python /app/code/backend/src/baserow/manage.py sync_templates --settings=cloudron.settings
+/app/code/env/bin/python /app/code/backend/src/baserow/manage.py sync_templates
 
 chown -R cloudron:cloudron /app/data

Currently building and testing.

nebulon

ah that makes much sense. I also tried to export the env vars in the start.sh to make sure it wasn't some supervisor thing, but I exported them too late!

nebulon

Finally Baserow 1.9 got out as a package. Apologies for the huge delay here, we will try to get this up to speed in the future like for other packages. Thanks @nigel_baserow and @BrutalBirdie for the help!

timconsidine

@nebulon thanks for doing this (delay not a problem, and not really much of one )

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

Baserow 1.9 and it's Dockerfile and packaging changes