Baserow 1.9 and it's Dockerfile and packaging changes

[nebulon]

I have taken another run at this and for a start, lets try to get 1.9 packaged, then see how we can sync up with @nigel_baserow suggestions on how to streamline the package/images.

At the moment, just building it for 1.9 results in a yarn nuxt error from RUN (cd web-frontend && yarn install && yarn build):

Entrypoint app = server.js server.js.map

ERROR in ./modules/database/store/view/bufferedRows.js 31:153
Module parse failed: Unexpected token (31:153)
File was processed with these loaders:
 * ./node_modules/babel-loader/lib/index.js
You may need an additional loader to handle the result of these loaders.
|  * ]
|  * ```
>  */export default(({service,customPopulateRow})=>{let lastRequestSource=null;const populateRow=row=>{if(customPopulateRow){customPopulateRow(row);}row._??={};// Matching rows for front-end only search is not yet properly
| // supported and tested in this store mixin. Only server-side search
| // implementation is finished.
 @ ./modules/database/store/view/gallery.js 1:0-76 1:322-334
 @ ./.nuxt/database.plugin.6555c3fe.js
 @ ./.nuxt/index.js
 @ ./.nuxt/server.js
 @ multi ./.nuxt/server.js

I am not sure at all what this means and what could be missing, does anyone have an idea here?

[nigel_baserow]

Apologizes for the delayed response above. We're happy to continue on with different ways of packaging Baserow upstream/downstream.

@nebulon Baserow 1.9 and prior was only built and tested against node 12. It looks like 1.8 and earlier versions happened to also work on node 14/16 present in the cloudron:3.2.0 image. However in 1.9 we introduced a change which accidentally breaks the build when not using node 12.

For Baserow 1.10 (not yet released) we've upgraded to using node 16 and fixed the issue above.

For now I think you will have to manually downgrade the node version in the Baserow cloudron image to get 1.9 working. Below is a patch for your https://git.cloudron.io/cloudron/baserow-app.git repo where I have:

1. Bumped the Baserow version to 1.9.1
2. Downgraded the image's node version to 12 to fix the build and ensure Baserow is running against its supported version of node.
3. Removed the MJML CLI install as it is no longer needed at runtime in version 1.9 for sending emails.

Index: Dockerfile
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/Dockerfile b/Dockerfile
--- a/Dockerfile	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/Dockerfile	(date 1650559344287)
@@ -16,13 +16,18 @@
 RUN ln -sf /dev/stdout /var/log/supervisor/supervisord.log
 RUN ln -sf /dev/stdout /app/code/supervisord.log
 
-RUN npm install -g mjml
-
-ARG BASEROW_VERSION=1.8.3
+ARG BASEROW_VERSION=1.9.1
 
 RUN curl -LSs "https://gitlab.com/bramw/baserow/-/archive/${BASEROW_VERSION}/baserow-${BASEROW_VERSION}.tar.gz" | tar -xz -C /app/code/ --strip-components 1 -f -
 RUN virtualenv -p python3 env
 RUN env/bin/pip install --no-cache -r backend/requirements/base.txt
+
+# Baserow 1.9 requires node 12. When Baserow 1.10 is released, this can be removed as it will support node 16 already present in cloudron/base:3.2.0.
+ARG NODEVERSION=12.22.2
+RUN mkdir -p /usr/local/node-${NODEVERSION} && \
+    curl -L https://nodejs.org/dist/v${NODEVERSION}/node-v${NODEVERSION}-linux-x64.tar.xz | tar Jxf - --strip-components 1 -C /usr/local/node-${NODEVERSION}
+ENV PATH /usr/local/node-${NODEVERSION}/bin:$PATH
+
 RUN (cd web-frontend && yarn install && yarn build)
 
 RUN (mkdir -p /app/code/cloudron/cloudron && touch /app/code/cloudron/cloudron/__init__.py)
Index: settings.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/settings.py b/settings.py
--- a/settings.py	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/settings.py	(date 1650557484604)
@@ -3,9 +3,6 @@
 
 MEDIA_ROOT = '/app/data/media'
 
-MJML_BACKEND_MODE = 'cmd'
-MJML_EXEC_CMD = 'mjml'
-
 FROM_EMAIL = os.environ['CLOUDRON_MAIL_FROM']
 EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend"
 EMAIL_USE_TLS = False

I won't have a chance today to manually test this image, however it now builds for me and should work.

[nebulon]

@nigel_baserow thanks for the quick response, I have adjusted the Dockerfile and it does build now with node 12.

For some reason it struggles to connect to redis upon installation, I will debug this tomorrow.

[nigel_baserow]

@nebulon

This is how we setup our redis connection with the Baserow django/celery backend:

REDIS_HOST = os.getenv("REDIS_HOST", "redis")
REDIS_PORT = os.getenv("REDIS_PORT", "6379")
REDIS_USERNAME = os.getenv("REDIS_USER", "")
REDIS_PASSWORD = os.getenv("REDIS_PASSWORD", "")
REDIS_PROTOCOL = os.getenv("REDIS_PROTOCOL", "redis")
REDIS_URL = os.getenv(
    "REDIS_URL",
    f"{REDIS_PROTOCOL}://{REDIS_USERNAME}:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}/0",
)

I'm guessing if there are redis problems then its due to the setup in https://git.cloudron.io/cloudron/baserow-app/-/blob/master/supervisor.conf

    REDIS_HOST='%(ENV_CLOUDRON_REDIS_HOST)s',
    REDIS_PORT='%(ENV_CLOUDRON_REDIS_PORT)s',
    REDIS_PASSWORD='%(ENV_CLOUDRON_REDIS_PASSWORD)s',

It looks like you instead provide a CLOUDRON_REDIS_URL with your redis addon, maybe using this instead will fix things?

So changing supervisor.conf to only have one env variable for redis being:

    REDIS_URL='%(ENV_CLOUDRON_REDIS_URL)s',

[nebulon]

@nigel_baserow thanks for the hints. I have tried the url as env variable but also even hardcoded the env vars to match yours without success.
The error remains:

Clearing Baserow's internal generated model cache...
Traceback (most recent call last):
  File "/app/code/env/lib/python3.8/site-packages/django_redis/cache.py", line 31, in _decorator
    return method(self, *args, **kwargs)
  File "/app/code/env/lib/python3.8/site-packages/django_redis/cache.py", line 117, in clear
    return self.client.clear()
  File "/app/code/env/lib/python3.8/site-packages/django_redis/client/default.py", line 434, in clear
    raise ConnectionInterrupted(connection=client) from e
django_redis.exceptions.ConnectionInterrupted: Redis ConnectionError: Error -2 connecting to redis:6379. Name or service not known.

During handling of the above exception, another exception occurred:

Which kinda looks like it doesn't even pick those up, as it tries to connect to redis:6379 which is not correct (the port is but the host would be redis-<appid>)

using the redis-cli works fine with the cloudron provided URL, so I guess it just didn't pick up the env vars by those names. Also I have seen the following warning:

WARNING: Baserow is configured to use a PUBLIC_BACKEND_URL of http://localhost:8000. If you attempt to access Baserow on any other hostname requests to the backend will fail as they will be from an unknown host.Please ensure you set PUBLIC_BACKEND_URL if you will be accessing Baserow from any other URL then http://localhost.
WARNING: Baserow is configured to use a default PUBLIC_WEB_FRONTEND_URL of http://localhost:3000. Emails sent by Baserow will use links pointing to http://localhost:3000 when telling users how to access your server. If this is incorrect please ensure you have set PUBLIC_WEB_FRONTEND_URL to the URL where users can access your Baserow server.

which might indicate none of the env vars are correctly picked up? Is there a chance they are differently named and somewhat translated by your Dockerfile start script?

[nebulon]

Hm clearly the env vars are not somehow translated looking at https://github.com/bram2w/baserow/blob/e1dd374010de5d7870f5e02503a5ed4dfa6bed19/backend/src/baserow/config/settings/base.py

Something off with forwarding the environment it seems.

[nigel_baserow]

@nebulon Figured out the problem!

In start.sh we are calling migrate and sync_templates before we have setup the REDIS_HOST etc env variables from the CLOUDRON_ ones. These commands are the ones you are seeing failing to connect to redis and not the actual backend processes of Baserow. Previously in version 1.8 and prior these commands did not connect to redis. Now they do (they clear some cached data from redis which potentially becomes invalid on Baserow upgrade/database changes).

Below is a patch where i've entirely removed the supervisor.conf environment section and just moved those variables setup into start.sh prior to any command usage. This way we can be sure the management commands run with the same and correct environment as the actual supervisord run processes.

Index: supervisor.conf
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/supervisor.conf b/supervisor.conf
--- a/supervisor.conf	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/supervisor.conf	(date 1650878179553)
@@ -2,14 +2,6 @@
 nodaemon = true
 logfile=/dev/null
 logfile_maxbytes=0
-environment =
-    DJANGO_SETTINGS_MODULE='cloudron.settings',
-    REDIS_HOST='%(ENV_CLOUDRON_REDIS_HOST)s',
-    REDIS_PORT='%(ENV_CLOUDRON_REDIS_PORT)s',
-    REDIS_PASSWORD='%(ENV_CLOUDRON_REDIS_PASSWORD)s',
-    PRIVATE_BACKEND_URL='http://localhost:8000',
-    PUBLIC_WEB_FRONTEND_URL='https://%(ENV_CLOUDRON_APP_DOMAIN)s',
-    PUBLIC_BACKEND_URL='https://%(ENV_CLOUDRON_APP_DOMAIN)s'
 
 [program:gunicorn]
 user=cloudron
Index: start.sh
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/start.sh b/start.sh
--- a/start.sh	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/start.sh	(date 1650878843914)
@@ -7,11 +7,19 @@
 fi
 source /app/data/.secret
 
+export REDIS_HOST="$CLOUDRON_REDIS_HOST"
+export REDIS_PORT="$CLOUDRON_REDIS_PORT"
+export REDIS_PASSWORD="$CLOUDRON_REDIS_PASSWORD"
+export DJANGO_SETTINGS_MODULE="cloudron.settings"
+export PRIVATE_BACKEND_URL="http://localhost:8000"
+export PUBLIC_WEB_FRONTEND_URL="https://$CLOUDRON_APP_DOMAIN"
+export PUBLIC_BACKEND_URL="https://$CLOUDRON_APP_DOMAIN"
+
 echo "==> Executing database migrations"
-/app/code/env/bin/python /app/code/backend/src/baserow/manage.py migrate --settings=cloudron.settings
+/app/code/env/bin/python /app/code/backend/src/baserow/manage.py migrate
 
 echo "==> Syncing templates"
-/app/code/env/bin/python /app/code/backend/src/baserow/manage.py sync_templates --settings=cloudron.settings
+/app/code/env/bin/python /app/code/backend/src/baserow/manage.py sync_templates
 
 chown -R cloudron:cloudron /app/data

With these changes I can now startup this image and it all works for me locally.

[BrutalBirdie]

@nigel_baserow I tried applying these patches but a normal git apply this.patch resulted in some errors.
I cleanup up the diffs and split them up. (It seems the intellij version does not work nicely with git)

diff --git a/supervisor.conf b/supervisor.conf
--- a/supervisor.conf	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/supervisor.conf	(date 1650878179553)
@@ -2,14 +2,6 @@
 nodaemon = true
 logfile=/dev/null
 logfile_maxbytes=0
-environment =
-    DJANGO_SETTINGS_MODULE='cloudron.settings',
-    REDIS_HOST='%(ENV_CLOUDRON_REDIS_HOST)s',
-    REDIS_PORT='%(ENV_CLOUDRON_REDIS_PORT)s',
-    REDIS_PASSWORD='%(ENV_CLOUDRON_REDIS_PASSWORD)s',
-    PRIVATE_BACKEND_URL='http://localhost:8000',
-    PUBLIC_WEB_FRONTEND_URL='https://%(ENV_CLOUDRON_APP_DOMAIN)s',
-    PUBLIC_BACKEND_URL='https://%(ENV_CLOUDRON_APP_DOMAIN)s'
 
 [program:gunicorn]
 user=cloudron

diff --git a/start.sh b/start.sh
--- a/start.sh	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/start.sh	(date 1650878843914)
@@ -7,11 +7,19 @@
 fi
 source /app/data/.secret
 
+export REDIS_HOST="$CLOUDRON_REDIS_HOST"
+export REDIS_PORT="$CLOUDRON_REDIS_PORT"
+export REDIS_PASSWORD="$CLOUDRON_REDIS_PASSWORD"
+export DJANGO_SETTINGS_MODULE="cloudron.settings"
+export PRIVATE_BACKEND_URL="http://localhost:8000"
+export PUBLIC_WEB_FRONTEND_URL="https://$CLOUDRON_APP_DOMAIN"
+export PUBLIC_BACKEND_URL="https://$CLOUDRON_APP_DOMAIN"
+
 echo "==> Executing database migrations"
-/app/code/env/bin/python /app/code/backend/src/baserow/manage.py migrate --settings=cloudron.settings
+/app/code/env/bin/python /app/code/backend/src/baserow/manage.py migrate
 
 echo "==> Syncing templates"
-/app/code/env/bin/python /app/code/backend/src/baserow/manage.py sync_templates --settings=cloudron.settings
+/app/code/env/bin/python /app/code/backend/src/baserow/manage.py sync_templates
 
 chown -R cloudron:cloudron /app/data

---

Currently building and testing.

[nebulon]

ah that makes much sense. I also tried to export the env vars in the start.sh to make sure it wasn't some supervisor thing, but I exported them too late!

[nebulon]

Finally Baserow 1.9 got out as a package. Apologies for the huge delay here, we will try to get this up to speed in the future like for other packages. Thanks @nigel_baserow and @BrutalBirdie for the help!

[timconsidine]

@nebulon thanks for doing this (delay not a problem, and not really much of one )