Baserow 1.9 and it's Dockerfile and packaging changes

nigel_baserow

Hey all,

I'm a dev working at Baserow and am currently reworking our Docker / CI for the next version of Baserow 1.9 which is not yet released. As part of this we will be providing a new Dockerhub baserow/baserow image which runs Baserow and all of it's services in one container. We are also hoping to change our cloudron image we provide and support (https://gitlab.com/bramw/baserow/-/blob/develop/deploy/cloudron/Dockerfile) to instead build from and be a slight variation of this new baserow/baserow image.

I do not believe that these changes if they happen will break the cloudron packaged Baserow at https://git.cloudron.io/cloudron/baserow-app, and it could simply bump its Baserow version to 1.9 to release successfully.

However this raises some questions about https://git.cloudron.io/cloudron/baserow-app which atm is a copy and slight edit of our own image. If we were to change our cloudron image to be a variation of the baserow/baserow image instead then our cloudron and the cloudron images would differ as follows:

The base image we are building from is ultimately debian:buster-slim, where-as cloudron-base is ubuntu:focal
Our new baserow/baserow image:
- Has new supervisord setup files and related bash scripts used as entrypoints etc.
- Now uses Caddy instead of nginx
- Removes the service mjml
Our new baserow/baserow image sets up exactly the correct versions of node, psql, redis, python etcetc we support. In contrast currently the cloudron base image 3.0.0 uses node 14 and 3.2.0 node 16 which we don't yet support for Baserow.
The baserow/baserow image only installs dependencies required to run Baserow, compared with using cloudron/base which installs a number of other dependencies not required for Baserow etc.

Options

From what I can tell these are our options:

We go ahead with the changes above to the Baserow provided cloudron image, no changes other than a version bump are made to the Cloudron Baserow image. As a result we end up with two quite different ways of packaging Baserow for Cloudron. Additionally the cloudron way still has the now unused mjml service and future changes to the Baserow services will make upgrading harder.
We go ahead with the Baserow side of the changes, and we also change how Cloudron packages Baserow to be identical (the Docker image becomes)

ARG FROM_IMAGE=baserow/baserow:1.9
FROM $FROM_IMAGE as image_base

ENV DATA_DIR=/app/data/

RUN apt-get remove -y postgresql postgresql-contrib

# TODO copy over the deploy/cloudron/cloudron_env.sh to the cloudron baserow repo, remove all of the other setup files which now
# just come baked into baserow/baserow:1.9
COPY deploy/cloudron/cloudron_env.sh /baserow/supervisor/env/cloudron_env.sh

We go with some sort of hybrid multi-stage image in the cloudron repo, which copies relevant parts from baserow/baserow:1.9 but is still built based FROM cloudron/base:3.0.0, which would look something like (i've cut out some setup sections):

ARG FROM_IMAGE=baserow/baserow:1.9
FROM $FROM_IMAGE as image_base
FROM cloudron/base:3.0.0@sha256:455c70428723e3a823198c57472785437eb6eab082e79b3ff04ea584faf46e92

ENV DOCKER_USER=cloudron
ENV DATA_DIR=/app/data

# VARIOUS APT-GET / REDIS SETUP STEPS ETC

# ========================
# = BASEROW
# ========================
COPY --chown=cloudron:cloudron --from=image_base /baserow/ /baserow/
# The copy above will have chowned the supervisor folder to cloudron user when it should
# be root.
RUN chown -R root:root /baserow/supervisor
# Virtualenvs are not portable and especially not so between image_base which is debian
# and focal (cloudron/base). We install our python venv here instead.
RUN rm /baserow/venv -rf && \
   python3 -m pip install --no-cache-dir --no-warn-script-location --disable-pip-version-check --upgrade pip==22.0.3 && \
   python3 -m pip install --no-cache-dir --no-warn-script-location --upgrade virtualenv==20.13.1 && \
   python3 -m virtualenv /baserow/venv
# hadolint ignore=SC1091
RUN . /baserow/venv/bin/activate && pip3 install --no-cache-dir -r /baserow/backend/requirements/base.txt

RUN cd /baserow/web-frontend && yarn install && yarn build && chown cloudron: -R /baserow/web-frontend/

COPY deploy/cloudron/cloudron_env.sh /baserow/supervisor/env/cloudron_env.sh
COPY deploy/all-in-one/baserow.sh /baserow.sh

ENTRYPOINT ["/baserow.sh"]
CMD ["start"]

We (Baserow) delete our own separate cloudron image and we entirely switch to using https://git.cloudron.io/cloudron/baserow-app . Additionally we could then pick one of the 3 options above and apply them to the cloudron baserow repo.

From Baserow's perspective of keeping our various Dockerfiles simple and easy to maintain option 2 would be our preferred one. My understanding is that as long as the Docker image running on cloudron follows a few conventions (put data in /app/data being the main one) then it should work correctly.

I'd love to get ideas and feedback from you guys on this. I'm also happy to help contributing any of the above changes to the cloudron baserow repo.

Thanks!

robi

I think the main for cloudron issue is realizing that all cloudron apps are based on the cloudron base image. This saves disk space and provides a stable single base image.

Having a new base image for a single app would add yet another base image used by only one app. Not ideal.

I'll let @staff chime in for a more official response.

girish

@nigel_baserow sorry for the late reply!

As @robi mentioned, all our apps are based on the same base image. This is mostly to maintain consistency and saves us a lot of developer time in terms of maintaining these images. The base image also needs to have many packages that are expected by the Cloudron tooling - the web terminal, upload, extract features all depend on the base image having various packages pre-installed.

In general, we are comfortable with the cloudron packaging and the upstream docker image diverging. From what we have seen, the goals of these images are quite different. Upstream images want to support a wide variety of configurations and have different optimizations in mind. The Cloudron image on the other hand is pre-configured (database, email, storage, auth etc) and has various security restrictions (read only file system, non-root user etc).

We are happy to make the changes needed to the cloudron packaging with your help (this is just a matter of adapting what you have written to the cloudron image, I think). So, I think 1) is the way to go. Another thing is that removing mjml service is not a problem at all. In fact, that is the advantage of cloudron packaging, the docker images can be thrown away and replaced with another as long as the docker image can handle the "data" from the previous version.

jeau

@girish @bram @nigel_baserow

Is there any chance to see the latest version of Baserow (1.9.1) on the Cloudron store? The version on the store is 1.8.2.

I don't know what to think about the request that opened this thread and the responses I just read on the baserow forum:

https://community.baserow.io/t/cloudron-release/339

jdaviescoates

@jeau said in Baserow 1.9 and it's Dockerfile and packaging changes:

latest version of Cloudron (1.9.1)

You mean the latest version of Baserow

jeau

@jdaviescoates you are right I correct

alex-a-soto

@girish said in Baserow 1.9 and it's Dockerfile and packaging changes:

@nigel_baserow sorry for the late reply!

As @robi mentioned, all our apps are based on the same base image. This is mostly to maintain consistency and saves us a lot of developer time in terms of maintaining these images. The base image also needs to have many packages that are expected by the Cloudron tooling - the web terminal, upload, extract features all depend on the base image having various packages pre-installed.

In general, we are comfortable with the cloudron packaging and the upstream docker image diverging. From what we have seen, the goals of these images are quite different. Upstream images want to support a wide variety of configurations and have different optimizations in mind. The Cloudron image on the other hand is pre-configured (database, email, storage, auth etc) and has various security restrictions (read only file system, non-root user etc).

We are happy to make the changes needed to the cloudron packaging with your help (this is just a matter of adapting what you have written to the cloudron image, I think). So, I think 1) is the way to go. Another thing is that removing mjml service is not a problem at all. In fact, that is the advantage of cloudron packaging, the docker images can be thrown away and replaced with another as long as the docker image can handle the "data" from the previous version.

Hello, thank you for building Cloudron, and all of the work that has gone into it.

Baserow is one of the applications that I use a lot on Cloudron and I intend to use it further in combination with n8n and other applications.

What's the status of updating Baserow to 1.9 in Cloudron?

Thank you for your time and support.

robi

@alex-a-soto said in Baserow 1.9 and it's Dockerfile and packaging changes:

Baserow is one of the applications that I use a lot on Cloudron

How do you use it?

girish

@alex-a-soto we will continue to update/maintain the package. Expect an update next week or so.

tobru

@girish do you have any plans to provide 1.9 soon?

nebulon

I have taken another run at this and for a start, lets try to get 1.9 packaged, then see how we can sync up with @nigel_baserow suggestions on how to streamline the package/images.

At the moment, just building it for 1.9 results in a yarn nuxt error from RUN (cd web-frontend && yarn install && yarn build):

Entrypoint app = server.js server.js.map

ERROR in ./modules/database/store/view/bufferedRows.js 31:153
Module parse failed: Unexpected token (31:153)
File was processed with these loaders:
 * ./node_modules/babel-loader/lib/index.js
You may need an additional loader to handle the result of these loaders.
|  * ]
|  * ```
>  */export default(({service,customPopulateRow})=>{let lastRequestSource=null;const populateRow=row=>{if(customPopulateRow){customPopulateRow(row);}row._??={};// Matching rows for front-end only search is not yet properly
| // supported and tested in this store mixin. Only server-side search
| // implementation is finished.
 @ ./modules/database/store/view/gallery.js 1:0-76 1:322-334
 @ ./.nuxt/database.plugin.6555c3fe.js
 @ ./.nuxt/index.js
 @ ./.nuxt/server.js
 @ multi ./.nuxt/server.js

I am not sure at all what this means and what could be missing, does anyone have an idea here?

nigel_baserow

Apologizes for the delayed response above. We're happy to continue on with different ways of packaging Baserow upstream/downstream.

@nebulon Baserow 1.9 and prior was only built and tested against node 12. It looks like 1.8 and earlier versions happened to also work on node 14/16 present in the cloudron:3.2.0 image. However in 1.9 we introduced a change which accidentally breaks the build when not using node 12.

For Baserow 1.10 (not yet released) we've upgraded to using node 16 and fixed the issue above.

For now I think you will have to manually downgrade the node version in the Baserow cloudron image to get 1.9 working. Below is a patch for your https://git.cloudron.io/cloudron/baserow-app.git repo where I have:

Bumped the Baserow version to 1.9.1
Downgraded the image's node version to 12 to fix the build and ensure Baserow is running against its supported version of node.
Removed the MJML CLI install as it is no longer needed at runtime in version 1.9 for sending emails.

Index: Dockerfile
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/Dockerfile b/Dockerfile
--- a/Dockerfile	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/Dockerfile	(date 1650559344287)
@@ -16,13 +16,18 @@
 RUN ln -sf /dev/stdout /var/log/supervisor/supervisord.log
 RUN ln -sf /dev/stdout /app/code/supervisord.log
 
-RUN npm install -g mjml
-
-ARG BASEROW_VERSION=1.8.3
+ARG BASEROW_VERSION=1.9.1
 
 RUN curl -LSs "https://gitlab.com/bramw/baserow/-/archive/${BASEROW_VERSION}/baserow-${BASEROW_VERSION}.tar.gz" | tar -xz -C /app/code/ --strip-components 1 -f -
 RUN virtualenv -p python3 env
 RUN env/bin/pip install --no-cache -r backend/requirements/base.txt
+
+# Baserow 1.9 requires node 12. When Baserow 1.10 is released, this can be removed as it will support node 16 already present in cloudron/base:3.2.0.
+ARG NODEVERSION=12.22.2
+RUN mkdir -p /usr/local/node-${NODEVERSION} && \
+    curl -L https://nodejs.org/dist/v${NODEVERSION}/node-v${NODEVERSION}-linux-x64.tar.xz | tar Jxf - --strip-components 1 -C /usr/local/node-${NODEVERSION}
+ENV PATH /usr/local/node-${NODEVERSION}/bin:$PATH
+
 RUN (cd web-frontend && yarn install && yarn build)
 
 RUN (mkdir -p /app/code/cloudron/cloudron && touch /app/code/cloudron/cloudron/__init__.py)
Index: settings.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/settings.py b/settings.py
--- a/settings.py	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/settings.py	(date 1650557484604)
@@ -3,9 +3,6 @@
 
 MEDIA_ROOT = '/app/data/media'
 
-MJML_BACKEND_MODE = 'cmd'
-MJML_EXEC_CMD = 'mjml'
-
 FROM_EMAIL = os.environ['CLOUDRON_MAIL_FROM']
 EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend"
 EMAIL_USE_TLS = False

I won't have a chance today to manually test this image, however it now builds for me and should work.

nebulon

@nigel_baserow thanks for the quick response, I have adjusted the Dockerfile and it does build now with node 12.

For some reason it struggles to connect to redis upon installation, I will debug this tomorrow.

nigel_baserow

@nebulon

This is how we setup our redis connection with the Baserow django/celery backend:

REDIS_HOST = os.getenv("REDIS_HOST", "redis")
REDIS_PORT = os.getenv("REDIS_PORT", "6379")
REDIS_USERNAME = os.getenv("REDIS_USER", "")
REDIS_PASSWORD = os.getenv("REDIS_PASSWORD", "")
REDIS_PROTOCOL = os.getenv("REDIS_PROTOCOL", "redis")
REDIS_URL = os.getenv(
    "REDIS_URL",
    f"{REDIS_PROTOCOL}://{REDIS_USERNAME}:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}/0",
)

I'm guessing if there are redis problems then its due to the setup in https://git.cloudron.io/cloudron/baserow-app/-/blob/master/supervisor.conf

    REDIS_HOST='%(ENV_CLOUDRON_REDIS_HOST)s',
    REDIS_PORT='%(ENV_CLOUDRON_REDIS_PORT)s',
    REDIS_PASSWORD='%(ENV_CLOUDRON_REDIS_PASSWORD)s',

It looks like you instead provide a CLOUDRON_REDIS_URL with your redis addon, maybe using this instead will fix things?

So changing supervisor.conf to only have one env variable for redis being:

    REDIS_URL='%(ENV_CLOUDRON_REDIS_URL)s',

nebulon

@nigel_baserow thanks for the hints. I have tried the url as env variable but also even hardcoded the env vars to match yours without success.
The error remains:

Clearing Baserow's internal generated model cache...
Traceback (most recent call last):
  File "/app/code/env/lib/python3.8/site-packages/django_redis/cache.py", line 31, in _decorator
    return method(self, *args, **kwargs)
  File "/app/code/env/lib/python3.8/site-packages/django_redis/cache.py", line 117, in clear
    return self.client.clear()
  File "/app/code/env/lib/python3.8/site-packages/django_redis/client/default.py", line 434, in clear
    raise ConnectionInterrupted(connection=client) from e
django_redis.exceptions.ConnectionInterrupted: Redis ConnectionError: Error -2 connecting to redis:6379. Name or service not known.

During handling of the above exception, another exception occurred:

Which kinda looks like it doesn't even pick those up, as it tries to connect to redis:6379 which is not correct (the port is but the host would be redis-<appid>)

using the redis-cli works fine with the cloudron provided URL, so I guess it just didn't pick up the env vars by those names. Also I have seen the following warning:

WARNING: Baserow is configured to use a PUBLIC_BACKEND_URL of http://localhost:8000. If you attempt to access Baserow on any other hostname requests to the backend will fail as they will be from an unknown host.Please ensure you set PUBLIC_BACKEND_URL if you will be accessing Baserow from any other URL then http://localhost.
WARNING: Baserow is configured to use a default PUBLIC_WEB_FRONTEND_URL of http://localhost:3000. Emails sent by Baserow will use links pointing to http://localhost:3000 when telling users how to access your server. If this is incorrect please ensure you have set PUBLIC_WEB_FRONTEND_URL to the URL where users can access your Baserow server.

which might indicate none of the env vars are correctly picked up? Is there a chance they are differently named and somewhat translated by your Dockerfile start script?

nebulon

Hm clearly the env vars are not somehow translated looking at https://github.com/bram2w/baserow/blob/e1dd374010de5d7870f5e02503a5ed4dfa6bed19/backend/src/baserow/config/settings/base.py

Something off with forwarding the environment it seems.

nigel_baserow

@nebulon Figured out the problem!

In start.sh we are calling migrate and sync_templates before we have setup the REDIS_HOST etc env variables from the CLOUDRON_ ones. These commands are the ones you are seeing failing to connect to redis and not the actual backend processes of Baserow. Previously in version 1.8 and prior these commands did not connect to redis. Now they do (they clear some cached data from redis which potentially becomes invalid on Baserow upgrade/database changes).

Below is a patch where i've entirely removed the supervisor.conf environment section and just moved those variables setup into start.sh prior to any command usage. This way we can be sure the management commands run with the same and correct environment as the actual supervisord run processes.

Index: supervisor.conf
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/supervisor.conf b/supervisor.conf
--- a/supervisor.conf	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/supervisor.conf	(date 1650878179553)
@@ -2,14 +2,6 @@
 nodaemon = true
 logfile=/dev/null
 logfile_maxbytes=0
-environment =
-    DJANGO_SETTINGS_MODULE='cloudron.settings',
-    REDIS_HOST='%(ENV_CLOUDRON_REDIS_HOST)s',
-    REDIS_PORT='%(ENV_CLOUDRON_REDIS_PORT)s',
-    REDIS_PASSWORD='%(ENV_CLOUDRON_REDIS_PASSWORD)s',
-    PRIVATE_BACKEND_URL='http://localhost:8000',
-    PUBLIC_WEB_FRONTEND_URL='https://%(ENV_CLOUDRON_APP_DOMAIN)s',
-    PUBLIC_BACKEND_URL='https://%(ENV_CLOUDRON_APP_DOMAIN)s'
 
 [program:gunicorn]
 user=cloudron
Index: start.sh
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/start.sh b/start.sh
--- a/start.sh	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/start.sh	(date 1650878843914)
@@ -7,11 +7,19 @@
 fi
 source /app/data/.secret
 
+export REDIS_HOST="$CLOUDRON_REDIS_HOST"
+export REDIS_PORT="$CLOUDRON_REDIS_PORT"
+export REDIS_PASSWORD="$CLOUDRON_REDIS_PASSWORD"
+export DJANGO_SETTINGS_MODULE="cloudron.settings"
+export PRIVATE_BACKEND_URL="http://localhost:8000"
+export PUBLIC_WEB_FRONTEND_URL="https://$CLOUDRON_APP_DOMAIN"
+export PUBLIC_BACKEND_URL="https://$CLOUDRON_APP_DOMAIN"
+
 echo "==> Executing database migrations"
-/app/code/env/bin/python /app/code/backend/src/baserow/manage.py migrate --settings=cloudron.settings
+/app/code/env/bin/python /app/code/backend/src/baserow/manage.py migrate
 
 echo "==> Syncing templates"
-/app/code/env/bin/python /app/code/backend/src/baserow/manage.py sync_templates --settings=cloudron.settings
+/app/code/env/bin/python /app/code/backend/src/baserow/manage.py sync_templates
 
 chown -R cloudron:cloudron /app/data

With these changes I can now startup this image and it all works for me locally.

BrutalBirdie

@nigel_baserow I tried applying these patches but a normal git apply this.patch resulted in some errors.
I cleanup up the diffs and split them up. (It seems the intellij version does not work nicely with git)

diff --git a/supervisor.conf b/supervisor.conf
--- a/supervisor.conf	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/supervisor.conf	(date 1650878179553)
@@ -2,14 +2,6 @@
 nodaemon = true
 logfile=/dev/null
 logfile_maxbytes=0
-environment =
-    DJANGO_SETTINGS_MODULE='cloudron.settings',
-    REDIS_HOST='%(ENV_CLOUDRON_REDIS_HOST)s',
-    REDIS_PORT='%(ENV_CLOUDRON_REDIS_PORT)s',
-    REDIS_PASSWORD='%(ENV_CLOUDRON_REDIS_PASSWORD)s',
-    PRIVATE_BACKEND_URL='http://localhost:8000',
-    PUBLIC_WEB_FRONTEND_URL='https://%(ENV_CLOUDRON_APP_DOMAIN)s',
-    PUBLIC_BACKEND_URL='https://%(ENV_CLOUDRON_APP_DOMAIN)s'
 
 [program:gunicorn]
 user=cloudron

diff --git a/start.sh b/start.sh
--- a/start.sh	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/start.sh	(date 1650878843914)
@@ -7,11 +7,19 @@
 fi
 source /app/data/.secret
 
+export REDIS_HOST="$CLOUDRON_REDIS_HOST"
+export REDIS_PORT="$CLOUDRON_REDIS_PORT"
+export REDIS_PASSWORD="$CLOUDRON_REDIS_PASSWORD"
+export DJANGO_SETTINGS_MODULE="cloudron.settings"
+export PRIVATE_BACKEND_URL="http://localhost:8000"
+export PUBLIC_WEB_FRONTEND_URL="https://$CLOUDRON_APP_DOMAIN"
+export PUBLIC_BACKEND_URL="https://$CLOUDRON_APP_DOMAIN"
+
 echo "==> Executing database migrations"
-/app/code/env/bin/python /app/code/backend/src/baserow/manage.py migrate --settings=cloudron.settings
+/app/code/env/bin/python /app/code/backend/src/baserow/manage.py migrate
 
 echo "==> Syncing templates"
-/app/code/env/bin/python /app/code/backend/src/baserow/manage.py sync_templates --settings=cloudron.settings
+/app/code/env/bin/python /app/code/backend/src/baserow/manage.py sync_templates
 
 chown -R cloudron:cloudron /app/data

Currently building and testing.

nebulon

ah that makes much sense. I also tried to export the env vars in the start.sh to make sure it wasn't some supervisor thing, but I exported them too late!

nebulon

Finally Baserow 1.9 got out as a package. Apologies for the huge delay here, we will try to get this up to speed in the future like for other packages. Thanks @nigel_baserow and @BrutalBirdie for the help!

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

Baserow 1.9 and it's Dockerfile and packaging changes

Options

Options