Cloudron Forum

Hey all,

I'm a dev working at Baserow and am currently reworking our Docker / CI for the next version of Baserow 1.9 which is not yet released. As part of this we will be providing a new Dockerhub baserow/baserow image which runs Baserow and all of it's services in one container. We are also hoping to change our cloudron image we provide and support (https://gitlab.com/bramw/baserow/-/blob/develop/deploy/cloudron/Dockerfile) to instead build from and be a slight variation of this new baserow/baserow image.

I do not believe that these changes if they happen will break the cloudron packaged Baserow at https://git.cloudron.io/cloudron/baserow-app, and it could simply bump its Baserow version to 1.9 to release successfully.

However this raises some questions about https://git.cloudron.io/cloudron/baserow-app which atm is a copy and slight edit of our own image. If we were to change our cloudron image to be a variation of the baserow/baserow image instead then our cloudron and the cloudron images would differ as follows:

• The base image we are building from is ultimately debian:buster-slim, where-as cloudron-base is ubuntu:focal
• Our new baserow/baserow image:
  • Has new supervisord setup files and related bash scripts used as entrypoints etc.
  • Now uses Caddy instead of nginx
  • Removes the service mjml
• Our new baserow/baserow image sets up exactly the correct versions of node, psql, redis, python etcetc we support. In contrast currently the cloudron base image 3.0.0 uses node 14 and 3.2.0 node 16 which we don't yet support for Baserow.
• The baserow/baserow image only installs dependencies required to run Baserow, compared with using cloudron/base which installs a number of other dependencies not required for Baserow etc.

Options

From what I can tell these are our options:

1. We go ahead with the changes above to the Baserow provided cloudron image, no changes other than a version bump are made to the Cloudron Baserow image. As a result we end up with two quite different ways of packaging Baserow for Cloudron. Additionally the cloudron way still has the now unused mjml service and future changes to the Baserow services will make upgrading harder.
2. We go ahead with the Baserow side of the changes, and we also change how Cloudron packages Baserow to be identical (the Docker image becomes)

ARG FROM_IMAGE=baserow/baserow:1.9
FROM $FROM_IMAGE as image_base

ENV DATA_DIR=/app/data/

RUN apt-get remove -y postgresql postgresql-contrib

# TODO copy over the deploy/cloudron/cloudron_env.sh to the cloudron baserow repo, remove all of the other setup files which now
# just come baked into baserow/baserow:1.9
COPY deploy/cloudron/cloudron_env.sh /baserow/supervisor/env/cloudron_env.sh

3. We go with some sort of hybrid multi-stage image in the cloudron repo, which copies relevant parts from baserow/baserow:1.9 but is still built based FROM cloudron/base:3.0.0, which would look something like (i've cut out some setup sections):

ARG FROM_IMAGE=baserow/baserow:1.9
FROM $FROM_IMAGE as image_base
FROM cloudron/base:3.0.0@sha256:455c70428723e3a823198c57472785437eb6eab082e79b3ff04ea584faf46e92

ENV DOCKER_USER=cloudron
ENV DATA_DIR=/app/data

# VARIOUS APT-GET / REDIS SETUP STEPS ETC

# ========================
# = BASEROW
# ========================
COPY --chown=cloudron:cloudron --from=image_base /baserow/ /baserow/
# The copy above will have chowned the supervisor folder to cloudron user when it should
# be root.
RUN chown -R root:root /baserow/supervisor
# Virtualenvs are not portable and especially not so between image_base which is debian
# and focal (cloudron/base). We install our python venv here instead.
RUN rm /baserow/venv -rf && \
   python3 -m pip install --no-cache-dir --no-warn-script-location --disable-pip-version-check --upgrade pip==22.0.3 && \
   python3 -m pip install --no-cache-dir --no-warn-script-location --upgrade virtualenv==20.13.1 && \
   python3 -m virtualenv /baserow/venv
# hadolint ignore=SC1091
RUN . /baserow/venv/bin/activate && pip3 install --no-cache-dir -r /baserow/backend/requirements/base.txt

RUN cd /baserow/web-frontend && yarn install && yarn build && chown cloudron: -R /baserow/web-frontend/

COPY deploy/cloudron/cloudron_env.sh /baserow/supervisor/env/cloudron_env.sh
COPY deploy/all-in-one/baserow.sh /baserow.sh

ENTRYPOINT ["/baserow.sh"]
CMD ["start"]

4. We (Baserow) delete our own separate cloudron image and we entirely switch to using https://git.cloudron.io/cloudron/baserow-app . Additionally we could then pick one of the 3 options above and apply them to the cloudron baserow repo.

From Baserow's perspective of keeping our various Dockerfiles simple and easy to maintain option 2 would be our preferred one. My understanding is that as long as the Docker image running on cloudron follows a few conventions (put data in /app/data being the main one) then it should work correctly.

I'd love to get ideas and feedback from you guys on this. I'm also happy to help contributing any of the above changes to the cloudron baserow repo.

Thanks!

Apologizes for the delayed response above. We're happy to continue on with different ways of packaging Baserow upstream/downstream.

@nebulon Baserow 1.9 and prior was only built and tested against node 12. It looks like 1.8 and earlier versions happened to also work on node 14/16 present in the cloudron:3.2.0 image. However in 1.9 we introduced a change which accidentally breaks the build when not using node 12.

For Baserow 1.10 (not yet released) we've upgraded to using node 16 and fixed the issue above.

For now I think you will have to manually downgrade the node version in the Baserow cloudron image to get 1.9 working. Below is a patch for your https://git.cloudron.io/cloudron/baserow-app.git repo where I have:

1. Bumped the Baserow version to 1.9.1
2. Downgraded the image's node version to 12 to fix the build and ensure Baserow is running against its supported version of node.
3. Removed the MJML CLI install as it is no longer needed at runtime in version 1.9 for sending emails.

Index: Dockerfile
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/Dockerfile b/Dockerfile
--- a/Dockerfile	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/Dockerfile	(date 1650559344287)
@@ -16,13 +16,18 @@
 RUN ln -sf /dev/stdout /var/log/supervisor/supervisord.log
 RUN ln -sf /dev/stdout /app/code/supervisord.log
 
-RUN npm install -g mjml
-
-ARG BASEROW_VERSION=1.8.3
+ARG BASEROW_VERSION=1.9.1
 
 RUN curl -LSs "https://gitlab.com/bramw/baserow/-/archive/${BASEROW_VERSION}/baserow-${BASEROW_VERSION}.tar.gz" | tar -xz -C /app/code/ --strip-components 1 -f -
 RUN virtualenv -p python3 env
 RUN env/bin/pip install --no-cache -r backend/requirements/base.txt
+
+# Baserow 1.9 requires node 12. When Baserow 1.10 is released, this can be removed as it will support node 16 already present in cloudron/base:3.2.0.
+ARG NODEVERSION=12.22.2
+RUN mkdir -p /usr/local/node-${NODEVERSION} && \
+    curl -L https://nodejs.org/dist/v${NODEVERSION}/node-v${NODEVERSION}-linux-x64.tar.xz | tar Jxf - --strip-components 1 -C /usr/local/node-${NODEVERSION}
+ENV PATH /usr/local/node-${NODEVERSION}/bin:$PATH
+
 RUN (cd web-frontend && yarn install && yarn build)
 
 RUN (mkdir -p /app/code/cloudron/cloudron && touch /app/code/cloudron/cloudron/__init__.py)
Index: settings.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/settings.py b/settings.py
--- a/settings.py	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/settings.py	(date 1650557484604)
@@ -3,9 +3,6 @@
 
 MEDIA_ROOT = '/app/data/media'
 
-MJML_BACKEND_MODE = 'cmd'
-MJML_EXEC_CMD = 'mjml'
-
 FROM_EMAIL = os.environ['CLOUDRON_MAIL_FROM']
 EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend"
 EMAIL_USE_TLS = False

I won't have a chance today to manually test this image, however it now builds for me and should work.

@nebulon Figured out the problem!

In start.sh we are calling migrate and sync_templates before we have setup the REDIS_HOST etc env variables from the CLOUDRON_ ones. These commands are the ones you are seeing failing to connect to redis and not the actual backend processes of Baserow. Previously in version 1.8 and prior these commands did not connect to redis. Now they do (they clear some cached data from redis which potentially becomes invalid on Baserow upgrade/database changes).

Below is a patch where i've entirely removed the supervisor.conf environment section and just moved those variables setup into start.sh prior to any command usage. This way we can be sure the management commands run with the same and correct environment as the actual supervisord run processes.

Index: supervisor.conf
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/supervisor.conf b/supervisor.conf
--- a/supervisor.conf	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/supervisor.conf	(date 1650878179553)
@@ -2,14 +2,6 @@
 nodaemon = true
 logfile=/dev/null
 logfile_maxbytes=0
-environment =
-    DJANGO_SETTINGS_MODULE='cloudron.settings',
-    REDIS_HOST='%(ENV_CLOUDRON_REDIS_HOST)s',
-    REDIS_PORT='%(ENV_CLOUDRON_REDIS_PORT)s',
-    REDIS_PASSWORD='%(ENV_CLOUDRON_REDIS_PASSWORD)s',
-    PRIVATE_BACKEND_URL='http://localhost:8000',
-    PUBLIC_WEB_FRONTEND_URL='https://%(ENV_CLOUDRON_APP_DOMAIN)s',
-    PUBLIC_BACKEND_URL='https://%(ENV_CLOUDRON_APP_DOMAIN)s'
 
 [program:gunicorn]
 user=cloudron
Index: start.sh
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/start.sh b/start.sh
--- a/start.sh	(revision 825192ff012217ba76f726de16cd7a5cc5ade91f)
+++ b/start.sh	(date 1650878843914)
@@ -7,11 +7,19 @@
 fi
 source /app/data/.secret
 
+export REDIS_HOST="$CLOUDRON_REDIS_HOST"
+export REDIS_PORT="$CLOUDRON_REDIS_PORT"
+export REDIS_PASSWORD="$CLOUDRON_REDIS_PASSWORD"
+export DJANGO_SETTINGS_MODULE="cloudron.settings"
+export PRIVATE_BACKEND_URL="http://localhost:8000"
+export PUBLIC_WEB_FRONTEND_URL="https://$CLOUDRON_APP_DOMAIN"
+export PUBLIC_BACKEND_URL="https://$CLOUDRON_APP_DOMAIN"
+
 echo "==> Executing database migrations"
-/app/code/env/bin/python /app/code/backend/src/baserow/manage.py migrate --settings=cloudron.settings
+/app/code/env/bin/python /app/code/backend/src/baserow/manage.py migrate
 
 echo "==> Syncing templates"
-/app/code/env/bin/python /app/code/backend/src/baserow/manage.py sync_templates --settings=cloudron.settings
+/app/code/env/bin/python /app/code/backend/src/baserow/manage.py sync_templates
 
 chown -R cloudron:cloudron /app/data

With these changes I can now startup this image and it all works for me locally.

@nebulon

This is how we setup our redis connection with the Baserow django/celery backend:

REDIS_HOST = os.getenv("REDIS_HOST", "redis")
REDIS_PORT = os.getenv("REDIS_PORT", "6379")
REDIS_USERNAME = os.getenv("REDIS_USER", "")
REDIS_PASSWORD = os.getenv("REDIS_PASSWORD", "")
REDIS_PROTOCOL = os.getenv("REDIS_PROTOCOL", "redis")
REDIS_URL = os.getenv(
    "REDIS_URL",
    f"{REDIS_PROTOCOL}://{REDIS_USERNAME}:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}/0",
)

I'm guessing if there are redis problems then its due to the setup in https://git.cloudron.io/cloudron/baserow-app/-/blob/master/supervisor.conf

    REDIS_HOST='%(ENV_CLOUDRON_REDIS_HOST)s',
    REDIS_PORT='%(ENV_CLOUDRON_REDIS_PORT)s',
    REDIS_PASSWORD='%(ENV_CLOUDRON_REDIS_PASSWORD)s',

It looks like you instead provide a CLOUDRON_REDIS_URL with your redis addon, maybe using this instead will fix things?

So changing supervisor.conf to only have one env variable for redis being:

    REDIS_URL='%(ENV_CLOUDRON_REDIS_URL)s',