Baserow 1.9 and it's Dockerfile and packaging changes
-
@girish said in Baserow 1.9 and it's Dockerfile and packaging changes:
@nigel_baserow sorry for the late reply!
As @robi mentioned, all our apps are based on the same base image. This is mostly to maintain consistency and saves us a lot of developer time in terms of maintaining these images. The base image also needs to have many packages that are expected by the Cloudron tooling - the web terminal, upload, extract features all depend on the base image having various packages pre-installed.
In general, we are comfortable with the cloudron packaging and the upstream docker image diverging. From what we have seen, the goals of these images are quite different. Upstream images want to support a wide variety of configurations and have different optimizations in mind. The Cloudron image on the other hand is pre-configured (database, email, storage, auth etc) and has various security restrictions (read only file system, non-root user etc).
We are happy to make the changes needed to the cloudron packaging with your help (this is just a matter of adapting what you have written to the cloudron image, I think). So, I think 1) is the way to go. Another thing is that removing mjml service is not a problem at all. In fact, that is the advantage of cloudron packaging, the docker images can be thrown away and replaced with another as long as the docker image can handle the "data" from the previous version.
Hello, thank you for building Cloudron, and all of the work that has gone into it.
Baserow is one of the applications that I use a lot on Cloudron and I intend to use it further in combination with n8n and other applications.
What's the status of updating Baserow to 1.9 in Cloudron?
Thank you for your time and support.
-
@alex-a-soto said in Baserow 1.9 and it's Dockerfile and packaging changes:
Baserow is one of the applications that I use a lot on Cloudron
How do you use it?
-
@alex-a-soto we will continue to update/maintain the package. Expect an update next week or so.
-
I have taken another run at this and for a start, lets try to get 1.9 packaged, then see how we can sync up with @nigel_baserow suggestions on how to streamline the package/images.
At the moment, just building it for 1.9 results in a yarn nuxt error from
RUN (cd web-frontend && yarn install && yarn build):Entrypoint app = server.js server.js.map ERROR in ./modules/database/store/view/bufferedRows.js 31:153 Module parse failed: Unexpected token (31:153) File was processed with these loaders: * ./node_modules/babel-loader/lib/index.js You may need an additional loader to handle the result of these loaders. | * ] | * ``` > */export default(({service,customPopulateRow})=>{let lastRequestSource=null;const populateRow=row=>{if(customPopulateRow){customPopulateRow(row);}row._??={};// Matching rows for front-end only search is not yet properly | // supported and tested in this store mixin. Only server-side search | // implementation is finished. @ ./modules/database/store/view/gallery.js 1:0-76 1:322-334 @ ./.nuxt/database.plugin.6555c3fe.js @ ./.nuxt/index.js @ ./.nuxt/server.js @ multi ./.nuxt/server.jsI am not sure at all what this means and what could be missing, does anyone have an idea here?
-
N nebulon marked this topic as a question on
-
Apologizes for the delayed response above. We're happy to continue on with different ways of packaging Baserow upstream/downstream.
@nebulon Baserow 1.9 and prior was only built and tested against node 12. It looks like 1.8 and earlier versions happened to also work on node 14/16 present in the cloudron:3.2.0 image. However in 1.9 we introduced a change which accidentally breaks the build when not using node 12.
For Baserow 1.10 (not yet released) we've upgraded to using node 16 and fixed the issue above.
For now I think you will have to manually downgrade the node version in the Baserow cloudron image to get 1.9 working. Below is a patch for your https://git.cloudron.io/cloudron/baserow-app.git repo where I have:
- Bumped the Baserow version to 1.9.1
- Downgraded the image's node version to 12 to fix the build and ensure Baserow is running against its supported version of node.
- Removed the MJML CLI install as it is no longer needed at runtime in version 1.9 for sending emails.
Index: Dockerfile IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/Dockerfile b/Dockerfile --- a/Dockerfile (revision 825192ff012217ba76f726de16cd7a5cc5ade91f) +++ b/Dockerfile (date 1650559344287) @@ -16,13 +16,18 @@ RUN ln -sf /dev/stdout /var/log/supervisor/supervisord.log RUN ln -sf /dev/stdout /app/code/supervisord.log -RUN npm install -g mjml - -ARG BASEROW_VERSION=1.8.3 +ARG BASEROW_VERSION=1.9.1 RUN curl -LSs "https://gitlab.com/bramw/baserow/-/archive/${BASEROW_VERSION}/baserow-${BASEROW_VERSION}.tar.gz" | tar -xz -C /app/code/ --strip-components 1 -f - RUN virtualenv -p python3 env RUN env/bin/pip install --no-cache -r backend/requirements/base.txt + +# Baserow 1.9 requires node 12. When Baserow 1.10 is released, this can be removed as it will support node 16 already present in cloudron/base:3.2.0. +ARG NODEVERSION=12.22.2 +RUN mkdir -p /usr/local/node-${NODEVERSION} && \ + curl -L https://nodejs.org/dist/v${NODEVERSION}/node-v${NODEVERSION}-linux-x64.tar.xz | tar Jxf - --strip-components 1 -C /usr/local/node-${NODEVERSION} +ENV PATH /usr/local/node-${NODEVERSION}/bin:$PATH + RUN (cd web-frontend && yarn install && yarn build) RUN (mkdir -p /app/code/cloudron/cloudron && touch /app/code/cloudron/cloudron/__init__.py) Index: settings.py IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/settings.py b/settings.py --- a/settings.py (revision 825192ff012217ba76f726de16cd7a5cc5ade91f) +++ b/settings.py (date 1650557484604) @@ -3,9 +3,6 @@ MEDIA_ROOT = '/app/data/media' -MJML_BACKEND_MODE = 'cmd' -MJML_EXEC_CMD = 'mjml' - FROM_EMAIL = os.environ['CLOUDRON_MAIL_FROM'] EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend" EMAIL_USE_TLS = FalseI won't have a chance today to manually test this image, however it now builds for me and should work.
-
@nigel_baserow thanks for the quick response, I have adjusted the Dockerfile and it does build now with node 12.
For some reason it struggles to connect to redis upon installation, I will debug this tomorrow.
-
This is how we setup our redis connection with the Baserow django/celery backend:
REDIS_HOST = os.getenv("REDIS_HOST", "redis") REDIS_PORT = os.getenv("REDIS_PORT", "6379") REDIS_USERNAME = os.getenv("REDIS_USER", "") REDIS_PASSWORD = os.getenv("REDIS_PASSWORD", "") REDIS_PROTOCOL = os.getenv("REDIS_PROTOCOL", "redis") REDIS_URL = os.getenv( "REDIS_URL", f"{REDIS_PROTOCOL}://{REDIS_USERNAME}:{REDIS_PASSWORD}@{REDIS_HOST}:{REDIS_PORT}/0", )I'm guessing if there are redis problems then its due to the setup in https://git.cloudron.io/cloudron/baserow-app/-/blob/master/supervisor.conf
REDIS_HOST='%(ENV_CLOUDRON_REDIS_HOST)s', REDIS_PORT='%(ENV_CLOUDRON_REDIS_PORT)s', REDIS_PASSWORD='%(ENV_CLOUDRON_REDIS_PASSWORD)s',It looks like you instead provide a CLOUDRON_REDIS_URL with your redis addon, maybe using this instead will fix things?
So changing
supervisor.confto only have one env variable for redis being:REDIS_URL='%(ENV_CLOUDRON_REDIS_URL)s', -
@nigel_baserow thanks for the hints. I have tried the url as env variable but also even hardcoded the env vars to match yours without success.
The error remains:Clearing Baserow's internal generated model cache... Traceback (most recent call last): File "/app/code/env/lib/python3.8/site-packages/django_redis/cache.py", line 31, in _decorator return method(self, *args, **kwargs) File "/app/code/env/lib/python3.8/site-packages/django_redis/cache.py", line 117, in clear return self.client.clear() File "/app/code/env/lib/python3.8/site-packages/django_redis/client/default.py", line 434, in clear raise ConnectionInterrupted(connection=client) from e django_redis.exceptions.ConnectionInterrupted: Redis ConnectionError: Error -2 connecting to redis:6379. Name or service not known. During handling of the above exception, another exception occurred:Which kinda looks like it doesn't even pick those up, as it tries to connect to
redis:6379which is not correct (the port is but the host would beredis-<appid>)using the redis-cli works fine with the cloudron provided URL, so I guess it just didn't pick up the env vars by those names. Also I have seen the following warning:
WARNING: Baserow is configured to use a PUBLIC_BACKEND_URL of http://localhost:8000. If you attempt to access Baserow on any other hostname requests to the backend will fail as they will be from an unknown host.Please ensure you set PUBLIC_BACKEND_URL if you will be accessing Baserow from any other URL then http://localhost. WARNING: Baserow is configured to use a default PUBLIC_WEB_FRONTEND_URL of http://localhost:3000. Emails sent by Baserow will use links pointing to http://localhost:3000 when telling users how to access your server. If this is incorrect please ensure you have set PUBLIC_WEB_FRONTEND_URL to the URL where users can access your Baserow server.which might indicate none of the env vars are correctly picked up? Is there a chance they are differently named and somewhat translated by your Dockerfile start script?
-
@nebulon Figured out the problem!
In
start.shwe are callingmigrateandsync_templatesbefore we have setup the REDIS_HOST etc env variables from the CLOUDRON_ ones. These commands are the ones you are seeing failing to connect to redis and not the actual backend processes of Baserow. Previously in version 1.8 and prior these commands did not connect to redis. Now they do (they clear some cached data from redis which potentially becomes invalid on Baserow upgrade/database changes).Below is a patch where i've entirely removed the supervisor.conf
environmentsection and just moved those variables setup intostart.shprior to any command usage. This way we can be sure the management commands run with the same and correct environment as the actual supervisord run processes.Index: supervisor.conf IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/supervisor.conf b/supervisor.conf --- a/supervisor.conf (revision 825192ff012217ba76f726de16cd7a5cc5ade91f) +++ b/supervisor.conf (date 1650878179553) @@ -2,14 +2,6 @@ nodaemon = true logfile=/dev/null logfile_maxbytes=0 -environment = - DJANGO_SETTINGS_MODULE='cloudron.settings', - REDIS_HOST='%(ENV_CLOUDRON_REDIS_HOST)s', - REDIS_PORT='%(ENV_CLOUDRON_REDIS_PORT)s', - REDIS_PASSWORD='%(ENV_CLOUDRON_REDIS_PASSWORD)s', - PRIVATE_BACKEND_URL='http://localhost:8000', - PUBLIC_WEB_FRONTEND_URL='https://%(ENV_CLOUDRON_APP_DOMAIN)s', - PUBLIC_BACKEND_URL='https://%(ENV_CLOUDRON_APP_DOMAIN)s' [program:gunicorn] user=cloudron Index: start.sh IDEA additional info: Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP <+>UTF-8 =================================================================== diff --git a/start.sh b/start.sh --- a/start.sh (revision 825192ff012217ba76f726de16cd7a5cc5ade91f) +++ b/start.sh (date 1650878843914) @@ -7,11 +7,19 @@ fi source /app/data/.secret +export REDIS_HOST="$CLOUDRON_REDIS_HOST" +export REDIS_PORT="$CLOUDRON_REDIS_PORT" +export REDIS_PASSWORD="$CLOUDRON_REDIS_PASSWORD" +export DJANGO_SETTINGS_MODULE="cloudron.settings" +export PRIVATE_BACKEND_URL="http://localhost:8000" +export PUBLIC_WEB_FRONTEND_URL="https://$CLOUDRON_APP_DOMAIN" +export PUBLIC_BACKEND_URL="https://$CLOUDRON_APP_DOMAIN" + echo "==> Executing database migrations" -/app/code/env/bin/python /app/code/backend/src/baserow/manage.py migrate --settings=cloudron.settings +/app/code/env/bin/python /app/code/backend/src/baserow/manage.py migrate echo "==> Syncing templates" -/app/code/env/bin/python /app/code/backend/src/baserow/manage.py sync_templates --settings=cloudron.settings +/app/code/env/bin/python /app/code/backend/src/baserow/manage.py sync_templates chown -R cloudron:cloudron /app/dataWith these changes I can now startup this image and it all works for me locally.
-
@nigel_baserow I tried applying these patches but a normal
git apply this.patchresulted in some errors.
I cleanup up the diffs and split them up. (It seems the intellij version does not work nicely with git)diff --git a/supervisor.conf b/supervisor.conf --- a/supervisor.conf (revision 825192ff012217ba76f726de16cd7a5cc5ade91f) +++ b/supervisor.conf (date 1650878179553) @@ -2,14 +2,6 @@ nodaemon = true logfile=/dev/null logfile_maxbytes=0 -environment = - DJANGO_SETTINGS_MODULE='cloudron.settings', - REDIS_HOST='%(ENV_CLOUDRON_REDIS_HOST)s', - REDIS_PORT='%(ENV_CLOUDRON_REDIS_PORT)s', - REDIS_PASSWORD='%(ENV_CLOUDRON_REDIS_PASSWORD)s', - PRIVATE_BACKEND_URL='http://localhost:8000', - PUBLIC_WEB_FRONTEND_URL='https://%(ENV_CLOUDRON_APP_DOMAIN)s', - PUBLIC_BACKEND_URL='https://%(ENV_CLOUDRON_APP_DOMAIN)s' [program:gunicorn] user=cloudrondiff --git a/start.sh b/start.sh --- a/start.sh (revision 825192ff012217ba76f726de16cd7a5cc5ade91f) +++ b/start.sh (date 1650878843914) @@ -7,11 +7,19 @@ fi source /app/data/.secret +export REDIS_HOST="$CLOUDRON_REDIS_HOST" +export REDIS_PORT="$CLOUDRON_REDIS_PORT" +export REDIS_PASSWORD="$CLOUDRON_REDIS_PASSWORD" +export DJANGO_SETTINGS_MODULE="cloudron.settings" +export PRIVATE_BACKEND_URL="http://localhost:8000" +export PUBLIC_WEB_FRONTEND_URL="https://$CLOUDRON_APP_DOMAIN" +export PUBLIC_BACKEND_URL="https://$CLOUDRON_APP_DOMAIN" + echo "==> Executing database migrations" -/app/code/env/bin/python /app/code/backend/src/baserow/manage.py migrate --settings=cloudron.settings +/app/code/env/bin/python /app/code/backend/src/baserow/manage.py migrate echo "==> Syncing templates" -/app/code/env/bin/python /app/code/backend/src/baserow/manage.py sync_templates --settings=cloudron.settings +/app/code/env/bin/python /app/code/backend/src/baserow/manage.py sync_templates chown -R cloudron:cloudron /app/data
Currently building and testing.
-
Finally Baserow 1.9 got out as a package. Apologies for the huge delay here, we will try to get this up to speed in the future like for other packages. Thanks @nigel_baserow and @BrutalBirdie for the help!
-
N nebulon has marked this topic as solved on
)