How to install npm modules?

[shrey]

Hi @nebulon. Thanks for the recommendation.

Although, that's a bit of an odd recommendation. Because, i would like to be able (and also expect) to use the tool (n8n) with pretty much all of its capabilities that are allowed for the self-hosted versions, being able to natively use any compatible npm package being one of them.

@nebulon said in How to install npm modules?:

If this is custom code then given the low complexity of those modules, I would suggest to just add the js file from them.

Also, this isn't very efficient or even feasible as firstly, reusability of code/scripts is not a big thing in n8n. So, the entire script will have to be included in every node, every time it's required. Secondly, that's not the normal way of doing things in n8n.

I was under the impression that this was supposed to be a simple formality (to send a request to include an npm package in the n8n modules) and not really a vetting process. If that's not the policy of Cloudron for the near-future, then kindly let me know.

Thanks.

[nebulon]

it is a simple change at first, but the more node modules we ship, the higher the chances that it will break if we update and we can't realistically test all this. Mostly this is also due to our experience with how often node_modules become unmaintained, or break compatibility. Like with other loose plugin systems this is becomes an update nightmare very soon, so we are a bit hesitant to add non-essential ones and especially not ones which are solidly maintained upstream.

[shrey]

@nebulon
Is it not possible to allow install/restrict the custom modules to the concerned Cloudron instance only, and not the common n8n Cloudron package?

[nebulon]

So far this was not possible to implement due to the way npm works and how n8n handles requires. This is actually the only reason why we have this workaround to include essential ones in the package.

There is always a fine line between being able to test updates and maintain packages vs. full customization with code changes by the sysadmin. We usually aim for a middleground where we don't lose the ability to test updates.

[girish]

Generally, I look into the modules a bit before including them into the package. For example, https://www.npmjs.com/package/generate-password hasn't been updated for 2 years. Is this safe anymore (given that it's generating passwords).

@shrey More often than not, it is best to include library functions in your code then use modules. We are a node shop ourselves, and we have actually removed almost 50% of the modules we used to depend on the past and just include them in our code base itself. The smaller simple library function style modules are in my experience very risky.

[girish]

To add, pwgen -1s 16 (pwgen is in the base image) gives you a strong password. Can also add -y for stronger.

[kastiell93]

@girish Hi! Can you add the firebase-admin package to n8n?

[nebulon]

Added now

[ebauger]

npm install stripe?

[ebauger]

Hi @nebulon!

What if we can install custom npm module from this line export NODE_FUNCTION_ALLOW_EXTERNAL=module1,modulel2,moduleN+1
in /app/data/env.sh

Here the modified Dockerfile I propose:

FROM cloudron/base:4.2.0@sha256:46da2fffb36353ef714f97ae8e962bd2c212ca091108d768ba473078319a47f4

RUN mkdir -p /app/pkg /app/code
WORKDIR /app/code

RUN apt-get update && \
    apt-get -y install graphicsmagick recutils asciidoctor pandoc musl && \
    rm -rf /var/cache/apt /var/lib/apt/lists

RUN ln -s /usr/lib/x86_64-linux-musl/libc.so /lib/libc.musl-x86_64.so.1

ARG N8N_VERSION=1.25.1

# Add a script to install npm packages from env.sh
COPY install_packages_from_env.sh /app/pkg/
RUN chmod +x /app/pkg/install_packages_from_env.sh

# n8n. handlebars and jsonata are just helpful modules that user can enable
RUN npm install n8n@${N8N_VERSION}  && \
    npm install handlebars@4.7.7 jsonata@2.0.2 marked@4.3.0 bwip-js@3.3.0 ajv-formats@2.1.1 odoo-xmlrpc@1.0.8 firebase-admin@11.10.1

# install custom npm modules from the env.sh on the NODE_FUNCTION_ALLOW_EXTERNAL
RUN /app/pkg/install_packages_from_env.sh /app/data/env.sh

RUN rm -rf /app/code/node_modules/n8n/dist/public && ln -s /run/public /app/code/node_modules/n8n/dist/public

ENV N8N_USER_FOLDER="/app/data/user"
ENV N8N_CONFIG_FILES="/app/data/configs/default.json"
ENV N8N_CUSTOM_EXTENSIONS="/app/data/custom-extensions"

ENV PATH=/app/code/node_modules/.bin:$PATH

COPY start.sh env.sh.template /app/pkg/

CMD [ "/app/pkg/start.sh" ]

And here's an example of what the install_packages_from_env.sh script might look like:

#!/bin/bash

# Script to install npm packages from env.sh

ENV_FILE=$1

if [ -f "$ENV_FILE" ]; then
    # Extract the line with NODE_FUNCTION_ALLOW_EXTERNAL and check if it's not commented
    PACKAGES_LINE=$(grep -E '^export NODE_FUNCTION_ALLOW_EXTERNAL=' $ENV_FILE)

    if [ ! -z "$PACKAGES_LINE" ]; then
        # Remove 'export ' prefix
        PACKAGES=$(echo $PACKAGES_LINE | sed 's/export NODE_FUNCTION_ALLOW_EXTERNAL=//')

        # Split the packages and install them if not already present
        IFS=',' read -ra ADDR <<< "$PACKAGES"
        for package in "${ADDR[@]}"; do
            if [ ! -d "./node_modules/$package" ]; then
                echo "Installing $package..."
                npm install $package
            else
                echo "$package is already installed."
            fi
        done
    else
        echo "No packages to install or line is commented out with #."
    fi
else
    echo "env.sh file not found"
fi

On (re)start of n8n, the npm modules should installed.

What do you think?

[nebulon]

I am not sure I follow the proposal 100% but the main root cause of not being able to install other node_modules is actually that apps run in a read-only filesystem. So I don't see how such a script would fix this problem, but maybe I miss a point in your proposal.

[ebauger]

@nebulon my proposal is to install the modules when we build the image from the Dockerfile. I open to any other solution.

The modules installed will be on the read-only filesystem. See the Diff from this source

...
# Add a script to install npm packages from env.sh
COPY install_packages_from_env.sh /app/pkg/
RUN chmod +x /app/pkg/install_packages_from_env.sh
...
# install custom npm modules from the env.sh on the NODE_FUNCTION_ALLOW_EXTERNAL
RUN /app/pkg/install_packages_from_env.sh /app/data/env.sh
...

[BrutalBirdie]

@ebauger
Docker images are build before runtime.
/app/data does only exist on runtime.
So the command:

RUN /app/pkg/install_packages_from_env.sh /app/data/env.sh

will not work.

But I understand where the intention is.

[girish]

Recently, with nodebb we started experimenting with making the node_modules writable (grudgingly). Maybe we can do the same for n8n.

[shrey]

@girish Certainly support this effort!

[girish]

This is implemented in https://docs.cloudron.io/apps/n8n/#custom-node-modules .

@shrey let me know if you get a chance to test it.

[shrey]

@girish Just tried it. Seems to work alright!