n8n Queue Mode / Worker support

[Finn]

I’d like to suggest adding support for n8n Queue Mode to the official n8n package on Cloudron.

This topic has already come up in older posts, and I think it is still very relevant. I understand that n8n itself only uses one CPU core per process. But that is exactly why Queue Mode would be useful: with multiple workers, executions and workflows can be distributed much better.

Especially when running Cloudron inhouse or on your own infrastructure, “just upgrade to a bigger VPS” is not always a real solution. Single-core performance cannot simply be increased endlessly. With larger amounts of data or heavier workflows, n8n can quickly become slow, hang, or crash.

In my opinion, Queue Mode would not even need to be enabled by default. It would already be very helpful if the official package included the required support so it could be enabled when needed.

I previously experimented with a custom version of the Cloudron n8n app, using AI, and managed to build a prototype with Queue Mode. I am not a Developper and just wandet to test if this would solve my perfomance issues wich it did. That makes me think it would be a real benefit to have this option available in the official package as well.

Overall, I believe this would make n8n on Cloudron much more robust for larger setups.

[joseph]

IIUC, you want n8n worker to run in the same machine. If this is what you want, https://docs.n8n.io/hosting/scaling/concurrency-control/ is better suited. Have you tried setting N8N_CONCURRENCY_PRODUCTION_LIMIT in the env file? See https://docs.cloudron.io/packages/n8n/#custom-env on how to set it. Concurrency is disabled by default .

[firmansi]

@joseph I think what finn means is different than concurrency control , it seems the queue mode mean there are two n8n servers in Cloudron and one can acts as controller and the other acts as workers, so I think what finn means that Cloudron team can add capability to existing n8n package to make the n8n package can act as workers, so when capacity getting bigger, then the load can be divided into several n8n instances instead of one big instance, so if there's a fail in one of the instance then it wont impact other instances or cause the controller UI hangs

[Finn]

@firmansi Yes, that is exactly what I mean.

Just to avoid a misunderstanding: I am not necessarily asking for multiple separate Cloudron n8n app instances to be connected manually.

What I would like to see is support for Queue Mode inside the existing official Cloudron n8n package. In my test setup, this worked as a single Cloudron app where the configuration allowed setting the number of workers. On app startup, the app then started the main n8n process plus the configured number of worker processes.

From a user perspective, this could still remain one Cloudron app:

• n8n main process for UI, webhooks and triggers
• Cloudron Redis as the queue backend
• configurable number of n8n worker processes
• worker count set through config/env
• workers started automatically when the app starts

@joseph This is different from N8N_CONCURRENCY_PRODUCTION_LIMIT.

Concurrency control can help prevent overload by limiting how many executions run at the same time. That is useful, and I will test it. But it does not solve the core issue I am seeing.

My workflows sometimes process more than 100,000 items. The workflows are already optimized with batching where possible, but without Queue Mode, the n8n UI still becomes unresponsive while executions are running. In many cases, it is basically luck whether the jobs finish or whether the whole n8n instance crashes and needs to be restarted.

With my Queue Mode test, batching was still required so the workload could be split properly across executions/workers. But the important difference was that the UI stayed responsive because the main process was not doing all the heavy execution work itself. This made the system much more usable and stable.

So I agree that concurrency control may improve stability in some cases, but it is not a replacement for Queue Mode. My request is about making the official Cloudron package capable of running n8n in this architecture, preferably with a simple way to configure the number of workers.

[Finn]

Just to add to this the main problem in my case is CPU load. A single n8n process can only effectively use one CPU core for execution work. So even if the server has more CPU cores available, one busy n8n process can still become the bottleneck. Queue Mode helps here because the execution work can be moved to multiple worker processes, allowing the workload to be distributed across CPU cores.

[james]

Hello @finn
We added the queue worker support https://forum.cloudron.io/post/125398
Documented here: https://docs.cloudron.io/packages/n8n/#queue-mode
Please leave some feedback since it sounds like you are using N8N quite a lot.