At the end of the day, I found the culpret.
Also upgraded to a bigger server.
While yes N8N will happily crash it's self - I had a few perfmance bottlenecks and for now N8N seems to be happy with the traffic. Here is a breakdown of the solution without setting up queue mode.
- N8N uses postgress - so bumping up the RAM for N8N is not enough that just grows the list of RAM eating active processes, you also have to go bump the RAM for Postgres
- I am using Baserow for logging and followup task Queueing. By default baserow only has 3 Auth workers which is not enough to cover heavy load. So I had to install the CLI and run the command cloudron env set --app baserow.draglabs.com BASEROW_AMOUNT_OF_GUNICORN_WORKERS=9
Conclusion - Step 1 Significantly improved the performance of N8N dropping normal RAM consumption by about 30%
Step 2 Caused active processes to complete 10X faster further reducing N8N footprint.
Now that I upgraded to 24 gigs I am only using 8