Edit: Added as a Community Guide : Monitoring and Alerting

---

Motivations

Since I started using Cloudron, I've had the same recurring pain point:
It lacks a built-in monitoring and alerting system to help me anticipate and fix issues before they break things.

Don't get me wrong, I love Cloudron and what it does. I just feel like it doesn't give us the best tools to avoid shooting ourselves in the feet regarding monitoring and best practices.

I mean, just a guide (like this one) is all I was asking for, from the perspective of a web developer who needs to efficiently monitor my infrastructure. I needed to be given some strong foundation about how to do that, without having to experiment by myself tons of solutions.

So, this guide aims to change that, and provide you with a 5mn guide to add monitoring to your Cloudron servers, with those goals in mind:

• Simple to set up
• Receive notifications when alerts occur
• Handle multiple servers and centralize them
• Freemium (starts for free, pay if you grow beyond limits)

Solution - Using Netdata

Netdata is the solution I retained for this, it ticked all my boxes, and was super simple to install.
I tried Grafana first, and ugh, wasted 2-3h trying to understand how to do the setup, eventually succeeded, to finally fail to set up notifications.

How to install in 5mn (step by step)

Netdata steps are super simple:

• Create a Netdata account
• Add your first Node, select "Linux" (path: Deploy/Operating Systems/Linux)
• Copy the command line provided (wget...)
• Paste and execute the command line in the Linux server you want to monitor (basically, your Cloudron server), you'll need to connect to it first (using SSH, most likely)
• Accept the various prompts (Netdata installs itself as a background service, and it nicely asks for your confirmation before doing that)
• You're done, check your Netdata dashboard, the data will start flowing in immediately

Video guide

Here is a 10mn video guide covering the installation part, and also a demo about how the notification system works when an alert is triggered.

Other solutions

Grafana

Grafana is theoretically supposed to do the same thing as Netdata, it's OSS, too.
But when it came to installing and configuring, I found the process so much harder (I kind of wish I had tried Netdata first).

Eventually, I got stuck with the notification system and tried out Netdata, which was a pleasant surprise of simplicity and efficiency.

Custom scripts

Other members have proposed using custom bash scripts to handle the monitoring part.

But I don't want to do that. I don't know what I don't know, and I for sure know I'm not a sysadmin expert, and I also know I don't want to become one.

So, relying on scripts that I would need to install and maintain myself is a big no-go. I'm pretty sure I can't do as good of a job as a specialized tool. And, clearly, Netdata showed me I was on point about that.

Figured it out.

Note: I manually created a n8n-uploads folder in my case.

In n8n, using the "Read/Write Files from Disk" action:

• File Path and Name: /app/data/n8n-uploads/file.jpeg

/app/data/file.jpeg works out of the box, but I wanted to store all my files in the same folder.

Here is a video:
https://www.loom.com/share/5e6bccdbd19842a489cc10f90ea871d4

@girish said in Cloudron backup import failed - Not enough disk space, SSL certificate errors:

@AmbroiseUnly said in Cloudron backup import failed - Not enough disk space, SSL certificate errors:

When restoring a backup for an app, the restoration tool should first check whether the backup is available online

When the backup is not found, the restore operation will fail and then you can try to restore again. But that's the theory... I guess it didn't work that way for you.

That's how it works, it's just that it does numerous operations before fetching the backup, and thus failing a long time after beginning the import process, although it could fail right away, if it was fetching the backup earlier.

Are you still facing any issues. I lost track of the exact situation since it's across many threads.

Eventually, I imported it Dry Run mode and that worked well, I then updated the IP address myself and all is good now. The restore feature got broken because of some DNS stuff I still don't understand (was fetching DNS configuration from DigitalOcean instead of AWS Route53, for unknown reason)

Is there any plan on adding AI-based projects to Cloudron?

I'm asking because I wanted to build something with OpenAI, and got stuck because the OpenAI API doesn't allow browsing the internet (unlike ChatGPT), which came as a surprise, and I'm looking to use a self-hosted solution instead.

https://taoteh1221.github.io/
https://github.com/taoteh1221/Open_Crypto_Tracker

Seems to be a good tool, completely open source (and free) to configure various things about crypto, including alerting of crypto markets (which are paid features in most places).

Just for alerts, this tool would be useful. But it does a lot of other things which might interest even more people.
It also runs on Raspberry PI.

MR done

On GitHub it still says "single user", but the https://docs.2fauth.app/ has been updated indeed.

The Self-hosted AI Starter Kit is an open-source template that quickly sets up a local AI environment. Curated by n8n, it provides essential tools for creating secure, self-hosted AI workflows.

https://github.com/n8n-io/self-hosted-ai-starter-kit

I believe this could be very interesting to support. n8n is already supported, but this tool seems like a completely different tool, with more pieces.

Already supports Docker Compose, so it seems it could be possible to add it as a new Cloudron App.

I'd typically want to spawn a new server and start testing things without bothering with all the installation details.

Also, providing some tips about optimizing its performances (using "Nvidia GPU") would be very helpful when selecting the server to host this app. I believe Cloudron might have a few partnerships with Cloud providers to help us pick a good choice for improved performances out of the box.

@fbartels said in How do you handle (auto/manual) updating apps with confidence?:

@joseph said in How do you handle (auto/manual) updating apps with confidence?:

based on the update schedule

Which just to have it said, could be adjusted to install a new version every hour.

https://docs.cloudron.io/updates/#update-schedule

For anyone interested in configuring proper monitoring on your Cloudron server, I wrote a guide about it, and I hope you'll find it useful!

I wished when this issue happened that I had been warned before the disk ran out of space, and that's what monitoring is for.

Just for the sake of documenting, here is what I ended up with, in my .env.sh file:

# ------------- NODE MODULES ---------------------

# Allow node modules to be used in code node - https://docs.n8n.io/hosting/configuration/#use-built-in-and-external-modules-in-the-code-node
# Allows usage of all builtin modules
export NODE_FUNCTION_ALLOW_BUILTIN=*

# FYI This is a SPACE separated list
export EXTRA_NODE_MODULES="cloudinary@2.2.0 node-fetch@3.1.1"

# Allow usage of external npm modules
# Those modules are auto-installed by Cloudron upon app restart, see https://forum.cloudron.io/topic/11946/how-to-install-npm-packages-that-n8n-app-can-use/4?_=1719382777426
# FYI This is a COMMA separated list
export NODE_FUNCTION_ALLOW_EXTERNAL=cloudinary,node-fetch

Postgresql contain 21 GB of data on my Cloudron server that only has a single app: Baserow

The baserow database itself only contains around a hundred records, nothing huge.
So, I wonder why the postgresql on that server is so huge. Can I login to it and navigate the PG DB directly? Or is there another simpler/smarter way to understand what's being stored there?

$ du -h --max-depth=1
21G	./postgresql
84K	./addons
197M	./mysql
20K	./oidc
518M	./mongodb
136K	./nginx
4.0K	./tls
16K	./logrotate.d
4.0K	./cifs
12K	./backup
4.0K	./acme
4.0K	./sshfs
236K	./update
8.0K	./diskusage
24K	./sftp
524K	./redis
207M	./logs
4.0K	./firewall
12K	./collectd
48M	./graphite
22G	.

Apparently, my n8n app is down with "n8n is starting up. Please wait" since yesterday and I'm noticing it just now.

The issue seems to be related to an update. I restarted the app and now it's stuck at Starting migration AddConstraintToExecutionMetadata1720101653148 for 15mn without anything happening in the logs and web app is still down.

Is that a known issue?

Jul 18 23:01:44 box:docker pullImage: {"status":"Extracting","progressDetail":{"current":1372,"total":1372},"progress":"[==================================================>] 1.372kB/1.372kB","id":"3f0342ae64ba"}
Jul 18 23:01:44 box:docker pullImage: {"status":"Extracting","progressDetail":{"current":1372,"total":1372},"progress":"[==================================================>] 1.372kB/1.372kB","id":"3f0342ae64ba"}
Jul 18 23:01:44 box:docker pullImage: {"status":"Pull complete","progressDetail":{},"id":"3f0342ae64ba"}
Jul 18 23:01:44 box:docker pullImage: {"status":"Digest: sha256:b7f6e86bb634b0fc68c2020ec7760e3de437db744ac474a70137b16590a496a4"}
Jul 18 23:01:44 box:docker pullImage: {"status":"Status: Downloaded newer image for cloudron/io.n8n.cloudronapp:20240718-050754-90025e1e0"}
Jul 18 23:01:44 box:docker downloaded image cloudron/io.n8n.cloudronapp:20240718-050754-90025e1e0 . error: false
Jul 18 23:01:44 box:tasks update 1511: {"percent":35,"message":"Cleaning up old install"}
Jul 18 23:01:44 box:apptask deleteContainer: deleting app containers (app, scheduler)
Jul 18 23:01:44 box:shell removeLogrotateConfig /usr/bin/sudo -S /home/yellowtent/box/src/scripts/configurelogrotate.sh remove 693caa42-bcf9-44c2-8f44-78f8676fe44c
Jul 18 23:02:18 box:services teardownAddons: Tearing down []
Jul 18 23:02:18 box:tasks update 1511: {"percent":45,"message":"Downloading icon"}
Jul 18 23:02:18 box:apptask downloadIcon: Downloading icon of io.n8n.cloudronapp@3.41.0
Jul 18 23:02:19 box:tasks update 1511: {"percent":60,"message":"Updating addons"}
Jul 18 23:02:19 box:services setupAddons: Setting up ["localstorage","postgresql","sendmail"]
Jul 18 23:02:19 box:services setupAddons: setting up addon localstorage with options {}
Jul 18 23:02:19 box:services setupLocalStorage
Jul 18 23:02:19 box:shell createVolume /usr/bin/sudo -S /home/yellowtent/box/src/scripts/mkdirvolume.sh /home/yellowtent/appsdata/693caa42-bcf9-44c2-8f44-78f8676fe44c/data
Jul 18 23:02:19 box:services setupAddons: setting up addon postgresql with options {}
Jul 18 23:02:19 box:services Setting up postgresql
Jul 18 23:02:19 box:services Setting postgresql addon config to [{"name":"CLOUDRON_POSTGRESQL_URL","value":"postgres://user693caa42bcf944c28f4478f8676fe44c:e86932a5f7a3594662665ca9471e8369daecdaf0047796c38a928558e2678cd16a39086932e427b94f104606d5d9534487d9d9020683d8e13bd3f2280a3385f5@postgresql/db693caa42bcf944c28f4478f8676fe44c"},{"name":"CLOUDRON_POSTGRESQL_USERNAME","value":"user693caa42bcf944c28f4478f8676fe44c"},{"name":"CLOUDRON_POSTGRESQL_PASSWORD","value":"e86932a5f7a3594662665ca9471e8369daecdaf0047796c38a928558e2678cd16a39086932e427b94f104606d5d9534487d9d9020683d8e13bd3f2280a3385f5"},{"name":"CLOUDRON_POSTGRESQL_HOST","value":"postgresql"},{"name":"CLOUDRON_POSTGRESQL_PORT","value":"5432"},{"name":"CLOUDRON_POSTGRESQL_DATABASE","value":"db693caa42bcf944c28f4478f8676fe44c"}]
Jul 18 23:02:19 box:services setupAddons: setting up addon sendmail with options {"supportsDisplayName":true}
Jul 18 23:02:19 box:services Setting up SendMail
Jul 18 23:02:19 box:services Setting sendmail addon config to [{"name":"CLOUDRON_MAIL_SMTP_SERVER","value":"mail"},{"name":"CLOUDRON_MAIL_SMTP_PORT","value":"2525"},{"name":"CLOUDRON_MAIL_SMTPS_PORT","value":"2465"},{"name":"CLOUDRON_MAIL_STARTTLS_PORT","value":"2587"},{"name":"CLOUDRON_MAIL_SMTP_USERNAME","value":"n8n@unly.org"},{"name":"CLOUDRON_MAIL_SMTP_PASSWORD","value":"a36d29838cf0ff6943b73e69475aa61a6bfda56caff91d43"},{"name":"CLOUDRON_MAIL_FROM","value":"n8n@unly.org"},{"name":"CLOUDRON_MAIL_DOMAIN","value":"unly.org"},{"name":"CLOUDRON_MAIL_FROM_DISPLAY_NAME","value":"my.n8n.unly.org"}]
Jul 18 23:02:19 box:tasks update 1511: {"percent":70,"message":"Creating container"}
Jul 18 23:02:19 box:apptask createContainer: creating container
Jul 18 23:02:19 box:shell getSwaps execArgs: swapon ["--noheadings","--raw","--bytes","--show=type,size,used,name"]
Jul 18 23:02:57 box:shell addLogrotateConfig /usr/bin/sudo -S /home/yellowtent/box/src/scripts/configurelogrotate.sh add 693caa42-bcf9-44c2-8f44-78f8676fe44c /tmp/693caa42-bcf9-44c2-8f44-78f8676fe44c.logrotate
Jul 18 23:02:57 box:apptask startApp: starting container
Jul 18 23:02:59 box:tasks update 1511: {"percent":90,"message":"Configuring reverse proxy"}
Jul 18 23:02:59 box:tasks update 1511: {"percent":100,"message":"Done"}
Jul 18 23:02:59 box:tasks setCompleted - 1511: {"result":null,"error":null}
Jul 18 23:02:59 box:tasks update 1511: {"percent":100,"result":null,"error":null}
Jul 18 23:02:59 box:taskworker Task took 177.89 seconds
Jul 19 09:10:35 Stopping n8n... "{ file: 'start.js', function: 'stopProcess' }"
Jul 19 09:10:35 2024-07-19T07:10:35.205Z | info | Received SIGTERM. Shutting down... "{ file: 'BaseCommand.js' }"
Jul 19 09:10:35 2024-07-19T07:10:35.211Z | debug | Shutting down component "Queue.pause()" "{ file: 'Shutdown.service.js', function: 'shutdownComponent' }"
Jul 19 09:10:35 2024-07-19T07:10:35.212Z | info |
Jul 19 09:10:35 2024-07-19T07:10:35.214Z | error | Error: There was an error shutting down n8n. "{ file: 'LoggerProxy.js', function: 'exports.error' }"
Jul 19 09:10:35 2024-07-19T07:10:35.214Z | error | TypeError: Cannot read properties of undefined (reading 'removeAllQueuedWorkflowActivations') "{ file: 'LoggerProxy.js', function: 'exports.error' }"
Jul 19 09:10:35 2024-07-19T07:10:35.216Z | debug | Shutting down component "License.shutdown()" "{ file: 'Shutdown.service.js', function: 'shutdownComponent' }"
Jul 19 09:10:35 2024-07-19T07:10:35.216Z | debug | Shutting down component "Push.onShutdown()" "{ file: 'Shutdown.service.js', function: 'shutdownComponent' }"
Jul 19 09:10:35 2024-07-19T07:10:35.217Z | debug | Shutting down component "ActiveWorkflowManager.removeAllTriggerAndPollerBasedWorkflows()" "{ file: 'Shutdown.service.js', function: 'shutdownComponent' }"
Jul 19 09:10:35 2024-07-19T07:10:35.217Z | debug | Shutting down component "ExecutionRecoveryService.shutdown()" "{ file: 'Shutdown.service.js', function: 'shutdownComponent' }"
Jul 19 09:10:35 2024-07-19T07:10:35.218Z | debug | Removing all trigger- and poller-based workflows "{\n file: 'ActiveWorkflowManager.js',\n function: 'removeAllTriggerAndPollerBasedWorkflows'\n}"
Jul 19 09:10:35 2024-07-19T07:10:35.219Z | debug | Shutting down component "AbstractServer.onShutdown()" "{ file: 'Shutdown.service.js', function: 'shutdownComponent' }"
Jul 19 09:10:35 2024-07-19T07:10:35.219Z | debug | Shutting down component "PruningService.shutdown()" "{ file: 'Shutdown.service.js', function: 'shutdownComponent' }"
Jul 19 09:10:35 2024-07-19T07:10:35.221Z | debug | Shutting down component "RedisClientService.disconnectClients()" "{ file: 'Shutdown.service.js', function: 'shutdownComponent' }"
Jul 19 09:10:35 2024-07-19T07:10:35.221Z | debug | [Pruning] Removing soft-deletion and hard-deletion timers "{ file: 'pruning.service.js', function: 'stopPruning' }"
Jul 19 09:10:35 box:taskworker Starting task 1518. Logs are at /home/yellowtent/platformdata/logs/693caa42-bcf9-44c2-8f44-78f8676fe44c/apptask.log
Jul 19 09:10:35 box:apptask run: startTask installationState: pending_restart runState: running
Jul 19 09:10:35 box:tasks update 1518: {"percent":10,"message":"Starting app services"}
Jul 19 09:10:35 box:tasks update 1518: {"percent":20,"message":"Restarting container"}
Jul 19 09:10:38 => Ensure directories
Jul 19 09:10:38 => Loading configuration
Jul 19 09:10:38 => Setting permissions
Jul 19 09:10:38 box:tasks update 1518: {"percent":80,"message":"Configuring reverse proxy"}
Jul 19 09:10:38 box:shell providerMatches execArgs: openssl ["x509","-noout","-subject","-issuer"]
Jul 19 09:10:38 box:reverseproxy providerMatches: subject=CN = *.unly.org domain=*.unly.org issuer=C = US, O = Let's Encrypt, CN = E5 wildcard=true/true prod=true/true issuerMismatch=false wildcardMismatch=false match=true
Jul 19 09:10:38 box:shell getCertificateDates execArgs: openssl ["x509","-startdate","-enddate","-subject","-noout"]
Jul 19 09:10:38 box:reverseproxy expiryDate: subject=CN = *.unly.org notBefore=Jun 13 02:10:26 2024 GMT notAfter=Sep 11 02:10:25 2024 GMT daysLeft=53.791505925925925
Jul 19 09:10:38 box:reverseproxy ensureCertificate: n8n.unly.org acme cert exists and is up to date
Jul 19 09:10:38 box:reverseproxy needsRenewal: false. force: false
Jul 19 09:10:38 box:shell isOscpEnabled execArgs: openssl ["x509","-in","/home/yellowtent/platformdata/nginx/cert/_.unly.org.cert","-noout","-ocsp_uri"]
Jul 19 09:10:38 box:reverseproxy writeAppLocationNginxConfig: writing config for "n8n.unly.org" to /home/yellowtent/platformdata/nginx/applications/693caa42-bcf9-44c2-8f44-78f8676fe44c/n8n.unly.org.conf with options {"sourceDir":"/home/yellowtent/box","vhost":"n8n.unly.org","hasIPv6":true,"ip":"172.18.19.176","port":5678,"endpoint":"app","redirectTo":null,"certFilePath":"/home/yellowtent/platformdata/nginx/cert/_.unly.org.cert","keyFilePath":"/home/yellowtent/platformdata/nginx/cert/_.unly.org.key","robotsTxtQuoted":"\"User-agent: *\\nDisallow: /\"","cspQuoted":null,"hideHeaders":[],"proxyAuth":{"enabled":false,"id":"693caa42-bcf9-44c2-8f44-78f8676fe44c","location":"/"},"upstreamUri":"","ocsp":true,"hstsPreload":false}
Jul 19 09:10:38 box:shell reload /usr/bin/sudo -S /home/yellowtent/box/src/scripts/restartservice.sh nginx
Jul 19 09:10:39 => Installing cloudinary@2.2.0 node-fetch@3.1.1 form-data@4.0.0
Jul 19 09:10:39 box:tasks update 1518: {"percent":100,"message":"Done"}
Jul 19 09:10:39 box:taskworker Task took 4.249 seconds
Jul 19 09:10:39 box:tasks setCompleted - 1518: {"result":null,"error":null}
Jul 19 09:10:39 box:tasks update 1518: {"percent":100,"result":null,"error":null}
Jul 19 09:10:40 => Healtheck error: Error: connect ECONNREFUSED 172.18.19.176:5678
Jul 19 09:10:50 => Healtheck error: Error: connect ECONNREFUSED 172.18.19.176:5678
Jul 19 09:10:54 [..................] - idealTree:code: sill idealTree buildDeps [..................] \ idealTree:code: sill idealTree buildDeps [..................] \ idealTree:code: sill idealTree buildDeps [..................] \ idealTree:code: sill idealTree buildDeps [..................] \ idealTree:code: sill idealTree buildDeps [..................] \ idealTree:code: sill idealTree buildDeps [..................] | idealTree:code: sill idealTree buildDeps [..................] | idealTree:code: sill idealTree buildDeps [..................] - idealTree:code: sill idealTree buildDeps [..................] | idealTree:code: sill idealTree buildDeps [#########.........] \ idealTree: timing idealTree Completed in 4204ms [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] \ reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] - reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] / reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [#####
Jul 19 09:10:54 #############] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t [##################] | reify:@msgpackr-extract/msgpackr-extract-darwin-arm64: t <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - up to date, audited 1657 packages in 15s <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - 185 packages are looking for funding <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - run `npm fund` for details
Jul 19 09:10:54 <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - 19 vulnerabilities (9 moderate, 3 high, 7 critical) <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - To address issues that do not require attention, run: <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - npm audit fix <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - To address all issues (including breaking changes), run: <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - npm audit fix --force
Jul 19 09:10:54 <30>1 2024-07-19T07:10:54Z unly-n8n 693caa42-bcf9-44c2-8f44-78f8676fe44c 1050 693caa42-bcf9-44c2-8f44-78f8676fe44c - Run `npm audit` for details.
Jul 19 09:10:55 => Starting N8N
Jul 19 09:10:57 User settings loaded from: /app/data/user/.n8n/config
Jul 19 09:10:57 Loaded config overwrites from /app/data/configs/default.json
Jul 19 09:11:00 => Healtheck error: Error: connect ECONNREFUSED 172.18.19.176:5678
Jul 19 09:11:00 2024-07-19T07:11:00.894Z | info | Initializing n8n process "{ file: 'start.js', function: 'init' }"
Jul 19 09:11:01 2024-07-19T07:11:01.209Z | debug | Lazy Loading credentials and nodes from n8n-nodes-base "{\n credentials: 356,\n nodes: 455,\n file: 'LoggerProxy.js',\n function: 'exports.debug'\n}"
Jul 19 09:11:01 2024-07-19T07:11:01.231Z | debug | Lazy Loading credentials and nodes from @n8n/n8n-nodes-langchain "{\n credentials: 15,\n nodes: 75,\n file: 'LoggerProxy.js',\n function: 'exports.debug'\n}"
Jul 19 09:11:01 2024-07-19T07:11:01.514Z | debug | [Concurrency Control] Service disabled "{ file: 'concurrency-control.service.js', function: 'log' }"
Jul 19 09:11:01 2024-07-19T07:11:01.528Z | info | n8n ready on 0.0.0.0, port 5678 "{ file: 'AbstractServer.js', function: 'init' }"
Jul 19 09:11:01 2024-07-19T07:11:01.594Z | warn | Migrations in progress, please do NOT stop the process. "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"
Jul 19 09:11:01 2024-07-19T07:11:01.595Z | info | Starting migration AddConstraintToExecutionMetadata1720101653148 "{ file: 'migrationHelpers.js', function: 'logMigrationStart' }"

Here are the full logs:
https://gist.github.com/Vadorequest/7f879c614d8d6a4e40597bdad1c6b9bc

Restarting the app didn't work.

There was an awaiting Ubuntu update, I restarted the whole server and n8n booted properly this time.
So, it seems to be related to Ubuntu update, which got stuck the whole startup process.
I'm not sure what can be done to avoid this kind of situation.

https://docs.cloudron.io/apps/ntfy/ mentions link to https://forum.cloudron.io/category/148/piwigo instead of https://forum.cloudron.io/category/158/ntfy

I also encountered "disk full" issue, and I was quite dumbfounded there was no email notification for this, that seems pretty basic as far as monitoring goes.

Cloudron is well-placed to add this functionality, and it would save us so much headaches.

I've met quite a lot of "disk full" issues recently, and I'm not alone (thankfully).

I wonder about the benefits for Cloudron to optionally reserve part of the disk with "fake data", essentially locking away a bit of the space available in the main hard drive.
That might sound dumb, but it has benefits.

The main goal of such a feature would be to avoid the situation where:

• There is no disk space left
• We can't get back any, as we don't know what data we could delete
• Or, we know what data could be deleted, but to do so we need to be able to write a tiny bit first (like what happened there)
• Due to the disk being full, we have very little flexibility about what we can do, and simple solutions are out of our reach

From experience, it's very hard to recover from that issue when the disk is fully saturated, unless you know exactly what data can be removed safely and how to do it, or can rescale your server on the fly. You usually learn about them while trying to fix the whole thing.

By having Cloudron users have their disk being full a bit faster, it helps discover the "Disk full" issue a bit faster, too.
But, when the disk is really full, and we don't know what to delete, we are really stuck. We can't even do a clean stop of active apps and run a Cloudron backup of the system, because there is no more disk space to do so.

On the other hand, by locking away a bit of the disk (enough to allow performing ops, and not too much not to be a hindrance), and releasing it upon request, it gives Cloudron the ability to:

• Crash as usual when the disk is full
• Admin becomes aware of the "disk full" issue (usually not immediately, but when online services stop working)
• Admin can use "Release unused disk space" (the new feature I'm talking about)
  • We might also provide a few strategy tips there, such as stopping the apps, release and then do a backup
• Cloudron releases some space by removing its "reserve"
• Disk is no longer entirely full and Admin can now run some operations, such as:
  • Run a Cloudron Backup
  • Restart the apps (if they were stopped or errored due to no disk space left)

This solution could provide both awareness of the root issue, and enough time/space to find a proper solution while continuing operations.

Typically, for a server of 40GB, I would imagine a "reserve" of 1-5GB, configurable by Admin, 1Gb could be the default. It's essentially a way of asking our future self about "how much time/space do you want to have to find a proper solution, once you've figured out you didn't anticipate your disk being saturated and have to rely on the reserve, before you run out of space again".

In a way, it's a bit like how cars work. They don't just suddenly stop when you run out of oil, they don't stop at all, but they vehemently signal you that you're on the reserve and need a refill real quick. Cloudron doesn't have a native monitoring system about the hard drive, but could give us an alternative that isn't as complex to implement.

root@n8n-ubuntu-4gb-nbg1-2:~# ping route53.amazonaws.com
PING route53.amazonaws.com (54.239.31.187) 56(84) bytes of data.
64 bytes from 54.239.31.187 (54.239.31.187): icmp_seq=1 ttl=235 time=92.7 ms

Indeed, after assigning an IP v4 the ping works.
It should be part of the Cloudron documentation that not having an IP v4 can lead to such troubles.
And/or adapt Cloudron install to reach AWS on IPv6, as fallback.

Yesterday, I mistakenly restarted an App by changing the allocated max CPU.

I hadn't planned on restarting the app at this time, and Cloudron didn't give me any hint about what would cause this change.

I believe the Cloudron UI should always prompt for a User confirmation whenever an action will trigger a restart (App restart, whole Instance restart), as it seems to be a best practice to avoid shooting ourselves in the feet.

@necrevistonnezr said in Guide: How to add monitoring to your Cloudron server for free (RAM/Disk/etc.), using Netdata (5mn setup):

This should go to https://docs.cloudron.io/guides/community/ - you can create a PR yourself!

Great suggestion, I'll do that now!

---

I looked at https://git.cloudron.io/cloudron/docs, and it doesn't provide a "Contributing" section or Contributing.md about how to provide a contribution.

It would help external contributors to get started autonomously while ensuring we're following the expected procedure. Right now, I'm a bit at a loss with how to contribute. Do I need to create an account? Should I just clone (or fork?) and just create a branch? I don't see where Pull Requests are located.

Plenty of questions that could be addressed with a short guide