Any issues with including NetData on the root server and as an app add-on?

[girish]

I looked into this a bit. The installation adds a custom apt repo and sets up automatic update stuff and installs a few other things. In general, it probably doesn't break anything but I find it hard to gauge what it could potentially break. My understanding is this is similar to DigitalOcean's and AWS monitoring tools (which are optionally installed on the server) to provide dashboards. We have not heard of people facing issues when they install the tools.

I think the best approach here is to deal with issues as they appear @imc67 . What do you think? I can't think of any better way. We can't possibly track netdata releases or their installer code to understand what all changes are happening.

[marcusquinn]

@girish I think the utility being sought here is Application/Service level resource usage data, as opposed to the overall general CPU/RAM that host monitoring presents. As in, when something is hogging resources, it can be identified, same for when something is allocated more than it needs.

[fbartels]

Using the install script is not even necessary, as netdata can just as well be run as a container itself. So a simple docker-compose.yaml with the following is enough:

version: '3'
services:
  netdata:
    image: netdata/netdata
    container_name: netdata
    pid: host
    network_mode: host
    restart: unless-stopped
    cap_add:
      - SYS_PTRACE
      - SYS_ADMIN
    security_opt:
      - apparmor:unconfined
    volumes:
      - ./netdataconfig/netdata:/etc/netdata
      - netdatalib:/var/lib/netdata
      - netdatacache:/var/cache/netdata
      - /etc/passwd:/host/etc/passwd:ro
      - /etc/group:/host/etc/group:ro
      - /proc:/host/proc:ro
      - /sys:/host/sys:ro
      - /etc/os-release:/host/etc/os-release:ro
      #- /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      - DOCKER_HOST=127.0.0.1:2375
  cetusguard:
    image: hectorm/cetusguard:v1
    network_mode: host
    read_only: true
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    environment:
      CETUSGUARD_BACKEND_ADDR: unix:///var/run/docker.sock
      CETUSGUARD_FRONTEND_ADDR: tcp://:2375
      CETUSGUARD_RULES: |
        ! Inspect a container
        GET %API_PREFIX_CONTAINERS%/%CONTAINER_ID_OR_NAME%/json

volumes:
  netdatalib:
  netdatacache:

Afterwards one can just create an app proxy to http://127.0.0.1:19999 and netdata can be "publicly" reached.

The above docker-compose.yaml actually comes from the netdata documentation.

[girish]

@fbartels ah nice, cetusguard is a great find. Haven't heard of that before! We have our own internal docker proxy protector which does similar.

[fbartels]

@girish yes, that one was new to me as well. I think they recommended a different proxy before, but i cannot remember which one.

[imc67]

@fbartels @girish is this a base for a Cloudron app candidate or even possible to startup as-is?

[girish]

For Netdata to work well, it has to run like it runs on the host itself i.e without a sandbox. If you see https://learn.netdata.cloud/docs/installing/docker#recommended-way or the compose file @fbartels posted, it is giving access to proc, etc, sys of the host and docker access. This is security risk (or not) depending on whether you trust netdata. Cloudron apps don't have access to any of these things.

[imc67]

@fbartels said in Any issues with including NetData on the root server and as an app add-on?:

Afterwards one can just create an app proxy to http://127.0.0.1:19999 and netdata can be "publicly" reached.

I just installed Netdata (stable) on another Cloudron production server. Installation went well, connection to Netdata cloud dashboard went well and indeed as @fbartels wrote, the "app proxy" works also to have a local only view!

Perfect!

[girish]

@imc67 Did you install as docker container or via the apt repos?

[imc67]

@girish via apt

[d1rk]

@imc67 I understood, that installing netdata via the Kickstart command is not a bad thing to do?

https://learn.netdata.cloud/docs/installing/one-line-installer-for-all-linux-systems

ofc, taking into account, that messing with the system like that deviates from what is suggested for obvious reasons by Cloudron and is in my responsibility.

[imc67]

@d1rk on 3 of my Cloudrons it is working without an issue but indeed that’s not a measure for succes

[robi]

It might be useful to use --disable-telemetry: Disable anonymous statistics. for the privacy conscious; as well as the local webUI.

[d1rk]

Thanks for your feedback. Yeah, that would have been my approach, anyway:

• disable any remote communication
• use local webUI
• install via kickstart (which results in apt-packages)

Ok, I would give it a go and see for myself.

[girish]

Looks like we got our first support ticket related to installing netdata. I have not verified this but it seems that installing netdata installs the nodejs package which ends up downgrading nodejs to v12 . This in turn prevents Cloudron from starting up.

It's better to install netdata via Docker, atleast it would prevent the above issue.

[d1rk]

Oh, interesting find. I have to check which version of node I have. Currently it seems to work fine.

Thanks for keeping us posted, here.

[imc67]

Uhoh how do we check the version? And how to install it via Docker?

Netdata is extremely useful and needed like having a live dashboard in your car. Without its hard to drive

[imc67]

@imc67 said in Any issues with including NetData on the root server and as an app add-on?:

Uhoh how do we check the version?

I just found out: node -v on the command line.

On my 3 Cloudron's it says v18.16.0

Is that OK @girish ?

[imc67]

BTW nice to know:

Netdata is the most energy-efficient monitoring tool for Docker-based systems

Dec 11, 2023: University of Amsterdam published a study related to the impact of monitoring tools for Docker based systems, aiming to answer 2 questions:

What is the impact of monitoring tools on the energy efficiency of Docker-based systems?
What is the impact of monitoring tools on the performance of Docker-based systems?
They tested ELK, Prometheus, Netdata and Zipkin, under 9 different configurations (Low, Mid, High Frequency vs. Low, Mid, High Workload, 3x3).

This is how Netdata stands:

Netdata excels in energy efficiency: "... Netdata being the most energy-efficient tool ...", as the study says.
Netdata excels in CPU Usage, RAM Usage and Execution Time, and has a similar impact in Network Traffic as Prometheus.
The study did not normalize the results based on the number of metrics collected. Given that Netdata usually collects singificantly more metrics than the other tools, Netdata managed to outperform the other tools, while ingesting a much higher number of metrics. Read the full study here: https://www.ivanomalavolta.com/files/papers/ICSOC_2023.pdf

source: https://github.com/netdata/netdata

[girish]

@imc67 said in Any issues with including NetData on the root server and as an app add-on?:

On my 3 Cloudron's it says v18.16.0

Is that OK @girish ?

It is correct version right now. If netdata has a nodejs dep, then it might conflict with Cloudron's. We update Cloudron's nodejs without thinking about other software.