Add ability to run VM like containers in Cloudron via Sysbox

[Lonkle]

@robi So there’s more. But the developers seem against it. Can you tell why?

[robi]

@lonk against it? where does it say that?

[murgero]

@lonk said in Add ability to run VMs in containers in Cloudron via Sysbox:

No no, I'm asking, what does this give us in a practical sense and how hard would it be to implement do you think?

imho - What I can see down the road is the ability for companies to run some applications without the need to officially packaging the app. This can be useful for in-house apps that use parts of the filesystem that is normally read-only for example.

[robi]

@murgero Yes, that is what is meant by the Legacy Apps point above.

[Lonkle]

@robi Does it accomplish this by running another layer on top of the already existing Docker layer then?

[murgero]

@lonk I believe sysbox is a different container engine?

[Lonkle]

@murgero said in Add ability to run VMs in containers in Cloudron via Sysbox:

@lonk I believe sysbox is a different container engine?

Oh, now that I re-look at everything. You're right, I think it's too late for a restructure now.

[robi]

@murgero No.

It's simply a different container runtime.

Docker remains the same, we just tell it to use sysbox vs the default runcby adding --runtime sysbox-runc to the docker command line or default config.

That's it.

Simple.

[murgero]

@robi said in Add ability to run VMs in containers in Cloudron via Sysbox:

container runtime.

isn't that the same thing as engine? Or is docker the engine and containerd is the runtime?

[robi]

@murgero said in Add ability to run VMs in containers in Cloudron via Sysbox:

isn't that the same thing as engine? Or is docker the engine and containerd is the runtime?

No.
Docker Engine is a product name that uses containerd (the container daemon) which relies on runc (run container) which is a CLI tool for spawning and running containers according to the OCI specification.

All have a different abstraction level.

Therefore sysbox-runc is an alternate runc that is more secure and offers all of the above benefits.

Docker Engine and containerd don't change, and accept a parameter to specify which runtime (runc) to use.

[Lonkle]

@robi Thanks for going so much further into detail. Why do you personally want this feature?

[robi]

@lonk Let me count the ways.

1. It makes Cloudron better in so many ways already described above
2. It would let me have a build env in Cloudron
3. It would let me have a VDI in Cloudron via Guacamole
4. It would speed development
5. It would let me run more non-packaged apps more easily
6. It would open other opportunities we haven't even explored yet.

[Lonkle]

@robi said in Add ability to run VMs in containers in Cloudron via Sysbox:

@lonk Let me count the ways.

1. It makes Cloudron better in so many ways already described above
2. It would let me have a build env in Cloudron
3. It would let me have a VDI in Cloudron via Guacamole
4. It would speed development
5. It would let me run more non-packaged apps more easily
6. It would open other opportunities we haven't even explored yet.

Okay, perfect, now why do you think the developer's seem opposed (since those are the pros and if there were no cons, fs anyone would do it)? Time and effort switching infrastructures would be my personal guess.

[robi]

@lonk

1. fear?
2. lack of confidence?
3. not understanding how simple it may be?
4. time looking into it?
5. goto #1

[Lonkle]

@robi Can we help with any of those reasons or is this a vendetta we should give up?

[robi]

@lonk Hard to say, it's been a relatively odd echo chamber in this thread, so without more feedback and clarity of the thinking, it's feeling quite neglected.

[A Former User]

@robi I'd be interested in this. Will I likely use it to its full potential? No way. But my use case I am interested in: GitLab CI on cloudron without getting in the way of the containers on Cloudron. This would help, If I understand what I have read correctly.

[robi]

@atrilahiji yes, I think that's a docker-in-docker use case which is much better as a more isolated container.

[A Former User]

@robi hopefully we get this then I am tired of running my runner on a VM I pay for. Ideally I'd like the only things I pay for to me electricity (my server sips power) and Cloudron Licencing

[robi]

My wording isn't quite correct, it's not full VMs. See below.

https://blog.nestybox.com/2019/09/13/system-containers.html

A Nestybox system container is an enhanced Docker container, designed to package not just applications but also low-level system software.

What type of system software are we talking about? Currently Systemd and Docker, but in the near future software such as Kubernetes, graphical display servers, and others.

The following figure illustrates the difference.

But can’t you do this on a regular Docker container? No you can’t. Not properly.

For example, in order to run Docker inside a regular container (i.e., Docker-in-Docker) you need to run the container in “privileged” mode. This significantly weakens isolation between the container and the underlying host, posing a strong security risk (especially if you don’t trust the workloads running inside the container).

But in some cases even privileged mode is not sufficient. For example, some system level programs read resource consumption information from the kernel (e.g., via the Linux /proc directory). In order for the program to work properly inside a container, such information must be provided relative to the resources assigned to the container itself, not the resources of the underlying host. A regular container does not do this, even when running in privileged mode.

Nestybox system containers are designed to solve these problems.

We can summarize the key properties of a Nestybox system container as:

• Runs low-level system workloads (as well as applications).

• Provides strong isolation from the underlying host.

• Presents a more complete abstraction of a virtual host to its workloads.

• Typically runs multiple applications within it (rather than just one app).

One way to look at it is that a regular container packages applications. In contrast, a Nestybox system container packages virtual host environments capable of running applications as well as system-level workloads.
See it work!

Use Cases

But why would you want to run such system-level software inside a container in the first place? I.e., Why do we need system containers?

There are several use cases.

For example, by virtue of running Docker inside the container (securely), the system container can be used for:

• CI/CD pipelines (where the need for a container to run another container arises).

• Docker sandboxing (e.g., to run multiple Docker instances with total isolation between them).

Our blog site contains articles with practical examples of such use cases.

In the near future, as we add support for more system-level workloads inside the system container, more use cases will open up.

In general, if you have a need for a virtual host that runs many of the same workloads that you could run on a VM, yet is faster and more efficient, then a Nestybox system container is a good fit.

Key Features and Benefits

Deployment with Docker (and Kubernetes)

This allows you to leverage the power of these amazing tools to build, deploy, and manage system containers. No need to learn new tools.

Fast & Efficient

Just like regular application containers.

Strong Container Isolation

Nestybox system containers always use the Linux user namespace.

This means the root user in the system container has full capabilities inside the system container, but none outside of it.

In addition, Nestybox system containers use exclusive Linux user namespace user-ID and group-ID mappings for each system container.

If a process inside the container escapes the container sandbox, it will find itself without privileges to access resources of the host or of other containers.
Image Flexibility

A Nestybox system container image can be created with Docker, just like any Docker container.

However, it typically is configured with an environment resembling a virtual host (e.g., process manager, multiple apps, docker, app containers, graphical display server, etc), although you can also configure it with a single system-level application (e.g., Docker) if you wish. It’s up to you to choose what’s in the image and the entry-point.
Portability

You can deploy Nestybox system containers on any Linux machine, whether it’s bare-metal, a local VM, or a cloud VM, in a data-center, your laptop, an edge device, or even an IoT device.

And as with any Docker container you have the flexibility to move the system container around as you wish. Just upload it to your repo and deploy it on the target machine with Docker.
Partially virtualized procfs

In Nestybox system containers, portions of the Linux procfs (/proc) are virtualized. The goal is to make the system container more closely resemble a real host or VM. For example, the /proc/uptime file returns the container’s uptime, not the underlying host’s uptime.

How does it work?

Nestybox system containers are made possible by Sysbox, our system container runtime.

Sysbox is software that installs on the Linux host machine, integrates with Docker (and soon Kubernetes), and works under the covers.

Users interact with Docker to create the system container image and deploy it, just as with application containers. The difference is that this image can now include system-level software such as Docker itself (for Docker-in-Docker), etc.

The following figure illustrates this.

Running the system container is simple, it only requires passing the --runtime=sysbox-runc flag to Docker:

$ docker run --runtime=sysbox-runc -it my-syscont-image

Under the covers, Sysbox takes care of setting up the system container abstraction so that it can properly run system level workloads.

It’s easy. And you avoid the need for unsecure privileged containers or complex container configurations.

Is it a VM?

No, it’s not. It’s an enhanced container. As with all containers, it uses OS-level virtualization and shares the Linux kernel with the rest of the system. In contrast, VMs use hardware-level virtualization (i.e., emulate hardware in software) and have a dedicated OS per VM.

The following figure illustrates the differences.

This gives system containers and VMs different properties. In particular system containers are faster, more efficient, and more portable (see above) but offer a lesser degree of isolation from the underlying host.

From a workload perspective however, Nestybox is working to make our system containers support as many workloads as VMs can run such that they can present a viable alternative to VMs in some scenarios.