Add ability to run VM like containers in Cloudron via Sysbox

[robi]

With a small change to default docker config, we can now use sysbox-runc as a runtime. Thanks Nestybox.com

This is awesome as it let's one run full machine images in docker containers, fast, rootless, secure.

How would you like to run the latest Ubuntu image in a container inside Cloudron? Docker in Docker? Cloudron in Docker?

https://blog.nestybox.com/2020/10/06/related-tech-comparison.html

Since I'm all into emerging technologies, they were also invited to give a talk at my meetup, https://BayLISA.org
..and everyone loved it.

One can run a full OS in a container now with no performance loss. Full speed.

This would also allow for Cloudron in Cloudron (CinC) as well as a properly secure Docker in Docker(DinD). (Inception)

These are the requirements, which all Cloudrons already have: https://github.com/nestybox/sysbox-ee/blob/master/docs/distro-compat.md with the only exception at least kernel 5.4.

[murgero]

@robi said in Add new OCI container runtime - Sysbox:

One can run a full OS in a container now!

Can you already technically do this? Even without root, one can get X11, browsers, etc etc etc in docker - am I missing something here?

[robi]

@murgero yes, you are missing a lot. read the blog

Containers were designed for apps.

Sysbox is designed for more than apps.

It's significant.

[Lonkle]

@mehdi said in Why do we have to push an image to a registry?:

https://git.cloudron.io/cloudron/box/-/commit/546e38132510e29792323a9947ac7cdf9aa55c98

The patch is in a commit in the master branch, so it will be in the next release

I only realized that after I added the patch cause I had to copy and paste from master code. I just got confused because he said "patch" - but I'm happy this is in master. I know you're a developer so it could help you too in the right scenario. And IIRC, you are one of the few that knows how to work with box code.

[robi]

Can't wait for Sysbox integration so you can have a dev cloudron running on any prod Cloudron in a container, completely isolated.

[Lonkle]

@robi said in Why do we have to push an image to a registry?:

Can't wait for Sysbox integration so you can have a dev cloudron running on any prod Cloudron in a container, completely isolated.

What, you're kidding me, that would help so much with box code changes. I've been making all of my box code changes in ssh via nano and...well, that method leaves something to be desired with how many changes all the files went through.

It's this "Sysbox integration" planned? Still not sure exactly what Sysbox is, but if it allows me to contribute more easily to Cloudron box code, I'M INNNN!!!

[robi]

Yes, I posted about it.

[mehdi]

It is not planned, I believe, as the devs have not given their opinion in the matter.

Also, one could technically already create a Cloudron-in-cloudron app today. The limit is not the containers, as you can run docker-in-docker. The limit is that it would be an infrastructure nightmare, like which cloudron should expose which ports on the main public IP, and such. Sysbox would not help at all in this matter.

[mehdi]

I do not think Cloudron should move to Sysbox.

• It is not necessary for cloudron apps to run "full systems" instead of just an app inside the container. I do not see any usecase for this
• ~350 github stars vs ~7300 => it's less maintained, there's less community : let's stick to the regular runc everyone uses, it will be easier to solve problems when they arise

[Lonkle]

@mehdi said in Add ability to run VMs in containers in Cloudron via Sysbox:

The limit is that it would be an infrastructure nightmare, like which cloudron should expose which ports on the main public IP, and such.

Well, running Cloudron inside of Cloudron as an app would just massively speed up development. That would be my only reason, so
wouldn't care at all about security. Just development speed.

[Lonkle]

@robi What does sysbox allow for beyond @mehdi's suggestion of just adjusting Cloudron itself into having the ability to be inside it's own container - which for me would allow for box code contributions to use the same development flow as my app development flow - so I'd contribute more to the project, given it is open-source.

I don't think I'd need something like Sysbox if that's all I want it for, right?

[Lonkle]

Yeah, I reread this and I see sysbox of just being needless overhead when Cloudron can just be adapted as an app (to easier develop on, it the developers ever alright that). Reminds me of the whole "you can only emulate the Apple OS on Apple Hardware" thing. But it would be cool if I could alter Cloudron a little bit to give it the ability to run inside another Cloudron. The dev benefits alone, so much more speedy for box changes. At least for external contributors. I don't have a build flow for box so if we're ever able to use our normal cloudron update for it, I'll be contributing a lot more to this project with feature forks.

[robi]

Who doesn't want strongly isolated containers without having to run actual VMs?

Nestybox empowers containers to act as virtual servers capable of running the same workloads as VMs (e.g., Systemd, Docker, Kubernetes, and even legacy apps).

​Currently this requires unsecure privileged containers plus complicated Docker images with tricky entrypoints and custom volume mounts.

No more.
Nestybox enables you to do this using:

• Simple Docker commands
• Simple Docker images
• Strongly Isolated Containers
• No Hardware Virtualization (VMs)

Use Cases:
Kubernetes-in-Docker

• Running Kubernetes clusters inside containers is very useful for development, testing, and CI/CD.
• It avoids the need for heavy and costly VMs or cloud-based clusters.
• There exist a few tools to run Kubernetes-in-Docker. However these use complex container images and very unsecure privileged containers.
• Nestybox fixes this, enabling you to deploy the cluster in containers using strong isolation and very simple container images that you fully control.

Lightweight VM

• Sysbox makes it easy to use containers as lightweight VMs. For example, a container image can include systemd, ssh, a Docker daemon, preloaded inner container images, etc. You have full root access inside the container, but no capabilities outside of it.
• You can pack 2x as many containers as VMs on the same machine and get the same performance. And you can provision them 10x faster than VMs.

Docker-in-Docker

• It's often useful to run Docker inside a container for development, testing, and CI/CD.
• Up to now, the only way to do this was to use very unsecure privileged containers or exposing the host's Docker socket into a container. Neither is ideal.
• Nestybox removes these limitations, enabling you to run Docker inside a container with total isolation from the host.
• You can even preload inner container images into the outer container using a Dockerfile or Docker commit.

Legacy Apps

• With Nestybox, legacy apps may be lift-and-shifted into containers, enabling them to operate within cloud-native frameworks without resorting to VMs. This voids the need for re-architecting such applications.

[Lonkle]

@robi Okay, I'm interested - compare the current system to this proposed system with some pros and cons?

[robi]

@lonk That would be great, try it out and see what breaks and where there are gaps.

[Lonkle]

@robi No no, I'm asking, what does this give us in a practical sense and how hard would it be to implement do you think?

[robi]

@lonk maybe read the thread again?

[Lonkle]

@robi So the pro is you can run your own OS completely inside a Docker container?

[robi]

@lonk yes, that was mentioned.

[Lonkle]

@robi So there’s more. But the developers seem against it. Can you tell why?

[robi]

@lonk against it? where does it say that?

[murgero]

@lonk said in Add ability to run VMs in containers in Cloudron via Sysbox:

No no, I'm asking, what does this give us in a practical sense and how hard would it be to implement do you think?

imho - What I can see down the road is the ability for companies to run some applications without the need to officially packaging the app. This can be useful for in-house apps that use parts of the filesystem that is normally read-only for example.

[robi]

@murgero Yes, that is what is meant by the Legacy Apps point above.

[Lonkle]

@robi Does it accomplish this by running another layer on top of the already existing Docker layer then?

[murgero]

@lonk I believe sysbox is a different container engine?

[Lonkle]

@murgero said in Add ability to run VMs in containers in Cloudron via Sysbox:

@lonk I believe sysbox is a different container engine?

Oh, now that I re-look at everything. You're right, I think it's too late for a restructure now.

[robi]

@murgero No.

It's simply a different container runtime.

Docker remains the same, we just tell it to use sysbox vs the default runcby adding --runtime sysbox-runc to the docker command line or default config.

That's it.

Simple.

[murgero]

@robi said in Add ability to run VMs in containers in Cloudron via Sysbox:

container runtime.

isn't that the same thing as engine? Or is docker the engine and containerd is the runtime?

[robi]

@murgero said in Add ability to run VMs in containers in Cloudron via Sysbox:

isn't that the same thing as engine? Or is docker the engine and containerd is the runtime?

No.
Docker Engine is a product name that uses containerd (the container daemon) which relies on runc (run container) which is a CLI tool for spawning and running containers according to the OCI specification.

All have a different abstraction level.

Therefore sysbox-runc is an alternate runc that is more secure and offers all of the above benefits.

Docker Engine and containerd don't change, and accept a parameter to specify which runtime (runc) to use.

[Lonkle]

@robi Thanks for going so much further into detail. Why do you personally want this feature?

[robi]

@lonk Let me count the ways.

1. It makes Cloudron better in so many ways already described above
2. It would let me have a build env in Cloudron
3. It would let me have a VDI in Cloudron via Guacamole
4. It would speed development
5. It would let me run more non-packaged apps more easily
6. It would open other opportunities we haven't even explored yet.

[Lonkle]

@robi said in Add ability to run VMs in containers in Cloudron via Sysbox:

@lonk Let me count the ways.

1. It makes Cloudron better in so many ways already described above
2. It would let me have a build env in Cloudron
3. It would let me have a VDI in Cloudron via Guacamole
4. It would speed development
5. It would let me run more non-packaged apps more easily
6. It would open other opportunities we haven't even explored yet.

Okay, perfect, now why do you think the developer's seem opposed (since those are the pros and if there were no cons, fs anyone would do it)? Time and effort switching infrastructures would be my personal guess.

[robi]

@lonk

1. fear?
2. lack of confidence?
3. not understanding how simple it may be?
4. time looking into it?
5. goto #1

[Lonkle]

@robi Can we help with any of those reasons or is this a vendetta we should give up?

[robi]

@lonk Hard to say, it's been a relatively odd echo chamber in this thread, so without more feedback and clarity of the thinking, it's feeling quite neglected.

[A Former User]

@robi I'd be interested in this. Will I likely use it to its full potential? No way. But my use case I am interested in: GitLab CI on cloudron without getting in the way of the containers on Cloudron. This would help, If I understand what I have read correctly.

[robi]

@atrilahiji yes, I think that's a docker-in-docker use case which is much better as a more isolated container.

[A Former User]

@robi hopefully we get this then I am tired of running my runner on a VM I pay for. Ideally I'd like the only things I pay for to me electricity (my server sips power) and Cloudron Licencing

[robi]

My wording isn't quite correct, it's not full VMs. See below.

https://blog.nestybox.com/2019/09/13/system-containers.html

A Nestybox system container is an enhanced Docker container, designed to package not just applications but also low-level system software.

What type of system software are we talking about? Currently Systemd and Docker, but in the near future software such as Kubernetes, graphical display servers, and others.

The following figure illustrates the difference.

But can’t you do this on a regular Docker container? No you can’t. Not properly.

For example, in order to run Docker inside a regular container (i.e., Docker-in-Docker) you need to run the container in “privileged” mode. This significantly weakens isolation between the container and the underlying host, posing a strong security risk (especially if you don’t trust the workloads running inside the container).

But in some cases even privileged mode is not sufficient. For example, some system level programs read resource consumption information from the kernel (e.g., via the Linux /proc directory). In order for the program to work properly inside a container, such information must be provided relative to the resources assigned to the container itself, not the resources of the underlying host. A regular container does not do this, even when running in privileged mode.

Nestybox system containers are designed to solve these problems.

We can summarize the key properties of a Nestybox system container as:

• Runs low-level system workloads (as well as applications).

• Provides strong isolation from the underlying host.

• Presents a more complete abstraction of a virtual host to its workloads.

• Typically runs multiple applications within it (rather than just one app).

One way to look at it is that a regular container packages applications. In contrast, a Nestybox system container packages virtual host environments capable of running applications as well as system-level workloads.
See it work!

Use Cases

But why would you want to run such system-level software inside a container in the first place? I.e., Why do we need system containers?

There are several use cases.

For example, by virtue of running Docker inside the container (securely), the system container can be used for:

• CI/CD pipelines (where the need for a container to run another container arises).

• Docker sandboxing (e.g., to run multiple Docker instances with total isolation between them).

Our blog site contains articles with practical examples of such use cases.

In the near future, as we add support for more system-level workloads inside the system container, more use cases will open up.

In general, if you have a need for a virtual host that runs many of the same workloads that you could run on a VM, yet is faster and more efficient, then a Nestybox system container is a good fit.

Key Features and Benefits

Deployment with Docker (and Kubernetes)

This allows you to leverage the power of these amazing tools to build, deploy, and manage system containers. No need to learn new tools.

Fast & Efficient

Just like regular application containers.

Strong Container Isolation

Nestybox system containers always use the Linux user namespace.

This means the root user in the system container has full capabilities inside the system container, but none outside of it.

In addition, Nestybox system containers use exclusive Linux user namespace user-ID and group-ID mappings for each system container.

If a process inside the container escapes the container sandbox, it will find itself without privileges to access resources of the host or of other containers.
Image Flexibility

A Nestybox system container image can be created with Docker, just like any Docker container.

However, it typically is configured with an environment resembling a virtual host (e.g., process manager, multiple apps, docker, app containers, graphical display server, etc), although you can also configure it with a single system-level application (e.g., Docker) if you wish. It’s up to you to choose what’s in the image and the entry-point.
Portability

You can deploy Nestybox system containers on any Linux machine, whether it’s bare-metal, a local VM, or a cloud VM, in a data-center, your laptop, an edge device, or even an IoT device.

And as with any Docker container you have the flexibility to move the system container around as you wish. Just upload it to your repo and deploy it on the target machine with Docker.
Partially virtualized procfs

In Nestybox system containers, portions of the Linux procfs (/proc) are virtualized. The goal is to make the system container more closely resemble a real host or VM. For example, the /proc/uptime file returns the container’s uptime, not the underlying host’s uptime.

How does it work?

Nestybox system containers are made possible by Sysbox, our system container runtime.

Sysbox is software that installs on the Linux host machine, integrates with Docker (and soon Kubernetes), and works under the covers.

Users interact with Docker to create the system container image and deploy it, just as with application containers. The difference is that this image can now include system-level software such as Docker itself (for Docker-in-Docker), etc.

The following figure illustrates this.

Running the system container is simple, it only requires passing the --runtime=sysbox-runc flag to Docker:

$ docker run --runtime=sysbox-runc -it my-syscont-image

Under the covers, Sysbox takes care of setting up the system container abstraction so that it can properly run system level workloads.

It’s easy. And you avoid the need for unsecure privileged containers or complex container configurations.

Is it a VM?

No, it’s not. It’s an enhanced container. As with all containers, it uses OS-level virtualization and shares the Linux kernel with the rest of the system. In contrast, VMs use hardware-level virtualization (i.e., emulate hardware in software) and have a dedicated OS per VM.

The following figure illustrates the differences.

This gives system containers and VMs different properties. In particular system containers are faster, more efficient, and more portable (see above) but offer a lesser degree of isolation from the underlying host.

From a workload perspective however, Nestybox is working to make our system containers support as many workloads as VMs can run such that they can present a viable alternative to VMs in some scenarios.