VPN tunnel for apps

[Lonkle]

So, the "Cloudron CLI" has a bunch of important functions that the REST API doesn't, yet the REST APIs documentation says the CLI uses the REST API. Are there just undocumented REST API functions that the Cloudron CLI client has?

My main goal is that I have a Docker container (https://github.com/dperson/openvpn-client) that runs external to Cloudron (named "vpn") on the VPS (though I could build it into an actual Cloudron app if needed). And I need to run my Cloudron apps on demand through that container using the --net=container:vpn argument when using Docker Run.

The CLI will allow me to update the VPN as needed if I turn it into a Cloudron app, but I haven't found a way within Cloudron CLI or it's REST API to add that argument --net=container:vpn before it starts an Dockerized app.

Does anyone have any ideas?

[mehdi]

@Lonk I think there is currently no function in the API to do what you need

I saw you post on the other thread, asking about the docker addon, and there might be a way to use it to manually start apps the way you want, but it would demand more docker trickery than I have to figure out how ^^

However, I think the cloudron team said somewhere that they were thinking about how to allow the apps to communicate directly together, so I think what you are trying to achieve may integrate with their feature roadmap (even though, I must admit I am not very clear on the details of what you're trying to do ^^)

[nebulon]

Generally it is not advised to install anything on the side on the same server. Neither through docker nor otherwise. The main reason is that we cannot test such configurations and setups and thus it will eventually break during an update. The only way to have it working reliably would be through making a Cloudron app package, as you already mentioned as an option.

I may not fully understand your vpn usage there, however please note, that apps on Cloudron already run within their own local network.

Regarding the CLI tool and the API, this discrepancy is only there since we haven't yet documented that whole API. But the cli as well as the actual dashboard are fully built on top of the REST API. So far the usage of the API directly simply wasn't asked for a lot, thus the lack of wholesome docs

[girish]

To take a guess : The use case is that you run a OpenVPN client container which connects to some VPN service (not the existing app which is OpenVPN server). Then you make some or all apps use the same network as this openvpn container. This way all traffic goes via the VPN. Pretty cool, if this is what @Lonk had in mind

I think recently @p44 asked here how to hide TTRSS's querying IP. This is possibly an answer to that question.

I wonder if this should really be done at the server level instead of per-app. Maybe @Lonk can explain his use case further.

[Lonkle]

@nebulon Thank you so much for the reason for the discrepancy. I was hoping that was it. I can just use Charles to RE your web app and grab the calls for myself since you already have a documented way to authenticate with the API.

@girish That is exactly correct! I've actually forked https://github.com/dperson/openvpn-client to allow for choosing your own .ovpn (instead of it being static, I have 10 for example) in one Docker container which I can pass an argument to at runtime -e "" to connect to x VPN service or router. It works really well locally via Docker, even multiple Docker containers can "mirror" it's network via -net=container:vpn-client and they all can the VPN network simultaneously or one by one (which is what I was hoping to accomplish via Cloudron).

But thanks to @nebulon's description of the Cloudron REST API. Despite these calls being undocumented, I can just stop the VPN container when not in use, and start it when it's needed (I'm going to recode my fork to pull from a static URL for the .ovpn file it uses and change that externally).

But that still doesn't solve the main problem of being able to have Cloudron app's use the same network config AKA Docker's -net=container:app-name equivalent as the VPN Client Cloudron app.

So I may be out of luck and have to code a vpn-client on the network context that is also running cloudron, and globally change networks instead of per app like the Docker container made possible. Unless you guys were interested in this specific use case. I'd be more than happy to complete my https://github.com/dperson/openvpn-client fork to create as an actual Cloudron add-on along with my other enhancements (choose your own VPN to connect). If @mehdi was saying is true and you wanted Cloudron apps to be able to connect to each other down the road, this would be a great first use case. I'll do as much of the work as you'll let me. I really like Cloudron, so I'd be more than happy to plus I'd get the benefit of running on the VPN's network per cloudron app instead of running it at the root level (which, like @nebulon mentioned, may break things down the road with Cloudron itself).

@mehdi I installed the Docker Custom Build Service in my Cloudron but this was a problem that could only be tackled from outside a Docker container.

[Lonkle]

@girish Also, I read @p44's thread and yes, this would resolve his request asl well.

[Lonkle]

@mehdi There’s scarce documentation on the Docker Cloudron add-on as well as the net_admin Cloudron capability. Between the two of those I should be able to get this working...maybe. The documentation is p spotty so I can’t know for sure until I do a lot of experimentation. @nebulon did confirm that I can start and stop Cloudron apps using private API calls so I no longer need to worry about the OpenVPN client Docker app (that I’m turning into a Cloudron app) switching VPNs.

I just have one last obstacle and that’s making one Cloudron container using the network of another Cloudron container. This is very easy using Docker. One or many apps can use the same network as the OpenVPN-Client. You just need to stop and start them passing the argument of “use the OpenVPN-Client‘S network.

Cloudron is built heavily around Docker principles so maybe, just maybe the net_admin capability will work for that. Otherwise, the Docker add-on should work despite it forcing me to use pure Docker instead of Cloudron apps to connect the network to the VPN Client Docker app.

Last option is to side-install both the OpenVPN Client Docker and another Docker to connect to its network but Cloudron administrators say that’s not a stable way to accomplish this and will probably break during an update.

[Lonkle]

My final option that @girish hinted in his last post (“should this be done an the app level”) would work is building my own custom Cloudron app, combining a Wordpress Docker with the OpenVPN Client Docker and use the trick @nebulon implied would work (restarting the custom Cloudron app I make from within the Cloudron Private API itself to switch VPNs so I don’t have to touch any real code in the OpenVPN Client Docker).

But in doing so, I have to work on combining two images that weren’t designed to be combined, and I lose Wordpress updates from the Cloudron App Store.

Is this the direction you guys think I should go in for this if I want this done? Or can you think of a way of getting one Cloudron app use another Cloudron app’s network in the same way Docker allows.

This project I want to finish in the next couple weeks and if this is the only way possible, I’ll get to it. But I wanted the staffs advice / input before I start this very custom / non-ideal path.

[Lonkle]

So, @mehdi in another thread showed me the "semi-open source" code for Cloudron. I don't know how I'll be able to test it if it doesn't run if I clone the repo with no changes, but I can start to get some changes in there and see if I can add what I need and then do a merge request. Is that what would work best for you guys?

[Lonkle]

I know I've written an exhausting amount, but thought I'd write a quick and simple question - so since this is easily doable within Docker? Would it not be as easy as Cloudron allowing for the --net=container:vpn-client to be passed to the application it's about to run?

I envision this as a setting in the Cloudron App Settings pages. Saying "Have this app use another Cloudron app's network configuration (we recommend leaving this disabled and only using it if you need to): " and then a dropdown of other sites that have the net_admin Cloudron capability which my custom openvpn-client Docker converted to Cloudron app most certainly will have.

[Lonkle]

UPDATE: Combining Dockers didn't work. Too complex to combine Docker images.

Next to experiment either:

• With the Docker Cloudron "Add-on" which appears I can do what I need to though not ideally within all of Cloudron's app container's (looks like I can build the OpenVPN-Client and then another x Docker and run that under the network already running in the OpenVPN-Client).
Ref: https://docs.cloudron.io/custom-apps/addons/#docker

• Use the net_admin Cloudron "capability" to and see if I can get my OpenVPN-Client Docker installed successfully while using that capability.
Ref: https://docs.cloudron.io/custom-apps/manifest/#capabilities

Can't decide which to try first.

[p44]

@Lonk @girish Yes, Is almost the same scenario, but for my knowledge seems bit complex to apply

[Lonkle]

@p44 That’s what this discussion is about. To make this possible for the average user, possibly even adding it to the store (just like the OpenVPN Server App was created). After we figure out if Cloudron wants to support this at the inter-app level (and if they’re going to mirror Docker arguments since that seems like the easiest solution for them to achieve this) remains in question.

You’re right that as of now it’s too complex for the average user. But my goal is to both make it possible and easy for all users. Just need to know how the Cloudron developers feel before figuring out how to proceed.

If they want to work with me on making this a feature Cloudron can support, I’ll do as much of the legwork as possible and then I’ll ping ya on your thread lettin’ you know what your best solution is depending on the outcome of this thread. ️

[girish]

Still sifting through our 5.6 release bucket list, so I am trying to wrap my head around this a bit.

@Lonk IIUC, the feature requested here is to pipe the network traffic of some apps via a VPN connection. I love the idea! I can see use cases like the ttrss one but also for any of the "pulling" apps like emby metadata download, torrent, searx, scapers etc.

I think this involves two steps (which you already figured). Step 1 is to package the openvpn client app for Cloudron. I think you only need net_admin for this but we can easily add new caps depending on what is needed.

Step 2 is to then have some configuration to let apps use this container as the networking layer. If you ignore the UI/UX, this is really just a one-line change: https://git.cloudron.io/cloudron/box/-/blob/master/src/docker.js#L311 will be changed to the VPN client app's container. If we had a way to identify vpn clients, we can just have a dropdown of vpn clients as the available 'networks' in the app's configuration UI. And that should be it.

(I hope I understand your requirements correctly.)

[Lonkle]

@girish You understand the use case perfectly, I’m genuinely grateful you took the time to read and understand my posts on this thread (I know they were long). ️ I’m actually almost done with Step 1. Using the net_admin capability was key so I should be finished with that tonight. Step 2 would be a slight change in Cloudron’s manifest so as to identify an app as a ‘network client’ and then you could easily have a drop down at the app in Cloudron to list all other Cloudron apps flagged as a “network client” (of which the OpenVPN Client I’m making would be the first) to choose from. The Cloudron app switching it’s network connection would need to be restarted for the change to take effect I believe (but am not on that).

Again, I’m kind of blown away by you guys being so engaging with us (your users) so if there’s anything else I can do aside from finishing building my fork of the VPN Client for Cloudron, let me know!

[girish]

@Lonk Thanks for the encouragement and let us know if you need help finishing up the client part.

Do you know how docker behaves when it comes to routing? There is a docker network attach that lets us attach the container to multiple networks. This is required on Cloudron because the addons like databases are in a network named cloudron. With your app, we will attach the app to the vpn client and cloudron network. I wonder how we can "route" all internet traffic via the vpn client. I can't find any obvious docs for this.

[mehdi]

I think automatically detecting and proposing "network providers" apps in a dropdown would not be very easy... Plus, I think a lot of other apps would benefit from "providing" services to other apps, like a minio instance could provide object storage for other apps.

But this is a big undertaking in my opinion.

In the meantime, simply providing an UI to add custom docker options manually would do the job I think. It would not be as easy to use, but with a little bit of doc it should be doable even for someone who doesn't know docker inside and out. Plus, it would obviously allow a lot of other customizations.

[girish]

I think maybe adding fields to manifest etc to identify network client might be an overkill right now. What I was thinking is just to have an API to set an existing app id as the container's network. This is easy for the backend to do, we have to simply issue docker network attach. For the frontend dropdown, it can simply filter out network providers based on the manifest appstore id for now (hardcoded to openvpn client). Should be fine.

But I think the bigger thing to be figured out is how the routing works.

[Lonkle]

@girish Would docker connect work better in this regard? And also, I agree with the hardcoded dropdown to "openvpn-client" (manifest appstore id).

This is how I do it locally with just Docker:

docker run -it --cap-add=NET_ADMIN --device /dev/net/tun --dns 8.8.8.8 --name openvpn-client -e "CONNECTION_TIMEOUT=-1" -p 3000:3000 lonk/openvpn-client:latest

That starts the OpenVPN Client Docker and connects it to vpn.conf (a hardcoded VPN server). I set the DNS because the VPN can't seem to get the DNS otherwise for some reason, and I set the port because the other container needs that port to be open.

So then I connect my next Docker container to the VPN Client container at runtime:

docker run --name chrome -"CONNECTION_TIMEOUT=-1" --net=container:openvpn-client viridiancloud/chrome:latest

This works perfectly locally, the chrome container runs everything through the VPN if I pass the --net=container:openvpn-client on runtime. I'm trying to get the puppeteer container to use the network of the openvpn-client while it's still running now. Experimenting with docker network connect now on my localhost.

I may need to create a network somehow and assign the OpenVPN Client Docker to that. Looking into it now so we can further understand how to get this working in the cloudron network context (or in addition to it since containers can connect to multiple networks at once).

[girish]

@Lonk said in Better documentation for the API:

Would docker connect work better in this regard

Is this a new command? Atleast, I don't have this in my docker 19.03.12.

$ docker connect
docker: 'connect' is not a docker command.
See 'docker --help'

edit: don't mind me, I guess they are just typos. docker network connect and docker network attach, I guess. I wasn't aware of this connect subcommand.

edit2: there is no, docker network attach I meant docker network connect all the time.