VPN tunnel for apps

[Lonkle]

@girish said in VPN tunnel for apps:

Step 2 is to then have some configuration to let apps use this container as the networking layer. If you ignore the UI/UX, this is really just a one-line change: https://git.cloudron.io/cloudron/box/-/blob/master/src/docker.js#L311 will be changed to the VPN client app's container. If we had a way to identify vpn clients, we can just have a dropdown of vpn clients as the available 'networks' in the app's configuration UI. And that should be it.

https://github.com/lonkle/openvpn-client-cloudron

I am finished with the MVP, it'll be ready for the App Store as soon as I include instructions on install to show the user installing it how to get their .ovpn in the /app/data folder for the OpenVPN Client to pick it up and some other small polishes (like using a wildcard to pick up whatever .ovpn file they put into /app/data and adding the necessary one-liner automatically to the file for it to be compatible with Cloudron's network. But, as of now, every time you change OVPN servers, you have to restart the app. I'm going to use the Cloudron API to do this manually but I haven't found a way to not restart the app yet and just disconnect + connect to a different server on-demand, just at initial runtime.

Oh, and when you need an .ovpn for testing container network connections, let me know! But I'm gonna take a stab (no promises, nervous at how different the Cloudron base code is gonna be ) at getting this working myself on the Cloudron side. Thanks for pointing me in the right direction!

[Lonkle]

Okay, I'm close to finalizing the Cloudron side of things. But I can't seem to find a function like

getContainerIdByAppId('com.joelstickney.openvpn-client')

@girish Do you know any functions off-hand or maybe some files that I should be lookin’ into? I need the Open VPN client's (sub)Container ID so I can apply it to the other (sub)Containers network(s).

[Lonkle]

I wish I had a better deployment system for cloudron:box changes, this is much harder than regular app development. But that's probably a testament to how well Cloudron's app "subcontainer" packaging system is built.

[Lonkle]

I did find a getAll(callback) and even a more specific getByFqdn(fqdn, callback) lines I may be able to use to get the container ID of my Open VPN Client.

[Lonkle]

I'm now able to connect any container to any network I want on app startup based on the app's FQDN or ID. But even if I create a separate network than cloudron and stick the OpenVPN Client in there with another app. The other app doesn't inherit the OpenVPN Client's IP like it does when you connect the connectors directly on the bridge network using the docker run --net command.

There may be a firewall issue though. Cloudron does have a very specific subnet and maybe I can't escape it without running into a firewall.

@girish @nebulon - any ideas for me on this one?

[girish]

Oh wow, you are moving way faster than we can take a look at this. Great progress. Please give us a day or two to look into this! mon/tue is always a bit slower given all the app updates, support requests etc that pile up over the weekend

[Lonkle]

@girish No problem at all. I can work on porting some other apps in the meantime. Really love the platform you guys created.

And I might be able to figure the answer to my own question. I made an Ubuntu Graphical 18.04 Virtual Machine and installed Cloudron on that so I can develop for the “above subcontainer level” (AKA: box) app code. I am tho running into the two app restriction on localhost - I wish that limitation didn’t apply to custom apps developers are making.

Anyway, thank you again for being as responsive as you have been and don’t mind me updating as I go. Respond whenever you think you can. ️ You guys have done some really amazing work; both of you are inspirations tbh.

[girish]

@Lonk If you are working on app, please do leave a note in the corresponding app's wishlist topic. Just to make sure all our work does not overlap. Thanks.

[girish]

BTW, does the vpn client have some sort of web ui? Would be great to have something where a user can just login and provide connection information. We can look into the openvpn server app for inspiration.

[Lonkle]

@girish
Goals for v0.4.0:
• Modify docker.js to use the container:vpnContainerID NetworkMode for all apps choosing to run their traffic through the VPN (presumably a drop-down box in the config option of an app which will need to be coded in the dashboard, hardcoded to the ovpn-client's manifest ID). Note: Need to remove hostname and any port bindings so that the container takes on the attributes of the VPN otherwise the NetworkMode will conflict with the VPN Client's Network.

Goals for v0.5.0:
• Add an HTTP API endpoint to change and convert any .ovpn file a user supplies via POST into the container and disconnect from current to connect to the newly added .ovpn

Goals for v1.0.0 (I'd consider it ready to upload to the official Cloudron Appstore at this point):
• Add a full web interface at the base URL of the app which uses the API endpoint in v0.5 to allow a user to submit any .ovpn file and convert it to what Cloudron needs to straight from a web interface (so they don't have to use the API or manually add it in the terminal / file manager, which would be confusing to new users using Cloudron).

[Lonkle]

@girish And I will post on each app's wishlist forum post when I start work on converting.

I'm just really stuck on this NetworkMode code. No matter what I do, the NetworkMode stays cloudron even after restarting the container and hardcoding it to use the NetworkMode container:vpncontainerid. Effectively making the app useless. The NetworkMode needs to change, but it's like once the NetworkMode is attached to an app, it stays on there no matter what. Still experimenting as that's the last piece of this puzzles aside from finding a function that can take a cloudron-manifest-id and translate it to a Docker Container ID (so I don't have to docker network inspect and then hardcode it's ID).

[mehdi]

@Lonk Are you sure you are re-building the app container, not just re-starting it ?

[Lonkle]

@mehdi My OpenVPN Client doesn't require any more edits to work on Cloudron. Or are you referring to the "other container" that I'm trying to attach to my OpenVPN Client (to then change it's public IP which I just check with curl icanhazip.com every new experiment)? So all apps / cloudron containers I just restart from the Cloudron dashboard since they are the one's who's NetworkMode needs to be container:vpn-docker-constainer-id-or-name-but-really-just-id (it's hard to parse containters and networks because docker.js has to sent the app names to their container ids to prevent container naming conflicts) - which is a box problem that needs to be solved (particularly docker.js).

I see a lot of mentions of appContainer references and subcontainer references in REing the docker.js code. But they're all just regular containers (nothing is special about them except a flag is set if it's.a subcontainer) so that must be a Cloudron nomenclature.

I think (really just hope) I found my issue:
function containersCreate(req, res, next) { safe.set(req.body, 'HostConfig.NetworkMode', 'cloudron'); // overwrite the network the container lives in

Now why are you doing that to me @girish?

[Lonkle]

THAT WAS IT! Well, there's more to box code changes than just proving the vpn client connection works. But @girish was overriding the network name in dockerproxy.js which took an entire day to figure out! but really

Alright, @mehdi, I'm glad you pushed me enter into box code realms. I'm still intimidated by it. But I've got (sub)container creation down pretty well now.

[Lonkle]

This would be faster using Cloudron on localhost in my VM as I installed it on yesterday to do box code. But, the two app limit thing was too hard to deal with.

[Lonkle]

One of the worst parts of box code is that all these containers (aside from contained cloudron services and cloudron itself) have hshes for names. So, when trying to connect one to another and then inspecting the attributes of all of them is so confusing.

[Lonkle]

To get this app store ready, it might even need modifications to the nginx reverse proxy. @mehdi Maybe you can help? Is there a way, in cloudron to open more than one port? The manifest seems to be a way to publish a random port to forward to the specified-in-the-manifest port

Example:

Using docker container ls --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}" -a - I can look at all the "published" ports. They all seems to be going through a nginx reverse proxy to reach port 80 / 443. And their original port that forwards to that is random (?).

NAMES ----------------------------------- PORTS
0b993ea5-85fc-4465-af5f-a1cd5ea5aeb7 127.0.0.1:43641->8080/tcp
57c94244-112d-4b98-abc4-0f15d6b07ca7 127.0.0.1:40875->8000/tcp
10a9ae64-689e-4765-a990-89a3d5e400d2 127.0.0.1:44749->80/tcp

[Lonkle]

Maybe if I hardcode the bind address to 0.0.0.0. Cloudron hardcodes it to localhost (likely for security purposes).

[Lonkle]

Sorry about the spam, I just started learning Docker two weeks ago now when I started building this app, so it's been all very foreign to me. Had to learn so many weird networking things.

[Lonkle]

I'm having a reverse proxy battle. Everything works now, but the "connected-to-the-vpn-client" app can't access the outside world. Cloudron doesn't publish ports in the standard way, there's some proxy between it all, and I have to RE that to get the apps connected to the openvpn-client to use the proxy somehow. It should just work by default looking at the code, but - there's always something.