VPN tunnel for apps

[Lonkle]

@robi said in VPN tunnel for apps:

That means Nginx goes into the OpenVPN container, no?
Rules for reverse-proxying are something along the lines of:
if coming from internal IPs, do X
if coming from cloudron IP, do Y
if coming from elsewhere, do Z to the App.

I'm new to nginx, literally only using it's .conf files to try to get this working as a mere proof of concept. But so far I'm only familiar with proxy_pass for it taking incoming 80, 443 ports to pass it to the internal Docker port (and IP...I think but I don't *think this is an IP issue, though it might be).

[Lonkle]

@robi But if I wanted to expose all of it's ports, it would actually be more of a box docker.js thing with the "exposed ports" parameter being fed to gContainer.createContainer. I should be able to do that after I give my mind a lil break.

[Lonkle]

Okay, after hardcode exposing ALL internal ports of the OpenVPN container, now the last thing is adding a second nginx reverse proxy which would mean this:

My exposedPort for the Gucamole app I'm installing to connect to the OpenVPN Client app exposes 8080. It gets a Docker internal httpPort of 32455. Nginx already correctly creates a nginx proxy config from web incoming port 80 and 443 to 32455 (somehow, presumably DNS TXT records, it even gets an SSL cert).

So, I need to have nginx after that continue to proxy it to 8080. Which normally is Docker's job (with it's binding port) but when connecting two container's networks together, it doesn't do that job with the dockerode library for some Docker-y reason. I wonder...can I reverse proxy it in the same file as the original reserve proxy server. Or do I have to create a .conf and an entirely new one?

[Lonkle]

If I have the same nginx reverse proxy "listen" to it's own forwarded port (32455), then I could proxy_pass again to the real internal port and IP (8080) of Guacomole (the test app I'm now using to connect to the vpn client app)...or maybe I need another reverse proxy residing at Internal-IP-of-Guacomole:32455 to then proxy_pass again to 8080.

[robi]

The problem there is that the stuff in the container doesn't have access outside the container, so it's hard to drop updates to the host Nginx as another .conf file.

You could tell box to do it via how it already happens for new apps. Must be an API for it.

[Lonkle]

@robi I keep trying to experiment on how to shoehorn a second Ngnix proxy into this to get it all up and running 100% with no success.

Inter-communication between the two containers would be nice, but tbh I could hardcode that if I knew how to setup the second reverse nginx inside of reverseproxy.js. Which, honestly, I can't figure out how.

[robi]

maybe we need a few more eyes from @nebulon @girish @mehdi for more options

[Lonkle]

I know the answer lies here:

https://git.cloudron.io/cloudron/box/-/blob/master/src/reverseproxy.js#L463

I can configure a second file for a second nginx server to proxy app.httpPort to app.manifest.httpPort - but, like, I don't know how to change that it's hardcoded to listen only on 80 and 443. Plus, I don't know how to actually get this hypothetical second nginx reverse proxy running to "take" the new config within the container that's connecting to the VPN. But that's the last step here on how to make this work.

[mehdi]

@mehdi said in VPN tunnel for apps:

(Disclaimer: I am no expert on docker networking magic)

I think instead of trying to find a way to expose the ports from the OpenVPN container, you should instead try and find a way to make it work directly from the app container itself.

Even if you do make it work for the exposedPort (which, I think, refers to the main web port exposed by the app, the one which is behind the reverse proxy, to expose its web interface and such, that's why there's only one: there's only one web interface), you're gonna have trouble with the extra ports (the ones defined in the manifest here https://docs.cloudron.io/custom-apps/manifest/#tcpports ) for apps that use these.

I really think this. You should try finding a way to make the app container's default route be through the OpenVPN network, but still be connected to cloudron's regular network so it can interact with nginx and stuff. There is no good reason to make all the traffic, even the local one, be through OpenVPN.

@Lonk is there a repo with your box changes ? Maybe I'll have time to take a look this WE

[Lonkle]

@mehdi You only have to change one file (docker.js) so I’ll send that to you today!

But the app has no other issues. It is on the Cloudron network. It has everything network-wise that the OpenVPN container has which means there are no issues anywhere. The only issue is that Nginx doesn’t take that into account because it inherits all network ports and modes from the OVPN Client it’s connected to.

I actually have an idea though to prove that everything works except the web interface using the app terminal.

I will also be sending you my OVPN client in chat so you can see how it works.

[Lonkle]

@mehdi Thanks for your willingness to help me out. You were my inspiration for working on this. Felt right since you had to dive into box code to add the NET_ADMIN capability.

[Lonkle]

@mehdi Oh, and just to make it clear, the app is running fine even in the terminal. If I install the LAMP app for instance and connect it to the OpenVPN Client. I can go to LAMP's terminal and type curl 127.0.0.1:80 and it will give me the LAMP Welcome page, but from the IP address of the OpenVPN Client. It'll just never get past the "Starting..." stage without a 200 response and Nginx is routing it to the wrong location. Maybe I could try hardcoding it to the right port after it installs it's config.

Like, this works. But I don't know how to get NGINX to route it to the port I need it to route to. Docker is supposed to bind the app.httpPort (say 34567) to the exposedPort (say 80). But that Docker binding command has to be removed to connect the containers so NGINX has to bind them manually somehow and I don't know how to work with Nginx.

[Lonkle]

I'm near positive the way to route NGINX properly is to to add another reverse proxy which leads from the Docker randomized app.http port to the exposedPort (AKA app.manifest.httpPort). I'm not sure how though. But I can clearly access the exposedPort's data from either terminal.

In the terminal of either of the two network-bonded apps (the LAMP or the VPN Client) - I can curl (localhost) the exposed ports of either app (the VPN Client or the LAMP) and get back the correct HTML pages.

It's just that the reverse proxy is forwarding it to a port that Docker wasn't allowed to bind. So with no binding, that means no NGINX web page.

[Lonkle]

For people just joining and for me to read this in the future:

Cloudron has a user-defined network cloudron that it uses for all of it's apps and services. I'm connecting my OpenVPN Client app directly to another app (LAMP for testing) using NetworkMode which is the official way to do this. When doing so, both apps share the exact same network space (including both being connected to the cloudron network) and can, in fact, even talk to each other directly (I need to set the OpenVPN Client's exposed port to something ridiculous like in the 50000s so it doesn't conflict with any regular app's commonly exposed ports - but in the end the OpenVPN Client will expose all of it's internal network ports -P for compatibility with all other apps connecting to it - but for now, this will do since I'm just testing).

Now, the NGINX Reverse Proxy resides on the main box level of Cloudron and routes to the randomized Docker binding port of the app which forwards it to it's "real" port (I honestly don't know where the IP translation takes place...I can't find it).

So, solution, place a second NGINX proxy at the randomized Docker app.http port and have that nginx server be inside of the internal network of the LAMP testing app so it can then internally forward the web request to it's exposedPort which is, also, port 80 (for LAMP anyway).

[Lonkle]

^^^^ -- Does anyone know how to do this?

[Lonkle]

@mehdi said in VPN tunnel for apps:

There is no good reason to make all the traffic, even the local one, be through OpenVPN.

All local traffic is local, it doesn't go through the VPN. All of the connected-to-the-vpn-client apps run indefinitely. Eventually after Cloudron says "Starting..." it switches the message to "Not Responding" but the app will continue indefinitely because there's nothing actually wrong with it. It's connecting to all the local services it needs, it's just, this feature was designed without consideration of Cloudron's network. That's why from within the app's terminal, I can access everything locally and everything externally (the open web) from Cloudron and the web sees my VPN Client's IP Address instead of Cloudrons. This is a perfectly running app. Except...I can't expose it's internal IP:ExposedPort to NGINX to have it actually pass the health check (meaning the web app isn't being routed to the right place and I think I need a second NGINX reverse proxy to get me routed to the app's exposed port). It gets as far as app.httpPort and then because there is no binding for the app.mainfest.httpPort (AKA: ExposedPort), it just gets stuck on the web side of things.

I can curl all I want into it the LAMP welcome. But can't access it from outside the container itself.

[Lonkle]

Quick example of all local processes running for the second container:

Oct 02 12:05:28 2020-10-02 16:05:28,718 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Oct 02 12:05:28 2020-10-02 16:05:28,719 INFO success: redis-service entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Oct 02 12:05:40 [GET] /healthcheck

It just gets stuck at the health check because NGINX is giving it a 502 Bad Gateway since port 80 is not bound to port app.httpPort. That's what I still need to figure out and I think what's best is @robi's double nginx server suggestion. I just don't know how to set up NGINX inside of the LAMP container from box code though I'm sure it's possible.

[Lonkle]

I still think the solution lies in this function:

https://git.cloudron.io/cloudron/box/-/blob/master/src/reverseproxy.js#L463

But that function is setting NGINX config listening on the server's public IP port. I need NGINX inside of the container to forward to the app.manifest.httpPort (in LAMP's case it's port 80).

Or maybe it doesn't need to be inside of the container...since if I SSH into the VPS and do curl ip-address-of-lamp:80 then it returns the HTML just fine.

[Lonkle]

SUCCESS!!!

I'm so close. I can manually edit the NGINX configuration of the LAMP-test app and get it's web app working using it's local IP (not 127.0.0.1:app.httpPort, but 172.18.0.3:80 - the port 80 is what LAMP uses). Both web pages come up for their each individual domain names. WOOOO!

It still doesn't make it past the "starting..." step though. That might be because I'm editing the NGINX file and then restarting the app to make it's web interface work. If the NGINX file was properly created from the beginning the check might pass.

[Lonkle]

Which...now I need to find the function to get the IP address of the VPN container (the app connected to it share's it's IP address).

[mehdi]

@Lonk said in VPN tunnel for apps:

@mehdi said in VPN tunnel for apps:

There is no good reason to make all the traffic, even the local one, be through OpenVPN.

All local traffic is local, it doesn't go through the VPN.

Sorry, I meant: There is no good reason to make all the traffic, even the local one, be through the OpenVPN client container.

[mehdi]

What you should try to achieve is making only the app's outgoing traffic go through OpenVPN, and the connexion between it and Nginx and stuff stay exactly the same as it currently is

[Lonkle]

@mehdi said in VPN tunnel for apps:

What you should try to achieve is making only the app's outgoing traffic go through OpenVPN, and the connexion between it and Nginx and stuff stay exactly the same as it currently is

That is currently the case as far as I can tell. All incoming traffic bypasses the VPN entirely. Outgoing traffic uses the VPN's connection. But the NGINX needs an incredibly minor change. There's no way around that.

[Lonkle]

But the bigger deal is that it works now!

Okay, I know it says not responding but it actually is fully working (web page and everything). The "not responding" status is something I think will fix itself when I get the Nginx IP proxy_pass working. Because it's sort of a race condition as to when I start the app and when I manually rewrite the Nginx IP. And I'll never go fast enough to beat the "Not responding message".

[Lonkle]

I just need a function like getAppByManifestId(manifestID) - and this will be a fully functioning proof of concept. And then another function for getLocalIPAddressByApp(app).

Oh, and I need to change some slight NGINX config, make it more dynamic (Cloudron hardcodes it's proxy_pass to 127.0.0.1 rn and it needs to be dynamic).

[Lonkle]

@girish The post above this one, do you know off the top of your head how to get those values?

[girish]

You can't getAppByManifestId since there can be many apps for a given manifest id. I think you have to store whatever you want in the database in the apps table. Then when you get the app, you can get that information (like say the network id). Sorry, if my answer is totally off since I didn't read the full thread completely.

[Lonkle]

@girish I had actually nailed that issue out actually as a matter of fact. The only thing I have left is getLocalIPAddressByApp(app) - but maybe the app object already stores its internal IP (I didn’t think it did, but it stores a lot)?

[Lonkle]

@girish As for the first function, it was mainly due to me needing to find a way to identify the VPN client app by the only unique identifier I knew. I’ve hardcoded it for my use case rn...but that won’t work for the production version now that I think about it - so I’ll still need to get app data using the manifest.id (com.joelstickney.openvpn-client-cloudron). So when the app that needs to connect to it starts, it attaches to the container ID of the Open VPN client on containerCreate.

Maybe right now I’ll store the things I need in ENV variables and find a way to use the “apps” object to parse through all of them till I find the VPN client.

[Lonkle]

@girish I think you’re right about using the app table. I’ll try to RE how you use it to populate the apps variable.

And no worries not following along with this thread. It’s been really difficult for me to get this working so I used this as kind of note-taking. Now that I’m nearly finished, people won’t get as bugged by my posting here every time I discover something new about the way box works.

[Lonkle]

Oh wait, the container object, would contain the IP and the container ID. I could intercept that right after it’s created. 🧐

[Lonkle]

@girish said in VPN tunnel for apps:

getAppByManifestId

I guess the proper function name then would be getAppsByManifestId - the point is to get the possibly multiple VPN Clients installed to populated the drop down box in other app’s configs asking which VPN Client app they would like to share a network with. So this is a needed function and I can parse through the app DB to collect the data.

I found your function to getIDByIPAddressand I’m basically using it as a reference to write the opposite function now getIPAddressById. Would you like me to make this a function or would you like me to just put the code in the NginxAppConfig function the singular time I need to use it? It’s needed to make all of this work.

I need you to know of the singular limitation of the VPN Client app though:

• Multiple apps can connect to a singular VPN Client app unless two apps exposed httpPorts are the same. So you couldn’t connect two of the same app or different apps that use the same web port. I cannot find a way around this except to install a second VPN Client. Which I don’t think is a bad solution given how small my app is. But that will have to be enforced in the drop down box via a new function. I don’t plan on touching dashboard code tho so I’ll leave that up to you. But when you get to that step, tell me and I’ll write you a function to only show you the VPN clients whose connected app.manifest.httpPorts don’t conflict with the app being configured in the dashboard to protect users from port conflicts.

[Lonkle]

@girish (or @nebulon) I'm finished with the OpenVPN Client app, and with the docker.js and reverseproxy.js changes needed to make it work. Right now, it's hardcoded so that if any app being installed with a domain name containing "vpnconnect" it will connect to the OpenVPN Client if there is one (you'd need to change this hardcoded if conditional to be the dropdown box value where you choose which VPN Client to connect on the Cloudron Dashboard that @girish suggested - since you can install as many as you want and thus connect as many as you want to them, a dropdown will be needed).

Everything is working perfectly now. But, my code to get the IP Address and the container name of the OpenVPN Client is...not great.

Ideally, on starting the VPN Client app, it would (if I could have figured it out) add it's IP Address and container name somewhere easy for other apps to retrieve those two pieces of data and connect to the client since those are the only two values needed. But instead I changed the docker.js - createSubcontainer() function and also when writing the Nginx .conf in reverseproxy.js - writeAppNginxConfig() - I parse the DB for those VPNClient values and get them - but it's messy code. Could I send you those files for you to look over? I don't know another way to write the code - but my original goal was to load VPNClientIPAddress and VPNClientContainerName into every app object on initialization if an app exists on Cloudron who's manifest.id incudes "vpnclient". I'm sure that's possible, but I couldn't find where you load the app objects values. I could RE some more and find it, but this works so I'm just gonna use it.

So now that this is working with no issues, where should I go from here to:

1. Make this an official app on the App Store?
2. Work with you guys to make the box changes that are needed to make the app work?

[Lonkle]

Note: I'm still working on a simple web interface to change the .ovpn file rather than having to replace the file in the Cloudron File Manager (the web interface will do it for you). When the file is changed in the web interface, the container / app will restart via a Cloudron API call meaning the user would have to provide their Cloudron token unless there's another way to restart the container from within it (but I plan on just disconnecting and reconnecting to the new file with no container restart - but for an MVP that will work). Right now, you have to change the file in the Cloudron File Manager and restart the app yourself.

[Lonkle]

For anyone interested in the full VPN Client app's code, I pasted it here: https://github.com/lonkle/openvpn-client-cloudron

Still unfinished. Gonna complete 1.0 and then move it to Cloudron's Git.

[Lonkle]

Quick Ideas for the web interface I wanted to jot down:

• Easy access to common VPN providers
• Attaching VPN app to other app's from within the app itself (using the Cloudron API)
• Saving previous connections for later use
• Not having to restart itself (the app) every connection change (there's no avoiding needing to restart an app that's connecting to it though, that's a limitation of docker unless docker network connect --link vpnclientcontainername cloudron container2 would work to mitigate this - it did not in my testing but I know more now, I'll test this again at some point).

[girish]

@Lonk I am not sure changes to the reverse proxy code are required at all. Maybe I didn't get a full grasp of your feature request, so let's take a step back.

• The reverse proxy is only for inbound traffic. It's not used for outbound traffic. It's just a demuxer for incoming http requests to the app containers. My understanding of the feature you suggested was that we want say TTRSS, to use the tunnel for outbound traffic like say when fetching RSS feeds. This means that https://reader.domain.com itself still points to Cloudron but any outbound traffic originating from the container goes via this tunnel.

• You shouldn't filter by app id etc in docker. Instead, we have to first create a networkId entry in the apps table. Then, app.networkId is then simply the network id to be used in docker.js when creating the container. We don't even have to filter by openvpnclient app etc. The filtering will be done by the dashboard code (i.e in the UI).

I can help out a bit more but I want to first double check if we are in sync about this reverse proxy stuff.

[girish]

@Lonk said in VPN tunnel for apps:

@mehdi said in VPN tunnel for apps:

What you should try to achieve is making only the app's outgoing traffic go through OpenVPN, and the connexion between it and Nginx and stuff stay exactly the same as it currently is

That is currently the case as far as I can tell. All incoming traffic bypasses the VPN entirely. Outgoing traffic uses the VPN's connection. But the NGINX needs an incredibly minor change. There's no way around that.

It seems @mehdi nailed it already What is the minor change you require in reverse proxy?

[Lonkle]

@girish Well, @mehdi is correct that that should be how it works. But it has always worked this way.

The problem was never trying to run incoming traffic through the VPN. The problem lies in that if you connect two app's NetworkMode’s together, they have to share the same IP and it removes all network binds to the secondary app (so your healthcheck and nginx routing can't do anything with the info it usually uses). So, the reverse proxy change detects if an app is connected to a VPN Client and if it is, it uses the Docker actual / internal IP and the secondary app’s actual exposed port to route the incoming traffic to.

It can’t use localhost:randomizedport (127.0.0.1:56343) unless we used a second nginx proxy to forward its randomized Docker port app.httpPort to its real IP and real exposed port. Docker, in normal circumstances, binds these things for all apps unless their “NetworkMode” is another container - which it is in this case, to make the VPN function.

Did I properly explain that / does it make sense?

Note: I'm positive there's a very complicated solution to this that I mapped out and is too ridiculous to implement. But there is a caveat, which is that multiple apps using the same exact exposedPort cannot co-exist on the "same network" / connected-to-the-vpn-container. But that's kind of the point of the OpenVPN Client sharing it's network connection, sharing the IP. So, we could restrict one app to connect to one VPN rn to remove "Error: Port Collision" user errors - because you can always install more than One VPN client and connect it to the same VPN. Less work. Don't really see a use case for a a lot of the "same exposed port" apps being used with the same vpn client, but we can let the community decide on that later?

[Lonkle]

Also, of note, I use some ridiculous exposed port (29875 or something like that) for the VPN app's web interface itself (it's exposedPort) so it'll never conflict with an actual app on the store that tend to use 80 or 8080.

[Lonkle]

@girish said in VPN tunnel for apps:

You shouldn't filter by app id etc in docker. Instead, we have to first create a networkId entry in the apps table. Then, app.networkId is then simply the network id to be used in docker.js when creating the container. We don't even have to filter by openvpnclient app etc. The filtering will be done by the dashboard code (i.e in the UI).

100% agreed. I'm only doing that now as a proof of concept because, tbh, I'm not sure where to put this option in the dashboard code (plus dashboard code is harder to work with). So, my hack to filter the other apps to crab the containerId of the OpenVPN Client (and using a hardcoded app.fqdn conditional to connect to it), I'll use for now until you have time to get together and talk about how to implement this on the front end.

[Lonkle]

TODO: Last two things.
• I WILL: Add backend interface to allow a logged in user to submit any .ovpn file via a file chooser and convert it to what Cloudron needs after submitting the new .ovpn.

• @girish MIGHT HAVE TIME TO DO: Modify Dashboard and appdb.js to add the vpnclient's vpnContainerID and internalIP to use in NetworkMode for any apps choosing to route only their outgoing traffic through the VPN (presumably a drop-down box in the config option of an app which will need to be coded in the dashboard, codes to the ovpn-client's manifest ID - which allows for getting the containerID and internalIP of the vpnclient app).

DONE: Add an HTTP API endpoint to change and convert any .ovpn file a user supplies via POST into the container and disconnect from current to connect to the newly added .ovpn

DONE: Add a front end web interface login page at the base URL of the app which allows for logging in to the app via Cloudron's LDAP

[Lonkle]

LDAP integration was surprisingly easy to add after I realized cloudron update doesn't expose new add-ons added to the manifest and I had to cloudron uninstall and then cloudron install.

[Lonkle]

To anyone who really needs this right now. I just finished with the beta. I've tested it, but there is one manual step needed (you have to copy and paste the "containerName" of your app into a file in the OpenVPN Client File Manager) until @girish has time after 6.0 to add some Dashboard polish to the whole thing in which case the only thing you need to do is add an .ovpn file to it's backend, which you simply login with the same credentials you login with Cloudron (since it has LDAP integration) and it'll do the rest. The @girish Dashboard code I think he's adding is a dropdown box to all installed OpenVPN client app's to choose which one you want to connect to (right now, like I mentioned, you have to manually add that to the file inside of the VPN Client's filesystem which isn't very intuitive)".

[Lonkle]

Oh, and it may require like 4 or 5 patches to box code. Forgot about that. It's fully functional tho, a little unpolished, a lot unpolished. But everything works. @girish and I will work together to integrate it properly at some point after 6.0. My patches run at "start" time, so the fact they're inefficient isn't too big of a deal, but just know that somewhere down the line, @girish and I will add it properly into a stable version of Cloudron.

[Lonkle]

@lonk said in VPN tunnel for apps:

Oh, and it may require like 4 or 5 patches to box code. Forgot about that. It's fully functional tho, a little unpolished, a lot unpolished. But everything works. @girish and I will work together to integrate it properly at some point after 6.0. My patches run at "start" time, so the fact they're inefficient isn't too big of a deal, but just know that somewhere down the line, @girish and I will add it properly into a stable version of Cloudron.

What an accomplishment this was for me back then. I like that my first post in the forums is this crazy hellscape of Cloudron and Docker development jargon. I also wonder if this will ever help anyone down the road. Either way, I'm glad this whole thing is archived, it's p nostalgic for me. ️