VPN tunnel for apps

Lonkle

So, the "Cloudron CLI" has a bunch of important functions that the REST API doesn't, yet the REST APIs documentation says the CLI uses the REST API. Are there just undocumented REST API functions that the Cloudron CLI client has?

My main goal is that I have a Docker container (https://github.com/dperson/openvpn-client) that runs external to Cloudron (named "vpn") on the VPS (though I could build it into an actual Cloudron app if needed). And I need to run my Cloudron apps on demand through that container using the --net=container:vpn argument when using Docker Run.

The CLI will allow me to update the VPN as needed if I turn it into a Cloudron app, but I haven't found a way within Cloudron CLI or it's REST API to add that argument --net=container:vpn before it starts an Dockerized app.

Does anyone have any ideas?

mehdi

@Lonk I think there is currently no function in the API to do what you need

I saw you post on the other thread, asking about the docker addon, and there might be a way to use it to manually start apps the way you want, but it would demand more docker trickery than I have to figure out how ^^

However, I think the cloudron team said somewhere that they were thinking about how to allow the apps to communicate directly together, so I think what you are trying to achieve may integrate with their feature roadmap (even though, I must admit I am not very clear on the details of what you're trying to do ^^)

nebulon

Generally it is not advised to install anything on the side on the same server. Neither through docker nor otherwise. The main reason is that we cannot test such configurations and setups and thus it will eventually break during an update. The only way to have it working reliably would be through making a Cloudron app package, as you already mentioned as an option.

I may not fully understand your vpn usage there, however please note, that apps on Cloudron already run within their own local network.

Regarding the CLI tool and the API, this discrepancy is only there since we haven't yet documented that whole API. But the cli as well as the actual dashboard are fully built on top of the REST API. So far the usage of the API directly simply wasn't asked for a lot, thus the lack of wholesome docs

girish

To take a guess : The use case is that you run a OpenVPN client container which connects to some VPN service (not the existing app which is OpenVPN server). Then you make some or all apps use the same network as this openvpn container. This way all traffic goes via the VPN. Pretty cool, if this is what @Lonk had in mind

I think recently @p44 asked here how to hide TTRSS's querying IP. This is possibly an answer to that question.

I wonder if this should really be done at the server level instead of per-app. Maybe @Lonk can explain his use case further.

Lonkle

@nebulon Thank you so much for the reason for the discrepancy. I was hoping that was it. I can just use Charles to RE your web app and grab the calls for myself since you already have a documented way to authenticate with the API.

@girish That is exactly correct! I've actually forked https://github.com/dperson/openvpn-client to allow for choosing your own .ovpn (instead of it being static, I have 10 for example) in one Docker container which I can pass an argument to at runtime -e "" to connect to x VPN service or router. It works really well locally via Docker, even multiple Docker containers can "mirror" it's network via -net=container:vpn-client and they all can the VPN network simultaneously or one by one (which is what I was hoping to accomplish via Cloudron).

But thanks to @nebulon's description of the Cloudron REST API. Despite these calls being undocumented, I can just stop the VPN container when not in use, and start it when it's needed (I'm going to recode my fork to pull from a static URL for the .ovpn file it uses and change that externally).

But that still doesn't solve the main problem of being able to have Cloudron app's use the same network config AKA Docker's -net=container:app-name equivalent as the VPN Client Cloudron app.

So I may be out of luck and have to code a vpn-client on the network context that is also running cloudron, and globally change networks instead of per app like the Docker container made possible. Unless you guys were interested in this specific use case. I'd be more than happy to complete my https://github.com/dperson/openvpn-client fork to create as an actual Cloudron add-on along with my other enhancements (choose your own VPN to connect). If @mehdi was saying is true and you wanted Cloudron apps to be able to connect to each other down the road, this would be a great first use case. I'll do as much of the work as you'll let me. I really like Cloudron, so I'd be more than happy to plus I'd get the benefit of running on the VPN's network per cloudron app instead of running it at the root level (which, like @nebulon mentioned, may break things down the road with Cloudron itself).

@mehdi I installed the Docker Custom Build Service in my Cloudron but this was a problem that could only be tackled from outside a Docker container.

Lonkle

@girish Also, I read @p44's thread and yes, this would resolve his request asl well.

Lonkle

@mehdi There’s scarce documentation on the Docker Cloudron add-on as well as the net_admin Cloudron capability. Between the two of those I should be able to get this working...maybe. The documentation is p spotty so I can’t know for sure until I do a lot of experimentation. @nebulon did confirm that I can start and stop Cloudron apps using private API calls so I no longer need to worry about the OpenVPN client Docker app (that I’m turning into a Cloudron app) switching VPNs.

I just have one last obstacle and that’s making one Cloudron container using the network of another Cloudron container. This is very easy using Docker. One or many apps can use the same network as the OpenVPN-Client. You just need to stop and start them passing the argument of “use the OpenVPN-Client‘S network.

Cloudron is built heavily around Docker principles so maybe, just maybe the net_admin capability will work for that. Otherwise, the Docker add-on should work despite it forcing me to use pure Docker instead of Cloudron apps to connect the network to the VPN Client Docker app.

Last option is to side-install both the OpenVPN Client Docker and another Docker to connect to its network but Cloudron administrators say that’s not a stable way to accomplish this and will probably break during an update.

Lonkle

My final option that @girish hinted in his last post (“should this be done an the app level”) would work is building my own custom Cloudron app, combining a Wordpress Docker with the OpenVPN Client Docker and use the trick @nebulon implied would work (restarting the custom Cloudron app I make from within the Cloudron Private API itself to switch VPNs so I don’t have to touch any real code in the OpenVPN Client Docker).

But in doing so, I have to work on combining two images that weren’t designed to be combined, and I lose Wordpress updates from the Cloudron App Store.

Is this the direction you guys think I should go in for this if I want this done? Or can you think of a way of getting one Cloudron app use another Cloudron app’s network in the same way Docker allows.

This project I want to finish in the next couple weeks and if this is the only way possible, I’ll get to it. But I wanted the staffs advice / input before I start this very custom / non-ideal path.

Lonkle

So, @mehdi in another thread showed me the "semi-open source" code for Cloudron. I don't know how I'll be able to test it if it doesn't run if I clone the repo with no changes, but I can start to get some changes in there and see if I can add what I need and then do a merge request. Is that what would work best for you guys?

Lonkle

I know I've written an exhausting amount, but thought I'd write a quick and simple question - so since this is easily doable within Docker? Would it not be as easy as Cloudron allowing for the --net=container:vpn-client to be passed to the application it's about to run?

I envision this as a setting in the Cloudron App Settings pages. Saying "Have this app use another Cloudron app's network configuration (we recommend leaving this disabled and only using it if you need to): " and then a dropdown of other sites that have the net_admin Cloudron capability which my custom openvpn-client Docker converted to Cloudron app most certainly will have.

Lonkle

UPDATE: Combining Dockers didn't work. Too complex to combine Docker images.

Next to experiment either:

• With the Docker Cloudron "Add-on" which appears I can do what I need to though not ideally within all of Cloudron's app container's (looks like I can build the OpenVPN-Client and then another x Docker and run that under the network already running in the OpenVPN-Client).
Ref: https://docs.cloudron.io/custom-apps/addons/#docker

• Use the net_admin Cloudron "capability" to and see if I can get my OpenVPN-Client Docker installed successfully while using that capability.
Ref: https://docs.cloudron.io/custom-apps/manifest/#capabilities

Can't decide which to try first.

p44

@Lonk @girish Yes, Is almost the same scenario, but for my knowledge seems bit complex to apply

Lonkle

@p44 That’s what this discussion is about. To make this possible for the average user, possibly even adding it to the store (just like the OpenVPN Server App was created). After we figure out if Cloudron wants to support this at the inter-app level (and if they’re going to mirror Docker arguments since that seems like the easiest solution for them to achieve this) remains in question.

You’re right that as of now it’s too complex for the average user. But my goal is to both make it possible and easy for all users. Just need to know how the Cloudron developers feel before figuring out how to proceed.

If they want to work with me on making this a feature Cloudron can support, I’ll do as much of the legwork as possible and then I’ll ping ya on your thread lettin’ you know what your best solution is depending on the outcome of this thread. ️

girish

Still sifting through our 5.6 release bucket list, so I am trying to wrap my head around this a bit.

@Lonk IIUC, the feature requested here is to pipe the network traffic of some apps via a VPN connection. I love the idea! I can see use cases like the ttrss one but also for any of the "pulling" apps like emby metadata download, torrent, searx, scapers etc.

I think this involves two steps (which you already figured). Step 1 is to package the openvpn client app for Cloudron. I think you only need net_admin for this but we can easily add new caps depending on what is needed.

Step 2 is to then have some configuration to let apps use this container as the networking layer. If you ignore the UI/UX, this is really just a one-line change: https://git.cloudron.io/cloudron/box/-/blob/master/src/docker.js#L311 will be changed to the VPN client app's container. If we had a way to identify vpn clients, we can just have a dropdown of vpn clients as the available 'networks' in the app's configuration UI. And that should be it.

(I hope I understand your requirements correctly.)

Lonkle

@girish You understand the use case perfectly, I’m genuinely grateful you took the time to read and understand my posts on this thread (I know they were long). ️ I’m actually almost done with Step 1. Using the net_admin capability was key so I should be finished with that tonight. Step 2 would be a slight change in Cloudron’s manifest so as to identify an app as a ‘network client’ and then you could easily have a drop down at the app in Cloudron to list all other Cloudron apps flagged as a “network client” (of which the OpenVPN Client I’m making would be the first) to choose from. The Cloudron app switching it’s network connection would need to be restarted for the change to take effect I believe (but am not on that).

Again, I’m kind of blown away by you guys being so engaging with us (your users) so if there’s anything else I can do aside from finishing building my fork of the VPN Client for Cloudron, let me know!

girish

@Lonk Thanks for the encouragement and let us know if you need help finishing up the client part.

Do you know how docker behaves when it comes to routing? There is a docker network attach that lets us attach the container to multiple networks. This is required on Cloudron because the addons like databases are in a network named cloudron. With your app, we will attach the app to the vpn client and cloudron network. I wonder how we can "route" all internet traffic via the vpn client. I can't find any obvious docs for this.

mehdi

I think automatically detecting and proposing "network providers" apps in a dropdown would not be very easy... Plus, I think a lot of other apps would benefit from "providing" services to other apps, like a minio instance could provide object storage for other apps.

But this is a big undertaking in my opinion.

In the meantime, simply providing an UI to add custom docker options manually would do the job I think. It would not be as easy to use, but with a little bit of doc it should be doable even for someone who doesn't know docker inside and out. Plus, it would obviously allow a lot of other customizations.

girish

I think maybe adding fields to manifest etc to identify network client might be an overkill right now. What I was thinking is just to have an API to set an existing app id as the container's network. This is easy for the backend to do, we have to simply issue docker network attach. For the frontend dropdown, it can simply filter out network providers based on the manifest appstore id for now (hardcoded to openvpn client). Should be fine.

But I think the bigger thing to be figured out is how the routing works.

Lonkle

@girish Would docker connect work better in this regard? And also, I agree with the hardcoded dropdown to "openvpn-client" (manifest appstore id).

This is how I do it locally with just Docker:

docker run -it --cap-add=NET_ADMIN --device /dev/net/tun --dns 8.8.8.8 --name openvpn-client -e "CONNECTION_TIMEOUT=-1" -p 3000:3000 lonk/openvpn-client:latest

That starts the OpenVPN Client Docker and connects it to vpn.conf (a hardcoded VPN server). I set the DNS because the VPN can't seem to get the DNS otherwise for some reason, and I set the port because the other container needs that port to be open.

So then I connect my next Docker container to the VPN Client container at runtime:

docker run --name chrome -"CONNECTION_TIMEOUT=-1" --net=container:openvpn-client viridiancloud/chrome:latest

This works perfectly locally, the chrome container runs everything through the VPN if I pass the --net=container:openvpn-client on runtime. I'm trying to get the puppeteer container to use the network of the openvpn-client while it's still running now. Experimenting with docker network connect now on my localhost.

I may need to create a network somehow and assign the OpenVPN Client Docker to that. Looking into it now so we can further understand how to get this working in the cloudron network context (or in addition to it since containers can connect to multiple networks at once).

girish

@Lonk said in Better documentation for the API:

Would docker connect work better in this regard

Is this a new command? Atleast, I don't have this in my docker 19.03.12.

$ docker connect
docker: 'connect' is not a docker command.
See 'docker --help'

edit: don't mind me, I guess they are just typos. docker network connect and docker network attach, I guess. I wasn't aware of this connect subcommand.

edit2: there is no, docker network attach I meant docker network connect all the time.

Lonkle

@girish said in VPN tunnel for apps:

there is no, docker network attach I meant docker network connect all the time.

Okay, that makes a lot of sense, when you mentioned the "network" attach command, I figured I had some reading up to do.

I edited my post to fix my typo and remove mentions of "docker attach".

Lonkle

@girish I have finished converting the app into a Cloudron app and have pushed it to the Docker Hub, there appears to be one issue with these two lines in the start.sh file:

mkdir -p /dev/net
[[ -c /dev/net/tun ]] || mknod -m 0666 /dev/net/tun c 10 200

There seems to be an issue with this part of the starting script. Likely related to this docker run argument that I run locally --device /dev/net/tun. This is the exact error I get when attempting to run it on Cloudron (despite it successfully running locally with strictly Docker).

ERROR: Cannot ioctl TUNSETIFF tun: Operation not permitted (errno=1)

What do you think on the Cloudron side of things there?

Lonkle

@Lonk Note: I submitted the DockerHub link for you here (https://www.cloudron.io/app-developer.html) to pull and test yourself locally via docker run -it --cap-add=NET_ADMIN --device /dev/net/tun --dns 8.8.8.8 image/name (which will succeed, just try curl icanhazip.com in the Docker container and it will see a different IP address). Then check the logs on cloudron when trying to install to it and you'll see the issues with /net/tun that don't exist locally.

girish

@Lonk Have you pushed the package source code somewhere?

girish

For the tun device stuff, @mehdi solved it by creating the device in Dockerfile - https://git.cloudron.io/cloudron/openvpn-app/-/blob/master/Dockerfile#L30 (RUN mknod /app/code/net-tun c 10 200) and then using dev-node /app/code/net-tun option in openvpn.conf

mehdi

It's quite a long time ago now that I wrote the openvpn-server app, but I remember that this ton device stuff was hell (Plus, Cloudron did not have support for capabilities like NET_ADMIN back then, so I had to implement that in the server itself^^)

@Lonk in your command line, you forward the host's /dev/net/tun device to the container, which cloudron does not do, so the problem probably comes from there.

If @girish's suggestion of using the same trick as the openvpn-server app does not work, I'd be happy to take a look.

Lonkle

I worked on this for a few more hours tonight and after editing the starting script and DOCKERFILE (and a dumb mess up with CloudronManifest.json), I now have it 100% working within Cloudron.

Thank you so much for your help @girish / @mehdi!

There's a bunch of polish now that needs to be made. Proper healthcheck since this doesn't have a web admin exposed yet. I'll be making a custom web admin to change out VPN files (.ovpn) and credentials to make this a proper web app for Cloudron. For now, I'm just going to fake the healthcheck in testing just to see if we can connect containers using @girish Cloudron's code edit idea to make another container use my OpenVPN Client's Container's Network (akin to docker run's --net=container:vpn-client command).

PS. @girish, I haven't published the code on Github yet because I've hardcoded my home's VPN credentials in there since this is just a proof of concept. I gave you guy's the DockerHub link through this form: https://www.cloudron.io/app-developer.html - so that only you guys would know the link to pull it to Cloudron. Once I make the VPN configurable, I'll be releasing the source. I'll fix the health check in the morning with v0.3. Until then, feel free to pull the image to Cloudron to install for your own testing purposes (the idea you had to make it connect to other Cloudron apps). If you don't have time to, or are unable, I can take a stab at editing the Cloudron code directly. Let me know! ️

Lonkle

Alright, v0.3 release notes:

• Uses cloudron:2.0 base image now.
• Added apache to start the process of configuring a web interface (it'll need to do a few things like add dev-node /app/code/net-tun to the openvpn-server.ovpn files and convert them into vpn.conf files (which is stored in app/data so it's persistent and can change at anytime through an API endpoint I'm going to write...for now, you can change the file yourself and add it in the Cloudron File Manager and then restart the app).
• Fixed the healthcheck so that Cloudron now runs it as a full fledged app

This is at the point where I'm ready to upload it to Github tonight and I'll post the link. I just need to revoke my personal OVPN Server certificate first before I do that and I have some more testing when it comes to the web interface and modifying files.

@girish Do you have any questions from me in order to make your side of things (Cloudron's base code) connect to this container's network to other apps on Cloudron? Were you able to find and pull my testing image from the DockerHub through the form I submitted - which I did get a response, as I'm now in the contributor program and I'm happy to be because developing with Cloudron is just getting more and more fun. I have some ideas for Wordpress which is where my real expertise is. I'll finish this project and then move onto that after you guys release 6.0 (since I know it's coming with Wordpress changes).

girish

@Lonk Thanks! It might be easier for me to test with the package source. On Cloudron, you will need the manifest file to install and not just the docker image anyway.

I am happy to look into the Cloudron side of things whenever the package is semi-ready. Thanks for all the work!

Lonkle

@girish My DockerHub link you can pull to Cloudron and install comes with the Cloudron manifest So, rn, it’s working perfectly as an app in Cloudron now, fully operational (just doesn’t have a web interface to change the OVPN file yet but contains everything you need to test it, directly connects to my homes VPN until I revoke the cert which I didn’t want to do until you were done testing). I’ll post the source on GitLab today minus the hardcoded .ovpn file. But can I send you that specific file personally so you can use it for testing. You just place it in the /app/data folder and restart the OpenVPN Client app and your IP for that container will magically change. So you can then test connecting another app to it network-wise (a la a cloudron eqivalent to docker run's --net=container:vpn-client runtime argument) and then curl icanhazip.com the “connected-to-the-vpn-client” app to see if it gets the same IP address as the OpenVPN Client Cloudron app.

Does all of that make sense?

Lonkle

@girish I removed the pre-configured .ovpn file that is in the DockerHub version of 0.3 and added the full source code to Github. I tried adding it to Cloudron's Gitlab but it didn't let me create a repository for some reason so I just used Github.

https://github.com/lonkle/openvpn-client-cloudron

If you need my preconfigured .ovpn file for testing instead of creating your own (you can add your converted .ovpn file and rename it to vpn.conf to the root directory before you install or after the install with the Cloudron File Manager, but if you do that you have to restart the app for it to take effect), then let me know so I can send it to you privately. I'll revoke my pre-configured one when you're done testing if you want to go that route (pun intended ).

Lonkle

Goals for v0.5.0:

• Add an HTTP API endpoint to change and convert any .ovpn file a user supplies via POST into the container and disconnect from current to connect to the newly added .ovpn

Goals for v1.0.0 (I'd consider it ready to upload to the official Cloudron Appstore at this point):

• Add a full web interface at the base URL of the app which uses the API endpoint in v0.5 to allow a user to submit any .ovpn file and convert it to what Cloudron needs to straight from a web interface (so they don't have to use the API or manually add it in the terminal / file manager, which would be confusing to new users using Cloudron).

@girish Neither of these final features will be needed for you to do your testing whenever you'd like to connect one or multiple containers networks to the the "OpenVPN Client v0.3" app's network. So let me know a good way to send you my preconfigured vpn.conf file for you to use to test the feature. In the end, the API backend will automatically convert any .ovpn files supplied by a user via the API or web interface into a format Cloudron needs; mine is just pre-configured for ya so you don't have to make your own just to test networking between the containers.

Lonkle

@p44 I built the OpenVPN Client you can use to change your IP for your app. @girish offered to add a custom feature to Cloudron specifically to allow Cloudron apps to network connect to my OpenVPN Client app. So, not much longer now! I might try to dive into the network code myself but I'll admit, Cloudron's base code intimidates me.

mehdi

@Lonk Give it a try, it's not that hard ! I had to do it for the OpenVPN server, it's only fair that you do it for the OpenVPN client

Lonkle

@mehdi Alright alright, you've convinced me - Is the flow similar to app creation to build it locally? Won't I need another license to start working on Cloudron locally though? So, the cloud can stay up, and I can develop this multiple app network VPN connection code locally? Normally, I could just use the free license for local dev, but I literally need three apps to test this because coincidentally this modification needs a total of 3 simultaneously running apps to test, the Open VPN Client App, another app to connect to it's network (so one to one connection is tested and confirmed), and then another app to simultaneously (3 apps needed to be installed at once to test) connect to the same Open VPN Client App's network.

I also just read on Gitlap that some of Cloudron is closed source, hmm, I should focus my feature requests on the closed source aspects I literally can't change myself and then merge request it.

Lonkle

@girish said in VPN tunnel for apps:

Step 2 is to then have some configuration to let apps use this container as the networking layer. If you ignore the UI/UX, this is really just a one-line change: https://git.cloudron.io/cloudron/box/-/blob/master/src/docker.js#L311 will be changed to the VPN client app's container. If we had a way to identify vpn clients, we can just have a dropdown of vpn clients as the available 'networks' in the app's configuration UI. And that should be it.

https://github.com/lonkle/openvpn-client-cloudron

I am finished with the MVP, it'll be ready for the App Store as soon as I include instructions on install to show the user installing it how to get their .ovpn in the /app/data folder for the OpenVPN Client to pick it up and some other small polishes (like using a wildcard to pick up whatever .ovpn file they put into /app/data and adding the necessary one-liner automatically to the file for it to be compatible with Cloudron's network. But, as of now, every time you change OVPN servers, you have to restart the app. I'm going to use the Cloudron API to do this manually but I haven't found a way to not restart the app yet and just disconnect + connect to a different server on-demand, just at initial runtime.

Oh, and when you need an .ovpn for testing container network connections, let me know! But I'm gonna take a stab (no promises, nervous at how different the Cloudron base code is gonna be ) at getting this working myself on the Cloudron side. Thanks for pointing me in the right direction!

Lonkle

Okay, I'm close to finalizing the Cloudron side of things. But I can't seem to find a function like

getContainerIdByAppId('com.joelstickney.openvpn-client')

@girish Do you know any functions off-hand or maybe some files that I should be lookin’ into? I need the Open VPN client's (sub)Container ID so I can apply it to the other (sub)Containers network(s).

Lonkle

I wish I had a better deployment system for cloudron:box changes, this is much harder than regular app development. But that's probably a testament to how well Cloudron's app "subcontainer" packaging system is built.

Lonkle

I did find a getAll(callback) and even a more specific getByFqdn(fqdn, callback) lines I may be able to use to get the container ID of my Open VPN Client.

Lonkle

I'm now able to connect any container to any network I want on app startup based on the app's FQDN or ID. But even if I create a separate network than cloudron and stick the OpenVPN Client in there with another app. The other app doesn't inherit the OpenVPN Client's IP like it does when you connect the connectors directly on the bridge network using the docker run --net command.

There may be a firewall issue though. Cloudron does have a very specific subnet and maybe I can't escape it without running into a firewall.

@girish @nebulon - any ideas for me on this one?

girish

Oh wow, you are moving way faster than we can take a look at this. Great progress. Please give us a day or two to look into this! mon/tue is always a bit slower given all the app updates, support requests etc that pile up over the weekend

Lonkle

@girish No problem at all. I can work on porting some other apps in the meantime. Really love the platform you guys created.

And I might be able to figure the answer to my own question. I made an Ubuntu Graphical 18.04 Virtual Machine and installed Cloudron on that so I can develop for the “above subcontainer level” (AKA: box) app code. I am tho running into the two app restriction on localhost - I wish that limitation didn’t apply to custom apps developers are making.

Anyway, thank you again for being as responsive as you have been and don’t mind me updating as I go. Respond whenever you think you can. ️ You guys have done some really amazing work; both of you are inspirations tbh.

girish

@Lonk If you are working on app, please do leave a note in the corresponding app's wishlist topic. Just to make sure all our work does not overlap. Thanks.

girish

BTW, does the vpn client have some sort of web ui? Would be great to have something where a user can just login and provide connection information. We can look into the openvpn server app for inspiration.

Lonkle

@girish
Goals for v0.4.0:
• Modify docker.js to use the container:vpnContainerID NetworkMode for all apps choosing to run their traffic through the VPN (presumably a drop-down box in the config option of an app which will need to be coded in the dashboard, hardcoded to the ovpn-client's manifest ID). Note: Need to remove hostname and any port bindings so that the container takes on the attributes of the VPN otherwise the NetworkMode will conflict with the VPN Client's Network.

Goals for v0.5.0:
• Add an HTTP API endpoint to change and convert any .ovpn file a user supplies via POST into the container and disconnect from current to connect to the newly added .ovpn

Goals for v1.0.0 (I'd consider it ready to upload to the official Cloudron Appstore at this point):
• Add a full web interface at the base URL of the app which uses the API endpoint in v0.5 to allow a user to submit any .ovpn file and convert it to what Cloudron needs to straight from a web interface (so they don't have to use the API or manually add it in the terminal / file manager, which would be confusing to new users using Cloudron).

Lonkle

@girish And I will post on each app's wishlist forum post when I start work on converting.

I'm just really stuck on this NetworkMode code. No matter what I do, the NetworkMode stays cloudron even after restarting the container and hardcoding it to use the NetworkMode container:vpncontainerid. Effectively making the app useless. The NetworkMode needs to change, but it's like once the NetworkMode is attached to an app, it stays on there no matter what. Still experimenting as that's the last piece of this puzzles aside from finding a function that can take a cloudron-manifest-id and translate it to a Docker Container ID (so I don't have to docker network inspect and then hardcode it's ID).

mehdi

@Lonk Are you sure you are re-building the app container, not just re-starting it ?

Lonkle

@mehdi My OpenVPN Client doesn't require any more edits to work on Cloudron. Or are you referring to the "other container" that I'm trying to attach to my OpenVPN Client (to then change it's public IP which I just check with curl icanhazip.com every new experiment)? So all apps / cloudron containers I just restart from the Cloudron dashboard since they are the one's who's NetworkMode needs to be container:vpn-docker-constainer-id-or-name-but-really-just-id (it's hard to parse containters and networks because docker.js has to sent the app names to their container ids to prevent container naming conflicts) - which is a box problem that needs to be solved (particularly docker.js).

I see a lot of mentions of appContainer references and subcontainer references in REing the docker.js code. But they're all just regular containers (nothing is special about them except a flag is set if it's.a subcontainer) so that must be a Cloudron nomenclature.

I think (really just hope) I found my issue:
function containersCreate(req, res, next) { safe.set(req.body, 'HostConfig.NetworkMode', 'cloudron'); // overwrite the network the container lives in

Now why are you doing that to me @girish?

Lonkle

THAT WAS IT! Well, there's more to box code changes than just proving the vpn client connection works. But @girish was overriding the network name in dockerproxy.js which took an entire day to figure out! but really

Alright, @mehdi, I'm glad you pushed me enter into box code realms. I'm still intimidated by it. But I've got (sub)container creation down pretty well now.

Lonkle

This would be faster using Cloudron on localhost in my VM as I installed it on yesterday to do box code. But, the two app limit thing was too hard to deal with.

Lonkle

One of the worst parts of box code is that all these containers (aside from contained cloudron services and cloudron itself) have hshes for names. So, when trying to connect one to another and then inspecting the attributes of all of them is so confusing.

Lonkle

To get this app store ready, it might even need modifications to the nginx reverse proxy. @mehdi Maybe you can help? Is there a way, in cloudron to open more than one port? The manifest seems to be a way to publish a random port to forward to the specified-in-the-manifest port

Example:

Using docker container ls --format "table {{.ID}}\t{{.Names}}\t{{.Ports}}" -a - I can look at all the "published" ports. They all seems to be going through a nginx reverse proxy to reach port 80 / 443. And their original port that forwards to that is random (?).

NAMES ----------------------------------- PORTS
0b993ea5-85fc-4465-af5f-a1cd5ea5aeb7 127.0.0.1:43641->8080/tcp
57c94244-112d-4b98-abc4-0f15d6b07ca7 127.0.0.1:40875->8000/tcp
10a9ae64-689e-4765-a990-89a3d5e400d2 127.0.0.1:44749->80/tcp

Lonkle

Maybe if I hardcode the bind address to 0.0.0.0. Cloudron hardcodes it to localhost (likely for security purposes).

Lonkle

Sorry about the spam, I just started learning Docker two weeks ago now when I started building this app, so it's been all very foreign to me. Had to learn so many weird networking things.

Lonkle

I'm having a reverse proxy battle. Everything works now, but the "connected-to-the-vpn-client" app can't access the outside world. Cloudron doesn't publish ports in the standard way, there's some proxy between it all, and I have to RE that to get the apps connected to the openvpn-client to use the proxy somehow. It should just work by default looking at the code, but - there's always something.

Lonkle

Okay, so there's a problem here:

appPortBindings.hostPort needs to be set for all of Cloudron's containers or Cloudron turns them off. The only reason I know my OpenVPN client works is because I curl icanhazip.com before box stops the container for not passing all of it's "checks."

But this is a special case. To connect to the VPN Client - the app container can't have a hostPort as it inherits that from the VPN Client as part of the connection. But Cloudron requires the App published to have a port (I hardcode it not to), so even though the app works, Cloudron will stop it from running eventually once it fails enough checks until I figure out how to deal with this reverse proxy thing works.

Lonkle

I just have to figure out how to let appdb.js know that there's an actual http port set, it's just not in the **connected-to-vpn-client's`` app / container, it's defined in the OpenVPN Client's app / container.

Barring figuring that out though, I have to have the openvpn-client app mirror the ports that are exposed for every app connnected to it (right now, I'm just letting one container connect since Cloudron doesn't support more than one exposed port). Ahhh, so basically the OpenVPN client's http port has to be variable depending on the connecting app (or apps if people want to connect more than one other app to it at a time - I only personally need one app connected to it). So that's manifest.js code to also modify to get this to work.

Lonkle

I'm climbing a mountain too tall for me rn.

marcusquinn

The first step in making any tunnel is digging a hole

Lonkle

Nice metaphor for a VPN tunnel app. I’m so close to get this working. I found the SQL that inserts the httpPort into the DB so I’m going to try to have the container “attaching to the openvpn client” inherit the port and port type from the VPN Client while still having it be discoverable by the database. Let’s hope this works.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

VPN tunnel for apps