Hide Cloudron (Cloudfront, CDN, multiple IPs)

[Prinzhorn]

Let's say we use Cloudron to host multiple client projects on a single instance. Each client has their own domain. We don't want clients to know that we are running Cloudron and that multiple projects are running on the same machine (let's just take this as a given).

For a PoC I've set up Cloudron and installed Matomo. We are using Matomo for Log Analytics only, I've written a script that imports the logs into the app. The Matomo instance will host data for different clients, since Matomo offers multi-user support, that should work as expected.

Is there any good way to hide Cloudron? E.g. hitting the IP directly gives "You are seeing this page because the DNS record of _ is set to this server's IP but Cloudron has no app configured for this domain."

I've tried putting the Matomo App behind Cloudfront. That would give us the level of concealment we want (given we properly protect/hide the origin server). However, I keep running into one wall after the other. Matomo just doesn't like the mix of being accessed on localhost by nginx, but then getting things like an Origin header from my Cloudfront distribution. And I really don't want to write Lambda@Edge to clean this all up.

Another option we considered was attaching multiple IP addresses to a single instance (e.g. EC2). Then each client's domain would point to a different IP. But that sounds overly complex and we'd still need to be careful not to leak the information through things like DNS enumeration (clients might be on subdomains of us, e.g. "client-a.matomo.example.com").

Can you recommend any way other than setting up multiple separated Cloudron instances (which kind of defeats the purpose)?

Thanks!

[nebulon]

To hide the IP as you mentioned, putting a CDN like Cloudfront in front of it, is the way to go. Generally all apps should work in such an environment, but maybe we miss some configs for Matomo there. Do you have any more details on your Matomo issue and what it does not like about this?

[Prinzhorn]

Thanks for the reply. At first I was just naively putting Cloudfront in front of it without any special configuration (default behavior). It looked like this:

All the CSS and JS served via index.php (e.g./index.php?module=Proxy&action=getCss&cb=0fa66222ec3d70db44b49a08b83c8b97) serve the same login page as text/html instead of the actual CSS/JS.

Trying to sign in resulted in:

Error: Form security failed, token miss match. Please reload the form and check that your cookies are enabled.

So I went ahead and explicitly allowed "MATOMO_SESSID" in my Cloudfront distribution. Now I can see "set-cookie" actually making it through. However, the page is still busted and now login gives me:

Error: Form security failed, invalid origin. If you previously connected using https, please ensure you are connecting over a secure (SSL/TLS) connection and try again.

I assume because of CSRF protection via "Origin" (and/or "Referer") header. Which obviously both contain the Cloudfront host/origin, which Matomo knows nothing about (and the browser knows nothing about what's happening beyond Cloudfront). That's the part I think I'd have to rewrite using Lambda@Edge.

I've also put in test.php in my Matomo app and diffed them (the request to the origin vs through Cloudfront):

<?php
var_dump($_SERVER);
?>

That's how I noticed the cookie was missing. Looking at it again it looks like Cloudfront is also not forwarding the "Accept" header, which might explain CSS/JS not being served correctly? Assuming Matomo even looks at them. But I don't see how that would affect the login issue with the origin validation.

Is there something I'm doing wrong? Or would the nginx config on your end need to be adjusted to make this work?

[girish]

@Prinzhorn that broken page can also happen if you use an adblocker. Here is my fresh install without any Cloudfront.

Disabling adblocker makes it work. Can you confirm this on your side? In fact, the first time I loaded the login page, it did not show the adblocker warning, it only appeared later (i.e the screenshot looked very similar to yours).

[girish]

@Prinzhorn OK, so I tried with cloudflare and it just works (tm). By some magic, I don't even have to disable the adblocker. Unfortunately, I don't have much idea about Cloudfront.

[Prinzhorn]

@girish well, I found it. It wasn't related to adblock (I had it disabled) but Cloudfront. I had to enable query strings in the distribution behavior as well. That was stupid, I just now realized from the access logs that Cloudfront would just access "index.php" without the CSS/JS params.

I've now enabled everything: header, query strings and cookies:

This gives me:

If I set "headers" to "none" I'm still left with the login issue, which I attribute to CSRF/Origin/Referer. Or maybe I've still configured something wrong? Can you put Cloudfront in front of your Matomo and login via *.cloudfront.net?

@girish did you change anything about the distribution or did you only enter the origin (your Cloudron Matomo domain) and click save? Can you actually log in though?

[girish]

@Prinzhorn said in Hide Cloudron (Cloudfront, CDN, multiple IPs):

@girish did you change anything about the distribution or did you only enter the origin (your Cloudron Matomo domain) and click save? Can you actually log in though?

Er, in case you misread, I did not test with Cloudfront! I tested with Cloudflare. Cloudflare can also hide your IP address.

[Prinzhorn]

@girish well, between Cloudron, Cloudfront and Cloudflare it's easy to misread ;). We are locked into AWS, so no Cloudflare for this particular project.

Can you also do something about the "a new user can only post once every 360s" for me? It kind of hinders this discussion right now.

Edit: did you use an entirely unrelated Domain in Cloudflare or how did that work? Because I'm working with the *.cloudfront.net domain right now, vs my matomo.cloudron.example.com

Also can you actually log in through Cloudflare?

[girish]

I also found you have to add this to config.ini.php for cloudflare use here:

; Uncomment line below if you use CloudFlare
;proxy_client_headers[] = HTTP_CF_CONNECTING_IP

[girish]

@Prinzhorn said in Hide Cloudron (Cloudfront, CDN, multiple IPs):

Can you also do something about the "a new user can only post once every 360s" for me? It kind of hinders this discussion right now.

I think that should be gone now. New user restrictions go away with an upvote which I have done so now.

[Prinzhorn]

@girish but it's not behind Cloudflare? It's behind nginx? The question is how much does Matomo actually know what's happening in front of nginx.

[girish]

@Prinzhorn said in Hide Cloudron (Cloudfront, CDN, multiple IPs):

Edit: did you use an entirely unrelated Domain in Cloudflare or how did that work? Because I'm working with the *.cloudfront.net domain right now, vs my matomo.cloudron.example.com

On Cloudflare atleast you can keep the original domain. So, I am working with matomo.cloudron.example.com.

Also can you actually log in through Cloudflare?

Yes. You can try if you like https://matomo2.cloudron.site/ (this link won't work for future readers). You can use admin/changeme

And server IP is hidden:

$ host  matomo2.cloudron.site
matomo2.cloudron.site has address 104.21.4.130
matomo2.cloudron.site has address 172.67.132.25
matomo2.cloudron.site has IPv6 address 2606:4700:3030::ac43:8419
matomo2.cloudron.site has IPv6 address 2606:4700:3037::6815:482

[Prinzhorn]

@girish thanks, it does indeed work. But I think what Cloudflare does is entirely different from how it works through Cloudfront. So maybe someone else will be able to help us out. Also I wonder what your next Let's Encrypt re-new in Cloudron will do? Wouldn't it fail because now you're not resolving back to the Cloudron machine? You'd need to set up the Cloudflare self-signed cert for full ssl, right?

[girish]

@Prinzhorn said in Hide Cloudron (Cloudfront, CDN, multiple IPs):

@girish but it's not behind Cloudflare? It's behind nginx? The question is how much does Matomo actually know what's happening in front of nginx.

(As you probably know), cloudflare and cloudfront and very different kind of proxies. Cloudflare has a "limitation" that the domain also has to be hosted there (in my case, cloudron.site is hosted on cloudflare). Cloudfront does not have this limitation.

TBH, I don't completely understand the issue you are facing. I have to try Cloudfront to understand better. I am going to try this shortly.

[girish]

@Prinzhorn said in Hide Cloudron (Cloudfront, CDN, multiple IPs):

Also I wonder what your next Let's Encrypt re-new in Cloudron will do? Wouldn't it fail because now you're not resolving back to the Cloudron machine? You'd need to set up the Cloudflare self-signed cert for full ssl, right?

Cloudflare requires the domain to be hosted on cloudflare This means then that we can certs via Let's Encrypt DNS automation. So, the site always has valid certs even when behind cloudflare...

[Prinzhorn]

@girish yes, they appear to be completely different. The main issue and my main point is that I would access Matomo from a domain that neither Matomo nor Cloudron know about. A domain that points to the Cloufront distribution. Thanks for looking into this!

[girish]

@Prinzhorn I ran into same issue as you when using Cloudfront. So, the issue is that Cloudfront is trying to serve the files incorrectly.

When accessing directly, see that the content-type is text/css:

When accessing via cloudfront, the content-type changes to text/html, I have no idea why:

This is the reason the resources don't load properly on the browser. The css/js everything is treated as html.

[girish]

At this point, I feel I don't understand how Cloudfront works at all. Isn't it just a caching+proxying service? Clearly, it's doing a lot more than that.

[robi]

@girish it's not a proxy, it's a dumb CDN.
https://www.freecodecamp.org/news/do-not-use-s3-for-static-assets/

Hence it's used for assets not fronting applications.

[girish]

@robi said in Hide Cloudron (Cloudfront, CDN, multiple IPs):

Hence it's used for assets not fronting applications.

you the man! I fundamentally misunderstood the service. So, @Prinzhorn you can't easily front matomo with Cloudfront. I think if you want to hide the IP, we have to go another route.

To take a step back, is your only issue that http://ip is showing some Cloudron page?