@girish well, between Cloudron, Cloudfront and Cloudflare it's easy to misread ;). We are locked into AWS, so no Cloudflare for this particular project.

Can you also do something about the "a new user can only post once every 360s" for me? It kind of hinders this discussion right now.

Edit: did you use an entirely unrelated Domain in Cloudflare or how did that work? Because I'm working with the *.cloudfront.net domain right now, vs my matomo.cloudron.example.com

Also can you actually log in through Cloudflare?

Let's say we use Cloudron to host multiple client projects on a single instance. Each client has their own domain. We don't want clients to know that we are running Cloudron and that multiple projects are running on the same machine (let's just take this as a given).

For a PoC I've set up Cloudron and installed Matomo. We are using Matomo for Log Analytics only, I've written a script that imports the logs into the app. The Matomo instance will host data for different clients, since Matomo offers multi-user support, that should work as expected.

Is there any good way to hide Cloudron? E.g. hitting the IP directly gives "You are seeing this page because the DNS record of _ is set to this server's IP but Cloudron has no app configured for this domain."

I've tried putting the Matomo App behind Cloudfront. That would give us the level of concealment we want (given we properly protect/hide the origin server). However, I keep running into one wall after the other. Matomo just doesn't like the mix of being accessed on localhost by nginx, but then getting things like an Origin header from my Cloudfront distribution. And I really don't want to write Lambda@Edge to clean this all up.

Another option we considered was attaching multiple IP addresses to a single instance (e.g. EC2). Then each client's domain would point to a different IP. But that sounds overly complex and we'd still need to be careful not to leak the information through things like DNS enumeration (clients might be on subdomains of us, e.g. "client-a.matomo.example.com").

Can you recommend any way other than setting up multiple separated Cloudron instances (which kind of defeats the purpose)?

Thanks!

@girish I don't think I understand the problem here. There is no "cloudron domain" that could be leaked. I create a CloudFront distribution, say example.cloudfront.net (or whatever custom domain I configure in CloudFront). This distribution forwards requests with this host to my Cloudron box (over plain HTTP). And there I have a custom nginx config that connects example.cloudfront.net to the Matomo container. There is no other domain involved.

I think the way Cloudron works I still need at least one domain like matomo.cloudron.example.com connected to the app (what you called the "cloudron domain"). Because Cloudron wouldn't know that my custom nginx config already handles connection of the domain. But I don't use it and Matomo doesn't know it exists. Matomo also doesn't know about CloudFront. To Matomo it is indistinguishable from the original setup. Because Matomo is not behind CloudFront, Matomo is behind Cloudron's nginx. How could it leak?

The only thing I need to take care of is that if someone finds the Cloudron Domain (which I don't need, but Cloudron does), they cannot correlate it to the same Matomo that is served through CloudFront. I think the way I'll do this is by using a domain specifically for Cloudron (e.g. not cloudron.my-company.com but my-cloudron.com), so that it cannot be found by enumeration of our subdomains (not sure how well you can protect DNS). And if the custom nginx config takes precedence over Cloudron's own, then I think I can also disable the custom domain and return a 400 instead of forwarding requests to Matomo? So someone "discovering" the domain would still not know it's actually the same Matomo instance served via CloudFront (CloudFront doesn't leak the origin).

Is there something I'm misunderstanding?

@girish did you get a chance to look at my recent post? Is this something Cloudron is interested in supporting (connecting domains to an app without TLS)? Alternatively (and more flexible) will https://forum.cloudron.io/topic/6713/where-can-i-put-custom-nginx-conf-that-won-t-be-overwritten go somewhere? You said

I think to properly support this, we can look into this for the next release

If there was a way to customize the nginx config then I could wire everything up myself. That would probably be both easier to implement on your end (doesn't even need a UI in the first iteration) and also way more flexible for different use-cases. Maybe some sort of fs based heuristic where you include custom nginx configs for each app in a specific location or that follow a specific name pattern?

When a CDN is placed in front of an app, the app needs to understand there is a CDN in front of it. The app has to be specifically coded and have configuration options to support a CDN.

I think all these things you've listed apply to your nginx in the same way they apply to any reverse proxy you put in front of it (including CloudFront). And you've already solved them, otherwise Matomo and other apps would respond with localhost:8080 in the HTML and break all the time.

Finally, the Origin header is only needed for CORS i.e when a tracked website is making a call into matomo. It doesn't come into picture when loading matomo's login screen. It might appear to come into picture, but this is only because we have a separate domain when using a CDN and this goes back to figuring out if matomo supports a CDN based setup (this, I don't know and I cannot find any information in their docs or the forum).

No and yes. As I've assumed earlier I'm running in Origin based CSRF protection and the Matomo source confirms that https://github.com/matomo-org/matomo/blob/e4a7b579622da9f585a9e662efc114b08b6fe08d/core/Nonce.php#L132-L139

Thanks for all the information regarding how nginx and Matomo interact. I've been able to successfully access Matomo through CloudFront by including Origin: https://matomo.cloudron.example.com in the origin request. However, login redirects me from foo.cloudfront.net to matomo.cloudron.example.com, which is not surprising, since that's the Host header Matomo sees.

From all the information I have I think all I need is the following: a way to associate a domain with an app without all the DNS/TLS setup that Cloudron otherwise needs. The only downside would be that it wouldn't be full SSL and traffic between EC2 and CloudFront would be unencrypted inside the AWS network.

So instead of

https://matomo.cloudron.example.com -> nginx -> docker

it would be

https://foo.cloudfront.net -> CloudFront -> http://foo.cloudfront.net -> nginx -> docker

what I have right now is

https://foo.cloudfront.net -> CloudFront -> https://matomo.cloudron.example.com -> nginx -> docker

I can configure CloudFront to forward the Host header. So there just needs to be an additional nginx config to forward my arbitrary host to the Matomo container. Matomo itself doesn't know about this, but would see the CloudFront Host + Origin header and be happy. Right now it doesn't know about foo.cloudfront.net, because CloudFront looks to nginx like any user accessing Matomo at matomo.cloudron.example.com. With this new setup there cannot be a leak of matomo.cloudron.example.com, because it isn't part of the request or response at all and Matomo is not aware of its existence. As you said enable_trusted_host_check is disabled and I just need a second trusted host (trusted by nginx) that nginx forwards to the container.

I'm sorry, but this is just wrong. How is https://www.freecodecamp.org/news/do-not-use-s3-for-static-assets/ relevant to what we're talking about here at all? First they say that Cloudfront does not support brotli (ok?) and their conclusion is that Cloudfront is too expensive? The article say nothing about dynamic content, it sounds more like a random collection of facts S3 / Cloudfront for SEO?

I'm not saying Cloudfront is amazing or anything. And I'm far from being an expert. But saying it's not possible to serve dynamic content is completely misleading. And backing that by a random source that is entirely unrelated doesn't make your argument more convincing.

https://aws.amazon.com/cloudfront/dynamic-content/

If you are serving dynamic content such as web applications or APIs directly from an Amazon Elastic Load Balancer (ELB) or Amazon EC2 instances to end users on the internet, you can improve the performance, availability, and security of your content by using Amazon CloudFront as your content delivery network. With Amazon CloudFront, your end users connections are terminated at CloudFront locations closer to them, which helps in reducing the overall round trip time required to establish a connection.

https://aws.amazon.com/blogs/networking-and-content-delivery/dynamic-whole-site-delivery-with-amazon-cloudfront/

For Amazon Elastic Compute Cloud (Amazon EC2) origins serving dynamic and/or static content, you should also consider using CloudFront because it offers a number of additional benefits. In this blog post, we will provide an overview of the performance, security, and cost benefits you get when using CloudFront to serve dynamic and/or static assets from Amazon EC2.

You'll find countless sources, both official and unofficial:

https://docs.aws.amazon.com/whitepapers/latest/best-practices-wordpress/dynamic-content.html
https://blog.shikisoft.com/serving-dynamic-website-with-amazon-cloudfront/

@girish regarding stuff being served as text/html, I already solved that as described above. I had to configure Cloudfront to pass along query strings as well, or else Cloudron will only get index.php as URL without the CSS/JS params.

Let's take a step back. How does Cloudron's nginx talk to Matomo? From what I can tell Matomo does not know that the domain I'm using to access it (e.g. matomo.cloudron.example.com) even exists. Only localhost is configured as trusted_hosts[]. So I assume nginx already does some rewriting to make it work at all, correct? E.g. rewrite the known domain to localhost? Does that include things like Origin header? Or am I on the wrong track?

@girish yes, they appear to be completely different. The main issue and my main point is that I would access Matomo from a domain that neither Matomo nor Cloudron know about. A domain that points to the Cloufront distribution. Thanks for looking into this!

@girish thanks, it does indeed work. But I think what Cloudflare does is entirely different from how it works through Cloudfront. So maybe someone else will be able to help us out. Also I wonder what your next Let's Encrypt re-new in Cloudron will do? Wouldn't it fail because now you're not resolving back to the Cloudron machine? You'd need to set up the Cloudflare self-signed cert for full ssl, right?

@girish but it's not behind Cloudflare? It's behind nginx? The question is how much does Matomo actually know what's happening in front of nginx.

@girish well, between Cloudron, Cloudfront and Cloudflare it's easy to misread ;). We are locked into AWS, so no Cloudflare for this particular project.

Can you also do something about the "a new user can only post once every 360s" for me? It kind of hinders this discussion right now.

Edit: did you use an entirely unrelated Domain in Cloudflare or how did that work? Because I'm working with the *.cloudfront.net domain right now, vs my matomo.cloudron.example.com

Also can you actually log in through Cloudflare?

@girish well, I found it. It wasn't related to adblock (I had it disabled) but Cloudfront. I had to enable query strings in the distribution behavior as well. That was stupid, I just now realized from the access logs that Cloudfront would just access "index.php" without the CSS/JS params.

I've now enabled everything: header, query strings and cookies:

This gives me:

If I set "headers" to "none" I'm still left with the login issue, which I attribute to CSRF/Origin/Referer. Or maybe I've still configured something wrong? Can you put Cloudfront in front of your Matomo and login via *.cloudfront.net?

@girish did you change anything about the distribution or did you only enter the origin (your Cloudron Matomo domain) and click save? Can you actually log in though?

Thanks for the reply. At first I was just naively putting Cloudfront in front of it without any special configuration (default behavior). It looked like this:

All the CSS and JS served via index.php (e.g./index.php?module=Proxy&action=getCss&cb=0fa66222ec3d70db44b49a08b83c8b97) serve the same login page as text/html instead of the actual CSS/JS.

Trying to sign in resulted in:

Error: Form security failed, token miss match. Please reload the form and check that your cookies are enabled.

So I went ahead and explicitly allowed "MATOMO_SESSID" in my Cloudfront distribution. Now I can see "set-cookie" actually making it through. However, the page is still busted and now login gives me:

Error: Form security failed, invalid origin. If you previously connected using https, please ensure you are connecting over a secure (SSL/TLS) connection and try again.

I assume because of CSRF protection via "Origin" (and/or "Referer") header. Which obviously both contain the Cloudfront host/origin, which Matomo knows nothing about (and the browser knows nothing about what's happening beyond Cloudfront). That's the part I think I'd have to rewrite using Lambda@Edge.

I've also put in test.php in my Matomo app and diffed them (the request to the origin vs through Cloudfront):

<?php
var_dump($_SERVER);
?>

That's how I noticed the cookie was missing. Looking at it again it looks like Cloudfront is also not forwarding the "Accept" header, which might explain CSS/JS not being served correctly? Assuming Matomo even looks at them. But I don't see how that would affect the login issue with the origin validation.

Is there something I'm doing wrong? Or would the nginx config on your end need to be adjusted to make this work?