SOLVED Cloudron dropping packets from router

[ReachableCEO]

[Jan 7 19:13] IPTables Packet Dropped: IN=ens18 OUT= MAC=56:28:f3:9f:0b:4b:26:4a:3c:83:cd:5b:08:00 SRC=10.251.50.252 DST=10.251.50.2 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=0 DF PROTO=TCP SPT=31182 DPT=9443 WINDOW=65228 RES=0x00 SYN URGP=0

10.251.50.252 is the router IP address (I have an HA router pair .252/.253 with a VIP of .254)

localuser@app:~$ ip ro
default via 10.251.50.254 dev ens18 proto static
10.251.50.0/24 dev ens18 proto kernel scope link src 10.251.50.2

The router is a VM (Opnsense) . I have 80/443 NAT to 10.251.50.2 which is the IP of the VM.

Key thing to note: This VM was moved from OVH (proxmox to proxmox) and used to have a public IP address directly on the VM.

I have split horizon DNS , so externally:

root@tsys-dc-01:~# dig @8.8.8.8 my.knownelement.com +short
104.182.29.21
root@tsys-dc-01:~#

and internally:

C:\Users\reach>nslookup my.knownelement.com
Server:  pihole1
Address:  10.251.37.5

Non-authoritative answer:
Name:    my.knownelement.com
Address:  10.251.50.2


C:\Users\reach>

I use pihole for internal DNS, it resolves to the Univention domain controllers which host authorative zones with wildcard records for all the cloudron hosted apps set to 10.251.50.2.

HSTS is preventing me from accessing the web ui where I imagine I can fix this issue somewhat easily? I seem to recall seeing some mention in the UI about adding upstream IP addresses or something. Alas the cert expired ...

Websites prove their identity via certificates, which are valid for a set time period. The certificate for my.knownelement.com expired on 1/5/2022.

Can I disable HSTS from the CLI? Make some manual iptables modification to that LE can do it's validation so I can get back in and fix things?

Please help.

I have been professionally administering Linux/Windows/Networks since 2001. Very comfortable at the CLI.

[robi]

@reachableceo said in Cloudron dropping packets from router:

disable HSTS

https://virtuallywired.io/2020/01/02/fix-error-you-cannot-visit-site-right-now-because-the-website-uses-hsts/

[ReachableCEO]

@robi Thank you! That allowed me to get back into the admin panel.

I am still unsure how to proceed and why iptables is blocking the router IP address.

[ReachableCEO]

Chasing the "failed ports" was a red herring.

root@app:/var/log# grep -c DPT=6443 syslog
1197
root@app:/var/log#

root@app:/var/log# grep -c DPT=9443 syslog
840
root@app:/var/log#

Turns out those were health checks from haproxy on the firewall for k0s (which was on old virtual machines , one of which was using 10.251.50.2). Still researching the issue, but at least I've cleared noise from syslog now!

[nebulon]

@reachableceo did you manually configure iptables prior or after installation of Cloudron? By default no IP ranges should be blocked. All not necessary ports for Cloudron will be closed however. Some more details here https://docs.cloudron.io/security/#cloud-firewall

Further just to double check, do you have configs made through the Cloudron dashboard about blocked IP ranges https://docs.cloudron.io/networking/#firewall ?

[ReachableCEO]

@nebulon The only adjustment I made was to allow webmin/netdata ports.

I did check for blocked IP addresses in the web ui.

The iptables messages were a red herring (the 6443/9443 port connections were health checks from the haproxy instance on the router).

I am able to connect to the server internally. For some reason I am unable to access it via the internet through the NAT/firewall. I am currently building a plain Ubuntu VM with Apache on 10.251.50.3 (10.251.50.1 is my legacy web server I'm migrating off from, I NAT to haproxy directly on the router and haproxy terminate SSL , handles lets encrypt and forwards to the VM).

The rules for 10.251.50.1,2,3 are exactly the same . Each of those has it's own corresponding external IP. I need to look at tcpdump on the router and cloudron and see if I can trace the packets from outside in.

[ReachableCEO]

This can be closed/archived/resolved. The issue was with my upstream ISP router ACLs. Adjusted those and all is working now.

[girish]

@reachableceo thanks for the update!