Cloudron CLI and NGINX Routing Weirdness

[Lonk]

This has happened to me a few times and I could never figure out the pattern but here's the issue:

• Developing an app and want to start fresh instead of update
• cloudron uninstall
• cloudron build && cloudron install
• Location: [same location as before]
• NGINX gives me 503 errors and app never passes health check

Repairing the app doesn't help, restarting, uninstalling and reinstalling. Nothing. BUT, I can change the domain name and then it suddenly starts working.

So I ran into it again today when testing @girish CLI's new update. And found the solution, even though I think this is a bug that should be fixed:

• Developing an app and want to start fresh instead of update
• cloudron uninstall
• Additional Step: Go to Cloudron Services -> Restart NGINX
• cloudron build && cloudron install
• Location: [same location as before]
• 200: SUCCESS

My guess is NGINX cached the domain somehow and is listening to that cached domain's routing rather than the new updated file it writes to route the new installation correctly. That's why restarting NGINX in between uninstall and install works (or using a different domain name, since there would be no cache there).

Can anyone else reproduce this?

[robi]

that just sounds like the uninstall isn't cleaning up properly.

Also try a nginx reload instead of restart. If that works, adding that to the uninstall flow should fix it.

[Lonk]

@robi said in Cloudron CLI and NGINX Routing Weirdness:

that just sounds like the uninstall isn't cleaning up properly.

Also try a nginx reload instead of restart. If that works, adding that to the uninstall flow should fix it.

That is exactly what that sounds like and not even a cloudron repair works to fix it so it's solely a NGINX issue.

I would have to CURL to the server's API in the middle to do the NGINX restart. It's not in there API docs but there is one for that. There's also some really interesting API commands that are undocumented, like, really surprising ones.

[Lonk]

@robi That's exactly what it was. I reproduced the error and read the NGINX logs:

conflicting server name ignored when trying to access my domain, which then sends me a 503.

This is very likely a time thing, but that cache should be flushed immediately because sometimes only on initial install can certain features be tested.

[robi]

@Lonk nice, thanks for valiadating.

My gut feelings are working well!

[Lonk]

But is there anything we can do about it. @girish, do you think this is a bug very easy to fix, otherwise I can just keep altering domain names when I uninstall and re-install (the restarting NGINX server isn't surefire but changing domain names is ).

[Lonk]

My "band-aid" solution doesn't even work anymore. Even tried to reboot the server. Nginx has literally cached the domain name thinks it already exists when being called. I'll just have to alternate domain names when doing re-installations until the cache expires. I'm just glad I know why I've been encountering that error and really glad it doesn't affect cloudron update.

[robi]

@Lonk that's more likely your browser cache, not nginx.

[Lonk]

@robi I tested it with different browsers. And the Nginx logs directly say “domain name conflict”. It is something cached in NGINX, I think. But if it is is related to browser caching. I’ll test it more thoroughly - it’s incredibly easy to reproduce.

[robi]

great, then you have direct evidence.

what happens if you reload nginx config?
what happens if you restart nginx w/ same config?
what happens if you restart nginx w/ diff config?

[Lonk]

@robi said in Cloudron CLI and NGINX Routing Weirdness:

what happens if you reload nginx config?
what happens if you restart nginx w/ same config?
what happens if you restart nginx w/ diff config?

The NGINX config file for the new installation is correctly written so that's not the problem.

It's just the domain hits nginx and then it goes to read the .conf file - but there are two for the same domain (in a cache somewhere I can't find), and it conflicts and sends me a 503 gateway error since NGINX doesn't know whether to choose the old .conf or the new .conf. Just gotta find where in the code that box feeds nginx the .conf file. Probably somewhere in reverseproxy.js.

[robi]

@Lonk it doesn't read a config file on access, it loads it into memory at start time and if there's multiple entries, they override each other the latest one being the one that is taken, otherwise the load fails with an error.

There should be a way to dump the running (mem) config of nginx and monitor it during the uninstall process.

[Lonk]

@robi You see this is one of those things that I don't have enough motivation to fix. I'll just avoid the flow entirely by constantly using different names to avoid the cache and rotate back when it somehow magically leaves the cache - unless @girish and / or @nebulon fix it one day or encounter it themselves and fix it. I just don't have the motivation to fix this, though usually I do. I'm just so done with NGINX man, the Open VPN Client scarred me. I never wanna go into reverseproxy.js ever again.

[Lonk]

Okay, maybe I will try to figure it out. I just tried setting it back to my preferred domain name and it failed because of domain name conflicts sooooo, I'm going to have to figure this out now. Dang it.

Edit: It's been a day so any cache would be gone, maybe I'll just do a re-nstall of Cloudron itself.

[Lonk]

Well, it's definitely a NGINX issue:

root@server:~# sudo nginx -t
nginx: [warn] conflicting server name "vpn.viridiancloud.com" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "vpn.viridiancloud.com" on [::]:80, ignored
nginx: [warn] conflicting server name "vpn.viridiancloud.com" on 0.0.0.0:443, ignored
nginx: [warn] conflicting server name "vpn.viridiancloud.com" on [::]:443, ignored
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

I tried clearing the cache. And restarting the NGINX service. But maybe they store the nginx cache in a different place. I hate working with nginx.

[Lonk]

Gave up and am re-installing. NGINX just wasn't fixing itself. I just will make sure not to uninstall an app from the command line because it forgets to clear some NGINX cache out.

[robi]

I don't see why there's a conflict there, and v6 likely isn't needed.

What happens when you switch the domain to vpn2?

@girish should have some insight.

[Lonk]

@robi It has nothing to do with the VPN, the VPN has a front end client and it can be off, but even if it's off NGINX still conflicts with itself like it was in some kind of cache I couldn't find and I wanted the domain name back on the app so I re-installed.

[robi]

@Lonk never said it did.

[Lonk]

@robi said in Cloudron CLI and NGINX Routing Weirdness:

What happens when you switch the domain to vpn2?

If I switched the domain to vpn2 it would just work. The domain name is being cached by NGINX in a way I don't understand so I just re-installed.