Cloudron CLI and NGINX Routing Weirdness

[Lonk]

This has happened to me a few times and I could never figure out the pattern but here's the issue:

• Developing an app and want to start fresh instead of update
• cloudron uninstall
• cloudron build && cloudron install
• Location: [same location as before]
• NGINX gives me 503 errors and app never passes health check

Repairing the app doesn't help, restarting, uninstalling and reinstalling. Nothing. BUT, I can change the domain name and then it suddenly starts working.

So I ran into it again today when testing @girish CLI's new update. And found the solution, even though I think this is a bug that should be fixed:

• Developing an app and want to start fresh instead of update
• cloudron uninstall
• Additional Step: Go to Cloudron Services -> Restart NGINX
• cloudron build && cloudron install
• Location: [same location as before]
• 200: SUCCESS

My guess is NGINX cached the domain somehow and is listening to that cached domain's routing rather than the new updated file it writes to route the new installation correctly. That's why restarting NGINX in between uninstall and install works (or using a different domain name, since there would be no cache there).

Can anyone else reproduce this?

[robi]

that just sounds like the uninstall isn't cleaning up properly.

Also try a nginx reload instead of restart. If that works, adding that to the uninstall flow should fix it.

[Lonk]

@robi said in Cloudron CLI and NGINX Routing Weirdness:

that just sounds like the uninstall isn't cleaning up properly.

Also try a nginx reload instead of restart. If that works, adding that to the uninstall flow should fix it.

That is exactly what that sounds like and not even a cloudron repair works to fix it so it's solely a NGINX issue.

I would have to CURL to the server's API in the middle to do the NGINX restart. It's not in there API docs but there is one for that. There's also some really interesting API commands that are undocumented, like, really surprising ones.

[Lonk]

@robi That's exactly what it was. I reproduced the error and read the NGINX logs:

conflicting server name ignored when trying to access my domain, which then sends me a 503.

This is very likely a time thing, but that cache should be flushed immediately because sometimes only on initial install can certain features be tested.

[robi]

@Lonk nice, thanks for valiadating.

My gut feelings are working well!

[Lonk]

But is there anything we can do about it. @girish, do you think this is a bug very easy to fix, otherwise I can just keep altering domain names when I uninstall and re-install (the restarting NGINX server isn't surefire but changing domain names is ).

[Lonk]

My "band-aid" solution doesn't even work anymore. Even tried to reboot the server. Nginx has literally cached the domain name thinks it already exists when being called. I'll just have to alternate domain names when doing re-installations until the cache expires. I'm just glad I know why I've been encountering that error and really glad it doesn't affect cloudron update.

[robi]

@Lonk that's more likely your browser cache, not nginx.

[Lonk]

@robi I tested it with different browsers. And the Nginx logs directly say “domain name conflict”. It is something cached in NGINX, I think. But if it is is related to browser caching. I’ll test it more thoroughly - it’s incredibly easy to reproduce.

[robi]

great, then you have direct evidence.

what happens if you reload nginx config?
what happens if you restart nginx w/ same config?
what happens if you restart nginx w/ diff config?

[Lonk]

@robi said in Cloudron CLI and NGINX Routing Weirdness:

what happens if you reload nginx config?
what happens if you restart nginx w/ same config?
what happens if you restart nginx w/ diff config?

The NGINX config file for the new installation is correctly written so that's not the problem.

It's just the domain hits nginx and then it goes to read the .conf file - but there are two for the same domain (in a cache somewhere I can't find), and it conflicts and sends me a 503 gateway error since NGINX doesn't know whether to choose the old .conf or the new .conf. Just gotta find where in the code that box feeds nginx the .conf file. Probably somewhere in reverseproxy.js.

[robi]

@Lonk it doesn't read a config file on access, it loads it into memory at start time and if there's multiple entries, they override each other the latest one being the one that is taken, otherwise the load fails with an error.

There should be a way to dump the running (mem) config of nginx and monitor it during the uninstall process.

[Lonk]

@robi You see this is one of those things that I don't have enough motivation to fix. I'll just avoid the flow entirely by constantly using different names to avoid the cache and rotate back when it somehow magically leaves the cache - unless @girish and / or @nebulon fix it one day or encounter it themselves and fix it. I just don't have the motivation to fix this, though usually I do. I'm just so done with NGINX man, the Open VPN Client scarred me. I never wanna go into reverseproxy.js ever again.

[Lonk]

Okay, maybe I will try to figure it out. I just tried setting it back to my preferred domain name and it failed because of domain name conflicts sooooo, I'm going to have to figure this out now. Dang it.

Edit: It's been a day so any cache would be gone, maybe I'll just do a re-nstall of Cloudron itself.

[Lonk]

Well, it's definitely a NGINX issue:

root@server:~# sudo nginx -t
nginx: [warn] conflicting server name "vpn.viridiancloud.com" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "vpn.viridiancloud.com" on [::]:80, ignored
nginx: [warn] conflicting server name "vpn.viridiancloud.com" on 0.0.0.0:443, ignored
nginx: [warn] conflicting server name "vpn.viridiancloud.com" on [::]:443, ignored
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

I tried clearing the cache. And restarting the NGINX service. But maybe they store the nginx cache in a different place. I hate working with nginx.

[Lonk]

Gave up and am re-installing. NGINX just wasn't fixing itself. I just will make sure not to uninstall an app from the command line because it forgets to clear some NGINX cache out.

[robi]

I don't see why there's a conflict there, and v6 likely isn't needed.

What happens when you switch the domain to vpn2?

@girish should have some insight.

[Lonk]

@robi It has nothing to do with the VPN, the VPN has a front end client and it can be off, but even if it's off NGINX still conflicts with itself like it was in some kind of cache I couldn't find and I wanted the domain name back on the app so I re-installed.

[robi]

@Lonk never said it did.

[Lonk]

@robi said in Cloudron CLI and NGINX Routing Weirdness:

What happens when you switch the domain to vpn2?

If I switched the domain to vpn2 it would just work. The domain name is being cached by NGINX in a way I don't understand so I just re-installed.

[robi]

@Lonk yes, you said this already.

what's different in nginx -t or other relevant place?

[Lonk]

@robi said in Cloudron CLI and NGINX Routing Weirdness:

@Lonk yes, you said this already.

what's different in nginx -t or other relevant place?

Well nginx -t said conflicting server name "vpn.viridiancloud.com" four times or so when I used it in the command line, and since that gave me no new information I gave up. Pick your battles!

[robi]

@Lonk sigh, the question was what's different with vpn2...

[Lonk]

@robi said in Cloudron CLI and NGINX Routing Weirdness:

@Lonk sigh, the question was what's different with vpn2...

Well, changing the location, it creates a completely new container including network / port changes. But repairing the Cloudron app, didn't fix it so it wasn't the container or the network itself. Hmm, maybe I'll still look more into this problem cause I still am curious, I just need to ask a question first on the dev forum about remote SQLing into Cloudron's App DB first before I dive further as there is one last thing I need to solve it.

[Lonk]

Well, at least I can say with confidence that backing up, then re-installing Cloudron completely fixes the issue. But I did a full restart of the server and that didn't fix it so it's not exactly a "cache" issue. I'll do more testing since I can repoduce it.

[Lonk]

But it could just be me, can any @appdev reproduce this:

• So you're developing an app and want to start fresh instead of cloudron update to see your app installed as a new user
• cloudron uninstall
• cloudron build && cloudron install
• Location: [same location as before the uninstall]
• NGINX gives me 503 errors and app never passes health check

If it's just me, then I have to debug this issue in a different way.

[mehdi]

@Lonk I never encountered the issue, and this is a workflow I often use. Maybe it has something to do with your networking changes on your cloudron you did for the VPN thing ?

[Lonk]

@mehdi That’s what I’m thinking it must be - it’s happening with my VPN Client since that’s what I’ve been working. But the box code treats the OpenVPN Client like normal and only treats the apps connected to it special.....wait, what if the cache of the domain exists because an app was connected to my OpenVPN Client app so it’s still “somewhere” in memory even though technically that container is no longer referencing an active container.

That’s a great idea and I think that may be the cause. Because it would persist reboots even, the app connected to the VPN Client attempting to connect. I’m just hoping I can get some advice on how to access Cloudron’s internal app DB via Remote SQL (like you can with individual apps) so I can debug this issue entirely.

I want to release the OpenVPN Client on the store in 2021. So I gotta make sure it’s perfect - and your comment just sparked a thought process that very likely could be the problem and I was dismissing it in my head for silly reasons. Thanks. I’ll reproduce it now that I know it’s just me, and see if I can fix it.

I really just needed one other person on the latest Cloudron perform those steps and have it go perfectly for me to know it was my app (and box changes).