Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Cloudron CLI and NGINX Routing Weirdness

    App Packaging & Development
    3
    28
    560
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • Lonkle
      Lonkle last edited by Lonkle

      This has happened to me a few times and I could never figure out the pattern but here's the issue:

      • Developing an app and want to start fresh instead of update
      • cloudron uninstall
      • cloudron build && cloudron install
      • Location: [same location as before]
      • NGINX gives me 503 errors and app never passes health check

      Repairing the app doesn't help, restarting, uninstalling and reinstalling. Nothing. BUT, I can change the domain name and then it suddenly starts working.

      So I ran into it again today when testing @girish CLI's new update. And found the solution, even though I think this is a bug that should be fixed:

      • Developing an app and want to start fresh instead of update
      • cloudron uninstall
      • Additional Step: Go to Cloudron Services -> Restart NGINX
      • cloudron build && cloudron install
      • Location: [same location as before]
      • 200: SUCCESS

      My guess is NGINX cached the domain somehow and is listening to that cached domain's routing rather than the new updated file it writes to route the new installation correctly. That's why restarting NGINX in between uninstall and install works (or using a different domain name, since there would be no cache there).

      Can anyone else reproduce this?

      1 Reply Last reply Reply Quote 0
      • robi
        robi last edited by

        that just sounds like the uninstall isn't cleaning up properly.

        Also try a nginx reload instead of restart. If that works, adding that to the uninstall flow should fix it.

        Life of Advanced Technology

        Lonkle 1 Reply Last reply Reply Quote 0
        • Lonkle
          Lonkle @robi last edited by Lonkle

          @robi said in Cloudron CLI and NGINX Routing Weirdness:

          that just sounds like the uninstall isn't cleaning up properly.

          Also try a nginx reload instead of restart. If that works, adding that to the uninstall flow should fix it.

          That is exactly what that sounds like and not even a cloudron repair works to fix it so it's solely a NGINX issue.

          I would have to CURL to the server's API in the middle to do the NGINX restart. It's not in there API docs but there is one for that. There's also some really interesting API commands that are undocumented, like, really surprising ones.

          1 Reply Last reply Reply Quote 0
          • Lonkle
            Lonkle last edited by Lonkle

            @robi That's exactly what it was. I reproduced the error and read the NGINX logs:

            conflicting server name ignored when trying to access my domain, which then sends me a 503.

            This is very likely a time thing, but that cache should be flushed immediately because sometimes only on initial install can certain features be tested.

            robi 1 Reply Last reply Reply Quote 1
            • robi
              robi @Lonkle last edited by

              @Lonk nice, thanks for valiadating.

              My gut feelings are working well! 👍

              Life of Advanced Technology

              1 Reply Last reply Reply Quote 1
              • Lonkle
                Lonkle last edited by

                But is there anything we can do about it. @girish, do you think this is a bug very easy to fix, otherwise I can just keep altering domain names when I uninstall and re-install (the restarting NGINX server isn't surefire but changing domain names is 😅).

                1 Reply Last reply Reply Quote 0
                • Lonkle
                  Lonkle last edited by

                  My "band-aid" solution doesn't even work anymore. Even tried to reboot the server. Nginx has literally cached the domain name thinks it already exists when being called. I'll just have to alternate domain names when doing re-installations until the cache expires. I'm just glad I know why I've been encountering that error and really glad it doesn't affect cloudron update.

                  robi 1 Reply Last reply Reply Quote 0
                  • robi
                    robi @Lonkle last edited by

                    @Lonk that's more likely your browser cache, not nginx.

                    Life of Advanced Technology

                    Lonkle 1 Reply Last reply Reply Quote 0
                    • Lonkle
                      Lonkle @robi last edited by

                      @robi I tested it with different browsers. And the Nginx logs directly say “domain name conflict”. It is something cached in NGINX, I think. But if it is is related to browser caching. I’ll test it more thoroughly - it’s incredibly easy to reproduce.

                      1 Reply Last reply Reply Quote 0
                      • robi
                        robi last edited by

                        great, then you have direct evidence.

                        what happens if you reload nginx config?
                        what happens if you restart nginx w/ same config?
                        what happens if you restart nginx w/ diff config?

                        Life of Advanced Technology

                        Lonkle 1 Reply Last reply Reply Quote 0
                        • Lonkle
                          Lonkle @robi last edited by

                          @robi said in Cloudron CLI and NGINX Routing Weirdness:

                          what happens if you reload nginx config?
                          what happens if you restart nginx w/ same config?
                          what happens if you restart nginx w/ diff config?

                          The NGINX config file for the new installation is correctly written so that's not the problem.

                          It's just the domain hits nginx and then it goes to read the .conf file - but there are two for the same domain (in a cache somewhere I can't find), and it conflicts and sends me a 503 gateway error since NGINX doesn't know whether to choose the old .conf or the new .conf. Just gotta find where in the code that box feeds nginx the .conf file. Probably somewhere in reverseproxy.js.

                          robi 1 Reply Last reply Reply Quote 0
                          • robi
                            robi @Lonkle last edited by

                            @Lonk it doesn't read a config file on access, it loads it into memory at start time and if there's multiple entries, they override each other the latest one being the one that is taken, otherwise the load fails with an error.

                            There should be a way to dump the running (mem) config of nginx and monitor it during the uninstall process.

                            Life of Advanced Technology

                            Lonkle 1 Reply Last reply Reply Quote 0
                            • Lonkle
                              Lonkle @robi last edited by Lonkle

                              @robi You see this is one of those things that I don't have enough motivation to fix. I'll just avoid the flow entirely by constantly using different names to avoid the cache and rotate back when it somehow magically leaves the cache - unless @girish and / or @nebulon fix it one day or encounter it themselves and fix it. I just don't have the motivation to fix this, though usually I do. I'm just so done with NGINX man, the Open VPN Client scarred me. I never wanna go into reverseproxy.js ever again. 😂

                              1 Reply Last reply Reply Quote 0
                              • Lonkle
                                Lonkle last edited by Lonkle

                                Okay, maybe I will try to figure it out. I just tried setting it back to my preferred domain name and it failed because of domain name conflicts sooooo, I'm going to have to figure this out now. Dang it.

                                Edit: It's been a day so any cache would be gone, maybe I'll just do a re-nstall of Cloudron itself.

                                1 Reply Last reply Reply Quote 0
                                • Lonkle
                                  Lonkle last edited by

                                  Well, it's definitely a NGINX issue:

                                  root@server:~# sudo nginx -t
                                  nginx: [warn] conflicting server name "vpn.viridiancloud.com" on 0.0.0.0:80, ignored
                                  nginx: [warn] conflicting server name "vpn.viridiancloud.com" on [::]:80, ignored
                                  nginx: [warn] conflicting server name "vpn.viridiancloud.com" on 0.0.0.0:443, ignored
                                  nginx: [warn] conflicting server name "vpn.viridiancloud.com" on [::]:443, ignored
                                  nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
                                  nginx: configuration file /etc/nginx/nginx.conf test is successful
                                  

                                  I tried clearing the cache. And restarting the NGINX service. But maybe they store the nginx cache in a different place. I hate working with nginx. 😂

                                  1 Reply Last reply Reply Quote 0
                                  • Lonkle
                                    Lonkle last edited by

                                    Gave up and am re-installing. NGINX just wasn't fixing itself. I just will make sure not to uninstall an app from the command line because it forgets to clear some NGINX cache out.

                                    1 Reply Last reply Reply Quote 0
                                    • robi
                                      robi last edited by

                                      I don't see why there's a conflict there, and v6 likely isn't needed.

                                      What happens when you switch the domain to vpn2?

                                      @girish should have some insight.

                                      Life of Advanced Technology

                                      Lonkle 2 Replies Last reply Reply Quote 0
                                      • Lonkle
                                        Lonkle @robi last edited by

                                        @robi It has nothing to do with the VPN, the VPN has a front end client and it can be off, but even if it's off NGINX still conflicts with itself like it was in some kind of cache I couldn't find and I wanted the domain name back on the app so I re-installed.

                                        robi 1 Reply Last reply Reply Quote 0
                                        • robi
                                          robi @Lonkle last edited by

                                          @Lonk never said it did.

                                          Life of Advanced Technology

                                          1 Reply Last reply Reply Quote 0
                                          • Lonkle
                                            Lonkle @robi last edited by

                                            @robi said in Cloudron CLI and NGINX Routing Weirdness:

                                            What happens when you switch the domain to vpn2?

                                            If I switched the domain to vpn2 it would just work. The domain name is being cached by NGINX in a way I don't understand so I just re-installed.

                                            robi 1 Reply Last reply Reply Quote 0
                                            • robi
                                              robi @Lonkle last edited by

                                              @Lonk yes, you said this already.

                                              what's different in nginx -t or other relevant place?

                                              Life of Advanced Technology

                                              Lonkle 1 Reply Last reply Reply Quote 0
                                              • Lonkle
                                                Lonkle @robi last edited by

                                                @robi said in Cloudron CLI and NGINX Routing Weirdness:

                                                @Lonk yes, you said this already.

                                                what's different in nginx -t or other relevant place?

                                                Well nginx -t said conflicting server name "vpn.viridiancloud.com" four times or so when I used it in the command line, and since that gave me no new information I gave up. Pick your battles! 😂

                                                robi 1 Reply Last reply Reply Quote 0
                                                • robi
                                                  robi @Lonkle last edited by

                                                  @Lonk sigh, the question was what's different with vpn2...

                                                  Life of Advanced Technology

                                                  Lonkle 1 Reply Last reply Reply Quote 0
                                                  • Lonkle
                                                    Lonkle @robi last edited by Lonkle

                                                    @robi said in Cloudron CLI and NGINX Routing Weirdness:

                                                    @Lonk sigh, the question was what's different with vpn2...

                                                    Well, changing the location, it creates a completely new container including network / port changes. But repairing the Cloudron app, didn't fix it so it wasn't the container or the network itself. Hmm, maybe I'll still look more into this problem cause I still am curious, I just need to ask a question first on the dev forum about remote SQLing into Cloudron's App DB first before I dive further as there is one last thing I need to solve it.

                                                    1 Reply Last reply Reply Quote 0
                                                    • Lonkle
                                                      Lonkle last edited by

                                                      Well, at least I can say with confidence that backing up, then re-installing Cloudron completely fixes the issue. But I did a full restart of the server and that didn't fix it so it's not exactly a "cache" issue. I'll do more testing since I can repoduce it.

                                                      1 Reply Last reply Reply Quote 0
                                                      • Lonkle
                                                        Lonkle last edited by

                                                        But it could just be me, can any @appdev reproduce this:

                                                        • So you're developing an app and want to start fresh instead of cloudron update to see your app installed as a new user
                                                        • cloudron uninstall
                                                        • cloudron build && cloudron install
                                                        • Location: [same location as before the uninstall]
                                                        • NGINX gives me 503 errors and app never passes health check

                                                        If it's just me, then I have to debug this issue in a different way.

                                                        mehdi 1 Reply Last reply Reply Quote 0
                                                        • mehdi
                                                          mehdi App Dev @Lonkle last edited by

                                                          @Lonk I never encountered the issue, and this is a workflow I often use. Maybe it has something to do with your networking changes on your cloudron you did for the VPN thing ?

                                                          Lonkle 1 Reply Last reply Reply Quote 0
                                                          • Lonkle
                                                            Lonkle @mehdi last edited by

                                                            @mehdi That’s what I’m thinking it must be - it’s happening with my VPN Client since that’s what I’ve been working. But the box code treats the OpenVPN Client like normal and only treats the apps connected to it special.....wait, what if the cache of the domain exists because an app was connected to my OpenVPN Client app so it’s still “somewhere” in memory even though technically that container is no longer referencing an active container.

                                                            That’s a great idea and I think that may be the cause. Because it would persist reboots even, the app connected to the VPN Client attempting to connect. I’m just hoping I can get some advice on how to access Cloudron’s internal app DB via Remote SQL (like you can with individual apps) so I can debug this issue entirely.

                                                            I want to release the OpenVPN Client on the store in 2021. So I gotta make sure it’s perfect - and your comment just sparked a thought process that very likely could be the problem and I was dismissing it in my head for silly reasons. Thanks. I’ll reproduce it now that I know it’s just me, and see if I can fix it.

                                                            I really just needed one other person on the latest Cloudron perform those steps and have it go perfectly for me to know it was my app (and box changes).

                                                            1 Reply Last reply Reply Quote 0
                                                            • First post
                                                              Last post
                                                            Powered by NodeBB