Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


    Cloudron Forum

    • Register
    • Login
    • Search
    • Categories
    • Recent
    • Tags
    • Popular

    Solved Unable to enable robots.txt crawling

    Support
    2
    4
    202
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • robi
      robi last edited by

      First, I was fighting Surfer which kept changing the contents of what I put into robots.txt and it oddly replaces it with the App configuration security setting of no crawling.

      Then after changing that, I can set the file to Allow crawling, but it's still not able to crawl (according to google search console.

      Also: the "Disable indexing" link should perhaps be a toggle, that enables and disables vs just disables.

      Life of Advanced Technology

      robi 1 Reply Last reply Reply Quote 1
      • robi
        robi @robi last edited by robi

        @robi turns out while working on another app with the security settings unmodified it became clear that it needs to be blank to enable robots to crawl, vs changing it with the Allow commands.

        Totally not obvious, even for me.

        Life of Advanced Technology

        nebulon 1 Reply Last reply Reply Quote 1
        • nebulon
          nebulon Staff @robi last edited by

          @robi if the text input is empty, Cloudron's reverse proxy will not respond to the robots.txt request but will forward it to the app, so the app can control it. Is it possible that the app instance in your case also had a robots.txt which made this confusing?

          Also do you have suggestions for how to improve the docs then at https://docs.cloudron.io/apps/#robotstxt ?

          robi 1 Reply Last reply Reply Quote 0
          • robi
            robi @nebulon last edited by

            @nebulon Yes, it took a while to figure that out.

            The UI is the confusing part, since if you see that screen the first time and it is populated, you have no indication to clear it, or button to turn it off. The logical part is to change what it says, for ex Disallow to Allow, but that doesn't work either 😢

            So a UX improvement would be to have a toggle, which only displays the robots.txt proxy content if it's enabled and while off gives a hint as to what turning it on will actually do.

            That way the breadcrumbs lead to the loaf of understanding and satisfaction 😋 🍞

            Life of Advanced Technology

            1 Reply Last reply Reply Quote 1
            • First post
              Last post
            Powered by NodeBB