Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


Skip to content
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Search
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse
Brand Logo

Cloudron Forum

Apps | Demo | Docs | Install
  1. Cloudron Forum
  2. Support
  3. Unable to enable robots.txt crawling

Unable to enable robots.txt crawling

Scheduled Pinned Locked Moved Solved Support
4 Posts 2 Posters 782 Views 2 Watching
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • robiR Offline
    robiR Offline
    robi
    wrote on last edited by
    #1

    First, I was fighting Surfer which kept changing the contents of what I put into robots.txt and it oddly replaces it with the App configuration security setting of no crawling.

    Then after changing that, I can set the file to Allow crawling, but it's still not able to crawl (according to google search console.

    Also: the "Disable indexing" link should perhaps be a toggle, that enables and disables vs just disables.

    Conscious tech

    robiR 1 Reply Last reply
    1
    • robiR robi

      First, I was fighting Surfer which kept changing the contents of what I put into robots.txt and it oddly replaces it with the App configuration security setting of no crawling.

      Then after changing that, I can set the file to Allow crawling, but it's still not able to crawl (according to google search console.

      Also: the "Disable indexing" link should perhaps be a toggle, that enables and disables vs just disables.

      robiR Offline
      robiR Offline
      robi
      wrote on last edited by robi
      #2

      @robi turns out while working on another app with the security settings unmodified it became clear that it needs to be blank to enable robots to crawl, vs changing it with the Allow commands.

      Totally not obvious, even for me.

      Conscious tech

      nebulonN 1 Reply Last reply
      1
      • robiR robi

        @robi turns out while working on another app with the security settings unmodified it became clear that it needs to be blank to enable robots to crawl, vs changing it with the Allow commands.

        Totally not obvious, even for me.

        nebulonN Offline
        nebulonN Offline
        nebulon
        Staff
        wrote on last edited by
        #3

        @robi if the text input is empty, Cloudron's reverse proxy will not respond to the robots.txt request but will forward it to the app, so the app can control it. Is it possible that the app instance in your case also had a robots.txt which made this confusing?

        Also do you have suggestions for how to improve the docs then at https://docs.cloudron.io/apps/#robotstxt ?

        robiR 1 Reply Last reply
        0
        • nebulonN nebulon

          @robi if the text input is empty, Cloudron's reverse proxy will not respond to the robots.txt request but will forward it to the app, so the app can control it. Is it possible that the app instance in your case also had a robots.txt which made this confusing?

          Also do you have suggestions for how to improve the docs then at https://docs.cloudron.io/apps/#robotstxt ?

          robiR Offline
          robiR Offline
          robi
          wrote on last edited by
          #4

          @nebulon Yes, it took a while to figure that out.

          The UI is the confusing part, since if you see that screen the first time and it is populated, you have no indication to clear it, or button to turn it off. The logical part is to change what it says, for ex Disallow to Allow, but that doesn't work either 😢

          So a UX improvement would be to have a toggle, which only displays the robots.txt proxy content if it's enabled and while off gives a hint as to what turning it on will actually do.

          That way the breadcrumbs lead to the loaf of understanding and satisfaction 😋 🍞

          Conscious tech

          1 Reply Last reply
          1
          Reply
          • Reply as topic
          Log in to reply
          • Oldest to Newest
          • Newest to Oldest
          • Most Votes


          • Login

          • Don't have an account? Register

          • Login or register to search.
          • First post
            Last post
          0
          • Categories
          • Recent
          • Tags
          • Popular
          • Bookmarks
          • Search