Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse

Cloudron Forum

Apps | Demo | Docs | Install

Unable to enable robots.txt crawling

Scheduled Pinned Locked Moved Solved Support
4 Posts 2 Posters 247 Views
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • robiR Offline
    robiR Offline
    robi
    wrote on last edited by
    #1

    First, I was fighting Surfer which kept changing the contents of what I put into robots.txt and it oddly replaces it with the App configuration security setting of no crawling.

    Then after changing that, I can set the file to Allow crawling, but it's still not able to crawl (according to google search console.

    Also: the "Disable indexing" link should perhaps be a toggle, that enables and disables vs just disables.

    Life of sky tech

    robiR 1 Reply Last reply
    1
  • robiR Offline
    robiR Offline
    robi
    replied to robi on last edited by robi
    #2

    @robi turns out while working on another app with the security settings unmodified it became clear that it needs to be blank to enable robots to crawl, vs changing it with the Allow commands.

    Totally not obvious, even for me.

    Life of sky tech

    nebulonN 1 Reply Last reply
    1
  • nebulonN Offline
    nebulonN Offline
    nebulon Staff
    replied to robi on last edited by
    #3

    @robi if the text input is empty, Cloudron's reverse proxy will not respond to the robots.txt request but will forward it to the app, so the app can control it. Is it possible that the app instance in your case also had a robots.txt which made this confusing?

    Also do you have suggestions for how to improve the docs then at https://docs.cloudron.io/apps/#robotstxt ?

    robiR 1 Reply Last reply
    0
  • robiR Offline
    robiR Offline
    robi
    replied to nebulon on last edited by
    #4

    @nebulon Yes, it took a while to figure that out.

    The UI is the confusing part, since if you see that screen the first time and it is populated, you have no indication to clear it, or button to turn it off. The logical part is to change what it says, for ex Disallow to Allow, but that doesn't work either 😢

    So a UX improvement would be to have a toggle, which only displays the robots.txt proxy content if it's enabled and while off gives a hint as to what turning it on will actually do.

    That way the breadcrumbs lead to the loaf of understanding and satisfaction 😋 🍞

    Life of sky tech

    1 Reply Last reply
    1

  • Login

  • Don't have an account? Register

  • Login or register to search.
  • First post
    Last post
0
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Login

  • Don't have an account? Register

  • Login or register to search.