Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse

Cloudron Forum

Apps | Demo | Docs | Install

Set robots.txt defaults to the following

Scheduled Pinned Locked Moved Feature Requests
robots
3 Posts 2 Posters 221 Views
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • S Offline
    S Offline
    subtlecourage
    wrote on last edited by girish
    #1

    User-agent:ia_archiver
    Disallow: /

    User-agent: archive.org_bot
    Disallow: /

    User-agent: *
    Disallow: /

    User-agent: Rogerbot
    User-agent: Exabot
    User-agent: MJ12bot
    User-agent: Dotbot
    User-agent: Gigabot
    User-agent: Baiduspider
    User-agent: Ezooms
    User-agent: Nutch
    User-agent: archive.org_bot
    User-agent: MJ12bot
    User-agent: YandexBot
    User-agent: AhrefsBot
    User-agent: HTTrack
    User-agent: Wget
    User-agent: Zeus
    User-agent: BLEXBot
    User-agent: burroboot
    User-agent: DOC
    User-agent: MJ12Bot
    User-agent: SemrushBot
    User-agent: spbot
    User-agent: UbiCrawler
    User-agent: Zao
    User-agent: Netsparker
    User-agent: sitecheck.internetseer.com
    User-agent: Zealbot
    User-agent: MSIECrawler
    User-agent: SiteSnagger
    User-agent: WebStripper
    User-agent: WebCopier
    User-agent: Fetch
    User-agent: Offline Explorer
    User-agent: Teleport
    User-agent: TeleportPro
    User-agent: WebZIP
    User-agent: linko
    User-agent: Microsoft.URL.Control
    User-agent: Xenu
    User-agent: larbin
    User-agent: libwww
    User-agent: ZyBORG
    User-agent: Download Ninja
    User-agent: grub-client
    User-agent: k2spider
    User-agent: NPBot
    User-agent: WebReaper
    User-agent: CyotekWebCrawler
    User-agent: Whizbang
    User-agent: UniverseBot
    User-agent: SlySearch
    Disallow: /

    fbartelsF 1 Reply Last reply
    0
  • fbartelsF Offline
    fbartelsF Offline
    fbartels App Dev
    replied to subtlecourage on last edited by
    #2

    Having this as "the default" would probably collide with some users using Cloudron to host public facing websites (myself included).

    Besides:

    @subtlecourage said in Set robots.txt defaults to the following:

    User-agent: *
    Disallow: /

    is already added to robots.txt of the app if you go into the "security" tab of a website and select "Disable indexing". What makes you think that bots that ignore * (meaning all user agents) will not index your website if you name their user agent individually?

    S 1 Reply Last reply
    2
  • S Offline
    S Offline
    subtlecourage
    replied to fbartels on last edited by
    #3

    @fbartels Optimism and hope? /s

    Great points.

    1 Reply Last reply
    2

  • Login

  • Don't have an account? Register

  • Login or register to search.
  • First post
    Last post
0
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Login

  • Don't have an account? Register

  • Login or register to search.