Skip to content

Instantly share code, notes, and snippets.

@edm00se

edm00se/robots.txt

Last active October 5, 2023 15:08
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save edm00se/0147d7d38fccc8cc938f4e8e798b4927 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save edm00se/0147d7d38fccc8cc938f4e8e798b4927 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. edm00se revised this gist Oct 5, 2023. 1 changed file with 16 additions and 1 deletion.
    17 changes: 16 additions & 1 deletion robots.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -1,2 +1,17 @@
    User-agent: GPTBot

    User-agent: CCBot
    Disallow: /

    User-agent: ChatGPT-User
    Disallow: /

    User-agent: GPTBot
    Disallow: /

    User-agent: Google-Extended
    Disallow: /

    User-agent: Omgilibot
    Disallow: /

    User-Agent: FacebookBot
    Disallow: /
  2. edm00se created this gist Aug 11, 2023.
    2 changes: 2 additions & 0 deletions robots.txt
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,2 @@
    User-agent: GPTBot

    Disallow: /