acidburnmonkey
14 lines
203 B
Plaintext
14 lines
203 B
Plaintext
User-agent: ia_archiver
|
|
Disallow: /
|
|
|
|
User-agent: archive.org_bot
|
|
Disallow: /
|
|
|
|
User-agent: Googlebot
|
|
User-agent: AdsBot-Google
|
|
Disallow: /
|
|
|
|
#Example of how to block all crawlers
|
|
User-agent: *
|
|
Disallow: /
|