initial commit done , pre peoduction

2025-08-06 01:03:53 -04:00
commit aa8ae37799
18 changed files with 607 additions and 0 deletions
@@ -0,0 +1,13 @@
+User-agent: ia_archiver
+Disallow: /
+
+User-agent: archive.org_bot
+Disallow: /
+
+User-agent: Googlebot
+User-agent: AdsBot-Google
+Disallow: /
+
+#Example of how to block all crawlers
+User-agent: *
+Disallow: /