####################### # 🤖 ALLOWED AI CRAWLERS ####################### # GPTBot – OpenAI's ChatGPT crawler User-agent: GPTBot Allow: / Crawl-delay: 10 # https://openai.com/gptbot # ChatGPT-User – Follows links clicked by ChatGPT users User-agent: ChatGPT-User Allow: / # ClaudeBot – Anthropic's Claude crawler User-agent: ClaudeBot Allow: / Crawl-delay: 10 # https://www.anthropic.com/claudebot # PerplexityBot – AI search engine crawler User-agent: PerplexityBot Allow: / Crawl-delay: 10 # https://www.perplexity.ai/robots.txt ####################### # ❌ BLOCKED IMAGE / AI SCRAPERS ####################### # Pinterest – Image scraper User-agent: Pinterestbot Disallow: / # Meta (Facebook / Instagram) User-agent: facebookexternalhit Disallow: / # Meta AI User-agent: MetaAI Disallow: / # TikTok / Douyin AI User-agent: Bytespider Disallow: / ####################### # ❌ BLOCKED AI ARCHIVERS / CRAWLERS ####################### # Common Crawl – massive open dataset for AI training User-agent: CCBot Disallow: / # ArchiveBox – self-hosted web archiver User-agent: ArchiveBox Disallow: / # Wayback Machine – Internet Archive User-agent: ia_archiver Disallow: / ####################### # 🔐 SHOPIFY PLATFORM PROTECTIONS ####################### User-agent: * Disallow: /admin Disallow: /cart Disallow: /orders Disallow: /checkouts/ Disallow: /checkout Disallow: /account ####################### # 🔗 SITEMAP FOR SEO ####################### Sitemap: https://tekaoriginals.com/sitemap.xml