User-Agent: * Disallow: /admin/api Disallow: /pages/footer Disallow: /pages/topbar Disallow: /link/* Disallow: /cdn-cgi/ Disallow: /learn Disallow: /payment Disallow: /error # SEO Fix: block URL parameters from indexing Disallow: /*?msg=* # `/_next/` is deliberately NOT blocked. `/_next/static/` holds the JS chunks, # fonts and logo SVG a renderer needs to see the page as a visitor does, and # Google's guidance is to let it crawl them; hashed asset URLs don't rank on # their own, so there was never anything to protect. It was disallowed here # between 2026-06-26 and 2026-08-13 — see the commit that removed these lines. # AI assistants and answer engines. These already matched the `*` group above — # nothing here newly grants them access — but naming them makes the permission a # deliberate, reviewable decision rather than a side effect of the wildcard, and # it survives a future default-deny (Cloudflare, for one, can block AI crawlers # at the edge for a whole zone). # # IMPORTANT: a crawler that matches a named group ignores the `*` group # completely (RFC 9309 §2.2.1 — most specific match wins, and only that group is # read). So the Disallow list below is a deliberate copy of the one above, not a # leftover. Any path added or removed above must be mirrored here, or these # crawlers will happily fetch what every other bot is blocked from. User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-User User-agent: Claude-SearchBot User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Amazonbot User-agent: Bytespider User-agent: cohere-ai Allow: / Disallow: /admin/api Disallow: /pages/footer Disallow: /pages/topbar Disallow: /link/* Disallow: /cdn-cgi/ Disallow: /learn Disallow: /payment Disallow: /error Disallow: /*?msg=* Sitemap: https://theleaply.com/sitemap.xml