# CreatorFlow Robots.txt # Last updated: 2026-10-31 # Default rules for all crawlers # No Content-Signal line here on purpose: Lighthouse reads it as an unknown directive and fails this file # ("robots.txt is not valid", SEO score 92, measured 2026-10-31). All three signals would say yes anyway. User-agent: * Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /auth/ Disallow: /_next/ Disallow: /dashboard/ Disallow: /blog/search.json # /*-scheduler, /[xx/]channels and /request-my-data[-new] are deliberately NOT # disallowed (removed 2026-09-29). They return 410 Gone (/request-my-data 301s to # /privacy). A robots block hides the 410 or 301 from Google, so the URLs stay # indexed. Let Google crawl them and drop them. # Query-parameter duplicate URLs (?lang=, ?ref=, ?from=, ?template=, ?plan=, # ?returnUrl=) are all handled by 301 redirects to the clean canonical in # middleware. We deliberately DO NOT Disallow them — same lesson as the # double-locale block below: a robots block HIDES the 301 from Google, which # freezes the URLs in GSC as "Alternate page with proper canonical tag" / # "Blocked by robots.txt" forever (validation can never pass because Google # can't re-crawl to see the redirect). Letting Google crawl them lets it follow # the 301, drop the duplicate, and pass any backlink equity (e.g. ?ref=producthunt). # Re-add a block only once they have fully dropped out of the index. # # Pagination/category params stay blocked (high-volume, low-value crawl paths): Disallow: /*?category= Disallow: /*?page= # /login and /signup are deliberately NOT disallowed (removed 2026-09-28). /login # and /xx/login serve `noindex, follow`; /login?, /xx-xx/login and # /account 301 to /login; /signup and /xx/signup 301 to /pricing. The robots block # hid all of that from Google, which left /login + /login?plan=* stuck as "Indexed, # though blocked by robots.txt" and ~200 login/signup URLs as "Blocked by # robots.txt". Letting Google crawl them lets it read the noindex and follow the # 301s. Do not re-add the block: it cannot remove a URL from the index, only freeze it. # Double-locale crawler artifacts (e.g., /es/pt/vs/x, /it/de/for/y) return 410 Gone # in middleware. We deliberately DO NOT Disallow them: a robots block hides the 410 # from Google, which kept ~167 of these stuck as "Indexed, though blocked by robots.txt". # Letting Google crawl them lets it see the 410 and deindex. (Re-add a block only once # they have fully dropped out of the index.) # AI crawlers are welcome. This group names them so the policy is explicit. # # Keep its rules the same as the `*` group. Under RFC 9309 a crawler obeys ONLY # the most specific group that matches its name and does NOT inherit from `*`. # 18 single-line groups that used to sit here held `Allow: /` and no `Disallow`, # which silently voided every Disallow above for those agents and opened /admin/, # /api/, /auth/ and /dashboard/ to them. A group is all-or-nothing. # audit-reports/seo-en-2026-10-31/check-robots.js fails if the two groups differ. User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: anthropic-ai User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Applebot-Extended User-agent: Amazonbot User-agent: Meta-ExternalAgent User-agent: Meta-ExternalFetcher User-agent: DuckAssistBot User-agent: MistralAI-User User-agent: cohere-ai User-agent: CCBot Allow: / Disallow: /admin/ Disallow: /api/ Disallow: /auth/ Disallow: /_next/ Disallow: /dashboard/ Disallow: /blog/search.json Disallow: /*?category= Disallow: /*?page= # LLM context files # See https://llmstxt.org/ for specification # llms.txt: https://creatorflow.so/llms.txt # Sitemap location - index + individual sitemaps for direct discovery Sitemap: https://creatorflow.so/sitemap.xml Sitemap: https://creatorflow.so/sitemap/navigation.xml Sitemap: https://creatorflow.so/sitemap/templates.xml Sitemap: https://creatorflow.so/sitemap/tools.xml Sitemap: https://creatorflow.so/sitemap/glossary.xml Sitemap: https://creatorflow.so/sitemap/guides.xml Sitemap: https://creatorflow.so/sitemap/examples.xml Sitemap: https://creatorflow.so/sitemap/help.xml # Blog (separate static Astro site, generates its own sitemap index) Sitemap: https://creatorflow.so/blog/sitemap-index.xml # IndexNow # Key: 26e8ba4bb06f4d2f81a71b5bcf704df2 # Verification: https://creatorflow.so/26e8ba4bb06f4d2f81a71b5bcf704df2.txt