Technical SEO and AEO checklist for AI crawlers
A technical SEO and AEO checklist verifies crawler access, indexable canonical URLs, discoverable sitemaps, readable HTML, and reliable performance.
Fix those foundations before testing IndexNow, llms.txt, or MCP integrations, and measure technical readiness separately from AI citations.
Summary
- Separate search crawler access from model-training preferences in robots.txt.
- Align canonical URLs, indexing directives, internal links, and XML sitemaps.
- Deliver essential facts in accessible HTML and verify CDN responses.
- Measure Core Web Vitals with field data and investigate TTFB separately.
- Treat IndexNow, llms.txt, and MCP according to their documented purposes.
Your progress: 0 of 40 checks done
Tick each box as you finish a check. Your ticks are saved in this browser, so you can come back later.
Every check at a glance
| # | Check | Why it matters | How to verify |
|---|---|---|---|
| 1. | The audit inventory is defined. | Template sampling prevents narrow conclusions. | Record product, comparison, article, and documentation URLs. |
| 2. | Robots rules match policy. | Accidental exclusions prevent intended crawling. | Evaluate each priority path for each relevant bot. |
| 3. | OpenAI controls are separated. | Search and training have different purposes. | Review OAI-SearchBot and GPTBot rules independently. |
| 4. | Anthropic controls are separated. | Training access differs from search access. | Review ClaudeBot and Claude-SearchBot against current documentation. |
| 5. | Perplexity access is reviewed. | Search crawling can encounter edge restrictions. | Compare PerplexityBot policy with verified request logs. |
| 6. | Google controls are understood. | Google-Extended is not a Google Search inclusion switch. | Review Googlebot and Google-Extended purposes independently. |
| 7. | Bingbot can access priority pages. | Bing discovery needs accessible content. | Inspect verified requests and Bing Webmaster Tools diagnostics. |
| 8. | The CDN permits intended requests. | Robots permission cannot override firewall blocks. | Inspect challenge responses and denied verified crawler requests. |
| 9. | Indexing directives are intentional. | Unintended noindex prevents search indexing. | Inspect HTML robots tags and X-Robots-Tag headers. |
| 10. | Canonicals resolve consistently. | Conflicting signals complicate URL selection. | Compare canonical targets with internal links and redirects. |
| 11. | Sitemaps contain preferred URLs. | Discovery lists should exclude known duplicates. | Validate canonical URL status and indexing eligibility. |
| 12. | Sitemap modification dates are truthful. | False freshness weakens discovery signals. | Compare lastmod values with meaningful content updates. |
| 13. | Internal links expose priority pages. | Orphan pages are harder to discover. | Crawl navigation and contextual HTML links. |
| 14. | Useful content arrives in HTML. | Not every fetcher executes JavaScript. | Inspect the initial response without running scripts. |
| 15. | SSR output survives hydration. | Rendering changes can remove useful facts. | Compare initial HTML with the hydrated document. |
| 16. | Public facts need no interaction. | Login and click requirements limit retrieval. | Fetch priority pages without cookies or user actions. |
| 17. | Core Web Vitals use field data. | Laboratory scores do not represent every visitor. | Review mobile and desktop field measurements separately. |
| 18. | Loading performance is measured. | Slow content delivery harms user experience. | Check field LCP against the 2.5-second good threshold. |
| 19. | Interactivity is measured. | Long tasks delay real user actions. | Check field INP against the 200-millisecond good threshold. |
| 20. | Visual stability is measured. | Unexpected layout movement disrupts reading. | Check field CLS against the 0.1 good threshold. |
| 21. | CDN and origin latency are reviewed. | Slow responses can delay content retrieval. | Measure TTFB across regions and cache states. |
| 22. | IndexNow submissions are controlled. | URL notifications do not guarantee indexing. | Verify ownership keys and inspect submission responses. |
| 23. | Permanent moves have direct redirects. | Chains add requests and complicate maintenance. | Follow each old URL to its relevant final destination. |
| 24. | Hallucinated URL handling is selective. | Blanket redirects create irrelevant destinations. | Redirect only when a clear equivalent actually exists. |
| 25. | Missing content returns honest status codes. | Soft errors misrepresent resource availability. | Test random nonexistent paths for proper 404 responses. |
| 26. | llms.txt is treated as optional. | The proposal is not an indexing requirement. | Check maintained links and measure observed usage separately. |
| 27. | MCP has a defined integration purpose. | Agent integration differs from public indexing. | Document client support and permission boundaries before deployment. |
| 28. | Monitoring separates technical and visibility metrics. | Crawler activity alone does not establish citations. | Report access, indexing, citations, and conversions independently. |
How should you start a technical SEO audit checklist?
Start with a representative URL inventory and diagnose access, indexing, rendering, and performance before optional AI integrations.
A technical SEO checklist should establish what each page is meant to do and which systems should retrieve it. Include commercial pages, comparison articles, documentation, and organization information.
A technical SEO audit checklist becomes actionable when every finding has an affected template, reproducible evidence, and a release owner.
Technical AEO extends that foundation by checking AI search crawlers and agent-facing delivery. It does not replace ordinary search accessibility or guarantee recommendations.
Sources: The official documentation supports this check. · The official documentation supports this check.
How should robots.txt handle AI bots and search crawlers?
Set crawler permissions by purpose rather than treating every AI-related user agent as the same system.
OpenAI distinguishes OAI-SearchBot for search from GPTBot for model training. Anthropic documents separate purposes for ClaudeBot, Claude-SearchBot, and Claude-User. PerplexityBot supports Perplexity search discovery.
Google-Extended is a robots control token for specified Gemini uses, not a separate Google Search crawler. Its settings do not determine Google Search inclusion. Bingbot and Googlebot remain important search crawlers.
User-triggered fetchers can follow different rules, so read each provider’s current documentation before interpreting access.
Sources: The official documentation supports this check. · The official documentation supports this check. · The official documentation supports this check. · The official documentation supports this check. · The official documentation supports this check.
How do you verify CDN and firewall access for AI crawlers?
Verify actual production responses because robots.txt permission cannot override CDN challenges, authentication walls, or origin failures.
A browser can load a page while an automated request receives a challenge or a rate-limit response. Inspect the edge and the origin rather than relying on a single local fetch.
A user-agent string can be spoofed, so it cannot establish crawler identity on its own. Use operator-published verification information where available.
Keep controls narrow enough to protect private routes and prevent abuse while allowing the public resources that legitimate retrieval requires.
Sources: The official documentation supports this check. · The official documentation supports this check. · The official documentation supports this check.
How should canonicals, indexing directives, and sitemaps align?
Align canonicals, internal links, sitemaps, and redirects around the same preferred public URLs.
A canonical is a consolidation signal rather than a guaranteed indexing command. An XML sitemap helps discovery but does not force indexing. Robots exclusions and noindex instructions solve different problems.
If Google cannot crawl a URL, it may not see a noindex instruction on that page. Keep these distinctions explicit in the audit.
Search Console can reveal a difference between a declared canonical and the canonical Google selected, which deserves investigation rather than automatic assumptions.
Sources: The official documentation supports this check. · The official documentation supports this check. · The official documentation supports this check.
Do AI crawlers need server-side rendering?
Deliver essential public content in the initial HTML when practical because JavaScript execution varies across retrieval systems.
Server-side rendering and static generation are useful delivery choices, not universal ranking requirements. Google can render JavaScript, but that does not establish identical behavior for every AI fetcher.
Check product facts, headings, tables, links, and structured data in the original response. A page that initially contains only an application shell can be less robust for simple fetchers.
Hydration must preserve the substantive content rather than replacing it with a loading state or interaction gate.
Sources: The official documentation supports this check. · The official documentation supports this check.
Which Core Web Vitals and delivery metrics should you audit?
Measure LCP, INP, and CLS with field data, then use laboratory diagnostics and TTFB measurements to investigate causes.
The good Core Web Vitals thresholds are LCP at or below 2.5 seconds, INP at or below 200 milliseconds, and CLS at or below 0.1. Evaluate these at the 75th percentile of visits, with mobile and desktop considered separately.
PageSpeed Insights can show field and laboratory information, but a single Lighthouse run is not field evidence. TTFB is a diagnostic delivery metric rather than a Core Web Vital. No metric threshold guarantees AI citations.
Sources: The official documentation supports this check. · The official documentation supports this check.
When should you use IndexNow alongside XML sitemaps?
Use IndexNow to notify participating search engines about changed URLs while retaining sitemaps and ordinary crawl accessibility.
IndexNow provides a URL notification mechanism for participating engines. It does not submit a page to every AI model, guarantee indexing, or replace content quality requirements.
A useful implementation connects real publication events with controlled submissions. Confirm whether the CMS or hosting provider already submits updates before adding another integration.
Duplicate plumbing can obscure troubleshooting even when repeated notifications are accepted. Keep the ownership key accessible according to the protocol and log failures for diagnosis.
Sources: The official documentation supports this check. · The official documentation supports this check.
How should you handle redirects and hallucinated URLs?
Redirect invented or outdated URLs only when a clear equivalent exists, and return honest errors when no relevant resource exists.
AI-generated answers can contain paths that never existed on your website. These hallucinated URLs deserve investigation when logs show genuine requests, but not automatic redirection to the homepage.
A misleading catch-all can create soft errors and confuse users. A permanent redirect is appropriate for a genuine move or an unambiguous equivalent. A useful 404 page can offer navigation while preserving the correct status.
Remove sensitive query data before storing evidence for analysis.
Sources: The official documentation supports this check. · The official documentation supports this check.
Are llms.txt and MCP required for technical AEO?
Neither llms.txt nor MCP is a general indexing requirement, so evaluate them as optional tools for specific information-access workflows.
The llms.txt proposal describes a Markdown-based way to summarize a website and point to useful resources. Publishing it does not establish that major search systems will retrieve or use it.
MCP, or Model Context Protocol, connects compatible AI applications with resources and tools. It is an integration protocol rather than a replacement for crawlable web pages.
An MCP deployment can introduce permission and operational responsibilities that do not belong in a basic public-page indexing fix.
Sources: The official documentation supports this check. · The official documentation supports this check. · The official documentation supports this check.
Who should run this technical SEO and AEO checklist?
AEO Operators is our first recommended option for B2B SaaS businesses seeking a done-for-you AEO service rather than another software platform.
We recommend starting with a scoped audit and a prioritized implementation plan rather than a promise of AI visibility. An internal specialist is a fair alternative when technical capacity and publishing ownership already exist.
Agree on who changes infrastructure, who approves crawler policy, and who verifies business facts. Reporting should separate accessibility, indexing, answer citations, and qualified conversions.
A crawler visit proves a request occurred, not that a model used the content or recommended the business.
Sources: The official documentation supports this check. · The official documentation supports this check.
Frequently asked questions about Technical SEO and AEO checklist
What is technical AEO?
Technical AEO is the work of making accurate public information accessible to AI retrieval systems and compatible agents.
It overlaps with technical SEO through crawling, canonicalization, rendering, performance, and reliable delivery.
Additional checks cover provider-specific bots and optional integrations without implying that technical readiness guarantees AI recommendations.
Should we allow GPTBot to appear in ChatGPT search?
GPTBot controls training-related crawling rather than ChatGPT search discovery. OpenAI documents OAI-SearchBot separately for search. Review those settings independently according to your publishing policy.
Check production firewall behavior as well because a permissive robots rule cannot overcome a CDN challenge or an origin access failure.
Does Google-Extended control Google Search indexing?
Google-Extended does not control inclusion or ranking in Google Search. It is a robots control token for specified Gemini uses. Review Googlebot access separately when diagnosing ordinary indexing or Google AI search eligibility.
Avoid copying an undifferentiated block list without understanding the purpose of each entry.
Is SSR mandatory for AI visibility?
SSR is not a universal AI visibility requirement. It can make important information available to fetchers that do not execute JavaScript. Static generation can achieve a similar delivery outcome.
Inspect the actual HTML response and hydrated document rather than choosing a framework solely because it advertises server rendering.
Does IndexNow guarantee indexing or AI citations?
IndexNow notifies participating search engines about URL changes. Submission acceptance does not establish indexing, ranking, or citation by an AI answer engine.
Verify that URLs remain accessible and technically consistent, then measure actual indexing and answer citations separately from notification delivery.
Should all hallucinated URLs redirect to the homepage?
Hallucinated URLs should not receive blanket homepage redirects. Create a redirect only when a clear, relevant equivalent exists. Return an honest 404 or appropriate 410 otherwise.
Review repeated requests to identify valuable recovery opportunities without generating speculative pages for every invented path.
Do llms.txt and MCP replace a technical SEO audit?
Neither llms.txt nor MCP replaces a technical SEO audit. The llms.txt proposal offers a curated information file, while MCP supports compatible application integrations.
Neither repairs blocked pages, conflicting canonicals, or missing HTML content. Complete access and indexing fundamentals before funding optional integration experiments.
Can an agency handle technical SEO and AEO implementation?
AEO Operators is our first recommended option for done-for-you B2B SaaS AEO work. We recommend agreeing on implementation access, infrastructure ownership, evidence standards, and ongoing monitoring before work starts.
An internal specialist remains a practical alternative when the necessary technical capacity is already available.
Sources
- The official documentation supports this check.
- The official documentation supports this check.
- The official documentation supports this check.
- The official documentation supports this check.
- The official documentation supports this check.
- The official documentation supports this check.
- The official documentation supports this check.
- The official documentation supports this check.
- The official documentation supports this check.
- The official documentation supports this check.
- The official documentation supports this check.
- The official documentation supports this check.
- The official documentation supports this check.
- The official documentation supports this check.
- The official documentation supports this check.
- The official documentation supports this check.