AI Technical SEO: What Your Site Needs for AI Search, Straight From the Docs

For Google, barely: its AI features guidance says a page only has to be indexed and eligible to show with a snippet to appear as a supporting link in AI Overviews or AI Mode. The real difference sits outside Google. Several AI companies run separate crawlers under separate robots.txt tokens, and each must reach your pages.

Thirteen tokens from six operators are mapped below: Google, OpenAI, Anthropic, Perplexity, Apple and Common Crawl. They fall into three jobs. Search crawlers decide whether a page can be cited, training tokens decide whether content may feed future models, and user-triggered fetchers act for one person's question, sometimes outside robots.txt.

As of October 28, 2026, portlandpeakseo.com/robots.txt names no AI crawler at all. It holds two User-agent: * groups, which RFC 9309 says must combine into one. Blocked paths are limited to /wp-admin/, with admin-ajax.php reopened, three WooCommerce upload folders and add-to-cart query URLs, so every token above may crawl public pages.

Google documents a two-step test: a reverse DNS lookup on the logged IP must return googlebot.com, google.com or googleusercontent.com, and a forward lookup on that name must return the same IP. Apple and Common Crawl support reverse DNS too, while OpenAI, Anthropic and Perplexity publish JSON lists of their address ranges.

Google processes JavaScript pages in three phases, crawling, rendering and indexing, and says a page may sit in the render queue for a few seconds or longer. The same page notes that not all bots can run JavaScript. The durable build puts key text, links and honest status codes in the first HTML response.

Google sets three targets: the main content painting within 2.5 seconds (LCP), responses to input under 200 milliseconds (INP) and layout shift under 0.1 (CLS). web.dev supplies the test that decides it: a page passes when all three hold at the 75th percentile of page loads, measured separately on mobile and desktop.

Google says no special schema.org structured data is needed for AI Overviews or AI Mode. Markup still answers to its general structured data policies: describe only content readers can see, keep it current, and do not expect a rich result simply because the Rich Results Test passes. Valid markup earns eligibility, not a promised display.

Google names four: nosnippet, data-nosnippet, max-snippet and noindex. Its robots meta specification says nosnippet reaches AI Overviews and AI Mode and also stops the page serving as a direct input to them. data-nosnippet is valid only on span, div and section elements, which suits one passage rather than a whole page.

AI tools earn their keep on four sorting jobs: grouping crawl exports, summarizing server logs, drafting redirect maps and drafting JSON-LD. Each turns thousands of rows into a short list a person can test. The detection itself still comes from a crawler, a log file or Search Console; the model organizes what those sources found.

Google's URL Inspection tool, not a language model, shows the HTML Googlebot actually received, and that gap sets the limit. AI software works only on the data it is given. It cannot see firewall rules, field speed data or rendered output it never fetched, and it can draft a robots.txt rule that blocks more than intended.

Run a five-step loop: export the evidence, let the model sort it, restate each finding as a testable claim, confirm it on the live site or in Search Console, then fix and re-check. Because the URL Inspection API allows 2,000 calls per site per day, large sites need a sampling plan for the confirm step.

Google counts them in overall search traffic, reported in the Performance report under the Web search type. AI feature clicks and impressions therefore blend into the same totals as classic results. Track indexed pages, crawl errors and field Core Web Vitals against a fixed baseline, and question any tool claiming to isolate AI clicks from Search Console alone.

Technical work sits inside all three monthly plans: $450 Local, $950 Growth and $2,500 Authority, each month to month with no setup fee or notice period. The plan descriptions list technical foundations on Local, technical repair on Growth and a full AI search visibility program on Authority, all priced on one public page.

Five companion pages split the wider subject so this one can stay on crawl, render, index and audit work. The basic definition, answer formatting, generative engine citations, Perplexity and tool comparisons each have their own page, listed below with technical and background reading.

Four Google reviews, averaging 5.0, sat on the Portland Peak SEO Business Profile at the October 23, 2026 count. Behind the quotes: Michael Richardson, posting for the construction and environmental services business MGI, and Jacob Levin, an attorney practicing in Redding, California, under the name Jacob Levin Law. Neither review is about technical audits specifically.