Why Isn't My Website Showing in Google or AI Search? An Indexing Checklist

Quick answer: Most businesses missing from AI answers have a simpler problem than "AI optimisation": their pages aren't crawled and indexed. Google says a page must be indexed and eligible for a snippet to appear in AI Overviews or AI Mode. Check your key pages in Google Search Console and Bing Webmaster Tools, unblock the right crawlers, submit a sitemap, and fix whatever the indexing reports flag.

Why can't AI search find my website?

When a business owner tells us "ChatGPT never mentions us" or "we're not in Google's AI Overview", the first thing we check isn't content or schema. It's whether the search engines behind those AI tools have found, crawled and stored the pages that matter. That stored copy is called the index.

AI answer engines mostly pull from search indexes, then write an answer and link to the pages they used. If your service, pricing or location page isn't in that index, the AI has nothing of yours to quote.

Google is clear about this. Its July 2026 guide to generative AI features says: "To be eligible to be shown in generative AI features on Google Search, a page must be indexed and eligible to be shown in Google Search with a snippet, fulfilling the Search technical requirements." There is no separate AI index to submit to. The same guide says its AI features are "rooted in our core Search ranking and quality systems."

So before you spend money on "GEO packages", confirm the basics.

Which search indexes do the main AI tools rely on?

Knowing which crawler feeds which AI tool tells you what to check.

OpenAI's crawler documentation is blunt: "Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, though can still appear as navigational links." Perplexity says PerplexityBot "is designed to surface and link websites in search results on Perplexity" and is not used to train AI models. Blocking either one by accident removes you from that tool's answers.

How do I check if my website is indexed?

Use these checks in order, from quickest to most reliable.

Step 1: Do a quick site: search

Type site:yourdomain.com into Google. If your main service pages are missing, you likely have a problem. Treat this as a rough signal only. Google's documentation says the site: operator "doesn't necessarily return all the URLs that are indexed" for a domain.

Step 2: Inspect your key URLs in Google Search Console

Set up Google Search Console (it's free), verify your domain, and paste each important URL into the URL Inspection tool. It shows whether the page is on Google, which canonical Google chose, and whether crawling was allowed. "Test live URL" fetches the page in real time, so you can confirm a fix.

Step 3: Read the Page indexing report

In Search Console, open Indexing > Pages. It lists every reason Google gives for not indexing URLs it knows about. Don't panic at a big "not indexed" number. Google says: "You should not expect all URLs on your site to be indexed, only the canonical pages." What matters is whether your important pages are in the indexed list.

Step 4: Check Bing Webmaster Tools

Verify your site at Bing Webmaster Tools. You can import your Google Search Console properties in a few clicks. Use its URL Inspection feature and Site Explorer to see which pages Bing holds. This is the index Copilot draws from, and many Indian businesses have never looked at it.

Step 5: Check AI crawler access

Open yourdomain.com/robots.txt and look for rules that block OAI-SearchBot, PerplexityBot, Bingbot or Googlebot. Security plugins, CDN "bot fight" settings and copied robots.txt templates often block them without anyone noticing. The free Pagelens AI Crawler Checker reads your robots.txt and shows which AI crawlers are allowed in seconds.

What do the "not indexed" reasons in Search Console mean?

These are the most common reasons on small-business sites, and what to do about each.

One important detail: robots.txt is not a way to hide a page from Google. Google's documentation says it "is not a mechanism for keeping a web page out of Google." Use noindex for pages you want kept out, and never use it on pages you want AI tools to cite.

How do I fix indexing problems, step by step?

Here is the order we work in when a client's key pages are missing. Most small sites can finish this in a week.

1. List your 10 to 20 money pages: home, services, location pages, pricing and your best guides. These must be indexed.
2. Remove accidental blocks. Check for noindex tags, robots.txt disallow rules, password protection on live pages, and CDN or firewall rules that challenge bots.
3. Make sure each page returns a 200 status and loads reliably. Google's minimum technical requirements are simple: Googlebot isn't blocked, the page works (HTTP 200), and it has indexable content.
4. Put important text in the HTML. If your prices, services or FAQs only appear after JavaScript runs, many crawlers won't see them (more on this below).
5. Submit an XML sitemap in both Search Console and Bing Webmaster Tools, and add a Sitemap: line to robots.txt. Google says it uses <lastmod> dates only if they are "consistently and verifiably accurate", and ignores <priority> and <changefreq>. So keep lastmod honest.
6. Link to key pages from your menu and from related pages. Pages with no internal links (orphan pages) are slow to be discovered.
7. Request indexing for each fixed money page with the URL Inspection tool. Google notes there is a daily limit and that a request "does not guarantee" indexing, so save it for the pages that matter.
8. Turn on IndexNow if your CMS supports it. It tells Bing and other participating engines about new or updated pages instantly. Google doesn't use it.
9. Improve thin or duplicate pages. Ten city pages with identical text invite "crawled, currently not indexed". Add real local detail.
10. Re-check in two weeks. Google says indexing typically takes "a day or so" after a request, but can take longer.

Can AI crawlers read JavaScript-heavy websites?

Google can. It crawls, renders and then indexes JavaScript pages, though rendering can be delayed. Many AI crawlers are less capable. A 2024 analysis of crawler traffic across Vercel's network found that "none of the major AI crawlers currently render JavaScript", including OpenAI's crawlers, Anthropic's ClaudeBot and PerplexityBot. Crawlers change, so re-test your own site, but the lesson holds.

This matters for Indian startups and D2C brands with single-page apps built in React or similar. If the raw HTML is nearly empty until scripts run, an AI crawler may see a blank page. Google itself says "server-side or pre-rendering is still a great idea because it makes your website faster for users and crawlers."

Quick test: open "View page source" (not Inspect) and search for a sentence from your main content. If it's missing, ask your developer about server-side rendering or pre-rendering.

How do I know if AI tools are now showing my pages?

For the first time, the search engines themselves now report on AI visibility.

• Google Search Console: Google announced Search Generative AI performance reports on June 3, 2026, and says that as of August 31, 2026 it has rolled them out to all websites worldwide. They show impressions from AI Overviews and AI Mode by page, country and date. Click data isn't included yet, and sites with very few AI impressions may not see the report.
• Bing Webmaster Tools: The AI Performance report, launched in public preview in February 2026, shows how often your pages are cited in Microsoft Copilot and Bing's AI summaries, which pages are cited, and the "grounding queries" the AI used to find them.
• ChatGPT, Perplexity and Gemini: These don't offer site-owner reports, so you need to test the questions your buyers ask. The free Pagelens AI visibility audit needs no signup: enter your domain and it runs buyer-style prompts across ChatGPT, Claude, Gemini and Perplexity, then shows your visibility score and which competitors are being cited instead. For single engines, see the free checkers on Pagelens tools.

What results can you expect once indexing is fixed?

Indexing is the entry ticket, not the whole game. AI tools still choose the clearest, most trustworthy answer. Our guide on how to get your business cited in AI search covers that next layer.

Here's what the full process has delivered for two OffBuddy clients, measured as the share of tracked buyer questions where the AI named the brand:

In both cases the work started with the basics in this guide, fixing crawler access and indexing, before content and authority work. Results depend on your category and competition, so treat these as examples, not promises.

An illustrative example (not a client): a Pune interior studio spends ₹40,000 on a new website, but the "discourage search engines" setting stays on after launch. No blog or Instagram effort can get it into Google or AI answers until that one setting is fixed.

What indexing mistakes should Indian businesses avoid?

• Launching a new site and leaving the staging noindex or "discourage search engines" setting on
• Copying a robots.txt from a template that blocks AI crawlers you actually want
• Creating dozens of near-identical city pages with no real local detail
• Redesigning the site and changing URLs without 301 redirects, so old indexed pages turn into 404s

If you're weighing what to spend on fixing this, our guide to what SEO costs in India explains typical price ranges.

Sources

1. Google Search Central: Optimizing for generative AI features in Google Search (updated July 2026)
2. Search Console Help: Generative AI performance report (Search)
3. Google Search Central Blog: Introducing Search Generative AI performance reports in Search Console (June 2026)
4. Bing Webmaster Blog: Introducing AI Performance in Bing Webmaster Tools (February 2026)
5. Google Search Central: AI features and your website
6. Google Search Central: Google Search technical requirements
7. Search Console Help: Page indexing report
8. Search Console Help: URL Inspection tool
9. Google Search Central: site: search operator
10. Google Search Central: Introduction to robots.txt
11. Google Search Central: Build and submit a sitemap
12. Google Search Central: Understand JavaScript SEO basics
13. OpenAI: Overview of OpenAI crawlers
14. Perplexity: Perplexity crawlers
15. IndexNow: Frequently asked questions
16. Vercel: The rise of the AI crawler (December 2024)

Written by Surender K, Co-Founder, OffBuddy

Surender K is Co-Founder of OffBuddy, a Bengaluru marketing and tech agency. He works on SEO, Google Ads and Meta Ads campaigns and the websites they point to, for businesses in India and the US.

Related reading

• How to Get Your Business Cited in AI Search Results (AEO Guide)
• How Much Does SEO Cost in India? (2026 Pricing Guide)
• OffBuddy SEO services