Technical SEO: How to Check Whether Google Can Crawl, Render, and Index a Page

If a page is missing from Google, start with its exact URL—not a site-wide checklist. A missing internal link, a crawl block, a redirect, and a noindex directive call for different fixes. Work through discovery, crawling, rendering, indexing, and search appearance separately. Google describes crawl, index, and serving as distinct parts of its Search pipeline (Google Search Central).

What technical SEO checks

Technical SEO asks whether search engines can find and process a page: its links, crawl instructions, server response, rendered content, and indexing signals. On-page SEO concerns the page’s content and HTML elements, such as headings and title tags. Off-page SEO concerns signals from elsewhere, such as links and mentions (HubSpot’s technical SEO guide). A page can pass the access checks without Google choosing to index or prominently serve it.

Start with one URL and one symptom

Suppose https://example.com/guides/crawl-basics/ does not appear where you expect in Google. Record that exact URL, then inspect it in Google Search Console’s URL Inspection tool. Ask which question the available observations can answer:

• Discovery: Is there an internal link or sitemap entry pointing to it?
• Crawling: Does Google report visiting it, and is crawling allowed?
• Rendering: Can Google access the page and the resources needed to render its content?
• Indexing: What indexing status and canonical does Google report?
• Appearance: If indexed, is it eligible for any intended search feature?

For example, “Discovered – currently not indexed” and “Crawled – currently not indexed” describe different reported states. Neither label, by itself, proves why Google has not indexed the URL (Digital Roop’s explanation of URL Inspection states).

Check discovery and crawl access

Find the guide through the site’s internal links, then check whether the current XML sitemap lists the preferred URL. Sitemaps help Google discover and prioritize URLs; they do not confine Google to crawling only listed URLs (Google Search Central). For an intended search page, use crawlable links and check that its sitemap entry matches the URL you mean to expose.

Next, inspect robots.txt for a rule covering the path. A crawl block is not a dependable indexing exclusion: Google may still index a URL it cannot crawl if links point to it (Ahrefs’ technical SEO guide). Check Search Console’s Crawl stats report for Google crawl activity, or server logs for recorded requests. Activity elsewhere on the site does not establish that Google fetched this particular guide.

Check the response and rendered page

Inspect what the exact URL returns. If it has moved permanently, use a 301 redirect to its destination; for a temporary move, use a 302. A removed page should return a true 404 rather than an error message served as a successful page (Google Search Central). After a change, request the original URL again and verify both its status and, where relevant, its destination.

Also inspect the content Google can access after rendering. A page response alone does not show whether required JavaScript, CSS, or images are available; those resources matter to Google’s rendering of the page (Backlinko’s technical SEO guide). These checks establish accessibility and what is returned, not an indexing decision.

Check index eligibility and the preferred URL

Review the page’s robots meta directive, duplicate versions, and canonical link. Use noindex for an accessible page that should stay out of search, but leave it crawlable so Google can read the directive (HubSpot). A page-level example, placed in the HTML <head>, is:

<meta name="robots" content="noindex">

For an intended indexable guide, check that it does not carry that directive. If the guide is the preferred version, its HTML could identify itself:

<link rel="canonical" href="https://example.com/guides/crawl-basics/">

Canonicalization affects crawling and indexing; it does not force Google to index the named URL (Google Search Central). Align the canonical and sitemap entry with the preferred, live, index-eligible URL. Then consult Search Console’s Page Indexing report and URL Inspection for Google’s reported status and canonical issues (Backlinko). Those are Google-specific diagnostics, not promises about a later decision or another search engine.

A robots rule that catches the wrong page

Imagine the guide is linked and listed in the sitemap, but the site currently has this overly broad rule:

User-agent: *
Disallow: /

It blocks crawling of the intended guide along with everything else. If the actual aim is to restrict an internal-search URL pattern, the illustrative replacement is narrower:

User-agent: *
Disallow: /internal-search/

After changing the file, verify the rule and the guide’s internal links, then check Google’s reported crawl status again. The corrected rule removes that particular crawl obstacle; it does not establish that Google will index the guide.

Now consider https://example.com/private/, a page users can reach but that should not appear in search. Do not add a matching robots block and assume it will disappear. Leave the page crawlable and place the noindex meta directive in its <head> so Google can encounter it. If the content actually requires restricted access, access controls are a different choice from a public, crawlable noindex page (Ahrefs). Recheck the rule, page response, directive, and Search Console’s reported status after the change.

Check appearance and performance separately

Once the access and indexing questions are clear, inspect structured data if you want the page to be eligible for a rich result. Markup can enable that appearance; it does not guarantee Google will show it (AIOSEO’s technical SEO overview). For performance, use the Core Web Vitals report for site-wide numbers and PageSpeed Insights for an individual page (Google Search Central). Neither check proves that the page is indexed or that its rankings will improve.