{"slug":"google-index","title":"Google Index: How It Works and How to Get Your Site Indexed","seoTitle":"Google Index: How to Get Your Site Indexed in 4 Steps","description":"Get your site into Google's index in 4 steps, see how long Google says indexing takes, check any URL, and fix every Page indexing status in Search Console.","banner":"/images/blog-content/google-index/hero.jpg","ogImage":"/images/og/google-index-og.jpg","url":"https://www.masstortmarketingagency.com/blogs/google-index","published":"2026-10-22","modified":"2026-10-22","author":{"name":"Tarun","role":"Founder, Mass Tort Marketing Agency"},"keywords":["google index","index web page","index website","how to get google to index my site","google index checker","request indexing","Page indexing report","crawled currently not indexed","discovered currently not indexed","how long does Google take to index"],"blocks":[{"type":"quote","text":"**Quick answer.** The Google index is Google's database of the web pages it has crawled, processed and chosen to keep; Google says it covers \"[hundreds of billions of webpages](https://www.google.com/search/howsearchworks/how-search-works/organizing-information/)\" and is \"well over 100,000,000 gigabytes\" in size. To get your site indexed: verify it in Google Search Console, submit your XML sitemap under **Sitemaps** and click **Submit**, use **URL Inspection** and **Request indexing** for key pages, then confirm the page shows \"URL is on Google\". Google says indexing can take from \"[a few days to a few weeks](https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl)\" and is never guaranteed."},{"type":"paragraph","text":"**Key takeaways**"},{"type":"list","ordered":false,"items":["Indexing is the middle stage of crawl, index and serve, and [Google says](https://developers.google.com/search/docs/fundamentals/how-search-works) not every page it processes gets indexed, even when the page follows the rules.","The sitemap button is **Submit**; **Request indexing** lives in URL Inspection and needs an owner or full user of the Search Console property.","Google's own range for a request is a day or so, sometimes a week or two, and Google states that indexing is never instant.","Most \"not indexed\" pages fail one of six gates: known, allowed, fetched, indexable, canonical, valued. That is our framework, covered in the triage section.","robots.txt does not keep a page out of Google; `noindex` does, and only if Google is allowed to crawl the page and see it.","AI Overviews and AI Mode can only cite pages that are indexed and eligible to show with a snippet, so indexing is the gate to AI search too."]},{"type":"paragraph","text":"You published a page and it is nowhere on Google. Maybe it is a brand-new site, a new practice-area or tort page, or a page that ranked last month and has now vanished. The first question is always whether the page is in Google's index, because Google can only show, rank or cite pages it has stored."},{"type":"paragraph","text":"This guide answers that question in the order you will need it. It starts with what the Google index is, then the four Search Console steps that get a site or page in front of Google, then how long Google says indexing takes. After that come checking a page's status, sitemaps and submission methods, a six-gate triage for pages that will not index, every Page indexing status, the controls that keep pages out, a playbook for law firm and mass tort sites, indexing for AI answers and other engines, tools, a monthly routine and an FAQ."},{"type":"paragraph","text":"It is written for law firm owners, the marketers who run their Search Console, and the developers who build their sites, but every section works for any website. Most guides on page one route every fix through a paid crawler. This one uses Google's free tools and Google's own documentation, with each fact linked and dated as of October 2026. If you want the wider strategy around it, read [the full mass tort law firm SEO guide](/blogs/mass-tort-law-firm-seo)."},{"type":"heading","level":2,"text":"What is the Google index?"},{"type":"paragraph","text":"The Google index is the database where Google stores the pages it has crawled, rendered and decided to keep, so it can serve them as search results. Google compares it to the index at the back of a book, with an entry for every word on every page it indexes."},{"type":"image","src":"/images/blog-content/google-index/01-crawl-index-serve.jpg","alt":"Diagram of how the Google index works: crawling, rendering, indexing and serving, with a filter between each stage","caption":"Google Search works in three stages, and not every page makes it through each one."},{"type":"paragraph","text":"Key terms are defined below as they come up; our [legal marketing glossary](/glossary) covers the wider vocabulary."},{"type":"heading","level":3,"text":"The Google index in Google's own words"},{"type":"paragraph","text":"Google describes the index in its own documentation, and the exact wording matters because several common beliefs about it are wrong. On its [How Search Works page](https://www.google.com/search/howsearchworks/how-search-works/organizing-information/), Google says: \"The Google Search index covers hundreds of billions of webpages and is well over 100,000,000 gigabytes in size. It's like the index in the back of a book - with an entry for every word seen on every webpage we index.\" That is the only size figure this guide uses. Older numbers that circulate on SEO blogs are not on Google's current page."},{"type":"paragraph","text":"Google's developer guide adds the technical picture. Information about a page and the cluster of similar pages it belongs to \"may be stored in the Google index, a large database hosted on thousands of computers,\" and then the sentence most site owners miss: \"[Indexing isn't guaranteed; not every page that Google processes will be indexed](https://developers.google.com/search/docs/fundamentals/how-search-works).\""},{"type":"paragraph","text":"Search Console's [Page indexing help](https://support.google.com/webmasters/answer/7440203?hl=en) defines the process from the site owner's side: \"Indexing is when Google finds (crawls) your page, then processes content of the page and puts the page into the Google index (indexes it), where the page may be eligible to appear in Google Search results, as well on as other Google services, like Discover.\" Note \"may be eligible\". Being indexed makes a page a candidate, nothing more."},{"type":"paragraph","text":"Google's [Programmable Search Engine help](https://support.google.com/programmable-search/answer/4513925?hl=en) compares it to \"an index in a library\" and says it \"lists all of the webpages that Google knows about.\" That is a simplification: Google knows many URLs it never indexes. Treat \"known\" and \"indexed\" as different states."},{"type":"heading","level":3,"text":"Crawling, indexing and serving: the three stages"},{"type":"paragraph","text":"Google Search works in three stages, crawling, indexing and serving, and Google says \"[not all pages make it through each stage](https://developers.google.com/search/docs/fundamentals/how-search-works).\" Understanding where a page drops out is most of the work of fixing it."},{"type":"paragraph","text":"Crawling starts with discovery. \"There isn't a central registry of all web pages,\" so Google keeps looking for new and updated URLs. A page becomes known in one of three ways: Google has visited it before, it is linked from a page Google already knows, or it is listed in a sitemap someone submitted."},{"type":"paragraph","text":"Once a URL is known, crawlers fetch it. Google's [How Search Works page](https://www.google.com/search/howsearchworks/how-search-works/organizing-information/) says crawlers \"automatically visit publicly accessible webpages and follow links on those pages,\" and that Google's systems then \"render the content of the page, just as a browser does.\" The developer guide specifies that Google \"renders the page and runs any JavaScript it finds using a recent version of Chrome.\""},{"type":"paragraph","text":"Indexing is where Google analyzes the text, images, video and key tags, and decides what to keep. This includes canonicalization. Google groups pages with the same or very similar content into a cluster and picks one as the [canonical](https://developers.google.com/search/docs/crawling-indexing/canonicalization), the version \"objectively the most complete and useful for search users.\" The others are duplicates and are crawled less often. The triage and status sections later in this guide cover duplicates in detail."},{"type":"paragraph","text":"Serving is the last stage: when someone searches, Google picks results from the index. Two limits apply to the whole chain. First, \"Google doesn't guarantee that it will crawl, index, or serve your page, even if your page follows the Google Search Essentials.\" Second, Google's [technical requirements](https://developers.google.com/search/docs/essentials/technical) for a page to be \"eligible to be indexed\" are only three: Googlebot isn't blocked, the page returns an HTTP 200 (success) status code, and it has indexable content. Google adds that \"most sites pass the technical requirements without even realizing it.\" When a page fails, the cause is usually one of those three or a quality judgment."},{"type":"heading","level":3,"text":"Indexed is not the same as ranked"},{"type":"paragraph","text":"A page can be crawled and not indexed, and indexed and not ranked; these are three separate outcomes. Crawled means Googlebot fetched it. Indexed means Google stored it as a candidate. Ranked means it appears in a useful position for a real query."},{"type":"paragraph","text":"Search Console's top verdict, \"URL is on Google\", only confirms the second. Google's [URL Inspection help](https://support.google.com/webmasters/answer/9012289?hl=en) is explicit: \"'URL is on Google' doesn't actually guarantee that your page will appear in Search results.\" Which indexed page appears, and where, is decided by ranking, and Google states that \"ranking is done programmatically\" and cannot be bought."},{"type":"table","header":["Outcome","What it means","Where you see it"],"rows":[["Crawled","Googlebot fetched the URL","Last crawl date in URL Inspection"],["Indexed","Stored and eligible to appear","\"URL is on Google\" verdict"],["Ranked","Shown for a real search","Performance report impressions"]]},{"type":"paragraph","text":"This guide is about the first two rows. If your page is indexed but gets no impressions, the problem has moved from indexing to relevance, competition and authority, which is the territory of [SEO for lawyers](/blogs/seo-for-lawyers) and of the \"Indexed but not ranking\" section later in this guide."},{"type":"heading","level":3,"text":"Google keeps more than one index"},{"type":"paragraph","text":"The web page index is the one site owners care about, but Google says it keeps \"multiple indexes of different types of information.\" Its [How Search Works page](https://www.google.com/search/howsearchworks/how-search-works/organizing-information/) says these are gathered \"through crawling, through partnerships, through data feeds being sent to us and through our own encyclopedia of facts, the Knowledge Graph.\" Its examples are books from major libraries, travel times from local public transport, and data from public sources like the World Bank."},{"type":"paragraph","text":"A search index also differs from a back-of-book index or an old website's A to Z page list. Those are compiled once, by hand, for a fixed set of pages. A search index is rebuilt continuously by software and is selective: pages are added, refreshed, merged into canonical clusters and dropped as the web changes."},{"type":"paragraph","text":"Images and videos have their own condition. Google's page on [controlling what you share with Search](https://developers.google.com/search/docs/crawling-indexing/control-what-you-share), updated 2026-10-21, states that \"Google only indexes images and videos that Googlebot is allowed to crawl.\""},{"type":"heading","level":3,"text":"What an \"index page\" means (index.html)"},{"type":"paragraph","text":"An \"index page\" is a different thing from a page in the Google index: it is the default file a web server returns for a directory. In the Apache HTTP Server, the [`DirectoryIndex` directive](https://httpd.apache.org/docs/2.4/mod/mod_dir.html) defaults to `index.html`, so a request for `http://example.com/docs/` returns `/docs/index.html` if that file exists."},{"type":"paragraph","text":"If you searched \"index web page\" for that meaning, the short answer is that `index.html` is just a file name. Being an index page has nothing to do with being indexed by Google. The two meet in one place: your homepage should be indexed once, at `/`, and not also at `/index.html`. If both URLs load the same content, Google treats them as duplicates and picks one. The canonical handling in the noindex, robots.txt and canonical section of this guide shows how to make that choice for Google."},{"type":"heading","level":3,"text":"Is Google indexing free?"},{"type":"paragraph","text":"Yes. Google indexing costs nothing, and Google's [How Search Works page](https://www.google.com/search/howsearchworks/how-search-works/organizing-information/) says \"Google never accepts payment to crawl a site more frequently.\" The developer guide adds that Google doesn't accept payment \"to crawl a site more frequently, or rank it higher.\" Search Console, the tool used for every step below, is described by Google as \"a free toolset.\""},{"type":"paragraph","text":"Ads are a separate system. A firm can pay for Google Ads placements, but that buys ad slots, not organic indexing or ranking. Some services sell \"fast indexing\" for a fee. Google's statement is the only verified fact here: it does not take payment to crawl a site more often, so no payment to anyone can buy faster Google crawling."},{"type":"heading","level":2,"text":"How to get your site indexed by Google in 4 steps"},{"type":"paragraph","text":"To get a website or web page indexed by Google, use Google Search Console, the free tool Google provides for site owners:"},{"type":"list","ordered":true,"items":["Add and verify your site in Google Search Console.","Find or create an XML sitemap.","Submit the sitemap in the Sitemaps report.","Request indexing for your most important URLs in URL Inspection."]},{"type":"paragraph","text":"Then confirm with URL Inspection."},{"type":"image","src":"/images/blog-content/google-index/02-four-step-search-console.jpg","alt":"Generic Search Console style mock-up showing the four steps to index a website: verify, sitemap, submit, request indexing","caption":"The four steps that get a new site or page in front of Google, with the exact button names."},{"type":"heading","level":3,"text":"Step 1: Add and verify your site in Search Console"},{"type":"paragraph","text":"Start by adding your site as a property in Search Console and proving you control it. Nothing else in this list works until verification succeeds. Google allows \"[up to 1,000 properties](https://support.google.com/webmasters/answer/34592?hl=en)\" per account, so agencies and multi-site firms rarely hit a ceiling."},{"type":"paragraph","text":"You choose between two property types. A **Domain property** covers \"all subdomains (m, www, and so on) and multiple protocols (http, https, ftp)\". It is entered without a `www` prefix and needs DNS verification, unless the site is on Blogger or Google Sites. A **URL-prefix property** covers only URLs that start with the exact prefix you enter, protocol included, so `https://www.example.com/` and `https://example.com/` are different properties."},{"type":"paragraph","text":"Google lists the [verification methods](https://support.google.com/webmasters/answer/9008080?hl=en) as HTML file upload, HTML tag, Google Analytics tracking code, Google Tag Manager, Google Sites or Blogger, and domain name provider. The domain name provider method \"is the only way to verify a Domain property,\" and HTML file upload works for URL-prefix properties only. You can add several methods as a backup."},{"type":"table","header":["Property type","What it covers","How to verify","Best for"],"rows":[["Domain","All subdomains and protocols","DNS record at your domain provider","The whole firm site in one view"],["URL-prefix","One exact protocol and host","HTML file, tag, Analytics, Tag Manager or DNS","A section, or when DNS access is blocked"]]},{"type":"paragraph","text":"Permissions decide who can do what. Google's [permissions help](https://support.google.com/webmasters/answer/7687615?hl=en) lists three levels: an **Owner** has \"full control\"; a **Full user** has \"view rights to all data and can take some actions\"; a **Restricted user** has \"simple view rights on most data\". \"A property must have at least one verified owner.\" Two rules matter for indexing. Google's [recrawl guide](https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl) says \"You must be an owner or full user of the Search Console property to be able to request indexing in the URL Inspection tool.\" And the [Sitemaps report help](https://support.google.com/webmasters/answer/7451001?hl=en) says you need owner permissions to submit a sitemap there."},{"type":"paragraph","text":"My recommendation for law firms: use a Domain property. It puts http and https, www and non-www, and any subdomain, such as an `es.` subdomain for Spanish pages or a campaign microsite on a subdomain, into one view, so nothing slips through. Keep verified ownership inside the firm, through the person who controls the domain's DNS, and give your agency full-user access. That way a change of vendor never locks the firm out of its own data. This is standard practice in [technical SEO for personal injury firms](/services/personal-injury-seo)."},{"type":"paragraph","text":"Expect a short wait before reports fill in. Google's [Search Console data help](https://support.google.com/webmasters/answer/96568?hl=en) says \"It can take up to a week to generate data for a newly created site, or a site newly added to Search Console.\" Empty reports on day one are normal."},{"type":"heading","level":3,"text":"Step 2: Find or create your XML sitemap"},{"type":"paragraph","text":"An XML sitemap is a file that lists the URLs you want Google to know about, and most sites already have one. Check three places first: type `/sitemap.xml` after your domain, then `/sitemap_index.xml`, then open `/robots.txt` and look for a line that starts with `Sitemap:`. Google's [sitemap guide](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap) documents that robots.txt line as a standard way to point crawlers at a sitemap. The section \"Three ways to find an existing sitemap\" later in this guide covers the full set of methods, including search operators."},{"type":"paragraph","text":"If none of those addresses returns a file, your platform can almost certainly create one. WordPress core, the common WordPress SEO plugins, Wix, Squarespace, Shopify and Webflow can all generate a sitemap. If you are planning or rebuilding a site, sitemap generation belongs on the requirements list for [building a law firm website](/blogs/law-firm-website-development), next to HTTPS and mobile layout."},{"type":"paragraph","text":"Two points from Google keep this step in proportion. On hosted platforms such as Blogger or WordPress, Google's [recrawl guide](https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl) says that \"In most cases, the platform submits your new content to search engines automatically.\" And Google's [sitemaps overview](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview) says you might not need a sitemap if your site is \"small\" (\"about 500 pages or fewer\"). You might need one if the site is large, new with few external links, or heavy on rich media or news."},{"type":"heading","level":3,"text":"Step 3: Submit the sitemap (the button says Submit)"},{"type":"paragraph","text":"Submit your sitemap in the Sitemaps report: open your property, go to the Sitemaps report under Indexing, paste the sitemap URL into **Add a new sitemap**, and click **Submit**. That is the whole flow, per [Google's Sitemaps report help](https://support.google.com/webmasters/answer/7451001?hl=en). If you have a sitemap index file, submit the index and Google will read the child sitemaps from it."},{"type":"paragraph","text":"The report then shows one of three outcomes. **Success** means Google read the file. **Couldn't fetch** means Google could not retrieve it, often because of a typo in the URL, a server error, or a firewall that blocks Googlebot. A readable sitemap with errors means Google fetched it but found problems in its contents, which the report lists. Google \"will periodically recrawl the sitemap,\" so you do not need to resubmit it every time you publish."},{"type":"paragraph","text":"Some guides tell you to click a \"request indexing\" button in the Sitemaps menu. There isn't one. The Sitemaps report has **Submit**; **Request indexing** is a separate button in URL Inspection, covered in Step 4."},{"type":"paragraph","text":"Keep expectations realistic. Google's [sitemap guide](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap) says \"Submitting a sitemap is merely a hint: it doesn't guarantee that Google will download the sitemap or use the sitemap for crawling URLs on the site.\" It helps discovery; it does not force indexing."},{"type":"paragraph","text":"Also add the sitemap to robots.txt with a line such as `Sitemap: https://example.com/sitemap.xml`. Google notes \"there's no limit to the number of sitemaps you can include\" there, and it is the documented fallback if you lack owner permissions. One older method no longer works: the \"ping\" URL that used to notify Google of a sitemap. Google [deprecated the sitemaps ping endpoint](https://developers.google.com/search/blog/2023/06/sitemaps-lastmod-ping) in 2023, and pings now return a 404 error. Ignore any guide that still recommends it."},{"type":"heading","level":3,"text":"Step 4: Request indexing for your most important URLs"},{"type":"paragraph","text":"Request indexing tells Google to crawl one specific URL now, and it is the right tool for a handful of important pages, not for a whole site. Open **URL Inspection** from the search bar at the top of Search Console, paste the full URL, and read the verdict. If the page is not on Google, or you have changed it, click **Request indexing**."},{"type":"paragraph","text":"Google then runs \"a quick check\" on the live page. Per the [URL Inspection help](https://support.google.com/webmasters/answer/9012289?hl=en), \"You cannot request indexing if the page is considered to be non-indexable in the live test.\" If the button fails, the live test result tells you why, such as a `noindex` rule, a robots.txt block or a server error. You can also click **Test live URL** first to see what Googlebot gets right now."},{"type":"paragraph","text":"There is a quota. Google says \"There is a daily limit to how many index requests you can submit,\" plus a separate daily limit on live inspections per property. Google does not publish the numbers, and any specific figure you see on a forum is someone's observation, not Google's limit. Spend requests on the pages that matter."},{"type":"paragraph","text":"Where to start on a new website: Google's [Page indexing help](https://support.google.com/webmasters/answer/7440203?hl=en) says \"the best first step is to request indexing of your homepage.\" A homepage that links clearly to your main sections gives Google a path to the rest. After that, request your key money pages: the main practice areas, a new tort page, the contact page."},{"type":"paragraph","text":"What not to do: do not request the same URL again and again. Google's [recrawl guide](https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl) says asking for a recrawl of the same URL multiple times \"won't get it crawled any faster.\" Google also \"automatically recrawls your pages, so don't bother asking for a recrawl unless there is an important change, and Google doesn't seem to have noticed it for a while (a week or more).\""},{"type":"paragraph","text":"For more than a few URLs, use the sitemap instead. Google's advice for many pages is to \"submit a sitemap … with the updated pages marked by `<lastmod>`.\" Google [uses `<lastmod>`](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap) \"if it's consistently and verifiably … accurate\", so the date must change only when the page content really changes. A CMS that stamps every page with today's date undermines that."},{"type":"heading","level":3,"text":"Confirm it worked"},{"type":"paragraph","text":"To confirm a page is indexed, inspect the URL again in URL Inspection and look for the verdict \"URL is on Google\". The other verdicts Google lists are \"URL is on Google, but has issues\", \"URL is not on Google\" and \"URL is unknown to Google\", which means \"Google hasn't seen this URL before.\""},{"type":"paragraph","text":"Then check one field that many people skip: Google-selected canonical. If it shows a different URL from the one you inspected, Google has folded your page into another page's cluster, and the other URL is the one that will appear in results. The [URL Inspection help](https://support.google.com/webmasters/answer/9012289?hl=en) notes that the canonical can be determined \"only in the indexed data\"; the live test cannot predict it."},{"type":"paragraph","text":"A `site:` search on Google is a quick sanity check, but it is approximate. Google's [`site:` documentation](https://developers.google.com/search/docs/monitor-debug/search-operators/all-search-site) says an indexed URL \"can show up in search results for site: queries … however it's not guaranteed,\" and that `site:https://www.example.com` and `site:https://example.com/` return different results. If a page is missing from `site:`, trust URL Inspection over the search. The next section after timelines covers every way to check index status, including the Page indexing report and checking hundreds of URLs at once."},{"type":"paragraph","text":"If the verdict is still \"URL is not on Google\" a day later, that is usually normal. How long you should wait is the next question."},{"type":"heading","level":2,"text":"How long does Google take to index a page?"},{"type":"paragraph","text":"Google documents ranges, not promises: a request usually takes \"a day or so\" but can take \"a week or two\", a new site \"a week or so\", and crawling in general \"a few days to a few weeks\". Some pages are never indexed."},{"type":"image","src":"/images/blog-content/google-index/03-indexing-timelines.jpg","alt":"Timeline chart comparing Google's documented indexing time ranges for requests, new sites, recrawls and duplicate clusters","caption":"Google's documented ranges for common indexing events, as of October 2026. Ranges, not guarantees."},{"type":"heading","level":3,"text":"Google's documented timelines"},{"type":"paragraph","text":"Google's documentation gives a time range for each common indexing event, and the table below collects them in one place, as of October 2026. Every row quotes Google's wording and links the page it comes from."},{"type":"table","header":["Process","Google's wording","Source"],"rows":[["URL Inspection request","\"a day or so\", up to \"a week or two\"","[URL Inspection help](https://support.google.com/webmasters/answer/9012289?hl=en)"],["New page or site","\"a week or so\"","[Page indexing help](https://support.google.com/webmasters/answer/7440203?hl=en)"],["Newly added content","\"a few days\"","[Page indexing help](https://support.google.com/webmasters/answer/7440203?hl=en)"],["Crawl of a newly known URL","\"up to a few weeks\"","[Page indexing help](https://support.google.com/webmasters/answer/7440203?hl=en)"],["Recrawl","\"a few days to a few weeks\"","[Recrawl guide](https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl)"],["Duplicate cluster after a fix","\"up to two weeks\"","[Canonical troubleshooting](https://developers.google.com/search/docs/crawling-indexing/canonicalization-troubleshooting)"],["robots.txt cache","up to 24 hours","[robots.txt spec](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)"],["Removals tool block","about six months","[Removals help](https://support.google.com/webmasters/answer/9689846?hl=en)"],["Change of Address redirects","keep for at least 180 days","[Change of Address help](https://support.google.com/webmasters/answer/9370220?hl=en)"],["Search generative AI control","1-2 days","[Control help](https://support.google.com/webmasters/answer/16908024?hl=en)"]]},{"type":"paragraph","text":"Read the table as ranges. \"A day or so\" is typical, not a service level. The duplicate-cluster row matters for law firms that publish similar city pages: even after you make a page more distinct, Google \"might hold pages in a duplicate cluster for up to two weeks.\" The robots.txt row explains why unblocking a section does not take effect at once: Google \"generally caches the contents of robots.txt file for up to 24 hours.\""},{"type":"paragraph","text":"Search Console's own reporting adds a separate delay. Google's [data help](https://support.google.com/webmasters/answer/96568?hl=en) says collected data \"should be available in 2-3 days,\" and a new property can take up to a week. So a page can be indexed before the Page indexing report shows it. URL Inspection reflects the indexed state more directly, which is why it is the tool to use for confirmation."},{"type":"heading","level":3,"text":"The October 2026 timing figures, as reported"},{"type":"paragraph","text":"At Search Central Live Deep Dive Europe in Barcelona on 2026-10-02, Google's Gary Illyes reportedly showed internal timing ranges for crawling, indexing and serving. Google's [event announcement](https://developers.google.com/search/blog/2026/07/search-central-live-deep-dive-europe-2026) confirms the event ran from September 30 to October 2, 2026, with Day 2 covering \"Advanced Indexing & Beyond.\" The figures themselves come from an attendee recap by John Campbell, relayed by [Search Engine Roundtable](https://www.seroundtable.com/google-crawling-indexing-serving-data-42225.html) (Barry Schwartz, 2026-10-05) and [Search Engine Journal](https://www.searchenginejournal.com/google-crawling-indexing-recovery-timing-ranges/591829/) (Matt Southern, 2026-10-04)."},{"type":"paragraph","text":"Three caveats come with them. Search Engine Journal noted that \"The slides don't appear on Google's Search Central events page as of publication.\" Illyes told Search Engine Roundtable afterwards: \"mind that this was an exercise to see if the audience can relate to the numbers we pulled internally and put in those slides.\" And, per the recap, he stressed that the processes are linked, so \"delays stack up.\" These are attributed, indicative numbers, not Google documentation and not a promise."},{"type":"paragraph","text":"As listed by Search Engine Roundtable:"},{"type":"table","header":["Process","Typical (reported)","Slowest (reported)"],"rows":[["Discovery (new URL)","about 20 hours","weeks to never"],["Refresh (known URL)","about 30 days","weeks to never"],["Sitemap processing","about 24 hours","up to 14 days, or never (quality)"],["robots.txt update","about 24 hours","25 hours"],["Indexing (end to end)","about 1.5 hours","months or never (quality)"],["Canonicalisation change","1-3 weeks","months (conflicting signals)"],["Site move","1-3 months","6 months to 1 year or more"],["Removal in Search Console (owner)","about 2 hours","24 hours"],["Title and snippet update","1-2 days","several weeks to months"],["Manual action removal","1-2 weeks","4-6 weeks, or much longer for dormant sites"],["Core update change","3-6 months to recover","6 months to 1 year (next core update)"]]},{"type":"paragraph","text":"The \"slowest\" column is the one to read. Discovery, sitemap processing and indexing all include \"never\", and two of those rows tie the outcome to quality. A short typical figure describes a page Google already wants, not a thin or near-duplicate one. If you quote these numbers, quote both columns, below Google's documented ranges."},{"type":"heading","level":3,"text":"Vendor estimates and why they differ"},{"type":"paragraph","text":"SEO tool vendors publish their own estimates, and they differ because each describes a different set of sites, not because any of them has inside knowledge. Ahrefs' guide, [How to Get Google to Index Your Website](https://ahrefs.com/blog/google-index/), updated Oct 31, 2025, summarizes: \"New sites: days to weeks. Established sites: hours to days.\" Semrush's indexing guide, dated June 2, 2026, says Google \"typically processes these requests within a few weeks.\" Both are vendor estimates, and both sit inside Google's documented ranges."},{"type":"paragraph","text":"Other guides promise indexing in minutes. Google's documentation never gives a minute-level figure; treat any such claim as marketing."},{"type":"paragraph","text":"For a first-party data point, here is what we saw on our own site. When we checked on October 22, 2026, Search Console's URL Inspection API reported all 8 pages we had published and submitted on September 15 as \"Submitted and indexed\"; three showed a last crawl on the request day. That is current status, not the date each page was first indexed. It is one site and eight pages, an example rather than a benchmark."},{"type":"heading","level":3,"text":"What speeds indexing up and what does not"},{"type":"paragraph","text":"You cannot make Google index a page, but you can remove the delays that are under your control. Google's [recrawl guide](https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl) states the principle: \"Our systems prioritize the fast inclusion of high quality, useful content.\" Everything that helps either makes the page easier to discover and fetch, or makes it more clearly worth keeping."},{"type":"paragraph","text":"What helps:"},{"type":"list","ordered":false,"items":["**Internal links from pages Google crawls often.** Google discovers URLs through links from pages it already knows. A link from your homepage or a busy practice hub gets a new page found faster than a sitemap entry alone.","**A sitemap with accurate `<lastmod>` dates.** Google uses `<lastmod>` when it is consistently accurate, which tells it which pages changed and are worth a fresh crawl.","**A fast, clean 200 response.** Redirect chains, slow servers and intermittent errors waste crawls. Google's [crawl rate guidance](https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate) also notes that \"You cannot request an increase in crawl rate,\" so the server has to earn it by staying fast and reliable.","**Unique, useful content.** A page that adds something the index lacks clears the quality gate. A page that repeats another page's content joins its duplicate cluster instead."]},{"type":"paragraph","text":"What does not help:"},{"type":"list","ordered":false,"items":["**Repeat requests.** Requesting the same URL several times \"won't get it crawled any faster.\"","**Payment.** Google \"never accepts payment to crawl a site more frequently.\"","**`<priority>` and `<changefreq>` tags.** Google's [sitemap guide](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap) says \"Google ignores `<priority>` and `<changefreq>` values.\"","**Pinging the sitemap URL.** The ping endpoint is retired and returns a 404 error.","**IndexNow, for Google specifically.** Google is not on [IndexNow's list of participating search engines](https://www.indexnow.org/searchengines.json). It can help with Bing and other engines, which later sections cover."]},{"type":"paragraph","text":"If Google's ranges have passed after links, a sitemap entry and one request, diagnose rather than re-request. Before you troubleshoot, confirm what Google actually has, which is the next section."},{"type":"heading","level":2,"text":"How to check if Google has indexed a page"},{"type":"paragraph","text":"Use URL Inspection in Google Search Console for a definitive answer on one URL, the Page indexing report for the whole site, and a `site:` search only as a rough first look."},{"type":"paragraph","text":"The table below compares the six ways to check index status. The first three answer \"is this one URL indexed?\" The last three answer \"how much of my site is indexed, and which groups of pages are missing?\""},{"type":"table","header":["Method","What it tells you","Limits","Best for"],"rows":[["`site:` search","Whether some indexed URLs on a domain or folder show in results","Not guaranteed to list every indexed URL; counts are rough","A quick first look"],["URL Inspection","Google's indexed version of one URL, with a verdict","One URL at a time; daily limit per property","A definitive answer on one page"],["Live test","Whether the current page could be indexed now","Cannot predict the canonical Google will pick","Checking a fix before requesting a crawl"],["Page indexing report","Indexed and not-indexed counts, with reasons","Example lists capped at 1,000 URLs","Site-wide health"],["Sitemaps report","Whether Google read each sitemap, and its status","Shows only sitemaps submitted in the report or API","Confirming Google can read your sitemaps"],["URL Inspection API","Index status for many URLs by script","2,000 queries per day per site","Monitoring hundreds of URLs"]]},{"type":"paragraph","text":"Never report an \"indexed\" count to anyone based on anything but Search Console."},{"type":"heading","level":3,"text":"The site: search and its limits"},{"type":"paragraph","text":"A `site:` search shows a sample of the pages Google has indexed for a domain, but Google says plainly that it is not a complete or guaranteed list. Its operator documentation says an indexed URL \"can show up in search results for site: queries … however it's not guaranteed\", and that if a URL is missing you should use URL Inspection instead ([Google Search Central, site: operator](https://developers.google.com/search/docs/monitor-debug/search-operators/all-search-site))."},{"type":"paragraph","text":"The same page warns that small differences in the query change the results: `site:https://www.example.com` does not return the same results as `site:https://example.com/`. Google's own Programmable Search help still suggests `site:mywebsite.com` as a way to see which pages are in the index ([Programmable Search Engine help](https://support.google.com/programmable-search/answer/4513925?hl=en)), which is why the habit persists."},{"type":"paragraph","text":"Useful quick queries are `site:example.com` for the whole domain and `site:example.com/practice-areas/` for one folder."},{"type":"paragraph","text":"Treat the \"About X results\" count as approximate. It moves from one search to the next and does not match the Page indexing report. My recommendation: use `site:` to spot an obvious problem, such as a brand-new domain with zero results, and then confirm everything in Search Console."},{"type":"heading","level":3,"text":"URL Inspection: reading the verdict"},{"type":"paragraph","text":"**URL Inspection** is the authoritative check for a single URL, because it reports what Google itself has stored for that address. Paste the full URL into the search bar at the top of Search Console. By default, the tool \"provides information about Google's indexed version of a specific page\"; that default view \"is not a live test\" ([Search Console help, URL Inspection](https://support.google.com/webmasters/answer/9012289?hl=en)). To see whether the page as it exists right now could be indexed, click **Test live URL**."},{"type":"paragraph","text":"The verdict at the top is one of four:"},{"type":"list","ordered":false,"items":["\"URL is on Google\": indexed and eligible to appear.","\"URL is on Google, but has issues\": indexed, with problems in enhancements such as structured data.","\"URL is not on Google\": known but not indexed, with a reason below.","\"URL is unknown to Google\": Google hasn't seen this URL before."]},{"type":"paragraph","text":"Google adds a warning worth repeating to any stakeholder: \"'URL is on Google' doesn't actually guarantee that your page will appear in Search results.\" Indexed means eligible, not ranked."},{"type":"paragraph","text":"Below the verdict, open the indexing details. The fields that matter most are **Last crawl** (in your local time), **Crawled as** (desktop or mobile), **Crawl allowed?**, **Page fetch**, **Indexing allowed?**, **User-declared canonical** and **Google-selected canonical**. One quirk catches many people: if robots.txt blocks the page, **Indexing allowed?** will always say \"Yes\", because Google can't see and respect any `noindex` on a page it cannot fetch. A \"Yes\" there is not proof the page is indexable."},{"type":"paragraph","text":"Also, canonical choice only appears in the indexed data: \"the live test cannot predict whether or not the tested version will be considered canonical.\"Google's inspection tools don't follow redirects ([Google Search Central, HTTP and network errors](https://developers.google.com/search/docs/crawling-indexing/http-network-errors)), so always inspect the final destination URL, not an old address that redirects to it. The fetches you trigger here come from the Google-InspectionTool crawler, which powers URL Inspection and the Rich Results Test ([Google's common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)), so your server logs will show that user agent rather than Googlebot for live tests."},{"type":"paragraph","text":"If the page says \"URL is on Google\" but you still cannot find it for its own title, Google says to check three places: the **Manual actions** report, the **Security issues** report, and **Removals**, in case someone blocked the URL temporarily."},{"type":"heading","level":3,"text":"The Page indexing report"},{"type":"paragraph","text":"The **Page indexing** report is the site-wide view: it splits every URL Google knows on your property into indexed and not indexed, and gives a reason for each group that is not indexed. The help article is titled \"Page indexing report\" ([Search Console help, Page indexing report](https://support.google.com/webmasters/answer/7440203?hl=en)). It replaced what Google introduced in 2018 as the \"Index Coverage report\", which launched with issue tracking and a \"validate fix\" flow ([Google Search Central Blog, 2018](https://developers.google.com/search/blog/2018/01/introducing-new-search-console)). Some of Google's own help text still says \"Coverage\", so you will see both names."},{"type":"paragraph","text":"The chart shows indexed and not-indexed totals over time, and a table below lists each not-indexed reason with a count and example URLs. Two caps apply. \"The list of example URLs in the report is limited to 1,000 items\", and the **View data about indexed pages** link shows \"an example list of up to 1,000 URLs that are indexed.\""},{"type":"paragraph","text":"The most useful filter is at the top left. Switch from \"All known pages\" to **All submitted pages**, or pick one sitemap, and the report compares what you asked Google to index with what it did index. That is the fastest way to see whether a new section is being picked up."},{"type":"paragraph","text":"Google says: \"Don't expect every URL on your site to be indexed\". Under \"What not to look for\" it lists \"100% coverage\" and \"Immediate indexing\". A site with many not-indexed URLs can be perfectly healthy if those URLs are redirects, parameter duplicates or pages you deliberately excluded."},{"type":"paragraph","text":"The reasons themselves, from \"Crawled - currently not indexed\" to \"Soft 404\", are explained one by one in the status section later in this guide."},{"type":"heading","level":3,"text":"The Sitemaps report"},{"type":"paragraph","text":"The **Sitemaps** report shows whether Google could read each sitemap you submitted and how many URLs it found in it. To submit, paste the sitemap URL into **Add a new sitemap** and click **Submit**. Statuses are \"Success\", \"Couldn't fetch\", or a readable sitemap that has errors ([Search Console help, Sitemaps report](https://support.google.com/webmasters/answer/7451001?hl=en))."},{"type":"paragraph","text":"The report shows up to 1,000 submitted sitemaps, and only those submitted through the report or the API; a sitemap listed only in robots.txt won't appear. Google \"will periodically recrawl the sitemap\", so you don't need to resubmit after every change. And \"a later failure or success to read a sitemap will not cause Google to 'forget' the information from previous reads\", so one bad fetch doesn't wipe out what Google already learned."},{"type":"paragraph","text":"When you see \"Couldn't fetch\", check three things in order:"},{"type":"list","ordered":true,"items":["Open the sitemap URL in a browser and confirm it returns a 200 status, not a redirect, a login page or a 404.","Confirm robots.txt doesn't block the sitemap's path.","Confirm the sitemap is in the property's scope: a sitemap only covers URLs in its own directory and below, unless you submit it through Search Console ([Google Search Central, build a sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap))."]},{"type":"heading","level":3,"text":"Crawl stats and the robots.txt report"},{"type":"paragraph","text":"Crawl stats shows whether Googlebot can reach your server reliably, and the robots.txt report shows whether Google can read your rules. Crawl stats lives under **Settings > Crawl stats** and is \"available only for root-level properties\" ([Search Console help, Crawl stats](https://support.google.com/webmasters/answer/9679690?hl=en))."},{"type":"paragraph","text":"Its host status panel covers the past 90 days in three categories: robots.txt fetching, DNS resolution and server connectivity. As an example threshold, Google says DNS resolution failing on more than 5% of requests in a day counts as an issue. The report also splits crawl requests by purpose: \"Discovery\" for URLs Google has never crawled, and \"Refresh\" for known URLs. After a launch, a rise in Discovery requests is a good sign Google has found the new section."},{"type":"paragraph","text":"The **robots.txt report** shows the robots.txt files for \"the top 20 hosts on your site\", when each was last crawled and any warnings or errors, for files fetched in the last 30 days. It lets you \"request a recrawl … for emergency situations\", such as after you remove a rule that blocked the whole site by mistake. It is available only for domain-level properties ([Search Console help, robots.txt report](https://support.google.com/webmasters/answer/6062598?hl=en))."},{"type":"paragraph","text":"Google's current list of reports includes the robots.txt report and has no robots.txt tester, not even in its legacy section ([Search Console help, reports at a glance](https://support.google.com/webmasters/answer/9133276?hl=en)). A few help pages still tell you to use \"the robots.txt tester\". Use the report."},{"type":"heading","level":3,"text":"Checking hundreds of URLs at once"},{"type":"paragraph","text":"For more than a handful of URLs, use the URL Inspection API, which returns the same index status as the tool, by script. Google launched it on January 31, 2022 ([Google Search Central Blog](https://developers.google.com/search/blog/2022/01/url-inspection-api)), and the current quota is 2,000 queries per day and 600 per minute for each site ([Search Console API limits](https://developers.google.com/webmaster-tools/limits))."},{"type":"paragraph","text":"Two other limits shape bulk checks. Search Console tables \"can show a maximum of 1,000 rows\" ([Search Console help, data timing](https://support.google.com/webmasters/answer/96568?hl=en)), so export by sitemap or folder rather than all at once. And Search Console keeps data for the last 16 months ([Google Analytics help](https://support.google.com/analytics/answer/13682862?hl=en)), so export anything you want to compare year over year."},{"type":"paragraph","text":"Many free \"Google index checker\" tools exist. They either run `site:`-style searches, which inherit every limit above, or call the same API with your Search Console login. In my view, the API itself, through a connector or a short script, is the better choice: it is first-party data, and you control what is stored."},{"type":"heading","level":2,"text":"XML sitemaps: how they help Google find pages"},{"type":"paragraph","text":"An XML sitemap lists the URLs you want Google to find and tells it when they last changed; Google treats it as a hint, not an instruction. In Google's words, \"Submitting a sitemap is merely a hint: it doesn't guarantee that Google will download the sitemap or use the sitemap for crawling URLs on the site\" ([Google Search Central, build a sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap))."},{"type":"paragraph","text":"A sitemap is one of three ways a URL becomes known to Google, alongside a previous visit and a link from a known page ([how Search works](https://developers.google.com/search/docs/fundamentals/how-search-works)), and the only one you fully control."},{"type":"heading","level":3,"text":"Do you need a sitemap?"},{"type":"paragraph","text":"Google says a small site may not need one, but most law-firm sites benefit anyway. Its sitemap overview says you might not need a sitemap if your site is \"small\" (\"about 500 pages or fewer\") and well linked. You might need one if the site is large, new with few external links, or heavy on rich media or news ([Google Search Central, sitemaps overview](https://developers.google.com/search/docs/crawling-indexing/sitemaps/overview))."},{"type":"paragraph","text":"My recommendation: every law-firm site should have one, even a 40-page site. New tort, practice-area and city pages are often published before anyone links to them from the menu or a hub, and a new domain usually has few external links. A sitemap gives those pages a discovery path from day one, at no cost."},{"type":"heading","level":3,"text":"Sitemap rules and limits"},{"type":"paragraph","text":"Sitemaps have hard size limits and a few rules Google applies strictly, summarized in the table below. The protocol itself, published at sitemaps.org, sets the base limits ([sitemaps.org protocol](https://www.sitemaps.org/protocol.html)); Google's documentation adds how Google reads them."},{"type":"table","header":["Rule","Limit or behavior","Source"],"rows":[["URLs per sitemap","50,000","sitemaps.org; Google"],["File size","50MB uncompressed (52,428,800 bytes); gzip allowed","sitemaps.org"],["Encoding","UTF-8","sitemaps.org"],["URL format","Fully qualified, absolute URLs","Google"],["Scope","Descendants of the sitemap's directory, unless submitted in Search Console","Google"],["Formats","XML, RSS, mRSS, Atom 1.0, plain text","Google"],["`<lastmod>`","Used if consistently and verifiably accurate","Google"],["`<priority>` and `<changefreq>`","Ignored by Google","Google"],["Sitemap index file","Up to 50,000 `<loc>` tags; same site, same directory or lower","Google"],["Index files per site","Up to 500 in Search Console","Google"],["Sitemap lines in robots.txt","No limit","Google"]]},{"type":"paragraph","text":"A few rules deserve a note. Google uses `<lastmod>` \"if it's consistently and verifiably … accurate\", and says outright: \"Google ignores `<priority>` and `<changefreq>` values.\" Its 2023 post on the retired sitemap ping repeated that Google \"still doesn't use the changefreq or priority elements at all\" ([Google Search Central Blog, 2023](https://developers.google.com/search/blog/2023/06/sitemaps-lastmod-ping)). So a plugin setting that marks your homepage \"priority 1.0\" does nothing for Google. An accurate `<lastmod>` does."},{"type":"paragraph","text":"Large sites split URLs across several sitemaps and list them in a sitemap index file, which \"may have up to 50,000 loc tags\"; you \"can submit up to 500 sitemap index files for each site\" ([Google Search Central, large sitemaps](https://developers.google.com/search/docs/crawling-indexing/sitemaps/large-sitemaps)). You can also list sitemaps in robots.txt with a line such as `Sitemap: https://example.com/my_sitemap.xml`, and \"there's no limit to the number of sitemaps you can include\"."},{"type":"paragraph","text":"A single entry in an XML sitemap is short. Written out line by line, one URL looks like this:"},{"type":"list","ordered":true,"items":["`<url>`","`<loc>https://www.example.com/torts/example-tort/</loc>`","`<lastmod>2026-10-20</lastmod>`","`</url>`"]},{"type":"paragraph","text":"Your CMS or SEO plugin generates this; your job is to check its output."},{"type":"heading","level":3,"text":"Three ways to find an existing sitemap"},{"type":"paragraph","text":"Most sites already have a sitemap, and you can usually find it in under a minute with one of three checks. Try them in order."},{"type":"list","ordered":true,"items":["**Guess the standard address.** Open `https://www.example.com/sitemap.xml` and `https://www.example.com/sitemap_index.xml` in a browser. Most CMSs and SEO plugins use one of the two. If you see a list of sitemap files, you have found an index; open each child file to see the URLs.","**Read robots.txt.** Open `https://www.example.com/robots.txt` and look for a line starting with `Sitemap:`. Google supports this declaration, so many sites list every sitemap there.","**Use search operators.** Try `site:example.com inurl:sitemap` or `site:example.com ext:xml`. These are practical tips rather than a Google-documented method, and like any `site:` query they can miss files, but they sometimes surface sitemaps with unusual names."]},{"type":"paragraph","text":"Then open the Sitemaps report to see what is already submitted. On an inherited site it often lists old sitemaps from a previous CMS that point to dead URLs; remove them."},{"type":"heading","level":3,"text":"CMS-generated sitemaps and the junk they include"},{"type":"paragraph","text":"Automatic sitemaps from WordPress, SEO plugins and site builders save time, but their defaults often list URLs you don't want indexed. Common offenders are tag and category archives, author pages, media attachment pages, thin auto-generated location pages and URLs that the same site marks `noindex`. On a law-firm site you may also find thank-you pages, intake form steps and paid landing pages in the default output."},{"type":"paragraph","text":"Each sends a mixed signal: the sitemap says \"index this\" while the page says otherwise. Google calls sitemap inclusion \"a weak signal\" for canonical choice, behind redirects and `rel=\"canonical\"` annotations ([Google Search Central, consolidate duplicate URLs](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls))."},{"type":"paragraph","text":"A clean sitemap includes only:"},{"type":"list","ordered":false,"items":["URLs that return a 200 status.","URLs that are indexable, with no `noindex` and no robots.txt block.","The canonical version of each page, not duplicates or alternate URLs.","No redirecting URLs, since \"Redirect URLs are not indexed, only the redirect target\" ([Page indexing report help](https://support.google.com/webmasters/answer/7440203?hl=en)).","No parameter URLs such as filter or sort variants.","A `<lastmod>` that changes only when the content changes."]},{"type":"paragraph","text":"Sitemap control, excluding post types and single URLs, is a basic requirement of [law firm web design](/services/web-design)."},{"type":"heading","level":3,"text":"Split sitemaps by content type"},{"type":"paragraph","text":"My recommendation is one sitemap per content type, all listed in a single sitemap index, so you can see indexing by group. When you filter the Page indexing report by one sitemap, it shows how many submitted URLs in that group are indexed. With one giant sitemap, a problem in one section hides inside the site-wide total."},{"type":"paragraph","text":"A practical split for a plaintiff firm:"},{"type":"list","ordered":false,"items":["`/sitemaps/practice-areas.xml` for core practice-area pages.","`/sitemaps/torts.xml` for individual tort pages.","`/sitemaps/locations.xml` for city and office pages.","`/sitemaps/blog.xml` for articles.","`/sitemaps/es.xml` for Spanish-language pages."]},{"type":"paragraph","text":"Submit the index file once in the Sitemaps report. Google reads the children from it, and each child appears in the report with its own status. If the locations sitemap shows 60 submitted and 12 indexed while torts shows 25 of 25, you know exactly where to look."},{"type":"paragraph","text":"Firms moving from one tort to many, as described in this guide to [scaling from one tort to a multi-litigation portfolio](/blogs/scaling-single-tort-to-multi-litigation), add whole sections at a time, and a per-section sitemap is the simplest way to watch each launch. It is also a standard part of any [mass tort SEO](/services/mass-tort-seo) setup."},{"type":"heading","level":3,"text":"Image, video, news and hreflang sitemaps"},{"type":"paragraph","text":"Google supports sitemap extensions for images, video, news and language versions, but most firms need only the standard sitemap plus, sometimes, hreflang. Image sitemaps allow \"up to 1,000 `<image:image>` tags\" in each `<url>`. News sitemaps should include only articles \"created in the last two days\", with up to 1,000 `<news:news>` tags per sitemap ([Google Search Central, image sitemaps](https://developers.google.com/search/docs/crawling-indexing/sitemaps/image-sitemaps); [news sitemaps](https://developers.google.com/search/docs/crawling-indexing/sitemaps/news-sitemap))."},{"type":"paragraph","text":"Video is the one with a hidden condition. Google indexes a video only if the watch page itself is indexed and the video has \"a valid thumbnail that's available at a stable URL\" ([Google Search Central, video best practices](https://developers.google.com/search/docs/appearance/video)). A video sitemap cannot rescue a video on a page Google has not indexed."},{"type":"paragraph","text":"For Spanish-language pages, hreflang can be declared in HTML, in HTTP headers or in a sitemap; the three methods are equivalent ([Google Search Central, localized versions](https://developers.google.com/search/docs/specialty/international/localized-versions)). The sitemap route is easiest when your CMS generates the alternate links for you."},{"type":"paragraph","text":"In my view, most firms need none of the news extension. The old sitemap ping is retired, as noted earlier."},{"type":"heading","level":2,"text":"Sitemaps vs Request Indexing vs IndexNow vs Indexing API"},{"type":"paragraph","text":"Sitemaps and Request Indexing are the two ways to prompt Google; IndexNow notifies Bing and other participating engines but not Google; the Indexing API is limited to job-posting and livestream pages. Mixing them up is one of the most common errors in indexing advice, so the comparison below lists what each one reaches and when to use it."},{"type":"image","src":"/images/blog-content/google-index/04-submission-methods.jpg","alt":"Comparison matrix of four ways to notify search engines: sitemap, Request indexing, IndexNow and the Indexing API","caption":"Four ways to tell search engines about URLs, and which engines each one reaches."},{"type":"table","header":["Method","Engines","Scope","Limits","Use it when"],"rows":[["XML sitemap","Google, Bing and others that read sitemaps","Every URL you want found","50,000 URLs or 50MB per file; a hint only","Always, for the whole site"],["Request indexing (URL Inspection)","Google","One URL per request","Daily limit per property, number not published","A key page is new or changed"],["IndexNow","Bing, Yandex, Seznam, Naver, Yep and other participants","Up to 10,000 URLs per POST","Not used by Google; 200 means received only","You want Bing and others told at once"],["Google Indexing API","Google","JobPosting or BroadcastEvent-in-VideoObject pages only","200 publish requests per day per project","You publish job postings or livestreams"],["Search Console API sitemap submit","Google","Sitemap files","Same rules as a sitemap","You automate sitemap submission"]]},{"type":"paragraph","text":"Sitemaps can also be submitted \"programmatically\" through the Search Console API. For Request indexing, Google says \"There is a daily limit to how many index requests you can submit\" and \"Submitting a request does not guarantee that the page will appear in the Google Index\" ([Search Console help](https://support.google.com/webmasters/answer/9012289?hl=en))."},{"type":"heading","level":3,"text":"Which method to use when"},{"type":"paragraph","text":"The right method depends on how many URLs changed and which engines you care about. These decision rules cover nearly every situation a law-firm site will face:"},{"type":"list","ordered":true,"items":["**New site.** Submit the sitemap index in the Sitemaps report, then request indexing for the homepage in URL Inspection.","**One important new or changed page.** Request indexing for that URL after the live test passes. Google won't accept a request for a page \"considered to be non-indexable in the live test\".","**Many changed pages.** Don't request each one. Update the sitemap with accurate `<lastmod>` dates; Google's own advice for many pages is to \"submit a sitemap … with the updated pages marked by `<lastmod>`.\"","**Bing and other engines.** Turn on IndexNow, usually through your CMS, SEO plugin or CDN, and keep a sitemap submitted in Bing Webmaster Tools too.","**Normal content pages.** Never use the Google Indexing API, however a plugin markets it."]},{"type":"paragraph","text":"Google does not publish the daily number, so save requests for pages that matter, such as a new tort page or one wrongly set to `noindex`."},{"type":"heading","level":3,"text":"Why the Indexing API is off-limits for most sites"},{"type":"paragraph","text":"Google's Indexing API is reserved for two page types, and using it for anything else risks losing access. The quickstart says the API \"can only be used to crawl pages with either JobPosting or BroadcastEvent embedded in a VideoObject.\" Submissions \"undergo rigorous spam detection\", and abuse, \"including the use of multiple accounts or other means to exceed usage quotas, may result in access being revoked\" ([Google Indexing API quickstart](https://developers.google.com/search/apis/indexing-api/v3/quickstart))."},{"type":"paragraph","text":"The default quota is 200 publish requests per day per project, resetting at midnight Pacific time, and the API \"is currently free to use\" ([Indexing API quota and pricing](https://developers.google.com/search/apis/indexing-api/v3/quota-pricing))."},{"type":"paragraph","text":"Some plugins and tools offer to push any URL through the Indexing API, and some guides describe it as \"primarily\" for jobs and livestreams. Google's documentation says \"only\". A practice-area page, blog post or tort page does not qualify, whatever a tool's settings screen allows."},{"type":"paragraph","text":"In my view, the one realistic exception for a law firm is a careers page whose open positions carry valid JobPosting markup. Everything else goes through sitemaps and Request indexing."},{"type":"heading","level":3,"text":"Where IndexNow fits"},{"type":"paragraph","text":"IndexNow is an open protocol that tells participating search engines a URL was added, updated or deleted; Google is not among the participants. indexnow.org describes it as \"a simple ping so that search engines know that a URL and its content has been added, updated, or deleted\", and its home page names support from \"Microsoft Bing, Naver, Seznam.cz, Yandex, Yep\" ([indexnow.org](https://www.indexnow.org/)). The machine-readable list has seven entries (bing, yandex, seznam, naver, yep, internetarchive and amazonbot) and does not include Google ([searchengines.json](https://www.indexnow.org/searchengines.json))."},{"type":"paragraph","text":"The mechanics are simple. You host a key of 8 to 128 characters as a UTF-8 text file, then send URLs, up to 10,000 per POST. An HTTP 200 response \"only indicates that the search engine has received your URL\" ([IndexNow documentation](https://www.indexnow.org/documentation)). The FAQ adds that a submission \"will be shared across all IndexNow-enabled search engines\", that it \"does not guarantee immediate indexing\", that every submitted URL \"counts toward your site's crawl quota\", and that you should keep using an XML sitemap alongside it ([IndexNow FAQ](https://www.indexnow.org/faq))."},{"type":"paragraph","text":"Bing calls IndexNow \"a free, open-source protocol\" and notes that many CMSs and CDNs offer built-in support ([Bing IndexNow](https://www.bing.com/indexnow)), so turning it on is often a single setting. It is worth doing for Bing coverage, which matters for the AI answers covered later in this guide. It won't change how Google indexes your pages."},{"type":"heading","level":2,"text":"Internal links and site structure for indexing"},{"type":"paragraph","text":"Google finds most new pages by following links from pages it already knows, so a page that nothing links to is slow to be found and easy to ignore. Google's crawlers \"automatically visit publicly accessible webpages and follow links on those pages\" ([How Google Search organizes information](https://www.google.com/search/howsearchworks/how-search-works/organizing-information/)), and a link from a known page is one of the three ways a URL becomes known ([how Search works](https://developers.google.com/search/docs/fundamentals/how-search-works))."},{"type":"heading","level":3,"text":"Links Google can follow"},{"type":"paragraph","text":"Google reliably follows only standard HTML links: an `a` element with an `href` attribute. Its documentation says that, generally, Google can only crawl a link built that way, and that links built with `onclick` handlers, or a `span` element given an `href`, are not reliably parsed ([Google Search Central, crawlable links](https://developers.google.com/search/docs/crawling-indexing/links-crawlable))."},{"type":"paragraph","text":"That rules out several patterns common on law-firm sites built with page builders:"},{"type":"list","ordered":false,"items":["Buttons that navigate with JavaScript `onclick` handlers and no `href`.","Mega-menus that build their links only after a script runs.","Image tiles or cards that are clickable through script but contain no `a` element with an `href`.","\"Load more\" buttons that reveal links to older articles only after a click."]},{"type":"paragraph","text":"Google renders pages with a headless Chromium, yet still calls server-side rendering or pre-rendering \"a great idea\" because \"not all bots can run JavaScript\" ([JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics)). Broken links waste the same discovery path; see our guide to [fixing broken links and redirects](/blogs/link-not-working)."},{"type":"heading","level":3,"text":"Hub-and-spoke structure and orphan pages"},{"type":"paragraph","text":"A hub-and-spoke structure, where one hub page links to every page on a topic and each of those links back, gives every page a short, crawlable path from the homepage. For a plaintiff firm, the hubs are usually practice areas, torts and locations. The hub links to each child; each child links back to the hub and to closely related siblings."},{"type":"paragraph","text":"You can see the pattern on a live site in our own [hub of per-tort pages](/mass-tort-leads): one page that links out to each tort page, so a new tort becomes reachable the moment it is added to the hub."},{"type":"paragraph","text":"At launch, link a new page from the homepage, a main menu item or an established hub. Google notes that the canonical page \"will be crawled most regularly; duplicates are crawled less frequently\" ([Google Search Central, canonicalization](https://developers.google.com/search/docs/crawling-indexing/canonicalization)), so link to the canonical URL, not a parameter or tracking variant."},{"type":"paragraph","text":"An orphan page is a page with no internal links pointing to it. It may still be in the sitemap, which is how Google can know it exists, but nothing on the site vouches for it. Orphans are common after redesigns."},{"type":"paragraph","text":"To find orphans, compare two lists. Export every URL in your sitemaps, then run a crawler from the homepage that follows only links. URLs in the sitemap that the crawler never reached are orphans. The crawler tools are covered in the tools section later in this guide. Fix each orphan by linking it from its hub, or remove it from the sitemap if it no longer deserves to exist."},{"type":"heading","level":3,"text":"Backlinks as a discovery signal"},{"type":"paragraph","text":"Links from other websites are another way Google discovers your pages, because Google follows links from any page it already knows ([how Search works](https://developers.google.com/search/docs/fundamentals/how-search-works)). A new tort page cited by a news site, a bar association directory or a co-counsel partner can be found through that link."},{"type":"paragraph","text":"Google publishes no figure on how much backlinks speed up indexing, and no threshold for how many a page needs. Treat them as a useful extra path, not a lever you can pull on a schedule."},{"type":"paragraph","text":"Avoid paid \"link indexer\" services. Google says it \"never accepts payment to crawl a site more frequently\" ([How Google Search organizes information](https://www.google.com/search/howsearchworks/how-search-works/organizing-information/)), and links bought or generated to manipulate rankings can fall under its spam policies ([Google spam policies](https://developers.google.com/search/docs/essentials/spam-policies))."},{"type":"heading","level":3,"text":"Using the Links report"},{"type":"paragraph","text":"The **Links** report in Search Console shows which of your pages are most linked, internally and from other sites, and it is the quickest built-in way to spot pages with too few internal links. Open it from the left menu under **Links**. Its tables \"are limited to 1,000 rows\", and external data is grouped by root domain ([Search Console help, Links report](https://support.google.com/webmasters/answer/9049606?hl=en))."},{"type":"paragraph","text":"Open **Top linked pages** in the internal links section and sort the list ascending. Important pages near the bottom, such as a new tort page with two internal links while an old blog post has two hundred, are the ones to fix first. Then check external links to see which pages other sites cite."},{"type":"paragraph","text":"Some guides send you to Google Analytics for this; the Links report is a Search Console report."},{"type":"paragraph","text":"If a page is known to Google, listed in a clean sitemap and linked from its hub, yet still shows as not indexed, structure is no longer the likely cause. The next step is a structured diagnosis that tests each possible blocker in order, starting with the six-gate triage below."},{"type":"heading","level":2,"text":"Not indexed? The six-gate triage for any URL"},{"type":"paragraph","text":"A page gets into the Google index only if it passes six gates in order: Google knows the URL, is allowed to crawl it, can fetch and render it, is permitted to index it, picks it as the canonical, and judges it worth keeping. Find the first gate it fails and fix that one first."},{"type":"image","src":"/images/blog-content/google-index/05-six-gate-triage.jpg","alt":"Decision tree for a page not indexed, testing six gates in order: known, allowed, fetched, indexable, canonical, valued","caption":"The six-gate triage: test each gate in order and fix the first one that fails."},{"type":"paragraph","text":"Order matters: rewriting content does nothing for a page Google cannot fetch. Google's own list of [technical requirements](https://developers.google.com/search/docs/essentials/technical) covers the first four gates in three conditions: Googlebot isn't blocked, the page returns an HTTP 200, and the page has indexable content. Google adds that \"most sites pass the technical requirements without even realizing it,\" which is why the last two gates explain so many stubborn cases. Each gate below gives the question, where to look, the statuses that point to it and the fix."},{"type":"heading","level":3,"text":"Gate 1: Known"},{"type":"paragraph","text":"The first question is whether Google has ever seen the URL at all, because a page Google has never discovered cannot be crawled, let alone indexed. Google finds pages it has visited before, pages linked from pages it knows, and pages listed in a submitted sitemap, because [\"there isn't a central registry of all web pages\"](https://developers.google.com/search/docs/fundamentals/how-search-works)."},{"type":"list","ordered":false,"items":["**Question:** Does Google know this URL exists?","**Where to look:** **URL Inspection** for the exact URL.","**Statuses that point here:** the verdict [\"URL is unknown to Google\"](https://support.google.com/webmasters/answer/9012289?hl=en) (\"Google hasn't seen this URL before\").","**Fix:** link to the page from an indexed page, list it in your sitemap, then use **Request indexing** once."]},{"type":"paragraph","text":"\"Discovered - currently not indexed\" is not a gate 1 problem: Google already knows that URL."},{"type":"heading","level":3,"text":"Gate 2: Allowed"},{"type":"paragraph","text":"The second question is whether anything stops Googlebot from requesting the page. The usual blockers are a robots.txt `Disallow` rule, a firewall or bot-protection service that challenges Googlebot, and a login wall that returns 401 or 403."},{"type":"list","ordered":false,"items":["**Question:** Is Googlebot permitted to request this URL?","**Where to look:** the **Crawl allowed?** field in URL Inspection, and the **robots.txt report** under Settings.","**Statuses that point here:** \"URL blocked by robots.txt\", \"Blocked due to unauthorized request (401)\", \"Blocked due to access forbidden (403)\".","**Fix:** remove or narrow the robots.txt rule, allow verified Googlebot through the firewall, and drop the password from public pages."]},{"type":"paragraph","text":"On 403s, Google's [Page indexing help](https://support.google.com/webmasters/answer/7440203?hl=en) is blunt: \"Googlebot never provides credentials, so your server is returning this error incorrectly.\" The [robots.txt report](https://support.google.com/webmasters/answer/6062598?hl=en) shows the file Google last fetched for your top 20 hosts and lets you request a recrawl of it \"for emergency situations.\""},{"type":"heading","level":3,"text":"Gate 3: Fetched and rendered"},{"type":"paragraph","text":"The third question is whether Google received a working page and could see its content after running the JavaScript. A crawl that ends in a server error, a timeout, a broken redirect or an empty shell fails here even when robots.txt allows it."},{"type":"list","ordered":false,"items":["**Question:** Did Googlebot get a 200 response with the real content in it?","**Where to look:** **Test live URL** in URL Inspection, then the rendered HTML and screenshot.","**Statuses that point here:** \"Server error (5xx)\", \"Redirect error\", \"Page indexed without content\".","**Fix:** stabilize the server, shorten redirect chains, and make sure the main content is in the HTML or renders without errors."]},{"type":"paragraph","text":"Two less obvious causes live at this gate. Googlebot for Search fetches only the [first 2MB of an HTML file](https://developers.google.com/search/docs/crawling-indexing/googlebot), and anything past that cutoff never reaches indexing. And Google [renders pages with a recent version of Chrome](https://developers.google.com/search/docs/fundamentals/how-search-works); without rendering, \"Google might not see that content.\""},{"type":"heading","level":3,"text":"Gate 4: Indexable"},{"type":"paragraph","text":"The fourth question is whether the page tells Google not to index it. A `noindex` rule in a meta tag or in an HTTP header stops indexing even when every earlier gate passes, and a page that looks empty or like an error can be treated as a soft 404."},{"type":"list","ordered":false,"items":["**Question:** Does the page permit indexing, and does it look like a real page?","**Where to look:** the **Indexing allowed?** field in URL Inspection, plus the response headers in the live test.","**Statuses that point here:** \"URL marked 'noindex'\", \"Soft 404\".","**Fix:** remove the `noindex` from the template, plugin setting or server header, and give thin or error-like pages real content or a real 404."]},{"type":"paragraph","text":"Google's [noindex documentation](https://developers.google.com/search/docs/crawling-indexing/block-indexing) says the rule can be set \"with either a `<meta>` tag or HTTP response header,\" so check both places. Since December 2025 Google's [documentation change log](https://developers.google.com/search/updates) has also said: \"If there's a possibility that you do want the page indexed, don't use a noindex tag in the original page code.\""},{"type":"heading","level":3,"text":"Gate 5: Canonical"},{"type":"paragraph","text":"The fifth question is whether Google chose this URL, or a near-copy of it, to represent the content. Google groups similar pages into a cluster and indexes one canonical; the others are recorded as duplicates, not errors."},{"type":"list","ordered":false,"items":["**Question:** Is this URL the version Google selected?","**Where to look:** **User-declared canonical** and **Google-selected canonical** in URL Inspection.","**Statuses that point here:** \"Duplicate without user-selected canonical\", \"Duplicate, Google chose different canonical than user\", \"Alternate page with proper canonical tag\".","**Fix:** send consistent signals (redirects, `rel=\"canonical\"`, internal links and sitemap entries all pointing at one URL), or make the page different enough to stand alone."]},{"type":"paragraph","text":"Google's [canonicalization guide](https://developers.google.com/search/docs/crawling-indexing/canonicalization) calls a declared canonical \"a hint, not a rule.\" After a fix, be patient: Google's [canonical troubleshooting page](https://developers.google.com/search/docs/crawling-indexing/canonicalization-troubleshooting) says it \"might hold pages in a duplicate cluster for up to two weeks,\" one of the documented ranges from the timeline section earlier."},{"type":"heading","level":3,"text":"Gate 6: Valued"},{"type":"paragraph","text":"The sixth question is whether Google thinks the page adds enough to be worth keeping. When the URL is known, allowed, fetched, indexable and canonical but still shows \"Crawled - currently not indexed\", the remaining explanation is usually value, not code."},{"type":"list","ordered":false,"items":["**Question:** Does the page offer something the index does not already have?","**Where to look:** the Page indexing report, then a side-by-side read of the page against similar pages on your site.","**Statuses that point here:** \"Crawled - currently not indexed\" with no technical problem found in gates 1 to 5.","**Fix:** improve, merge or remove the page, then link to it from stronger pages."]},{"type":"paragraph","text":"Google states that it [\"doesn't guarantee that it will crawl, index, or serve your page\"](https://developers.google.com/search/docs/fundamentals/how-search-works), and lists low content quality as a common indexing issue. Quality, spam policies and manual actions get the full treatment in the quality and spam section below."},{"type":"paragraph","text":"Quick recap, gate by gate, with the first tool to open:"},{"type":"list","ordered":true,"items":["Known: URL Inspection verdict (\"URL is unknown to Google\").","Allowed: Crawl allowed? field and the robots.txt report.","Fetched and rendered: Test live URL, rendered HTML and screenshot.","Indexable: Indexing allowed? field and the response headers.","Canonical: user-declared versus Google-selected canonical.","Valued: Page indexing report and an honest content comparison."]},{"type":"heading","level":2,"text":"Every Page indexing status, explained"},{"type":"paragraph","text":"The Page indexing report sorts every not-indexed URL into one of 15 reasons, plus two warnings for indexed pages; each one points to a different fix. The table lists all 17; the sections after it go deeper."},{"type":"paragraph","text":"If a term in the table is new, our [glossary of marketing terms](/glossary) defines the basics. The labels, including Google's hyphen style in \"Crawled - currently not indexed\", come from the [Page indexing report help page](https://support.google.com/webmasters/answer/7440203?hl=en). The \"first fix\" column is the first thing to check, not the only one."},{"type":"table","header":["Status","What it means","First fix"],"rows":[["Server error (5xx)","Server failed when Googlebot asked","Check server logs and capacity"],["Redirect error","Chain too long, loop, bad or overlong URL","Point straight at the final URL"],["URL blocked by robots.txt","A Disallow rule stops crawling","Remove or narrow the rule"],["URL marked 'noindex'","Page or header says do not index","Remove noindex if wanted"],["Soft 404","200 response that looks empty or missing","Add content or return real 404"],["Blocked due to unauthorized request (401)","Login required","Remove auth from public pages"],["Not found (404)","Page does not exist","Restore, redirect, or leave if intended"],["Blocked due to access forbidden (403)","Server refuses Googlebot","Allow Googlebot; check firewall"],["URL blocked due to other 4xx issue","Another client error code","Inspect the exact status code"],["Crawled - currently not indexed","Fetched, not kept for now","Improve or merge the page"],["Discovered - currently not indexed","Known, crawl postponed","Strengthen links; check server"],["Alternate page with proper canonical tag","Points to another canonical","Usually none needed"],["Duplicate without user-selected canonical","Duplicate, no canonical declared","Add a canonical tag"],["Duplicate, Google chose different canonical than user","Google ignored your canonical","Align all canonical signals"],["Page with redirect","URL redirects elsewhere","None if the redirect is intended"],["Indexed, though blocked by robots.txt (warning)","Indexed from links, never crawled","Unblock and use noindex"],["Page indexed without content (warning)","Indexed, but content not read","Run Test live URL"]]},{"type":"paragraph","text":"Keep the report in proportion. Google says [\"Don't expect every URL on your site to be indexed\"](https://support.google.com/webmasters/answer/7440203?hl=en) and lists \"100% coverage\" as something not to aim for. Act on rising counts, in unexpected statuses, on URLs you want indexed."},{"type":"heading","level":3,"text":"Crawled - currently not indexed"},{"type":"paragraph","text":"\"Crawled - currently not indexed\" means Google fetched the page and chose not to add it, for now. Google's wording is: [\"The page was crawled by Google but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.\"](https://support.google.com/webmasters/answer/7440203?hl=en)"},{"type":"paragraph","text":"Repeated requests do not change the decision. Google's [How Search Works guide](https://developers.google.com/search/docs/fundamentals/how-search-works) names low content quality as a common indexing issue, and the same help page reminds you that indexing is not immediate and not total. In practice the likely causes, once gates 1 to 5 pass, are thin pages, near-duplicates of pages Google already has, pages that are mostly template with little unique text, and pages that add little next to similar content already in the index."},{"type":"paragraph","text":"A law-firm example: near-identical city pages for one practice area, covered in the law firm playbook below."},{"type":"paragraph","text":"The fix sequence:"},{"type":"list","ordered":true,"items":["Compare the page with its closest siblings on your site. If most of the visible text is shared boilerplate, that is the likely cause.","Decide: improve it with genuinely distinct content, merge it into a stronger page with a 301, or remove it.","Add contextual internal links from strong, indexed pages that are relevant to it.","Fix template-level issues across the set, not one URL at a time. If you want help with this kind of work, see [technical SEO for tort pages](/services/mass-tort-seo).","Request indexing once for the improved URL, then wait."]},{"type":"heading","level":3,"text":"Discovered - currently not indexed"},{"type":"paragraph","text":"\"Discovered - currently not indexed\" means Google knows the URL but has not crawled it yet. Google explains: [\"Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl. This is why the last crawl date is empty on the report.\"](https://support.google.com/webmasters/answer/7440203?hl=en)"},{"type":"paragraph","text":"Two inputs decide when the crawl happens. Google's [crawl budget guide](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget) describes crawl budget as \"the set of URLs that Google can and wants to crawl\", built from the crawl capacity limit (how much your server can take) and crawl demand (how much Google wants the content). Server errors and 429 responses lower the capacity limit."},{"type":"paragraph","text":"Do not jump to \"crawl budget\" on a small site. The same guide says it is \"intended primarily\" for sites with over a million unique pages changing about weekly, sites with 10,000+ pages changing daily, or sites where a large share of URLs is stuck in this status. For a typical firm or small business site, suspect weak internal linking and low perceived value first: pages reachable only through the sitemap, buried several clicks deep, or one of many similar URLs. Check **Settings > Crawl stats** for server problems too."},{"type":"heading","level":3,"text":"Duplicate and canonical statuses"},{"type":"paragraph","text":"Three statuses describe duplicates, and only two of them may need work. \"Alternate page with proper canonical tag\" is usually fine: the page points to another canonical and Google agreed. The other two mean Google is making the choice itself."},{"type":"paragraph","text":"\"Duplicate without user-selected canonical\" means Google found duplicates and you declared no canonical, so Google picked one. \"Duplicate, Google chose different canonical than user\" means you declared a canonical and Google indexed a different URL. Google's help page adds a hard rule: [\"If the user-declared canonical is not similar to the current page, then Google won't ever choose that URL as canonical.\"](https://support.google.com/webmasters/answer/7440203?hl=en) Pointing a canonical at an unrelated page does not work."},{"type":"paragraph","text":"Context from Google's [canonicalization guide](https://developers.google.com/search/docs/crawling-indexing/canonicalization):"},{"type":"list","ordered":false,"items":["A declared canonical \"is a hint, not a rule.\" Google picks the version that is \"objectively the most complete and useful for search users\".","\"Some duplicate content on a site is normal and it's not a violation of Google's spam policies.\" There is no duplicate content penalty for ordinary duplication.","Language versions count as duplicates \"only if the primary content is in the same language\". Same-language regional variants need both canonicalization and hreflang."]},{"type":"paragraph","text":"Signal strength matters. Google's [guide to consolidating duplicate URLs](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls) ranks redirects and `rel=\"canonical\"` as \"a strong signal\" and sitemap inclusion as \"a weak signal.\" Parameter URLs such as `?sort=price` are one of Google's most common causes of large duplicate counts."},{"type":"paragraph","text":"Fix list:"},{"type":"list","ordered":true,"items":["Give every indexable page a self-referencing canonical with an absolute URL.","Use one method per page. Don't declare one canonical in HTML and another in a header.","Point internal links, the sitemap and redirects at the same URL you declare.","Don't use robots.txt, the removal tool or `noindex` to choose a canonical.","With hreflang, point each page's canonical to a page in the same language."]},{"type":"paragraph","text":"A city-page example: if your Dallas and Fort Worth pages share nearly all their text, Google may fold one into the other. Make each page locally specific, then confirm the result by [tracking local rankings by city](/blogs/local-rank-tracker) rather than by guessing."},{"type":"heading","level":3,"text":"Soft 404, 404 and other 4xx statuses"},{"type":"paragraph","text":"A soft 404 is a page that returns 200 but looks like an error or an empty page, and Google treats it as missing. Google's [HTTP status documentation](https://developers.google.com/search/docs/crawling-indexing/http-network-errors) says: \"For Google Search, an HTTP 2xx (success) status code doesn't guarantee indexing.\""},{"type":"paragraph","text":"Common sources are empty search results, expired listings and single-page apps that return 200 for every route. Google's [JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) suggests redirecting to a URL that returns a real 404 or adding `noindex` on error views."},{"type":"paragraph","text":"Google says it \"doesn't index URLs that return a 4xx status code, and URLs that are already indexed and return a 4xx status code are removed from the index\". The exception is 429, which Google treats as a server error. Both 404 and 410 fall under that rule; Google's documentation does not promise a speed difference between them, so pick whichever describes the page honestly."},{"type":"paragraph","text":"Two notes on the \"Not found (404)\" row. Google says \"there is no way to tell Googlebot to permanently forget a URL, although it will crawl it less and less often\", and the report shows only 404s from the past month. A 404 for a page you deleted on purpose is fine and needs no action."},{"type":"heading","level":3,"text":"Server errors, redirects and blocked requests"},{"type":"paragraph","text":"Server errors and broken redirects stop Google from reaching the content, and if they persist Google drops the URL. Google's [HTTP documentation](https://developers.google.com/search/docs/crawling-indexing/http-network-errors) says 5xx and 429 responses \"prompt Google's crawlers to temporarily slow down\", and \"Already indexed URLs are preserved in the index, but eventually dropped.\""},{"type":"paragraph","text":"Short outages are survivable. Google's [crawl rate guidance](https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate) allows 500, 503 or 429 for urgent cases lasting \"a couple of hours, or 1-2 days\", and warns that if Googlebot sees these codes on the same URL \"for multiple days, the URL may be dropped from Google's index.\""},{"type":"paragraph","text":"\"Redirect error\" covers four cases in the [Page indexing help](https://support.google.com/webmasters/answer/7440203?hl=en): a chain that was too long, a loop, a redirect URL past the maximum URL length, and a bad or empty URL in the chain. Google's crawlers follow up to 10 hops by default, but Google's inspection tools don't follow redirects at all, so inspect the final URL. 301 and 308 are \"a strong signal\" for the target; 302 and 307 are \"a weak signal.\" \"Page with redirect\" is not a problem, because redirect URLs are not indexed, only their targets."},{"type":"paragraph","text":"Blocked requests (401 and 403) usually mean a forgotten password, such as a staging login left on `/new-services/` after launch. Remove it, then use Validate fix."},{"type":"heading","level":3,"text":"Indexed, though blocked by robots.txt"},{"type":"paragraph","text":"This warning means Google indexed a URL it was never allowed to crawl, because other pages link to it. Google's words: [\"Google always respects robots.txt, but this doesn't necessarily prevent indexing if someone else links to your page\"](https://support.google.com/webmasters/answer/7440203?hl=en), and any snippet \"will probably be very limited\"."},{"type":"paragraph","text":"If you want the page out, Google's fix is to \"remove the robots.txt block and use 'noindex'.\" That sounds backwards until you see why: while robots.txt blocks the URL, Google cannot read the `noindex`. URL Inspection makes the trap visible, because when a page is blocked, [Indexing allowed? \"will always be 'Yes'\"](https://support.google.com/webmasters/answer/9012289?hl=en). If you want the page in, simply remove the block."},{"type":"paragraph","text":"The second warning, \"Page indexed without content\", means Google indexed the URL but could not read its content. It is often a fetch or rendering problem, such as content loaded by a script that failed or a server that returned a different response to Googlebot. Run **Test live URL** and compare the rendered HTML with what visitors see."},{"type":"heading","level":3,"text":"Using Validate fix"},{"type":"paragraph","text":"**Validate fix** asks Google to recheck the URLs affected by one issue after you have fixed it. Per the [Page indexing help](https://support.google.com/webmasters/answer/7440203?hl=en), validation \"can take several days or even longer,\" Google sends progress emails, only URLs with known instances of the issue are recrawled, and an issue closes 90 days after its last instance."},{"type":"paragraph","text":"When to click it:"},{"type":"list","ordered":false,"items":["After the fix is deployed to every affected URL, including template-level and server-level changes.","When the fix is binary and checkable, such as removing a `noindex`, lifting a robots.txt rule or fixing a 5xx."]},{"type":"paragraph","text":"When not to click it:"},{"type":"list","ordered":false,"items":["After a partial fix. Validation will fail on the URLs still broken and you start again.","For \"Crawled - currently not indexed\" pages you have not materially changed. Validation does not override a quality decision."]},{"type":"heading","level":2,"text":"noindex vs robots.txt vs canonical: what each controls"},{"type":"paragraph","text":"robots.txt controls crawling, `noindex` controls indexing, and `rel=\"canonical\"` suggests which duplicate to index; mixing them up is the most common reason pages stay in or out of Google by accident."},{"type":"image","src":"/images/blog-content/google-index/06-indexing-controls.jpg","alt":"Matrix comparing robots.txt, noindex, X-Robots-Tag, canonical, 404 or 410 and the Removals tool by what each one controls","caption":"Which control does what: crawling, indexing, canonical choice or temporary removal."},{"type":"paragraph","text":"The table compares the eight controls people reach for. Sources: Google's [robots.txt introduction](https://developers.google.com/search/docs/crawling-indexing/robots/intro), [noindex documentation](https://developers.google.com/search/docs/crawling-indexing/block-indexing), [Control what you share](https://developers.google.com/search/docs/crawling-indexing/control-what-you-share) page (updated 2026-10-21) and the [Removals tool help](https://support.google.com/webmasters/answer/9689846?hl=en)."},{"type":"table","header":["Control","Stops crawling?","Stops indexing?","Works on PDFs and files?","Use for"],"rows":[["robots.txt Disallow","Yes","No, URL can still appear","Yes, for crawling","Saving crawl on junk URLs"],["Meta robots noindex","No","Yes, once crawled","No, HTML only","Pages you want out"],["X-Robots-Tag: noindex","No","Yes, once crawled","Yes","PDFs, images, files"],["rel=\"canonical\"","No","Suggests one version","Via HTTP header only","Choosing among duplicates"],["301 redirect","No, target is crawled","Old URL replaced","Yes","Moved or merged pages"],["404 or 410","No","Yes, removed","Yes","Deleted content"],["Password protection","Yes, in effect","Yes","Yes","Private or staging content"],["Removals tool","No","Hides for about six months","Yes","Urgent temporary hiding"]]},{"type":"paragraph","text":"Three rows deserve a note. robots.txt \"is not a mechanism for keeping a web page out of Google\": a blocked URL \"can still appear in search results, but the search result won't have a description.\" `noindex` works only after Google crawls the page, and then Google \"will drop that page entirely from Google Search results, regardless of whether other sites link to it.\" The Removals tool blocks a URL \"for about six months\" and is described as \"the first step\" toward permanent removal, which still needs a 404 or 410, a password or `noindex`. Removal itself is covered in the law firm playbook below."},{"type":"heading","level":3,"text":"The noindex and Disallow trap"},{"type":"paragraph","text":"Combining `noindex` with a robots.txt `Disallow` on the same URL defeats the `noindex`, because Google never fetches the page to read it. Google's [noindex documentation](https://developers.google.com/search/docs/crawling-indexing/block-indexing) says the page \"must not be blocked by a robots.txt file\" for the rule to work."},{"type":"paragraph","text":"The result is the \"Indexed, though blocked by robots.txt\" warning described above: a URL known from links, indexed with no description. To keep a page out, allow crawling and use `noindex`. A minimal noindex tag is `<meta name=\"robots\" content=\"noindex\">` in the page `<head>`."},{"type":"paragraph","text":"Another old trick no longer works. Google's [2019 post on unsupported rules](https://developers.google.com/search/blog/2019/07/a-note-on-unsupported-rules-in-robotstxt) announced it would stop handling `noindex` inside robots.txt on September 1, 2019, noting such rules were \"contradicted by other rules in all but 0.001% of all robots.txt files.\" Google's docs now say it \"is not supported by Google.\""},{"type":"paragraph","text":"robots.txt itself is a formal standard: [IETF RFC 9309](https://www.rfc-editor.org/rfc/rfc9309.html), published September 2022, which extends the method \"originally defined by Martijn Koster in 1994.\" Google's [robots.txt specification](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) adds the rules that trip people up:"},{"type":"list","ordered":false,"items":["The file sits in the top-level directory, is UTF-8, and has a 500 KiB size limit.","Supported fields are user-agent, allow, disallow and sitemap; \"crawl-delay aren't supported\".","Path values are case-sensitive, so `/Thank-You/` and `/thank-you/` are different rules.","The most specific rule wins; between equal conflicting rules \"Google uses the least restrictive rule.\"","Google \"generally caches the contents of robots.txt file for up to 24 hours\", so changes are not instant.","If robots.txt returns 5xx, Google stops crawling the site for 12 hours, then uses the last good copy for up to 30 days."]},{"type":"paragraph","text":"A safe example that keeps crawlers out of a thank-you page folder, written line by line:"},{"type":"list","ordered":true,"items":["`User-agent: *`","`Disallow: /thank-you/` That stops crawling, not indexing; if the pages are already indexed, `noindex` them first and add the Disallow after they drop."]},{"type":"heading","level":3,"text":"X-Robots-Tag for PDFs and other files"},{"type":"paragraph","text":"The `X-Robots-Tag` HTTP header carries the same rules as the robots meta tag, and it is the only way to `noindex` files that have no HTML `<head>`. Google's [noindex documentation](https://developers.google.com/search/docs/crawling-indexing/block-indexing) says it \"can be used for non-HTML resources, such as PDFs, video files, and image files.\""},{"type":"paragraph","text":"Google's [robots meta tag documentation](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag) gives server examples for PDFs. For Apache, in `.htaccess` or `httpd.conf`:"},{"type":"list","ordered":true,"items":["`<Files ~ \"\\.pdf$\">`","`Header set X-Robots-Tag \"noindex, nofollow\"`","`</Files>`"]},{"type":"paragraph","text":"For NGINX, in the server configuration:"},{"type":"list","ordered":true,"items":["`location ~* \\.pdf$ {`","`add_header X-Robots-Tag \"noindex, nofollow\";`","`}`"]},{"type":"paragraph","text":"The valid rules include `noindex`, `nofollow`, `none` (\"Equivalent to noindex, nofollow\"), `nosnippet`, `max-snippet:[n]` and `unavailable_after:[date]`, among others. They are case-insensitive, and when rules conflict \"the more restrictive rule applies.\""},{"type":"paragraph","text":"The flip side is the accidental block. A `.htaccess` or server rule written for one folder can send `noindex` for the whole site, and the page HTML will look perfectly clean. When a page reports \"URL marked 'noindex'\" and you cannot find the tag in the source, check the response headers in the URL Inspection live test, then check server config, CDN rules and plugins."},{"type":"heading","level":3,"text":"The WordPress visibility checkbox"},{"type":"paragraph","text":"WordPress has a single setting that hides the whole site from search engines, and leaving it on after launch is one of the most common indexing failures. It lives at **Settings → Reading → Search Engine Visibility**, as the checkbox \"Discourage search engines from indexing this site,\" per the [WordPress documentation](https://wordpress.org/documentation/article/settings-reading-screen/)."},{"type":"paragraph","text":"What it does depends on the version. Since WordPress 5.3 it outputs a `noindex,nofollow` robots meta tag on every page; up to 5.2 it served `Disallow: /` in robots.txt instead. WordPress itself notes, \"It is up to search engines to honor your request.\""},{"type":"paragraph","text":"The usual story: a developer ticks the box on staging, the site launches, and weeks later nothing is indexed. URL Inspection reports \"URL marked 'noindex'\" on every page. Put the check in your [launch QA for new landing pages](/services/web-design). Other platforms and site builders have similar site-wide visibility settings, so look for one in any CMS before launch."},{"type":"heading","level":3,"text":"nofollow is a hint, not a block"},{"type":"paragraph","text":"`nofollow` does not stop Google from crawling or indexing a page; it is a hint. Google introduced `nofollow` in 2005, and in its [2019 announcement](https://developers.google.com/search/blog/2019/09/evolving-nofollow-new-ways-to-identify) made `nofollow`, `sponsored` and `ugc` all hints, adding: \"For crawling and indexing purposes, nofollow will become a hint as of March 1, 2020.\""},{"type":"paragraph","text":"Strictly, `nofollow` is a value of a link's `rel` attribute (or a page-level robots rule), not a standalone meta tag. Google says relying on it to block indexing \"was never recommended.\""},{"type":"paragraph","text":"For your own site, Google's [guidance on qualifying outbound links](https://developers.google.com/search/docs/crawling-indexing/qualify-outbound-links) is direct: \"For links within your own site, use the robots.txt disallow rule.\" If the aim is to keep a page out of the index, use `noindex`. Use `nofollow` for what it is for: outbound links you don't want to vouch for."},{"type":"heading","level":2,"text":"Rendering, mobile-first and server blockers"},{"type":"paragraph","text":"Google indexes what its smartphone crawler can fetch and render, so content that only loads after JavaScript runs, only exists on desktop, or sits behind a bot wall may never reach the index."},{"type":"heading","level":3,"text":"JavaScript rendering"},{"type":"paragraph","text":"Google can render JavaScript, but rendering is a separate, later step, and content that depends on it is at more risk. Google's [JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) describes three phases: crawling, rendering and indexing. A page may wait in the render queue \"for a few seconds, but it can take longer than that,\" and rendering uses \"a headless Chromium.\" Since 2019 Googlebot has been [evergreen](https://developers.google.com/search/blog/2019/05/the-new-evergreen-googlebot), running a current Chromium."},{"type":"paragraph","text":"Google still calls server-side rendering or pre-rendering \"a great idea\" because \"not all bots can run JavaScript.\" Its other rules: use the History API, not `#/` fragments, for routes; don't use JavaScript to change the canonical away from the original HTML; and remember that Google's [change log](https://developers.google.com/search/updates) notes non-200 pages may not be sent to rendering at all."},{"type":"heading","level":3,"text":"Mobile-first indexing"},{"type":"paragraph","text":"Google indexes the mobile version of your pages. Its [mobile-first indexing guide](https://developers.google.com/search/docs/crawling-indexing/mobile/mobile-sites-mobile-first-indexing) says Google \"uses the mobile version of a site's content, crawled with the smartphone agent, for indexing and ranking.\" On 2023-10-31 Google said the move, begun in 2016, was [\"now complete\"](https://developers.google.com/search/blog/2023/10/mobile-first-is-here)."},{"type":"paragraph","text":"Most crawl requests come from Googlebot Smartphone, and both Googlebot types obey one robots.txt token. Keep the same content and robots meta tags on mobile and desktop. In my view, the thing to check is content hidden or removed on small screens: disclaimers, FAQs or case-result sections cut from the mobile template are, for Google, not on the page."},{"type":"heading","level":3,"text":"The 2MB fetch limit"},{"type":"paragraph","text":"When crawling for Google Search, Googlebot fetches only the [first 2MB of an HTML file](https://developers.google.com/search/docs/crawling-indexing/googlebot) and the first 64MB of a PDF. The limit applies to uncompressed data, each CSS or JS file is fetched separately under the same limit, and only the downloaded part goes to indexing."},{"type":"paragraph","text":"Google's [March 2026 crawler post](https://developers.google.com/search/blog/2026/03/crawler-blog-post) adds that the count includes the HTTP header, that the fetch \"stops … exactly at the 2MB cutoff,\" and that 15MB is the default only for other crawlers that don't set a limit. Guides that cite a 15MB limit for Googlebot draw on [Google's 2022 explanation](https://developers.google.com/search/blog/2022/06/googlebot-15mb), which this replaces. Inline base64 images and huge inline scripts are the usual way a page passes 2MB."},{"type":"heading","level":3,"text":"Firewalls, bot protection and verifying Googlebot"},{"type":"paragraph","text":"Security tools that challenge or block unknown bots can block Googlebot too, so the rule is to allow verified Googlebot rather than challenge it. Google's [verification method](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot): run a reverse DNS lookup on the IP, check the hostname ends in `googlebot.com`, `google.com` or `googleusercontent.com`, then run a forward DNS lookup and confirm it returns the same IP. Alternatively, match the IP against Google's published IP-range files, which [moved on 2026-03-31](https://developers.google.com/search/blog/2026/03/crawler-ip-ranges) to `developers.google.com/crawling/ipranges/`."},{"type":"paragraph","text":"Expect a modest load: Google says Googlebot [shouldn't access most sites \"more than once every few seconds on average.\"](https://developers.google.com/search/docs/crawling-indexing/googlebot) Its crawlers support [HTTP/1.1 and HTTP/2](https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers), though HTTP/2 brings \"no ranking boost\", plus gzip, deflate and Brotli. If you truly need less crawling, return 503 or 429 briefly rather than 401 or 403, which Google says not to use for limiting crawl rate."},{"type":"paragraph","text":"When every technical gate in this section passes and a page is still not indexed, the remaining causes are quality, policy and intent. The next section starts there."},{"type":"heading","level":2,"text":"Quality, spam and security: when Google says no"},{"type":"paragraph","text":"Google chooses what to index, so a page that passes every technical check can still be left out if it adds little beyond pages Google already has, breaks a spam policy, or sits on a hacked site. Once the six gates in the triage come back clean, the remaining questions are about value, policy and trust rather than code."},{"type":"heading","level":3,"text":"The quality threshold"},{"type":"paragraph","text":"Google is direct that inclusion is selective. Its developer guide says \"Indexing isn't guaranteed; not every page that Google processes will be indexed,\" and it names [low content quality as one of three common indexing issues](https://developers.google.com/search/docs/fundamentals/how-search-works), alongside robots meta rules and site design that makes indexing difficult. The Page indexing help page adds that you should not expect every URL to be indexed and lists [\"100% coverage\" under what not to look for](https://support.google.com/webmasters/answer/7440203?hl=en). Google's recrawl documentation explains the bias: its systems [\"prioritize the fast inclusion of high quality, useful content.\"](https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl)"},{"type":"paragraph","text":"For law firms, one definition matters more than the rest. Google's helpful-content guidance says its systems give more weight to content with strong E-E-A-T \"for topics that could significantly impact the health, financial stability, or safety of people, or the welfare or well-being of society. We call these 'Your Money or Your Life' topics\" ([Google, creating helpful content](https://developers.google.com/search/docs/fundamentals/creating-helpful-content)). That page does not list legal topics by name. My view: injury claims and mass tort litigation plainly affect people's finances and safety, so attorney-reviewed pages with specific, current facts are the safest bet for getting indexed and staying there."},{"type":"heading","level":3,"text":"Doorway and scaled content policies"},{"type":"paragraph","text":"Two spam policies describe the page sets plaintiff firms are most tempted to build. Google's doorway examples include \"Having multiple domain names or pages targeted at specific regions or cities that funnel users to one page\" and \"Creating substantially similar pages that are closer to search results than a clearly defined, browseable hierarchy\" ([Google spam policies](https://developers.google.com/search/docs/essentials/spam-policies)). A hundred city pages that differ only by the city name fit the first example almost word for word."},{"type":"paragraph","text":"The scaled content policy covers \"many pages … generated for the primary purpose of manipulating search rankings and not helping users,\" and one listed example is \"Using generative AI tools or other similar tools to generate many pages without adding value for users.\" The same page, updated in August 2026, now defines spam to include \"attempting to manipulate generative AI responses in Google Search.\" The test Google applies is purpose and value, not the tool used to write the page. A page drafted with AI help and then checked, localized and reviewed by an attorney is a different object from a thousand unreviewed pages pushed out in an afternoon."},{"type":"heading","level":3,"text":"Manual actions and hacked content"},{"type":"paragraph","text":"A manual action is issued when \"a human reviewer at Google has determined that pages on the site are not compliant\" with its policies ([Manual actions report help](https://support.google.com/webmasters/answer/9044175?hl=en)). Two details from that page shape any cleanup plan. First, \"Fixing the issue on just some pages will not earn you a partial return to search results,\" so fix every affected page before you file. Second, \"Most reconsideration reviews can take several days or weeks,\" so plan paid coverage for the gap."},{"type":"paragraph","text":"Hacked content is \"any content placed on a site without permission, due to vulnerabilities in a site's security.\" The [Security issues report](https://support.google.com/webmasters/answer/9044101?hl=en) covers hacked content, malware and unwanted software, and social engineering, and affected pages can show a warning in results or an interstitial in the browser. A common scenario on law-firm WordPress sites: an outdated plugin lets an attacker inject thousands of spam pages, which then show up as new indexed URLs or new \"Crawled - currently not indexed\" entries. Patch, clean, then request a review. Google also changed its site reputation policy so that, [beginning August 30, 2026](https://developers.google.com/search/blog/2026/08/update-site-reputation-policy), manual actions under it work differently for searchers in the EEA than elsewhere."},{"type":"heading","level":3,"text":"Indexed but not ranking"},{"type":"paragraph","text":"If URL Inspection says \"URL is on Google\" but the page never shows for your target query, you have a ranking problem, not an indexing one. Google says plainly that [\"'URL is on Google' doesn't actually guarantee that your page will appear in Search results\"](https://support.google.com/webmasters/answer/9012289?hl=en), and that ranking \"is done programmatically\" with [no payment accepted to rank higher](https://developers.google.com/search/docs/fundamentals/how-search-works). Check four things in order: does the page match the search intent (a guide versus a contact page), is it deeper than the pages that do rank, do your own important pages link to it, and does anyone outside your site link to it. The [personal injury lawyer marketing guide](/personal-injury-lawyer-marketing) covers the ranking side in detail."},{"type":"heading","level":3,"text":"When a whole site drops out"},{"type":"paragraph","text":"When the indexed count falls to near zero across the board, work through a fixed order rather than guessing:"},{"type":"list","ordered":true,"items":["**Manual actions report.** A sitewide action explains everything at once.","**Security issues report.** Hacked or malware flags can strip visibility.","**robots.txt report.** Look for a deploy that shipped `Disallow: /` from a staging build.","**Response headers.** A sitewide `X-Robots-Tag: noindex` added by a server or CDN rule is invisible in the page source.","**Removals.** Another owner can file removals, since owners have [\"full control\"](https://support.google.com/webmasters/answer/7687615?hl=en) of a property. Check the Removals history for prefix requests you did not make.","**Crawl stats host status.** DNS and server connectivity failures show under [Settings > Crawl stats](https://support.google.com/webmasters/answer/9679690?hl=en), and a robots.txt file returning 5xx makes Google [stop crawling the site for the first 12 hours](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt)."]},{"type":"paragraph","text":"For scale, Google's Gary Illyes reportedly showed an owner removal in Search Console taking about 2 hours typically and 24 hours at the slowest, [as reported by Search Engine Roundtable](https://www.seroundtable.com/google-crawling-indexing-serving-data-42225.html); he called the slides \"an exercise,\" so treat that as indicative. The point is that an accidental removal can empty results fast, and the fix starts in the Removals report. Forum threads about a site \"removed completely\" from Google usually trace back to one of these six checks."},{"type":"paragraph","text":"Prevention is cheaper than recovery. Limit Owner access to the people who need it, review the user list when an agency or staff member leaves, and add a pre-deploy check that fails the release if robots.txt contains `Disallow: /` or the homepage returns a `noindex` header. Both checks take minutes to set up and catch the two most common self-inflicted drops."},{"type":"heading","level":2,"text":"How to remove pages and control what gets indexed"},{"type":"paragraph","text":"Keep a page out permanently with `noindex`, a 404 or 410, or a password; use the Removals tool only to hide a URL fast while one of those takes effect. Everything else in this section is about choosing which pages deserve to be indexed at all."},{"type":"heading","level":3,"text":"Pages to noindex on purpose"},{"type":"paragraph","text":"Some pages are useful to visitors and useless in search results. In my view these are the usual candidates on a law-firm site:"},{"type":"list","ordered":false,"items":["Thank-you and intake confirmation pages, which should never be a search entry point.","Internal site search results, which create endless thin URLs.","Paid landing page variants built for ad tests (more on these below).","Tag, date and author archives that repeat the same posts.","Staging and preview environments (better behind a password).","Print views and other alternate renderings of the same page."]},{"type":"paragraph","text":"Two cautions from Google. It does not recommend `noindex` to choose a canonical within a site, [\"because it will completely block the page from Search\"](https://developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls); use a canonical tag or a redirect for duplicates. And a December 2025 clarification in Google's [documentation change log](https://developers.google.com/search/updates) reads: \"If there's a possibility that you do want the page indexed, don't use a noindex tag in the original page code.\" Removing it later with JavaScript is not a reliable way back in."},{"type":"heading","level":3,"text":"The Removals tool is temporary"},{"type":"paragraph","text":"The Removals tool in Search Console blocks a URL from Google Search [\"for about six months\"](https://support.google.com/webmasters/answer/9689846?hl=en), and Google calls it \"the first step\" toward permanent removal. It works on a single URL or with \"Remove all URLs with this prefix,\" which is handy after a hack or a leaked staging folder. For permanent removal, Google lists three methods: return 404 or 410, require a password, or use `noindex`."},{"type":"paragraph","text":"One trap catches people: \"If your URL is unreachable by Google (404, 502/3) when you use this tool, it will assume that the page is gone, and your block request will expire.\" Do not rely on the tool to hold a page that is already erroring. URL Inspection does not remove anything either; it inspects and requests crawls. Google's updated [\"Control what you share\" page](https://developers.google.com/search/docs/crawling-indexing/control-what-you-share) (October 21, 2026) puts removing the content from your site first, as \"the best way to ensure that it won't appear in Google Search and anywhere else on the Internet,\" followed by password protection and `noindex`."},{"type":"table","header":["Goal","Use","Lasts"],"rows":[["Hide a URL today","Removals tool","About six months"],["Keep a live page out","`noindex` meta or header","While the rule stays"],["Delete the page","404 or 410","Permanent"],["Private content","Password","Permanent"]]},{"type":"paragraph","text":"A typical emergency, such as a staging folder or a leaked intake export that got indexed, runs like this:"},{"type":"list","ordered":true,"items":["In Removals, file a prefix request for the folder so results disappear while you work.","Put the folder behind a password or delete it, so the permanent fix is in place before the six months run out.","Do not add a robots.txt block for the folder: Google needs to recrawl the URLs to see the 404, 410 or `noindex`.","Re-inspect a sample of the URLs a few weeks later and confirm the removal holds without the temporary request."]},{"type":"heading","level":3,"text":"Outdated content on sites you do not own"},{"type":"paragraph","text":"When another site's page still shows an old snippet or cached text, the [Refresh Outdated Content tool](https://support.google.com/webmasters/answer/7041154?hl=en) is the route. It is for pages \"you do not own,\" and it fails if the page \"has not been removed or had significant changes.\" A typical case: an old directory listing still shows a former partner's name in Google's snippet even though the directory already updated the page. Ask the site owner to change or remove the page first, then submit the URL. If the content is still live and unchanged, the tool will not help."},{"type":"heading","level":3,"text":"Index bloat audits"},{"type":"paragraph","text":"Index bloat is the gap between what Google has indexed and what you actually want indexed. A quarterly audit takes an afternoon:"},{"type":"list","ordered":true,"items":["Export the list of URLs you want indexed: practice-area pages, tort pages, attorney bios, live blog posts.","Compare it with the Page indexing report's indexed count, filtered by each sitemap, and with a crawl of the site.","Sort the surplus by source. Google names the usual ones: [duplicates from filter or sort parameters](https://support.google.com/webmasters/answer/7440203?hl=en) (for example `sort=price`), redirects, and robots.txt rules. On law-firm sites, add date archives, media attachment pages and old campaign pages.","Act on each group: canonical or remove links for parameters, `noindex` for archives, redirect attachment pages to their parent, and retire dead campaign pages (covered in the playbook below)."]},{"type":"paragraph","text":"I would not set a target ratio. Google publishes none, and a small site with ten bloated URLs is in better shape than a large one with ten thousand."},{"type":"heading","level":2,"text":"Indexing playbook for law firm and mass tort sites"},{"type":"paragraph","text":"Plaintiff firms publish exactly the page types Google is pickiest about: time-sensitive tort pages, near-duplicate city pages, paid landing pages, Spanish versions and PDFs. Each needs its own indexing decision."},{"type":"image","src":"/images/blog-content/google-index/07-tort-page-launch.jpg","alt":"Timeline of a new mass tort page launch: publish, internal links, sitemap, request indexing, verify and monitor","caption":"A launch sequence for a new tort page: publish, link, list, request, verify and monitor."},{"type":"paragraph","text":"At Mass Tort Marketing Agency we build tort campaigns where a page that is not indexed when litigation news breaks loses the early organic window, so indexing is a step in the launch plan. See [our mass tort launch checklist](/resources/mass-tort-launch-checklist), or [tell us what you need](/get-started) if you want the launch handled."},{"type":"paragraph","text":"The table maps the page types most plaintiff sites carry to an indexing decision. Treat it as a starting point; your intake flow and ad setup may change a row."},{"type":"table","header":["Page type","Index?","How","Why"],"rows":[["Practice-area page","Yes","Sitemap + internal links","Core search entry point"],["Tort page","Yes","Sitemap + internal links","Time-sensitive demand"],["City page (unique)","Yes","Sitemap + internal links","Real local value"],["City page (template-only)","No","301 to hub","Doorway risk"],["Paid landing page variant","No","`noindex`","Duplicates organic page"],["Thank-you or intake confirmation","No","`noindex`","No search value"],["Spanish page","Yes","Sitemap + hreflang","Separate audience"],["Settlement or verdict PDF","Usually no","`X-Robots-Tag`","HTML summary ranks better"],["Attorney bio","Yes","Sitemap + internal links","Trust and name searches"],["Disclaimer or attorney-advertising page","Firm choice","Keep crawlable","Compliance reference"],["Blog post","Yes","Sitemap + internal links","Topic coverage"],["Expired campaign page","No","301 to hub","Litigation closed"]]},{"type":"paragraph","text":"The disclaimer row is a business call, not a technical one. Whether the page is indexed matters less than keeping it crawlable and linked so visitors can find it. Lawyer advertising rules are set by each state's rules of professional conduct, most of which build on ABA Model Rule 7.1 and the rules that follow it, so they vary by state; nothing here is legal advice. Our page on [mass tort marketing compliance](/mass-tort-marketing-compliance) covers the advertising side."},{"type":"heading","level":3,"text":"Launching a new tort page"},{"type":"paragraph","text":"A new tort page should follow the same sequence every time, in this order:"},{"type":"list","ordered":true,"items":["Publish with a 200 status and a self-referencing canonical.","Link it from the tort hub and the homepage, or a news section, the same day.","Add it to the tort sitemap with an accurate `<lastmod>`. Google uses `<lastmod>` only [\"if it's consistently and verifiably … accurate\"](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap).","In URL Inspection, click **Test live URL**, then **Request indexing**. You [\"cannot request indexing if the page is considered to be non-indexable in the live test\"](https://support.google.com/webmasters/answer/9012289?hl=en).","Re-inspect after a few days and confirm \"URL is on Google\" and that the Google-selected canonical matches yours.","Monitor the Page indexing report filtered by the tort sitemap."]},{"type":"paragraph","text":"After launch, update `<lastmod>` and request again only for significant changes, such as a new MDL ruling or a settlement. Google says not to ask for a recrawl [\"unless there is an important change, and Google doesn't seem to have noticed it for a while (a week or more).\"](https://support.google.com/webmasters/answer/7440203?hl=en) Expect the ranges in the timeline table earlier in this guide, not a fixed turnaround. Because organic indexing takes days at best, firms often run [mass tort PPC campaigns](/services/mass-tort-ppc) to cover the first weeks of a new tort."},{"type":"paragraph","text":"If the page is still \"URL is not on Google\" after two weeks, run the triage from earlier in this guide before requesting again. On tort pages the usual causes are specific: the page went live from a template that still carried a `noindex` from staging; the canonical points at a draft URL; the page is linked only from a menu that renders with JavaScript; or the copy is close to a page competitors and co-counsel already published, so Google has nothing new to index. Repeating the request will not help, since asking for the same URL multiple times [\"won't get it crawled any faster\"](https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl)."},{"type":"heading","level":3,"text":"Paid landing pages and AdsBot"},{"type":"paragraph","text":"Paid landing pages need two different crawlers treated differently. Google's Ads crawlers, AdsBot-Google and AdsBot-Google-Mobile, check landing page ad quality, and for them [\"The global user agent ( * ) is ignored.\"](https://developers.google.com/search/docs/crawling-indexing/google-special-case-crawlers) A `User-agent: *` block in robots.txt does not stop AdsBot; you would have to name it, which you should not do for pages you advertise."},{"type":"paragraph","text":"On the indexing side, `<meta name=\"robots\" content=\"noindex\">` [\"applies to search engine crawlers\"](https://developers.google.com/search/docs/crawling-indexing/robots-meta-tag). Google notes that non-search crawlers such as AdsBot-Google need rules targeted to them by name. So the safe setup for ad variants is: keep them crawlable, add the generic robots `noindex` so they stay out of organic results, and do not write any AdsBot-specific block. Our page on [PPC landing pages for law firms](/services/law-firm-ppc) covers the conversion side of these pages."},{"type":"paragraph","text":"The mistake to avoid is the reverse setup: putting ad variants under a `Disallow` in robots.txt to keep them out of search. That does not stop AdsBot, which ignores the `*` group, but it does stop Googlebot from reading the `noindex`, so a variant that picks up links can still be indexed without a description. Keep the decision in the page, not in robots.txt."},{"type":"heading","level":3,"text":"City and practice-area page sets"},{"type":"paragraph","text":"City pages are where firms most often drift into the doorway examples quoted earlier: \"pages targeted at specific regions or cities that funnel users to one page.\" The usual symptom is a pile of city URLs sitting in \"Crawled - currently not indexed,\" which Google describes as [\"crawled by Google but not indexed. It may or may not be indexed in the future\"](https://support.google.com/webmasters/answer/7440203?hl=en)."},{"type":"paragraph","text":"A city page earns its place when it carries facts a visitor in that city cannot get from the statewide page: the office address and the attorneys who work there, the local courts and how filing works there, local road or workplace hazards, and local process details such as police report requests. If you cannot write those for a city, merge it into a regional page and 301 the thin URL. Practices like [car accident lawyer marketing](/car-accident-lawyer-marketing) are where large city sets are most common, and where pruning usually helps most."},{"type":"heading","level":3,"text":"Syndicated tort content and co-counsel sites"},{"type":"paragraph","text":"Mass tort content often appears in three places: the firm's site, a co-counsel firm's site and a lead-gen microsite. Google clusters near-identical pages and picks one as canonical, the one it judges [\"objectively the most complete and useful for search users\"](https://developers.google.com/search/docs/crawling-indexing/canonicalization). A cross-domain canonical tag is \"a hint, not a rule,\" so the partner's copy may still be chosen; the report then shows \"Duplicate, Google chose different canonical than user.\""},{"type":"paragraph","text":"You have two clean options: write a genuinely different version for each site, or accept that the partner's copy may be the one indexed. Google says \"Some duplicate content on a site is normal and it's not a violation of Google's spam policies.\" The line is crossed only when copying is done at scale to manipulate rankings, which the scaled content policy covers."},{"type":"heading","level":3,"text":"Spanish-language pages and hreflang"},{"type":"paragraph","text":"Spanish pages are not duplicates of English ones. Google treats language versions as duplicates [\"only if the primary content is in the same language\"](https://developers.google.com/search/docs/crawling-indexing/canonicalization), so each Spanish page can and should be indexed in its own right. Link the pairs with hreflang, which [Google accepts in HTML, HTTP headers or a sitemap](https://developers.google.com/search/docs/specialty/international/localized-versions)."},{"type":"paragraph","text":"Codes are an ISO 639-1 language plus an optional region, so a US firm would use `en-US` and `es-US`, plus `x-default` for everyone else. In HTML that is `<link rel=\"alternate\" hreflang=\"es-US\" href=\"https://example.com/es/accidentes/\">` on both pages, each listing itself and the other. Two rules to keep: point each page's canonical at a URL in the same language, and remember that \"Google doesn't use hreflang or the HTML lang attribute to detect the language of a page.\" The visible text has to be Spanish."},{"type":"heading","level":3,"text":"Settlement PDFs and case-result documents"},{"type":"paragraph","text":"Google can [index PDFs](https://developers.google.com/search/docs/crawling-indexing/indexable-file-types) along with Word and other document formats, and Googlebot fetches up to [the first 64MB of a PDF](https://developers.google.com/search/docs/crawling-indexing/googlebot). To keep PDFs out, use the `X-Robots-Tag: noindex` HTTP header, which Google says works for non-HTML files such as PDFs. Google's own Apache example wraps `Header set X-Robots-Tag \"noindex, nofollow\"` in a `<Files ~ \"\\.pdf$\">` block; the NGINX version uses `add_header X-Robots-Tag \"noindex, nofollow\";` inside a PDF location block."},{"type":"paragraph","text":"My preference is to index an HTML summary page that links to the PDF, since HTML pages carry navigation, titles and context. State bar rules on publishing case results vary by state; check yours, and treat this as indexing advice, not legal advice."},{"type":"heading","level":3,"text":"Rebrands, mergers and domain moves"},{"type":"paragraph","text":"A merger or rebrand that changes the domain needs Search Console's [Change of Address tool](https://support.google.com/webmasters/answer/9370220?hl=en), which is for moves between domains or subdomains only. Do not use it for http to https or www to non-www changes. You \"must be an owner of both the old and new properties,\" the old homepage must 301 to the new one first, and Google says to \"Maintain the redirects for at least 180 days--longer if you still see any traffic to them from Google Search.\""},{"type":"paragraph","text":"Keep redirect chains short: Google says [ideally no more than 3 hops and fewer than 5](https://developers.google.com/search/docs/crawling-indexing/site-move-with-url-changes), and that for medium-sized sites \"it can take a few weeks or more\" for new URLs to replace old ones. Gary Illyes reportedly showed site moves taking 1–3 months typically and 6 months to a year or more at the slowest, [according to Search Engine Roundtable's recap](https://www.seroundtable.com/google-crawling-indexing-serving-data-42225.html); treat those as indicative, next to Google's documented range."},{"type":"heading","level":3,"text":"Retiring expired campaign pages"},{"type":"paragraph","text":"When litigation settles or closes, the campaign page needs one of three endings. If a live hub covers the same subject, 301 the page there; Google treats 301 as [\"a strong signal\"](https://developers.google.com/search/docs/crawling-indexing/http-network-errors) that the target should be processed. If nothing replaces it, return 404 or 410; Google removes already-indexed URLs that return 4xx. If people still search the tort, for example to check settlement status, I would keep the page and update it with the current status."},{"type":"paragraph","text":"Avoid redirecting every retired page to the homepage. A redirect to an unrelated page can be treated like a soft 404, since a 2xx status [\"doesn't guarantee indexing\"](https://developers.google.com/search/docs/crawling-indexing/http-network-errors) when the content does not match what was asked for."},{"type":"heading","level":3,"text":"Google Business Profile is a separate system"},{"type":"paragraph","text":"Map pack and local listings are managed in Google Business Profile, not in Search Console. A firm's website pages are crawled and indexed through Google Search, while its Business Profile is a separate listing with its own management tools. Fixing a Page indexing status does nothing for a missing office listing, and editing the Business Profile does nothing for an unindexed practice-area page. Check both when a local office \"disappears,\" and keep the website URL on the profile pointed at a live, indexable office page."},{"type":"heading","level":2,"text":"Indexing for AI Overviews, Bing and AI answers"},{"type":"paragraph","text":"Google's AI Overviews and AI Mode can only link to pages that are indexed and eligible for a snippet, and engines outside Google build their answers on their own indexes, so indexing is now the entry ticket to AI search as well. The work in this guide is the same work that earns an AI citation."},{"type":"heading","level":3,"text":"AI Overviews and AI Mode need an indexed page"},{"type":"paragraph","text":"Google's rule is short: \"To be eligible to be shown as a supporting link in AI Overviews or AI Mode, a page must be indexed and eligible to be shown in Google Search with a snippet,\" and [\"There are no additional technical requirements\"](https://developers.google.com/search/docs/appearance/ai-features), with no special schema needed. Both features may use a \"query fan-out\" technique, running related searches to gather sources. Google's [AI optimization guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) defines AEO and GEO and adds that you do not need llms.txt or \"AI text files,\" since \"Google Search itself doesn't use them,\" and that \"There's no requirement to break your content into tiny pieces.\""},{"type":"paragraph","text":"Clicks from these features count in the Performance report under the \"Web\" search type, and Google added [Search Generative AI performance reports](https://developers.google.com/search/blog/2026/06/gen-ai-performance-reports) on June 3, 2026, rolled out worldwide by August 31, 2026. Semrush's indexing guide makes the same point from the vendor side: pages not in Google's index \"won't appear in organic search results, AI Overviews, AI Mode, or Gemini.\" The trade-off to know is that `nosnippet` and a low `max-snippet` limit what Search can show, including in AI features."},{"type":"heading","level":3,"text":"The Search generative AI control and Google-Extended"},{"type":"paragraph","text":"Search Console now has a dedicated switch at **Settings > Search generative AI**. [\"As of August 31, 2026, we've rolled out this control to all websites worldwide,\"](https://support.google.com/webmasters/answer/16908024?hl=en) and it covers AI Overviews, AI Mode and generative AI features in Discover. The default is \"Include my site's links and content in Search generative AI features.\" If you exclude your site, \"You won't receive any traffic or impressions from these features.\" The control \"isn't used as a ranking or inclusion signal affecting other parts of Search,\" and changes take effect \"within 1-2 days,\" with some content taking longer."},{"type":"paragraph","text":"Google-Extended is a different thing. It is a robots.txt token for Gemini training and grounding, and Google says it [\"does not impact a site's inclusion in Google Search nor is it used as a ranking signal\"](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers). Blocking it does not remove you from AI Overviews. My recommendation for plaintiff firms: leave the Search generative AI control at its default. Opting out removes the site from AI Overviews and AI Mode entirely, with no ranking benefit elsewhere in Search."},{"type":"heading","level":3,"text":"Bing, Yahoo and DuckDuckGo"},{"type":"paragraph","text":"Outside Google, the Bing index also feeds other search engines. Yahoo says [\"Yahoo Search results come from the Yahoo web crawler (Slurp) and Bing's web crawler\"](https://help.yahoo.com/kb/SLN2213.html) and that some organic listings are \"powered by Bing.\" DuckDuckGo says its traditional links and images are ones [\"we largely source from Bing\"](https://duckduckgo.com/duckduckgo-help-pages/results/sources), alongside its own DuckDuckBot."},{"type":"paragraph","text":"The practical steps: verify the site in Bing Webmaster Tools and submit the same sitemap there, and turn on IndexNow, which Bing describes as [\"a free, open-source protocol\"](https://www.bing.com/indexnow) with built-in support in many CMSs and CDNs. Google does not take part in IndexNow, so it complements Search Console rather than replacing it. Check Bing Webmaster Tools itself for current submission limits."},{"type":"heading","level":3,"text":"AI crawlers outside Google"},{"type":"paragraph","text":"AI assistants run several crawlers each, and blocking the wrong one costs visibility without protecting anything. The table lists the main tokens as each operator describes them."},{"type":"table","header":["Bot","Operator","Purpose","Blocking effect"],"rows":[["OAI-SearchBot","OpenAI","ChatGPT search results","Not shown in ChatGPT search answers"],["GPTBot","OpenAI","Possible model training","Training only; search unaffected"],["PerplexityBot","Perplexity","Perplexity search results","Not surfaced in Perplexity search"],["Perplexity-User","Perplexity","User-requested fetches","Generally ignores robots.txt"],["Claude-SearchBot","Anthropic","Search result quality","Less visible in Claude search"],["Claude-User","Anthropic","User-requested fetches","Claude cannot fetch for users"],["ClaudeBot","Anthropic","Possible model training","Training only"],["Google-Extended","Google","Gemini training and grounding","No effect on Google Search"]]},{"type":"paragraph","text":"Sources: [OpenAI's bot page](https://platform.openai.com/docs/bots) (sites that opt out of OAI-SearchBot \"can still appear as navigational links,\" and robots.txt changes take \"~24 hours\"), [Perplexity's bot guide](https://docs.perplexity.ai/guides/bots) and [Anthropic's crawler article](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler), which notes that ClaudeBot is the training crawler and that blocking by IP address may not work reliably. A firm that wants AI visibility but no training use can allow the search bots and disallow GPTBot, ClaudeBot and Google-Extended. In robots.txt that is three short groups:"},{"type":"list","ordered":true,"items":["`User-agent: GPTBot` followed by `Disallow: /`","`User-agent: ClaudeBot` followed by `Disallow: /`","`User-agent: Google-Extended` followed by `Disallow: /`"]},{"type":"paragraph","text":"Leave OAI-SearchBot, PerplexityBot, Claude-SearchBot and Googlebot unblocked. Before blocking training bots, decide as a firm whether it matters: the pages are public marketing content, and some firms prefer to stay open. Whatever you choose, test the file in Search Console's robots.txt report so a typo does not block Googlebot as well."},{"type":"paragraph","text":"Our [AI search optimization](/services/ai-search-optimization) work starts from that crawl-access check, and [answer engine optimization for personal injury lawyers](/blogs/aeo-for-personal-injury-lawyers) covers the content side. Every engine above needs the same indexed, eligible page, and the last part of this guide turns that into a monthly monitoring routine."},{"type":"heading","level":2,"text":"Indexing tools and a monthly monitoring routine"},{"type":"paragraph","text":"Google Search Console and Bing Webmaster Tools are free and cover most indexing work; a desktop crawler and your server logs fill the gaps, and a short monthly check catches problems before they cost traffic."},{"type":"heading","level":3,"text":"Search Console and Bing Webmaster Tools"},{"type":"paragraph","text":"Search Console is the only tool that shows Google's own view of your URLs, so it is the starting point for every indexing question. Google calls it [\"a free toolset\"](https://www.google.com/search/howsearchworks/how-search-works/organizing-information/), and it has carried that name since Google decided in 2015 to [rebrand Google Webmaster Tools as Google Search Console](https://developers.google.com/search/blog/2015/05/announcing-google-search-console-new). Bing Webmaster Tools plays the same role for Bing, which also feeds part of Yahoo's and DuckDuckGo's results, as the previous section explained."},{"type":"paragraph","text":"Third-party crawlers simulate a search engine on your own site. They cannot tell you what Google indexed, but they find the causes (stray `noindex` tags, broken canonicals, orphan pages) at scale. The table lists the tools named in this guide. For third-party tools, check current pricing and plan limits with the vendor, as of October 2026; prices were not researched for this page."},{"type":"table","header":["Tool","Type","What it adds","Notes"],"rows":[["URL Inspection","Search Console","Google's verdict for one URL, live test, Request indexing","Daily request limit, number unpublished"],["Page indexing","Search Console","Indexed and not-indexed counts with reasons","Lists up to 1,000 example URLs per issue"],["Sitemaps","Search Console","Submitted sitemaps and their read status","Shows up to 1,000 submitted sitemaps"],["Removals","Search Console","Temporary block of a URL or prefix","Lasts about six months"],["Crawl stats","Search Console","Requests, responses, host status","Root-level properties only"],["robots.txt report","Search Console","Which robots.txt files Google fetched and errors","Domain-level properties only"],["Links","Search Console","Top linked pages, internal and external","Tables limited to 1,000 rows"],["Search generative AI control","Search Console","Include or exclude the site from AI features","Under Settings, site-wide"],["Bing Webmaster Tools","Engine console","Bing's view of your site, IndexNow support","Free; verify features in the tool"],["Screaming Frog SEO Spider","Desktop crawler","Crawls your site like a bot","Check pricing with the vendor"],["Sitebulb","Desktop or cloud crawler","Crawl audits with visual reports","Check pricing with the vendor"],["Ahrefs Webmaster Tools","Cloud audit","Site audit for verified sites","Check plan limits with the vendor"],["Semrush Site Audit","Cloud audit","Scheduled technical audits","Check plan limits with the vendor"],["Server log files","Raw data","Every request Googlebot actually made","Ask your host or CDN for access"]]},{"type":"paragraph","text":"Sources for the Google rows: [URL Inspection](https://support.google.com/webmasters/answer/9012289?hl=en), [data limits](https://support.google.com/webmasters/answer/96568?hl=en), [Sitemaps](https://support.google.com/webmasters/answer/7451001?hl=en), [Removals](https://support.google.com/webmasters/answer/9689846?hl=en), [Crawl stats](https://support.google.com/webmasters/answer/9679690?hl=en), [robots.txt report](https://support.google.com/webmasters/answer/6062598?hl=en), [Links](https://support.google.com/webmasters/answer/9049606?hl=en) and the [Search generative AI control](https://support.google.com/webmasters/answer/16908024?hl=en)."},{"type":"paragraph","text":"If you check hundreds of URLs, the [URL Inspection API](https://developers.google.com/webmaster-tools/limits) allows 2,000 queries per day and 600 per minute for each site. That is a separate quota from the manual Request indexing button, and the API only inspects; it cannot request indexing."},{"type":"heading","level":3,"text":"Crawlers and log files"},{"type":"paragraph","text":"A crawler shows what your site tells search engines, and your logs show what Googlebot actually did; together they explain most indexing gaps that Search Console only reports. Run a crawl from your homepage and compare its list of indexable URLs with your sitemap. Any URL in the sitemap that the crawl never reached is an orphan page: nothing links to it, so Google has to rely on the sitemap alone."},{"type":"paragraph","text":"A crawl also lists every page carrying `noindex` in the meta tag or the `X-Robots-Tag` header, every canonical that points somewhere else, and every redirect chain."},{"type":"paragraph","text":"Server logs answer a different question: did Googlebot request the page at all, and what status code did it get? Filter the logs by user agent, then confirm the hits are real. Anyone can fake the Googlebot name, so Google says to [verify Googlebot](https://developers.google.com/search/docs/crawling-indexing/verifying-googlebot) with a reverse DNS lookup to `googlebot.com`, `google.com` or `googleusercontent.com`, followed by a forward lookup that returns the original IP, or by matching the IP against Google's published IP-range files. A page with zero verified Googlebot requests has a discovery problem, not a quality problem."},{"type":"heading","level":3,"text":"The monthly indexing routine"},{"type":"paragraph","text":"A monthly check of eight items catches most indexing problems while they are still small. My recommendation is to put it on the calendar for the same day each month and record the numbers, because trends tell you more than any single snapshot. Search Console data [should be available in 2-3 days](https://support.google.com/webmasters/answer/96568?hl=en), so the last few days of any report are always incomplete."},{"type":"table","header":["Check","Where","Healthy","Act when"],"rows":[["Indexed page trend","Page indexing","Stable or rising with new content","Indexed count drops without a planned change"],["New not-indexed reasons","Page indexing","No new reason types","A reason appears or jumps in count"],["Submitted vs indexed per sitemap","Sitemaps and Page indexing","Most money pages indexed","A key sitemap lags far behind"],["Host status","Settings > Crawl stats","No host problems","robots.txt, DNS or server issues flagged"],["robots.txt fetch","robots.txt report","Fetched, no errors","Errors, warnings or an unexpected file"],["Manual actions and Security issues","Security and Manual Actions","No issues detected","Any issue listed"],["Spot check five money pages","URL Inspection","URL is on Google, right canonical","Not on Google, or Google chose another canonical"],["Crawl errors in Bing","Bing Webmaster Tools","No new errors","New errors on important URLs"]]},{"type":"paragraph","text":"Search Console sends email for some events, including progress notices after you use **Validate fix**, which [can take several days or even longer](https://support.google.com/webmasters/answer/7440203?hl=en). Make sure the person who owns the routine is a verified user on the property, so those emails reach someone who acts on them."},{"type":"paragraph","text":"For the spot check, choose the five pages that bring the most leads: usually the homepage, your top practice-area pages and any active tort or campaign page. Look at the verdict, the last crawl date and the Google-selected canonical every time."},{"type":"paragraph","text":"If something changed, run the six-gate triage from earlier in this guide on one affected URL before touching the whole site."},{"type":"heading","level":3,"text":"Launch checklist for a new page or site"},{"type":"paragraph","text":"Run this checklist before and right after any new page goes live; most failures it catches take minutes to fix before launch and weeks to recover from after. Each item maps to a section earlier in this guide."},{"type":"list","ordered":false,"items":["Confirm the URL returns a 200 status code, not a redirect or soft 404.","Set a self-referencing `rel=\"canonical\"` on the final URL.","Check there is no `noindex` in the robots meta tag or the `X-Robots-Tag` header.","Confirm robots.txt does not block the URL or its CSS and JavaScript.","Add the URL to the right sitemap with an accurate `<lastmod>` date.","Link to it from its hub page and from at least one frequently crawled page.","Verify the mobile HTML contains the same main content as desktop.","Keep the HTML under 2MB, since Googlebot stops fetching at that point.","Make sure your firewall or bot protection does not challenge verified Googlebot.","Uncheck the WordPress search engine visibility box on a new WordPress site.","Add reciprocal hreflang pairs if you publish a Spanish-language version.","Noindex paid landing-page variants while leaving AdsBot allowed.","Request indexing for the URL in URL Inspection after it passes the live test.","Confirm the verdict reads \"URL is on Google\" once Google processes it.","Check that the Google-selected canonical matches the URL you declared.","Add the page to your monthly spot-check list if it drives leads."]},{"type":"paragraph","text":"The 2MB item comes from Google's [Googlebot documentation](https://developers.google.com/search/docs/crawling-indexing/googlebot), which says Googlebot crawls the first 2MB of a supported file and the first 64MB of a PDF."},{"type":"paragraph","text":"If an outside firm builds or runs your pages, make this checklist part of the handoff. When you [vet a mass tort marketing agency](/how-to-choose-a-mass-tort-marketing-agency), ask to see its Page indexing reporting and how it handles new launches, and add it to the [questions to ask an agency before you hire it](/blogs/questions-to-ask-mass-tort-marketing-agency)."},{"type":"heading","level":2,"text":"Google indexing myths and outdated advice"},{"type":"paragraph","text":"Much indexing advice online describes tools Google has retired or rules it never followed; check any tip against Google's current documentation and its \"Last updated\" date. Most of Google's developer pages show that stamp at the bottom, and Google's [Search Central updates log](https://developers.google.com/search/updates) lists documentation changes by date."},{"type":"heading","level":3,"text":"Retired tools and renamed reports"},{"type":"paragraph","text":"Several tools that older guides still recommend no longer exist, and a few reports have new names; this dated list shows what changed and where to look instead."},{"type":"list","ordered":false,"items":["**2015:** Google announced it would [rebrand Google Webmaster Tools as Google Search Console](https://developers.google.com/search/blog/2015/05/announcing-google-search-console-new) after \"nearly ten years\" under the old name.","**2018:** the new Search Console [introduced the Index Coverage report](https://developers.google.com/search/blog/2018/01/introducing-new-search-console) with issue tracking and a validate-fix flow. The current help article is the [Page indexing report](https://support.google.com/webmasters/answer/7440203?hl=en). Google has not published a date for the rename, and some of its own pages still say \"Coverage\".","**September 1, 2019:** Google [stopped supporting `noindex` in robots.txt](https://developers.google.com/search/blog/2019/07/a-note-on-unsupported-rules-in-robotstxt), along with other unpublished rules.","**March 1, 2020:** `nofollow` became [a hint for crawling and indexing](https://developers.google.com/search/blog/2019/09/evolving-nofollow-new-ways-to-identify), not a directive.","**2022:** Google [deprecated the URL Parameters tool](https://developers.google.com/search/blog/2022/03/url-parameters-tool-deprecated) in Search Console, announced on March 28.","**2023:** Google [deprecated the sitemaps ping endpoint](https://developers.google.com/search/blog/2023/06/sitemaps-lastmod-ping), announced June 26; pings now return a 404 error.","**January 8, 2024:** the [crawl rate limiter tool was deprecated](https://developers.google.com/search/blog/2023/11/sc-crawl-limiter-byebye).","**Current:** Google's [Reports at a glance](https://support.google.com/webmasters/answer/9133276?hl=en) page lists the robots.txt report and no robots.txt tester, even in its legacy section.","**2026:** Googlebot for Search now [fetches the first 2MB of a URL](https://developers.google.com/search/blog/2026/03/crawler-blog-post) (64MB for PDFs). The [15MB figure from 2022](https://developers.google.com/search/blog/2022/06/googlebot-15mb) is now the default only for other Google crawlers.","**March 31, 2026:** Google [moved its crawler IP-range files](https://developers.google.com/search/blog/2026/03/crawler-ip-ranges) to `developers.google.com/crawling/ipranges/`, with redirects from the old location promised \"within 6 months\"."]},{"type":"heading","level":3,"text":"Myths that cost law firms indexed pages"},{"type":"paragraph","text":"Each myth below has a documented answer from Google, and acting on the myth usually leaves pages out of the index or keeps unwanted pages in it. The table pairs the claim with what Google's documentation says."},{"type":"table","header":["Myth","What Google says","Source"],"rows":[["Submit a sitemap with \"Request indexing\" in Sitemaps","The Sitemaps button is Submit; Request indexing is in URL Inspection","[Sitemaps report](https://support.google.com/webmasters/answer/7451001?hl=en)"],["URL Inspection removes pages","Removal is noindex, 404 or 410, a password, or the temporary Removals tool","[Removals](https://support.google.com/webmasters/answer/9689846?hl=en)"],["IndexNow alerts Google","Google is not on IndexNow's participant list","[indexnow.org list](https://www.indexnow.org/searchengines.json)"],["`<priority>` and `<changefreq>` guide Google","Google ignores both values","[Build a sitemap](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap)"],["robots.txt Disallow keeps a page out","Blocked URLs can still be indexed without a description","[robots.txt intro](https://developers.google.com/search/docs/crawling-indexing/robots/intro)"],["Any page can use the Indexing API","Only JobPosting or BroadcastEvent-in-VideoObject pages","[Indexing API](https://developers.google.com/search/apis/indexing-api/v3/quickstart)"],["Google indexes every page it knows","Indexing isn't guaranteed","[How Search works](https://developers.google.com/search/docs/fundamentals/how-search-works)"],["Paying speeds up indexing","Google never accepts payment to crawl more often","[Organizing information](https://www.google.com/search/howsearchworks/how-search-works/organizing-information/)"],["Crawl budget limits small sites","The guide targets very large or fast-changing sites","[Crawl budget](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget)"],["Duplicate content brings a penalty","Some duplication is normal, not a spam violation","[Canonicalization](https://developers.google.com/search/docs/crawling-indexing/canonicalization)"],["Removals is permanent","It lasts about six months","[Removals](https://support.google.com/webmasters/answer/9689846?hl=en)"],["Google-Extended controls AI Overviews","It doesn't affect Search inclusion","[Common crawlers](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers)"],["You need llms.txt for Google AI features","Google Search doesn't use such files","[AI optimization guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide)"]]},{"type":"paragraph","text":"Two of these cost law firms the most in practice. The Disallow myth leads teams to block a page in robots.txt to hide it, which also hides the `noindex` tag Google needed to see, so the URL can stay indexed with no description. The crawl budget myth sends small firms hunting for a server problem when Google's [crawl budget guide](https://developers.google.com/search/docs/crawling-indexing/large-site-managing-crawl-budget) is aimed at sites with a million or more pages that change weekly, or 10,000 or more that change daily. On a site with a few hundred pages, quality, duplication and discovery are the usual suspects."},{"type":"paragraph","text":"The AI myths are newer. Google's [AI optimization guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) says you don't need new machine-readable files, AI text files or special markup for Google's generative AI search, and that there is no requirement to break content into tiny pieces."},{"type":"paragraph","text":"These myths persist for simple reasons. Old posts get re-dated without being rewritten, tool vendors describe features in their own terms, and forum answers get copied long after the rule changed. The fix is a habit: before acting on a tip, open the Google page it relies on and read the \"Last updated\" stamp. If the tip names a tool or report you can't find in Search Console, it is probably out of date."},{"type":"paragraph","text":"Indexing is one technical layer of a larger plan. For how indexing fits with intake, paid media and tort selection, see our guide to [mass tort marketing strategies](/mass-tort-marketing-guide)."},{"type":"heading","level":2,"text":"Sources and methodology"},{"type":"paragraph","text":"This guide was researched in October 2026 from Google's own documentation, Search Console help pages, the sitemaps.org protocol, RFC 9309, indexnow.org and the bot pages of OpenAI, Perplexity and Anthropic; every fact links to its source."},{"type":"paragraph","text":"**How it was built.** Primary sources were read on 2026-10-22, and Google's \"Last updated\" stamps were noted where shown. Examples: the [recrawl guide](https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl) was last updated 2025-12-10, the [robots.txt specification](https://developers.google.com/search/docs/crawling-indexing/robots/robots_txt) 2026-08-31, [Control what you share](https://developers.google.com/search/docs/crawling-indexing/control-what-you-share) 2026-10-21 and the [Indexing API quickstart](https://developers.google.com/search/apis/indexing-api/v3/quickstart) 2026-10-16."},{"type":"paragraph","text":"**Secondary sources.** The October 2026 timing figures attributed to Google's Gary Illyes come from attendee and press recaps, mainly [Search Engine Roundtable](https://www.seroundtable.com/google-crawling-indexing-serving-data-42225.html) and [Search Engine Journal](https://www.searchenginejournal.com/google-crawling-indexing-recovery-timing-ranges/591829/). Google has not published the slides, so they are labeled as reported figures throughout and always sit next to Google's documented ranges."},{"type":"paragraph","text":"**First-party data.** When we checked on October 22, 2026, Search Console's URL Inspection API reported all 8 pages we had published and submitted on September 15 as \"Submitted and indexed\". That is current status, not the date each page was first indexed. An internal note also records that 12 Request indexing submissions on one property in one day raised no quota error. That is one observation, not a published limit; Google does not publish the number."},{"type":"paragraph","text":"**Competitors reviewed.** We read rankings.io's [\"How to Get Google to Index Your Site\"](https://rankings.io/blog/google-index/) (visible date May 22, 2025; schema modified 2026-01-30), [Ahrefs' indexing guide](https://ahrefs.com/blog/google-index/) (updated October 31, 2025) and Semrush's indexing guide (dated June 2, 2026). Where their guidance differs from Google's documentation, we followed Google. For readers comparing providers, there is also a page on [how a mass tort funnel compares with a general law firm SEO agency](/rankings-io-alternative)."},{"type":"paragraph","text":"**What changed in Google's documentation in 2026**"},{"type":"list","ordered":false,"items":["2026-02-03: the [Googlebot page](https://developers.google.com/search/docs/crawling-indexing/googlebot) documents the 2MB fetch limit for Search.","2026-03-31: [crawler IP ranges moved](https://developers.google.com/search/blog/2026/03/crawler-ip-ranges) to a new location.","2026-05-15: the [AI optimization guide](https://developers.google.com/search/docs/fundamentals/ai-optimization-guide) was added.","2026-06-03: [generative AI performance reports](https://developers.google.com/search/blog/2026/06/gen-ai-performance-reports) were announced for Search Console.","2026-07-10: [duplicate-cluster guidance](https://developers.google.com/search/docs/crawling-indexing/canonicalization-troubleshooting) was added, including a hold of up to two weeks.","2026-08-31: the [Search generative AI control](https://support.google.com/webmasters/answer/16908024?hl=en) reached all websites worldwide.","2026-09-24: Search Console added a [multimodal search-type filter](https://developers.google.com/search/blog/2026/09/web-multimodal-in-sc).","2026-10-16: the [Indexing API quota page](https://developers.google.com/search/apis/indexing-api/v3/quota-pricing) was updated.","2026-10-21: [Control what you share](https://developers.google.com/search/docs/crawling-indexing/control-what-you-share) added the robots meta tag, `data-nosnippet`, X-Robots-Tag and the Search generative AI control."]},{"type":"paragraph","text":"Mentions of bar advertising rules in this guide are general information, not legal advice; rules vary by state, so check with your state bar. Google changes its documentation often, so check each linked page's \"Last updated\" date before acting."}],"faqs":[{"question":"What is the Google index?","answer":"The Google index is the database where Google stores the pages it has crawled and chosen to keep, so it can show them in search results. Google describes it as being [like the index in the back of a book](https://www.google.com/search/howsearchworks/how-search-works/organizing-information/), covering hundreds of billions of webpages and well over 100,000,000 gigabytes. Google's developer guide says plainly that [\"Indexing isn't guaranteed; not every page that Google processes will be indexed\"](https://developers.google.com/search/docs/fundamentals/how-search-works)."},{"question":"How do I get Google to index my website?","answer":"To get Google to index your website, verify it in Search Console, find or create an XML sitemap, submit it, and request indexing for your key pages. In the [Sitemaps report](https://support.google.com/webmasters/answer/7451001?hl=en), paste the sitemap URL into **Add a new sitemap** and click **Submit**. Then open [URL Inspection](https://support.google.com/webmasters/answer/9012289?hl=en) for your homepage and most important pages and click **Request indexing**. Google says indexing typically takes \"a day or so\" but can take up to a week or two."},{"question":"How do I index a web page on Google?","answer":"To index a single web page, inspect its URL in Search Console and click **Request indexing** after the live test passes. Google's [URL Inspection help](https://support.google.com/webmasters/answer/9012289?hl=en) says the page must pass \"a quick check\", and you cannot request indexing if the live test finds the page non-indexable. Before you request, make sure the page returns a 200 status, has no `noindex`, is not blocked in robots.txt and is linked from another page on your site."},{"question":"How can I force Google to index my site?","answer":"You can't force Google to index a site; you can only make pages easy to find and worth keeping, then ask. Google's [recrawl guide](https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl) says \"Requesting a crawl does not guarantee that inclusion in search results will happen instantly or even at all\", and that requesting the same URL several times \"won't get it crawled any faster.\" What helps is removing technical blocks, linking new pages from crawled pages, submitting an accurate sitemap and publishing content that adds something. Google says its systems \"prioritize the fast inclusion of high quality, useful content.\""},{"question":"Is Google indexing free?","answer":"Yes. Google indexing is free, and so is Search Console, which Google calls [\"a free toolset\"](https://www.google.com/search/howsearchworks/how-search-works/organizing-information/). Google says it \"never accepts payment to crawl a site more frequently.\" There is no paid tier that gets your pages indexed faster or more completely."},{"question":"Can I pay Google to rank my website higher?","answer":"No. You cannot pay Google to rank higher in organic results. Google's [developer guide](https://developers.google.com/search/docs/fundamentals/how-search-works) says it doesn't accept payment \"to crawl a site more frequently, or rank it higher\" and that \"ranking is done programmatically.\" Paid placement exists only as ads, which are labeled and run through a separate auction. Running ads does not change how your pages are crawled, indexed or ranked organically."},{"question":"How long does it take Google to index a new website?","answer":"Google says it [\"can take a week or so\"](https://support.google.com/webmasters/answer/7440203?hl=en) for Google to start crawling and indexing a new page or site, and that the best first step for a new site is to request indexing of the homepage. Its [recrawl guide](https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl) adds that crawling \"can take anywhere from a few days to a few weeks.\" These are ranges, not guarantees. The timeline section earlier in this guide sets Google's documented ranges side by side with the figures reported from Google's October 2026 talk."},{"question":"Why isn't Google indexing my site?","answer":"Google isn't indexing your site because one of six gates is failing: the URL isn't known, isn't allowed, can't be fetched or rendered, isn't indexable, has another URL chosen as canonical, or isn't judged valuable enough. The [Page indexing report](https://support.google.com/webmasters/answer/7440203?hl=en) names the reason for each URL, such as \"URL marked 'noindex'\", \"Server error (5xx)\" or \"Crawled - currently not indexed\". Test the gates in order with URL Inspection and fix the first failure."},{"question":"How do I check if my website is indexed on Google?","answer":"The most reliable check is URL Inspection in Search Console, which shows a verdict such as \"URL is on Google\" or \"URL is not on Google\" for any page on your verified site. For site-wide counts, use the Page indexing report. A `site:` search gives a quick rough look, but Google says an indexed URL [\"can show up in search results for site: queries … however it's not guaranteed\"](https://developers.google.com/search/docs/monitor-debug/search-operators/all-search-site), so a missing result there proves nothing. Note also that [URL Inspection](https://support.google.com/webmasters/answer/9012289?hl=en) says \"URL is on Google\" doesn't guarantee the page appears for any query."},{"question":"What does \"Crawled - currently not indexed\" mean?","answer":"\"Crawled - currently not indexed\" means Google fetched the page but chose not to add it to the index for now. Google's [Page indexing help](https://support.google.com/webmasters/answer/7440203?hl=en) says: \"It may or may not be indexed in the future; no need to resubmit this URL for crawling.\" Common causes are thin or near-duplicate pages, such as city pages that only swap a place name. Improve the content, strengthen internal links from important pages, and check that the page isn't a weaker copy of another URL on your site."},{"question":"What does \"Discovered - currently not indexed\" mean?","answer":"\"Discovered - currently not indexed\" means Google knows the URL exists but hasn't crawled it yet. Google's [Page indexing help](https://support.google.com/webmasters/answer/7440203?hl=en) says that \"Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl,\" which is why the last crawl date is empty. On a small site it often reflects slow server responses or weak signals that the page matters. Check Crawl stats and link the page from crawled pages."},{"question":"How many URLs can I submit for indexing per day?","answer":"Google doesn't publish the number. Its [URL Inspection help](https://support.google.com/webmasters/answer/9012289?hl=en) says only that \"There is a daily limit of inspection requests for each property\" and a separate daily limit on live inspections. The [URL Inspection API](https://developers.google.com/webmaster-tools/limits) has a different, published quota of 2,000 queries per day per site, but it only inspects URLs and can't request indexing. For many pages, Google recommends a sitemap with accurate `<lastmod>` dates instead."},{"question":"Does submitting a sitemap guarantee indexing?","answer":"No. Google's sitemap documentation says [\"Submitting a sitemap is merely a hint: it doesn't guarantee that Google will download the sitemap or use the sitemap for crawling URLs on the site.\"](https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap) A sitemap helps Google discover URLs and notice updates, especially when `<lastmod>` dates are accurate. Every listed page still has to pass the same checks as any other: it must be fetchable, indexable, canonical and judged worth keeping."},{"question":"Does Google support IndexNow?","answer":"No. Google is not part of IndexNow. The protocol's machine-readable list of participants, [searchengines.json](https://www.indexnow.org/searchengines.json), names Bing, Yandex, Seznam, Naver, Yep, the Internet Archive and Amazonbot, and Google is not among them. IndexNow is still worth enabling for Bing and the other engines, and many content systems and CDNs offer it built in. For Google, use sitemaps with accurate `<lastmod>` dates and the Request indexing button in URL Inspection for individual priority pages."},{"question":"Can a law firm use the Google Indexing API?","answer":"Not for ordinary law-firm pages. Google's [Indexing API documentation](https://developers.google.com/search/apis/indexing-api/v3/quickstart) says it \"can only be used to crawl pages with either JobPosting or BroadcastEvent embedded in a VideoObject.\" Practice-area pages, tort pages, blog posts and attorney bios don't qualify. Google adds that submissions \"undergo rigorous spam detection\" and that abuse, \"including the use of multiple accounts or other means to exceed usage quotas, may result in access being revoked.\" A firm's job postings page could qualify if it uses JobPosting markup; everything else should go through sitemaps and URL Inspection."},{"question":"Why is my page indexed but not ranking?","answer":"Being indexed only makes a page eligible to appear; it doesn't decide where it ranks. Google's [URL Inspection help](https://support.google.com/webmasters/answer/9012289?hl=en) says \"'URL is on Google' doesn't actually guarantee that your page will appear in Search results.\" Ranking depends on relevance to the query, content quality, links and how competing pages compare. Check which queries the page earns impressions for in the Performance report, compare it with the pages that rank, and improve depth, internal links and local relevance. For law firms, that work is the core of [personal injury SEO](/services/personal-injury-seo)."},{"question":"How do I remove a page from Google's index?","answer":"To remove a page permanently, add `noindex`, return a 404 or 410 status, or put it behind a password; for speed, also use the Removals tool. Google's [Removals help](https://support.google.com/webmasters/answer/9689846?hl=en) says the tool blocks a URL \"for about six months\" and is only \"the first step\" toward permanent removal. Google's [noindex documentation](https://developers.google.com/search/docs/crawling-indexing/block-indexing) says the page must not be blocked by robots.txt, or Google never sees the rule. Google also says URLs that return a [4xx status are removed from the index](https://developers.google.com/search/docs/crawling-indexing/http-network-errors) once recrawled."},{"question":"Does robots.txt stop Google from indexing a page?","answer":"No. robots.txt controls crawling, not indexing. Google says robots.txt [\"is not a mechanism for keeping a web page out of Google\"](https://developers.google.com/search/docs/crawling-indexing/robots/intro); a blocked URL can still appear in results without a description if other pages link to it. Search Console reports that case as [\"Indexed, though blocked by robots.txt\"](https://support.google.com/webmasters/answer/7440203?hl=en), and Google's advice is to \"remove the robots.txt block and use 'noindex'.\" Use robots.txt to manage crawling of sections you don't need crawled, and `noindex` for pages that must stay out of results."},{"question":"What is an index page on a website?","answer":"An index page is the default file a web server returns when someone requests a folder rather than a named file, usually `index.html`. In Apache HTTP Server, the [`DirectoryIndex` directive](https://httpd.apache.org/docs/2.4/mod/mod_dir.html) defaults to `index.html`, so a request for a folder URL returns that file if it exists. The homepage of most sites is an index page in this sense. This meaning is unrelated to the Google index, although both share the word. An index page still needs to be crawled and indexed like any other URL to appear in Google."},{"question":"What is an example of an index page?","answer":"A request for `http://example.com/docs/` is a standard example: Apache's [documentation](https://httpd.apache.org/docs/2.4/mod/mod_dir.html) says it returns `/docs/index.html` if that file exists. Your homepage is the most common index page; visiting the bare domain serves the index file of the site's root folder. For search, make sure the folder URL and the explicit file URL don't both get indexed as duplicates; pick one and canonicalize to it."},{"question":"What is an index website?","answer":"\"Index website\" usually means a website that search engines have indexed, so its pages can appear in results. People also use the phrase for a site's index page (its default homepage file) or for sites with \"index\" in their name, but the search sense is the common one. To index your website on Google, follow the four steps: verify in Search Console, prepare a sitemap, [submit it in the Sitemaps report](https://support.google.com/webmasters/answer/7451001?hl=en), and request indexing for key pages."},{"question":"Is Yandex like a Russian Google?","answer":"Yandex is a separate search engine with its own crawler and its own index; it does not use Google's index. One practical difference for site owners is that Yandex takes part in IndexNow, which Google does not. [indexnow.org](https://www.indexnow.org/) lists Yandex among the engines that support the protocol, alongside Microsoft Bing, Naver, Seznam.cz and Yep, and its [participant list](https://www.indexnow.org/searchengines.json) does not include Google."},{"question":"Do I need to submit my site to Bing and Yahoo?","answer":"Submit to Bing through Bing Webmaster Tools; there is no separate submission for Yahoo. Yahoo's help pages say its results come from [\"the Yahoo web crawler (Slurp) and Bing's web crawler\"](https://help.yahoo.com/kb/SLN2213.html) and send site owners to Bing's Webmaster Help Center. DuckDuckGo says it [largely sources](https://duckduckgo.com/duckduckgo-help-pages/results/sources) its traditional links from Bing. Bing describes IndexNow as [\"a free, open-source protocol\"](https://www.bing.com/indexnow) that many content systems and CDNs support built in, so enabling it is an easy way to notify Bing when pages change."},{"question":"Does blocking Google-Extended remove my site from AI Overviews?","answer":"No. Google says Google-Extended [\"does not impact a site's inclusion in Google Search\"](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers); it controls use of your content for training Gemini models and grounding in Gemini Apps and Vertex AI. To keep your site out of AI Overviews and AI Mode, use the [Search generative AI control](https://support.google.com/webmasters/answer/16908024?hl=en) under **Settings > Search generative AI** in Search Console. Google says excluding your site means \"You won't receive any traffic or impressions from these features\", and that the control doesn't affect AI training. To remove content from Search entirely, use `noindex`."},{"question":"Is the Coverage report gone from Search Console?","answer":"The report still exists under a new name: the Page indexing report. Google introduced it as the [\"Index Coverage report\"](https://developers.google.com/search/blog/2018/01/introducing-new-search-console) in 2018, and the current help article is titled [\"Page indexing report\"](https://support.google.com/webmasters/answer/7440203?hl=en). Google has not published a date for the rename, and some of its own pages still use the old label; the [Reports at a glance](https://support.google.com/webmasters/answer/9133276?hl=en) page still says you can reach URL Inspection from \"the Coverage report\". If a guide tells you to open Coverage, look for Page indexing in the Indexing section of the menu."}],"related":["link-not-working","law-firm-website-development","local-rank-tracker","best-law-firm-websites"]}