Obtaining the complete list of pages from a website requires cross-referencing several data sources, as no isolated method covers all existing URLs. The XML sitemap, Search Console, crawling tools, and even server log files each provide a partial snapshot. Understanding the discrepancies between these sources allows for the selection of the right combination depending on the objective: SEO audit, migration, content inventory, or simple verification.
XML Sitemap, Search Console, and Crawl: What Each Source Actually Covers
The most common reflex is to open the sitemap.xml file of a domain. This file declares the URLs that the site owner wants to be crawled. It does not reflect the indexing status or the existence of orphan pages missing from the internal linking.
| Source | What It Lists | Main Limitation |
|---|---|---|
| XML Sitemap | URLs declared by the webmaster | Orphan pages and undeclared URLs missing |
| Google Search Console | URLs known to Google (crawled or indexed) | Export limited to 1,000 rows per report, data retained for 16 months |
| SEO Crawl (Screaming Frog, Sitebulb, etc.) | URLs accessible via internal linking | Pages protected by authentication or blocked by robots.txt are ignored |
| Server Logs | Any URL that has received at least one HTTP request | Requires technical access to the server, sometimes massive data volume |
The table highlights a often underestimated point: no single source guarantees a comprehensive inventory. A sitemap may omit URLs, the Search Console caps its exports, and a crawl only follows the links it discovers.
To illustrate this complementarity, one can consult the list of pages from Net Addict, which shows how a structured sitemap organizes URLs by categories and facilitates the reading of a site’s tree structure.
Discrepancies Between Processed Sitemap and Actually Indexed Pages
A common trap is to equate the processing of a sitemap by Google with the indexing of its content. A sitemap with a “successful” status only means that the file has been read, not that the URLs it contains have been crawled or added to the index.
Google then applies its own quality criteria and crawl budget to decide which URLs deserve a visit. Low-value pages, duplicate content, or unfavorable technical signals (high response times, noindex directives) can prevent indexing even if the URL is in the sitemap.
The lastmod Tag: A Freshness Signal Under Conditions
Google ignores the changefreq and priority tags in sitemaps. The lastmod tag is only taken into account if it reflects a substantial change in content. Artificial or cosmetic updates (changing a comma, updating a timestamp without altering the text) can diminish the trust placed in this signal.
A well-maintained sitemap remains a discovery tool, not a lever for forced indexing. To identify actually indexed URLs, the “Pages” report from the Search Console remains the reference, despite its export limit.
SEO Crawl vs. Google site: Operator
The search operator site:domaine.com in Google used to provide an estimate of the number of indexed pages. This method has lost reliability since Google no longer systematically displays the total number of results.
An SEO crawl tool scans the site by following internal and external links, then compiles the list of discovered URLs with their HTTP code, click depth, meta tags, and indexability status. The difference is structural:
- The site: operator queries Google’s index, which may contain deleted pages or exclude others that are perfectly accessible
- The SEO crawl scans the site as it exists at the moment, ignoring what Google knows or does not know
- Cross-referencing the two allows for the identification of indexed pages that have been removed from the site (404 errors in the index) and existing pages that are absent from the index
For a migration audit or content cleanup, the SEO crawl provides a more reliable foundation than the search operator.
IndexNow and Proactive Notification of Modified URLs
Bing offers a complementary approach through the IndexNow protocol. Rather than waiting for the crawler to pass, the webmaster directly notifies the added, modified, or deleted URLs. An IndexNow request can submit up to 10,000 URLs at once.
This protocol does not replace the sitemap, which remains the structural inventory of the site. It acts as a real-time update channel, particularly useful for sites with a high publication frequency or online stores whose product listings change regularly.
When to Combine Sitemap and IndexNow
The sitemap covers all existing URLs and serves as a reference for crawlers. IndexNow signals recent changes to expedite their consideration. The two complement each other without overlapping: the first offers the big picture, the second provides responsiveness.
Practical Method for Cross-Referencing Sources and Obtaining a Complete List
The most reliable approach is to merge data from several sources into a spreadsheet or audit tool, then identify the discrepancies.
- Export URLs from the XML sitemap (often accessible via domaine.com/sitemap.xml or domaine.com/sitemap_index.xml)
- Export known URLs from the Search Console (the “Pages” report, keeping in mind the 1,000-row export cap)
- Launch a complete crawl of the site with a dedicated tool to capture URLs accessible via internal linking
- Compare the three lists to isolate orphan URLs, pages not declared in the sitemap, and indexed URLs that are returning errors
URLs present in the crawl but absent from the sitemap are immediate candidates for addition. URLs present in the Search Console but returning a 404 error during the crawl signal a maintenance issue.
The complete inventory of a site’s pages remains an exercise in cross-referencing. The quality of the final list depends less on the chosen tool than on the rigor with which sources are cross-referenced, each compensating for the blind spots of the other.
