{Nadeem Haddadeen}

Arabic SEO Consultant, Python for SEO From Jordan

Crawling and indexing

XML sitemaps and sitemap indexes: what to include and how to check them

Distinguish a URL sitemap from a sitemap index, check the intended URLs and modification dates, and compare sitemap discovery with internal linking.

XML sitemaps and sitemap indexes: what to include and how to check them
Background photograph: Jakub Zerdzicki / Unsplash. Title design: Nadeem Haddadeen.

An XML sitemap gives search engines a list of URLs the website wants to surface. A sitemap index lists sitemap files. Opening the index and seeing several child sitemaps is normal; the page URLs are inside those children.

Use the sitemap as a reviewed publication output. A CMS generating it automatically can still include the wrong page types or omit an important section.

Recognise the two structures

A URL sitemap contains URL entries:

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/guides/shoe-sizing/</loc>
    <lastmod>2026-09-20</lastmod>
  </url>
</urlset>

A sitemap index uses sitemapindex with child sitemap locations. It is useful for separating larger collections or content types. Search Console can receive the index address, allowing the referenced files to be discovered through it.

Google’s sitemap implementation documentation describes supported formats and limits. For XML, each sitemap is limited to 50,000 URLs and 50 MB uncompressed; larger collections need multiple files.

Include the intended canonical pages

Check a sample from each sitemap. The listed pages should be the canonical URLs you want indexed, with working responses and appropriate indexing instructions. An old redirected path is usually less useful here than its current destination.

A sitemap does not force indexing. It also does not resolve a canonical contradiction in the page itself. If the file lists A while A declares B as canonical, investigate the intended version.

Keep modification dates meaningful

Use lastmod when you can provide a reliable date of a significant page update. Replacing every date with the current day on every sitemap request misrepresents what changed.

For a custom site, connect sitemap generation to the underlying publication data. For a CMS, inspect what the plugin or core feature produces. Frequent publishing may require more frequent refreshes, but a news-specific sitemap has separate requirements and should not be assumed necessary for every blog.

Submit and monitor the correct file

Submit the index or appropriate sitemap in Search Console’s Sitemaps area. Review fetch and processing results, then investigate the relevant child file when a problem is reported. A successful submission is not a confirmation that every contained page is indexed.

You can also reference a sitemap from robots.txt:

Sitemap: https://example.com/sitemap-index.xml

Use the real absolute address. A sitemap declaration is separate from Allow and Disallow rules.

Compare sitemap URLs with linked URLs

Import a sitemap URL set into List mode to check responses and metadata. Compare it with a normal linked crawl to identify pages known through the sitemap but absent from the discovered navigation.

For code that extracts WordPress sitemap URLs, use my existing Python sitemap extraction tutorial. The implementation is separate from deciding which URLs belong in the file.

My crawl and indexing analysis covers those comparisons. For large generated collections, programmatic SEO consulting connects sitemap rules with the underlying page and canonical policy.

Video lessons: SEO course session 6 · SEO course session 21. The videos are in Arabic.