Build the sitemap search engines are waiting for

A sitemap is a small file with an outsized reputation. It does not make pages rank, and it will not rescue a page nobody wants to link to. What it does is tell a crawler that a page exists and roughly when it last changed, which matters most for exactly the pages that are hardest to find: the ones buried three clicks deep, the ones nothing links to yet, and the ones that changed after the last crawl.

The field people get wrong is lastmod, and it is worth getting right because it is the only field a crawler treats as evidence. If lastmod updates on every deploy, every URL claims to have changed every week, the field stops carrying information, and it gets ignored. Set it to the date the page's content genuinely changed, and leave it alone otherwise. A sitemap that tells the truth about a handful of changes is worth more than one that shouts about all of them.

Priority and change frequency deserve less reverence than they usually get. They are hints, they are widely ignored, and no arrangement of them will promote a page. They are worth setting honestly and then forgetting about, rather than tuned as though they were a lever.

The bigger question a sitemap raises is which pages should be in it at all. It should list the pages you want indexed, in their canonical form, and nothing else. Including a URL that redirects, or one that carries a canonical tag pointing somewhere else, sends a crawler two contradictory instructions about the same page. That is a surprisingly common own goal and it costs crawl budget on sites large enough to have one.

For a small or medium site, generating this by hand once and updating it when the shape of the site changes is entirely reasonable. For anything that publishes regularly, the sitemap should be generated by the site itself from whatever it already knows about its own pages, because a hand maintained list will drift from reality within a month or two and nobody will notice.

Everything happens in this page. There is no crawl, which means it will not discover pages for you, but it also means your URL structure is not being handed to a third party before you have launched it.