Why Your Sitemap Lists URLs Your Own Pages Disown

Q: Why does my sitemap list URLs with a trailing slash when my canonical tags have none?

`@astrojs/sitemap` does not read your pages. It has no idea what your canonical tag says. It takes the list of routes your build generated, renders each one as a URL in whatever form it prefers, and writes that out.

Q: How do I strip the trailing slash from an Astro sitemap?

`serialize` runs per entry, so the file comes out of the build already consistent and there is no post-build rewrite step to forget.

A sitemap <loc> and a <link rel="canonical"> are both making exactly the same claim: this is the address of this page. They are written by different code, at different moments, from different inputs, and nothing in a normal build checks that they agree. When they disagree you have not published a preference. You have published an argument, and you have asked a crawler to settle it.

This site published that argument for months. Every page’s canonical tag was written without a trailing slash. The sitemap listed every URL with one. Both forms served a 200 with identical bytes and no redirect in either direction, so nothing on the server was breaking the tie. The file whose entire job is nominating URLs for indexing was nominating 22 URLs that each loaded a page saying “not me, the other one”.

What follows is the three distinct ways a static build produces that disagreement, the config that closes each one, and a script you can paste into your own project to find out which of the three you have. The three are separable: fixing the first does nothing for the second, and the third is invisible to both.

Two files, one page, two different addresses

Start with why this is worth an afternoon rather than a shrug.

If /Blog/post and /Blog/post/ both return 200 with the same bytes, they are two URLs for one document. Every crawler has to decide which one to keep, and the signals it uses are the canonical tag, the sitemap, the internal links, and whatever external links exist. Get those pointing at one address and the decision is trivial. Get them pointing at two and the crawler resolves it on its own terms — which may mean picking the form nothing links to, or splitting whatever link value the page has earned across two entries, or simply taking longer to make up its mind.

The stakes are concrete here. One URL on this domain, /Blog/youtube-shorts-automation-pi, carries 24 of the roughly 30 organic referrals this site has ever received. It is the only page that has ever demonstrably worked. Publishing a sitemap that nominates an address that page’s own <head> rejects is not a tidy-up item.

The fix is not “pick the pretty form”. The fix is: pick the form that is already indexed and already linked, and make every other signal agree with it. Here that was the slash-free form, because that is what the canonicals and every internal <a href> already used, so the sitemap was the one file that had to move.

The sitemap nominates the form it generates

Here is the part that surprises people. @astrojs/sitemap does not read your pages. It has no idea what your canonical tag says. It takes the list of routes your build generated, renders each one as a URL in whatever form it prefers, and writes that out. It nominates the URL form it generates, not the one your pages declare.

Astro’s own trailingSlash setting does not save you either. This project runs:

// astro.config.mjs
export default defineConfig({
  site: 'https://dankdev.com',
  trailingSlash: 'ignore',
  build: { format: 'directory' },

trailingSlash: 'ignore' is a dev-server routing policy — it says both forms are acceptable requests. build: { format: 'directory' } writes /Blog/<slug>/index.html, which is what makes both forms serve. Neither setting is a statement about what belongs in the sitemap.

The integration does hand you every entry before it is written, though, and that is the right place to intervene:

sitemap({
  serialize(item) {
    item.url = item.url.replace(/\/$/, '');
    return item;
  },
})

Four lines. serialize runs per entry, so the file comes out of the build already consistent and there is no post-build rewrite step to forget. Note what it does not do: it does not touch the root URL, because https://example.com/ and https://example.com are the same URL and every parser normalises them to the former. Only sub-paths have two forms.

I measured the before-state rather than trusting the story. On 2026-09-14 I deleted that serialize hook, rebuilt, and ran the audit script below against the result: 36 of the 37 <loc> values failed the canonical-match check, and the same 36 carried a trailing slash. The one that passed was the homepage, for the reason just given. Restoring the hook took all 37 back to zero.

Whatever generator you use, the shape of the fix is the same: find the hook that runs per URL and normalise there, not in a sed pass over the built XML.

A page can disown itself and still be nominated

The second failure is the same defect one level up, and the first fix does nothing for it.

Two URLs on this site — /electronics and /electronics/hello-esp32 — declare a canonical pointing at another domain. The hardware section moved to a sibling site in September 2026, and the old addresses could not simply start returning 404: retiring a URL properly needs 301s, 301s are server config, and server config was not in reach. So both paths still build, still return 200, and both hand their canonical to the new home.

They were still in the sitemap. Of course they were. The canonical tag lives in the rendered HTML, and the sitemap integration never opens the rendered HTML. It sees a route, and a route is a nomination. The result was a file that nominated for indexing two URLs whose own pages disavowed them in favour of a different origin: the same contradiction as the trailing slash, with a whole other domain involved.

The integration’s filter hook decides membership:

filter: (page) => !FOREIGN_CANONICAL_PATHS.has(toSitePath(page)),

The load-bearing decision is not the filter. It is where that set comes from:

// astro.config.mjs
import { foreignCanonicalPaths, toSitePath } from './src/lib/foreign-canonical.mjs';

const FOREIGN_CANONICAL_PATHS = foreignCanonicalPaths();

foreignCanonicalPaths() derives the set from one exported map of path to off-origin target, and the stub pages that render those two URLs import their canonical from the same map. The config, the pages and the tests read one definition.

The alternative — writing ['/electronics', '/electronics/hello-esp32'] straight into the config — is a second catalogue. Second catalogues rot, and they rot silently: the day someone adds a third off-origin canonical to a page, the page is correct, the sitemap is wrong, and nothing anywhere goes red. A hand-list is a promise that every future editor will remember a file they have no reason to open. Derivation costs about thirty lines and removes the promise.

One caveat worth stating, because it burned this repo: the first version of that module derived the list by reading canonical: out of a content collection. When the collection was later deleted, the derivation returned an empty set and quietly re-admitted both URLs. Derivation is only safer than a list when the thing you derive from cannot vanish without anyone noticing. The module is now an explicit map with a predicate over it, and the test suite fails if either URL reappears in the sitemap.

A file that is not a route does not exist

The third failure has nothing to do with canonicals, and neither hook above can see it.

In July 2026 two finished posts on this site — einvoice-conformance-engine and zero-dependency-libraries-npm — were written directly into a deployed release directory. They rendered. They were reachable. They had a date, a title, working markup. They were never added to the content collection the build reads, so the build never produced a route for them, so the sitemap generator never heard of them and no crawler was ever told they existed. They earned nothing for nineteen days, and not because of their quality. They were pulled back into src/content/blog/ on 2026-07-30, in a commit whose one job was bringing the repository back in line with what the server was actually serving.

The mechanism is worth stating flatly because it generalises past this stack. A sitemap is built from routes, not from files on the server. Anything you put into the deployed directory by hand — a rescued page, a landing page copied in during a hurry, a legacy file the migration script skipped — is a real page to anyone who has its URL and a non-existent page to the sitemap generator. The two indexes drift apart, and nothing tells you, because nothing is comparing them.

That is exactly what the script below compares.

Audit your own build

One file, no dependencies, no network. It reads your local build output and nothing else. Save it as sitemap-audit.mjs in your project root, build, and run it.

// sitemap-audit.mjs — does your sitemap nominate the URLs your pages claim to be?
// Zero dependencies, no network: it reads your local dist/ and nothing else.
//   npm run build && node sitemap-audit.mjs [dist-dir]
// Exits 1 if it finds anything, so it can sit in CI next to your build.
import { readdirSync, readFileSync, existsSync } from 'node:fs';
import { join, relative, sep } from 'node:path';

const DIST = process.argv[2] ?? 'dist';
const sitemapFile = ['sitemap-0.xml', 'sitemap.xml'].map((f) => join(DIST, f)).find(existsSync);
if (!sitemapFile) {
  console.error(`sitemap-audit: no sitemap-0.xml or sitemap.xml under ${DIST}/ — build first`);
  process.exit(2);
}

/** Any URL or path, reduced to the comparable site path: leading slash, no trailing slash. */
const sitePath = (u) => {
  const p = String(u).replace(/^https?:\/\/[^/]+/, '').replace(/[?#].*$/, '').replace(/\/+$/, '');
  return p.startsWith('/') ? p : `/${p}`;
};

// <loc> values in document order. No XML parser: <loc> takes no attributes and sitemap
// generators do not wrap it in CDATA. If yours does, this is the line to fix.
const locs = [...readFileSync(sitemapFile, 'utf8').matchAll(/<loc>\s*([^<\s]+)\s*<\/loc>/g)].map((m) => m[1]);
if (!locs.length) {
  console.error(`sitemap-audit: ${sitemapFile} contains no <loc> at all`);
  process.exit(2);
}

// Your origin, read out of the sitemap itself rather than configured a second time here.
const ORIGIN = new URL(locs[0]).origin;

/** Every built page, as the dist-relative directory that serves it. */
function builtPages(dir, out = []) {
  for (const entry of readdirSync(dir, { withFileTypes: true })) {
    const full = join(dir, entry.name);
    if (entry.isDirectory()) builtPages(full, out);
    else if (entry.name === 'index.html') out.push(full);
  }
  return out;
}

const CANONICAL_TAG = /<link\b[^>]*\brel=["']?canonical["']?[^>]*>/i;
const HREF = /\bhref=["']([^"']*)["']/i;

/** site path -> the canonical href that page declares, '' when it declares none. */
const pages = new Map();
for (const file of builtPages(DIST)) {
  const tag = CANONICAL_TAG.exec(readFileSync(file, 'utf8'));
  const dir = relative(DIST, file).split(sep).slice(0, -1).join('/');
  pages.set(sitePath(`/${dir}`), tag ? HREF.exec(tag[0])?.[1] ?? '' : '');
}

const offOrigin = (href) => {
  if (!href) return false;
  try { return new URL(href, ORIGIN).origin !== ORIGIN; } catch { return false; }
};

const nominated = new Set(locs.map(sitePath));
const unnominated = [];
const excluded = [];
for (const [path, href] of pages) {
  if (nominated.has(path)) continue;
  // A page that hands its canonical to another origin is SUPPOSED to be missing here.
  (offOrigin(href) ? excluded : unnominated).push(offOrigin(href) ? `${path}  ->  ${href}` : path);
}

const mismatched = [];
const trailing = [];
const ghosts = [];
for (const loc of locs) {
  const { pathname } = new URL(loc);
  if (pathname !== '/' && pathname.endsWith('/')) trailing.push(loc);
  const path = sitePath(loc);
  if (!pages.has(path)) { ghosts.push(loc); continue; }
  const href = pages.get(path);
  const declared = href ? new URL(href, ORIGIN).href : '';
  if (declared !== new URL(loc).href) {
    mismatched.push(`${loc}  ->  page says ${declared || '(no rel=canonical)'}`);
  }
}

const section = (label, rows) => {
  console.log(`${label}: ${rows.length}`);
  for (const row of rows) console.log(`    ${row}`);
};

console.log(`sitemap-audit: ${sitemapFile} — ${locs.length} <loc>, ${pages.size} built pages, origin ${ORIGIN}`);
section('(a) built pages NOT nominated', unnominated);
section('(b) locs whose page declares a different canonical', mismatched);
section('(c) locs with a trailing slash', trailing);
section('(d) locs with no built page behind them', ghosts);
section('    deliberately excluded (page declares an off-origin canonical)', excluded);

const problems = unnominated.length + mismatched.length + trailing.length + ghosts.length;
console.log(`${problems} problem(s).`);
process.exit(problems ? 1 : 0);

Three design notes, since you will be adapting it.

It takes the origin from the sitemap’s own first <loc> instead of asking you to configure your domain a second time — one fewer thing to get out of sync with the thing being checked. It compares canonical against <loc> after resolving both through new URL, which normalises https://example.com and https://example.com/ to the same string; that is correct, and it is why the homepage never appears in class (b) on a slash mismatch. And it separates class (a) from the “deliberately excluded” line: a page missing from the sitemap because it declares an off-origin canonical is working as designed, and lumping it in with genuine omissions would train you to ignore the output.

What a clean run prints here today

Run against this site’s build, immediately after npm run build:

sitemap-audit: dist/sitemap-0.xml — 80 <loc>, 82 built pages, origin https://dankdev.com
(a) built pages NOT nominated: 0
(b) locs whose page declares a different canonical: 0
(c) locs with a trailing slash: 0
(d) locs with no built page behind them: 0
    deliberately excluded (page declares an off-origin canonical): 2
    /electronics/hello-esp32  ->  https://dankbuild.com/electronics/hello-esp32/
    /electronics  ->  https://dankbuild.com/electronics/
0 problem(s).

That is a transcript, not a claim from memory. 82 built pages, 80 of them nominated, zero in all four problem classes. The gap between 82 and 80 is fully accounted for by the two stubs held back for declaring an off-origin canonical, which is the shape a healthy build has: zeros in every problem class, and a non-zero exclusion count you can explain on sight.

The final number sums the four problem classes and nothing else — the exclusion line is reported, never counted — and it is what the exit code is derived from. A guard in this repo re-runs the script on every build and compares its output to the block above, so if these numbers ever stop being true the build goes red rather than the page going stale.

What this audit cannot tell you

Honest limits, because a checker you trust past its range is worse than none.

It compares your build output against your build’s sitemap. Both come from the same run, so it cannot see the July failure described above at all: those two posts were never in dist/, they were in a deployed release directory. Point the script at what you actually serve — node sitemap-audit.mjs /var/www/example.com/current — and class (a) lights up immediately for every hand-placed file. Against dist/ it catches the narrower case where your build produced a page that your sitemap integration then dropped.

It reads <loc> with a regular expression. Sitemap generators do not put attributes or CDATA on that element, so this holds in practice, but it is a parser in the loosest sense and it is the first line to replace if you feed it something exotic. It only walks index.html, so if your build emits about.html rather than about/index.html you need to widen that check. It says nothing about <lastmod>, nothing about whether a nominated URL is actually reachable over HTTP, and nothing about noindex — a page can be nominated, self-canonical and still carry a robots meta tag telling crawlers to drop it. That is one more contradiction class, and this script does not look for it.

And none of this is a ranking claim. Removing a contradiction removes a contradiction. I have not measured a search effect from any of these three fixes and I am not going to imply one.

Make it a ratchet, not a fix

The last piece is the one people skip. All three problems above are the kind that come back, because they are produced by defaults, and defaults reassert themselves every time a dependency updates or a new route type appears.

So the checks live in the test suite, not in a person’s memory. astro/test/url-consistency.test.mjs is the ratchet for the first two: it asserts that every <loc> matches the canonical of the page it points at, that the origin in the config equals the one the shared module exports, and that neither off-origin path has re-entered the sitemap. A guard alongside this post pulls the script out of this very markdown file, runs it against the real build, and fails if any class is non-zero — so the numbers printed above cannot quietly stop being true while the prose still claims them.

That is the whole discipline: a sitemap is a set of claims about which URLs exist and what they are called, your pages are another set of claims about the same thing, and the only way they stay in agreement is if something fails loudly when they stop.

← Back to all posts

Follow new posts by RSS: paste dankdev.com/feed.xml into any feed reader. It is a plain XML file — no signup, no email address, no account. Posts land there the same day they go up, and there is nothing to unsubscribe from later.

Back to the top