Guides · Node.js & Express · Published 2026-09-13 · 3 min read

Serving sitemap.xml, robots.txt and ads.txt From Express — Generated From Content, Cached in Memory, and Configured by Environment Variables

Three small text endpoints every content site needs. How to build sitemap.xml from the content index with lastmod and alternates, keep robots.txt honest, serve ads.txt from an environment variable so ad networks can be added without a code change, and cache all three.

A content site has three plain-text endpoints that machines read and humans never see. They are easy to get slightly wrong: a sitemap that lists draft pages, a robots file that blocks CSS, an ads.txt that is a day behind the ad network's requirement. Serving them from Express, generated from the same data as the pages, keeps them correct without maintenance.

sitemap.xml

Build it from the content index, not by hand. For each language and each published article, emit a <url> with <loc> as the absolute URL, <lastmod> from the article's updated date, and, for pages that have translations, the xhtml:link rel="alternate" entries. Include the home page and category pages. Leave out anything unlisted, redirected or paginated, and never include URLs that return anything but 200.

app.get('/sitemap.xml', (req, res) => {
  res.type('application/xml').send(buildSitemap()); // cached, see below
});

Set the content type explicitly; some crawlers reject a sitemap served as text/html. If the site grows past tens of thousands of URLs, split it into a sitemap index, but a small site should keep the single file. The alternates format is described in a multi-language Express site with hreflang.

robots.txt

Keep it short and honest:

User-agent: *
Allow: /
Disallow: /admin/
Sitemap: https://example.com/sitemap.xml

Do not block the static folder; crawlers need CSS and JavaScript to render pages. Do not try to hide pages with Disallow; that only hides them from crawlers that obey, and they may still be indexed from links. For a redirect subdomain or an admin app, Disallow: / is correct because there is nothing to index. Serve it as text/plain.

ads.txt from an environment variable

Ad networks require ads.txt at the site root listing the seller accounts authorised to sell your inventory. The content changes when you add a network, and it is different per site. Rather than committing a file, read it from an environment variable at startup:

const ADS_TXT = process.env.ADS_TXT || '';
app.get('/ads.txt', (req, res) => {
  if (!ADS_TXT) return res.status(404).type('text/plain').send('');
  res.type('text/plain').send(ADS_TXT.replace(/\\n/g, '\n'));
});

Set the variable at deploy time with the lines separated by \n. Adding a network is then a redeploy with a new value, with no code change and no chance of committing one site's ads.txt to another site's repository. Some networks instead want ads.txt to redirect to a URL they host; a 301 from the same route handles that.

Caching all three

Generating the sitemap walks the content index; cheap, but not free on every request. Cache the generated string in memory and invalidate it when the content index reloads. Send Cache-Control: public, max-age=3600 on all three so edge caches and repeated crawler fetches do not hit the app. On Cloud Run this also reduces cold-start exposure; see what a small Cloud Run site really costs.

Verifying

Fetch each with curl -i after deploying and check the status, content type and first lines. Then submit the sitemap in Search Console, as covered in Search Console domain verification with Cloudflare, and check the ad network's dashboard for its ads.txt status a day later.

Common mistakes

Sitemap URLs with the wrong host. Build from a configured SITE_HOST, not from the request's Host header, which may be the internal service URL.
Blocking /img/ or /css/ in robots.txt.
A trailing newline missing from ads.txt. Some validators reject the last line.

Summary

Generate the sitemap from the content index with lastmod and alternates, keep robots.txt to a few honest lines with the sitemap URL, serve ads.txt from an environment variable, and cache all three with explicit content types. Check them with curl after every deploy.

express seo sitemap robots ads.txt nodejs

Related guides

Redirect link service with Firestore — Build a go.example.com redirect service in Node and Express backed by Firestore: one document per slug, a 302 to the target, a per-month click counter, and an admin list. Why routing every outbound link through it saves you when programs change.
Node TLS errors behind antivirus or a proxy — When npm, fetch or gcloud fail TLS verification on one machine only, something is intercepting HTTPS and re-signing it. How to see whose certificate you are being handed, why Node ignores the Windows store by default, and the one flag that fixes it safely.
Multi-language Express site with hreflang — How to structure a small Express and EJS site so that English lives at the root, other languages under a prefix, every page declares its alternates with hreflang and x-default, and the sitemap lists them, using a single config file and one content folder per language.
Search Console domain verification with Cloudflare — The fastest way to get a new domain into Search Console when DNS is on Cloudflare: choose a domain property, let Domain Connect add the TXT record, confirm, and submit the sitemap. What to do when the first sitemap fetch fails because the site is minutes old.