The sitemap error that wasn't about the sitemap

Search Console's 'Sitemap could not be read' banner above a diagram: a page's canonical and hreflang tags point at the Amplify preview host, crossed out in red, and are redirected in teal to www.jpmti2016.com, with a footer listing robots.txt 404, sitemap.xml 404, feed.xml 500, and indexed pages falling from 12 to 10

Search Console had one red banner for this site: Sitemap could not be read — General HTTP error, 1 instance. Underneath it, a quieter set of numbers: 17 discovered pages, four legacy URLs stuck in "Discovered – currently not indexed", four 404s, and an indexed count that had drifted from 12 to 10 over the summer while I kept publishing.

I assumed the sitemap had simply not survived a migration between two Next.js codebases. It hadn't. But that was the fourth most important thing wrong, and fixing it alone would have left the real damage in place. This post is the audit as I actually ran it — with curl, not with a dashboard — because the lesson generalizes: Search Console tells you where Google stumbled, not why. Ask the site directly.

TL;DR

  1. The sitemap error was real but shallow: /sitemap.xml and /robots.txt did not exist and fell into the locale catch-all, which served the 404 page with a 404 status.
  2. The damaging finding was invisible from the dashboard: every canonical and hreflang on the live site pointed at main.<app-id>.amplifyapp.com, because an Amplify env var held the preview URL. Google was being told the real pages were duplicates of a host nobody links to.
  3. The RSS feed returned 500 because it fetched and scraped its own pages at request time, which times out on serverless.
  4. The old site lived at /posts; the new one at /articles; no redirects bridged them. Every legacy link, including the ones Google remembered, was a 404.
  5. Fix the origin in code, gate indexability on the production branch, generate robots/sitemap/feed from the content library, and turn the redirect table into tested data.

The symptom versus the diagnosis

The first thing I did was not open the sitemap report. It was this:

for u in /robots.txt /sitemap.xml /feed.xml /posts /posts/free-self-hosted-web-analytics; do
  curl -sS -o /dev/null -w "$u %{http_code} %{content_type}\n" "https://www.jpmti2016.com$u"
done

/robots.txt 404 text/html; charset=utf-8
/sitemap.xml 404 text/html; charset=utf-8
/feed.xml 500
/posts 404 text/html; charset=utf-8
/posts/free-self-hosted-web-analytics 404 text/html; charset=utf-8

Four lines, four problems, none of them the one I expected to be fixing. Then I looked at what a crawler sees in the <head> of the homepage:

curl -s https://www.jpmti2016.com/ | grep -o '<link rel="canonical"[^>]*>\|<link rel="alternate"[^>]*>'

<link rel="canonical" href="https://main.df9x6nsrpiysq.amplifyapp.com"/>
<link rel="alternate" hrefLang="en" href="https://main.df9x6nsrpiysq.amplifyapp.com"/>
<link rel="alternate" hrefLang="es" href="https://main.df9x6nsrpiysq.amplifyapp.com/es"/>
<link rel="alternate" hrefLang="x-default" href="https://main.df9x6nsrpiysq.amplifyapp.com"/>
<link rel="alternate" type="application/rss+xml" href="https://main.df9x6nsrpiysq.amplifyapp.com//feed.xml"/>

That is the finding that mattered. A canonical tag is a page saying "the authoritative copy of me lives over there." Every page on www.jpmti2016.com was pointing over there — at the Amplify default domain for the main branch. Google's deduplication does exactly what it is told: it consolidates signals onto the declared canonical and drops the duplicate from the index. The slow slide from 12 indexed pages to 10 was not a content problem. It was the site voting against itself on every URL.

The double slash in the feed link was the tell for the cause: an env var with a trailing slash.

How one env var lied to every page

The site template reads its public origin from NEXT_PUBLIC_SITE_URL and uses it as Next's metadataBase, which is what turns relative canonical and hreflang paths into absolute URLs. Locally that variable is in .env.local. In Amplify Hosting it is set per app, and at some point the value became the preview URL, probably while checking the first deploy, and nobody looked at it again.

The fix was not to correct the value. It was to stop trusting a value at all. A site has exactly one public origin; it belongs in code:

// src/lib/site.js
export const SITE_URL = 'https://www.jpmti2016.com'

/**
 * True only for the build that serves the public domain. Amplify sets
 * AWS_BRANCH on every build; SITE_ENV=production lets a local build be
 * checked with the same behavior. Everything else is a preview and
 * must never be indexed.
 */
export function isProduction(env = process.env) {
  return env.AWS_BRANCH === 'main' || env.SITE_ENV === 'production'
}

Every canonical, hreflang, sitemap, feed and structured-data URL now derives from SITE_URL, whichever branch rendered the page. And the second export is what stops this class of bug from coming back in another shape: the staging preview also renders absolute URLs to the production domain, and it declares itself unindexable.

// src/app/[locale]/layout.jsx — generateMetadata()
metadataBase: new URL(SITE_URL),
robots: isProduction()
  ? { index: true, follow: true }
  : { index: false, follow: false },

// src/lib/robots.js
export function robotsRules({ production }) {
  if (!production) {
    return { rules: { userAgent: '*', disallow: '/' } }
  }
  return {
    rules: [{ userAgent: '*', allow: '/', disallow: ['/api/', '/thank-you'] }],
    sitemap: `${SITE_URL}/sitemap.xml`,
  }
}

A preview host can still be reached, but it now says noindex on every page and Disallow: / in its robots file, while its canonicals point home. That is the correct posture for any non-production environment, and it costs one boolean.

What the migration dropped

The previous site was a Pages Router app. It wrote public/sitemap.xml from a script hooked into the webpack build, shipped a hand-written public/robots.txt, and served its blog at /posts and /es/posts. The new App Router site kept the content and the design and quietly lost all three, because none of them were content — they were plumbing, and plumbing does not migrate itself.

Next's metadata routes replace the script. src/app/sitemap.js and src/app/robots.js are thin; the logic lives in pure modules that take already-loaded content and are unit-tested without touching MDX:

// src/app/sitemap.js
export default async function sitemap() {
  const articles = []
  const projects = []
  for (const locale of LOCALES) {
    articles.push(...(await getAllArticles(locale)))
    projects.push(...(await getAllProjects(locale)))
  }
  return buildSitemap({ articles, projects })
}

buildSitemap emits one entry per URL per locale, with alternates.languages so Google sees the same hreflang pairs in the sitemap that the pages declare in their heads. Seventy-two URLs, and the post-subscribe thank-you page deliberately absent.

The legacy URLs became data rather than config, so the mapping is something a test can read:

// src/lib/redirects.mjs
export const legacyRedirects = [
  { source: '/posts', destination: '/articles', permanent: true },
  { source: '/posts/:slug', destination: '/articles/:slug', permanent: true },
  { source: '/es/posts', destination: '/es/articles', permanent: true },
  { source: '/es/posts/:slug', destination: '/es/articles/:slug', permanent: true },
  // …
]

The verification for that table was the Search Console export itself. Every URL Google had flagged, followed to the end:

for p in /posts /es/posts /posts/6-free-resources-to-learn-Rust /posts/async-update-related-items; do
  curl -s -L -o /dev/null -w "$p -> %{http_code} %{url_effective}\n" "http://localhost:3100$p"
done

/posts -> 200 http://localhost:3100/articles
/es/posts -> 200 http://localhost:3100/es/articles
/posts/6-free-resources-to-learn-Rust -> 200 http://localhost:3100/articles/6-free-resources-to-learn-Rust
/posts/async-update-related-items -> 200 http://localhost:3100/articles/async-update-related-items

If Google flagged it and it now lands on a 200, that flag will clear on its own. The only one left as a 404 is /utils/gtag, a JavaScript module URL from the old site that was never a page. A 404 is the right answer for it.

The feed that scraped itself

The template's /feed.xml route worked like this: for every article, fetch() the article's own HTML over HTTP, load it into cheerio, pull the title, date and body out of the DOM, and add the item. Nineteen articles, nineteen server-side renders, sequentially, inside one request. On a laptop that takes a few seconds. On Amplify's compute it hits the function timeout, and CloudFront reports it as an empty 500.

The replacement is boring, which is the point. The article metadata is already exported from each MDX file, so the feed reads it directly and is generated once at build time:

// src/app/feed.xml/route.js
export const dynamic = 'force-static'

export async function GET() {
  const feed = new Feed({ title: SITE_NAME, id: SITE_URL, link: SITE_URL, language: 'en' })
  for (const article of await getAllArticles('en')) {
    const url = absoluteUrl('en', `/articles/${article.slug}`)
    feed.addItem({
      title: article.title,
      id: url,
      link: url,
      description: article.description,
      date: new Date(`${article.date}T00:00:00Z`),
    })
  }
  return new Response(feed.rss2(), { headers: { 'content-type': 'application/xml; charset=utf-8' } })
}

Items carry the description rather than the full body. Nothing in a feed reader has missed it.

Three Next.js traps I hit on the way

None of these are bugs. All of them are documented. All of them cost me a rebuild.

Layout alternates are replaced, not merged. The locale layout declared the RSS link under alternates.types, and every page declared its own alternates for canonical and hreflang. Next merges metadata shallowly per top-level key, so the page's object replaced the layout's whole alternates — and the RSS link was missing from every page that had a canonical, which is all of them. The RSS entry now rides inside the same helper that builds the canonical.

The opengraph-image file convention fills dynamic segments literally. With the site's default locale unprefixed (/about, not /en/about), an opengraph-image.jsx inside app/[locale]/ produces og:image URLs of the form /en/opengraph-image, and the /en/* redirect then bounces them. Functional, but sloppy for something a crawler fetches. The cards became plain route handlers instead, with the URL built explicitly.

The locale middleware skips any path with a dot. The matcher that keeps feed.xml and favicon.ico out of locale negotiation is the sensible '/((?!api|_next|_vercel|.*\\..*).*)'. That means a route at app/[locale]/og.png/route.js works at /es/og.png, where the locale is in the URL, and returns 404 at /og.png, because nothing rewrites it to /en/og.png. The route is /og now — no extension, and the content-type header does the talking.

/og                                                200 image/png
/es/og                                             200 image/png
/articles/deploying-nextjs16-on-amplify-ssr/og     200 image/png
/articles/no-such-post/og                          404

What AI crawlers get

The old robots file blocked GPTBot, CCBot and Google-Extended, which in 2022 read as prudence. For a site whose entire purpose is to be found and cited when someone hits the same error I hit, it reads as self-sabotage now. The new file explicitly allows the answer-engine crawlers — OpenAI's, Anthropic's, Perplexity's, Google's extended crawler, Apple's — and the * rule allows everyone else. Listing them by name changes nothing mechanically; it makes the policy legible in the served file.

Two small additions on top. An llms.txt at the root, generated from the same content library as the sitemap, gives an assistant a Markdown index with one line per article and its description, so an answer can cite the right URL instead of the homepage. And JSON-LD went in where Google documents it: Person and WebSite on the home page, BlogPosting plus BreadcrumbList on every article, with the author's profiles as sameAs. Every value in those objects is something the page already says; structured data is a restatement, not a place to make claims.

The Amplify gotcha at the very end

With everything deployed, one loose end remained: the default main.<app-id>.amplifyapp.com host still served an indexable copy of the site. Its canonicals now pointed home, so Google would consolidate correctly, but a redirect is cleaner. I added a rule in Amplify's rewrites and redirects with the source https://main.<app-id>.amplifyapp.com/<*>, target https://www.jpmti2016.com/<*>, type 301.

Nothing happened. No error, no redirect. The docs explain why, in an "Important" box that is easy to skim past:

Path components in domain-based source rules (such as "https://domain.com/path") are not supported and will cause the rule to be ignored without error.

A domain-based rule takes the bare origin as its source, and Amplify appends the path itself. Remove the wildcard from both sides and it works on the next request:

curl -sS -o /dev/null -w "%{http_code} -> %{redirect_url}\n" "https://main.df9x6nsrpiysq.amplifyapp.com/about"

301 -> https://www.jpmti2016.com/about

The same console page had one more thing worth touching: Amplify's auto-generated apex-to-www rule defaults to 302. Search Console lists that under "Page with redirect" forever. It is a dropdown; make it 301.

Rules that generalize

  • Audit with curl, not with the dashboard. Search Console reports what Google could not do. Fetching the same URLs yourself tells you why, in seconds, and surfaces the problems Google has not reported yet.
  • The public origin is code, not configuration. A site has one. Put it in a module, derive every absolute URL from it, and gate indexability on the branch that serves it. Previews get noindex and canonicals pointing home.
  • Plumbing does not migrate itself. When a site moves between frameworks, list what the old build generated — sitemap, robots, feed, redirects, headers — and check each one against the new deploy with a request, not a glance at the repo.
  • Prove redirects against the export. The list of URLs Google is stuck on is the test fixture. Follow each one to the end and demand a 200.
  • Read the "Important" box. Twice.

References

Related articles