Your Pages Exist. Google Just Doesn't Believe Them.
You ship a content site on Next.js. Lighthouse shows 100 for SEO. Every page has a title, a description, Open Graph tags and JSON-LD. Then you open Google Search Console and the Pages report shows a long grey bar: Not indexed.
Most indexing guides start with "write better content" and "build backlinks". Both matter, but they're the wrong first step. On a statically generated Next.js site, the most common reasons for missing pages are technical and fixable in an afternoon. Your sitemap points at URLs that 404, two URLs serve the same article, or a build quietly turned a page into an error.
Google doesn't trust a site that sends mixed signals. If 5% of the URLs in your sitemap are broken, Google trusts the whole sitemap less. If one article is reachable at two URLs that each call themselves canonical, Google picks one and labels the other a duplicate, which isn't always the one you wanted.
This guide walks through the audit we ran on this site. It found six tutorials returning 404, 31 pages reachable at two URLs, and 53 internal links pointing at the wrong one. Each section covers how to find the problem from the command line and how to fix it in code.
Step 1: Read the Search Console Report Correctly
Before changing code, open Indexing → Pages in Search Console and scroll to Why pages aren't indexed. Each reason points at a different layer of the problem:
| Reason in Search Console | What it usually means | Can code fix it? |
|---|---|---|
| Not found (404) | Sitemap or internal links point at dead URLs | Yes |
| Duplicate, Google chose different canonical | Same content at multiple URLs | Yes |
| Alternate page with proper canonical tag | Usually fine, if the canonical is the URL you want | Usually |
| Page with redirect | Links or sitemap point at redirecting URLs | Yes |
| Excluded by 'noindex' tag | A robots meta or header blocks it |
Yes |
| Crawled – currently not indexed | Google saw it and decided it wasn't worth indexing | Partly (quality) |
| Discovered – currently not indexed | Google knows the URL but hasn't crawled it yet | Partly (authority, crawl budget) |
The first five are bugs. The last two are judgment calls Google makes about your site. Fix the bugs first: they're cheap, and a cleaner signal often moves pages out of the "currently not indexed" buckets on its own.
Step 2: Check What Googlebot Actually Receives
Your local build isn't what Google sees. Start with the live site:
# Redirects and canonical host
for u in https://example.com/ https://www.example.com/ http://example.com/; do
curl -sI "$u" | grep -iE "^HTTP|location|x-robots-tag"
done
# robots.txt and sitemap
curl -s https://example.com/robots.txt
curl -s https://example.com/sitemap.xml | grep -c "<loc>"
You want exactly one final host. www and http should each send a single 301 or 308 to the canonical HTTPS origin, and no X-Robots-Tag: noindex header should appear anywhere.
Then check that every URL in your sitemap returns 200. This is the check most people skip, and it's the one that found our biggest problem:
curl -s https://example.com/sitemap.xml \
| grep -o "<loc>[^<]*" | sed 's/<loc>//' \
| while read u; do
code=$(curl -s -o /dev/null -w "%{http_code}" "$u")
[ "$code" != "200" ] && echo "$code $u"
done
Any line printed here is a URL you're asking Google to index that can't be indexed. On our site it printed six tutorials, all returning 404.
Step 3: Silent 404s from Broken Frontmatter
The six broken pages existed. The .mdx files were in the repo, generateStaticParams listed them, and the sitemap included them. So why 404?
The App Router page wrapped content loading in a try/catch that called notFound() on any error:
// ❌ Every error becomes a 404, including bugs in your own content
export default async function DocPage({ params }: PageProps) {
const { category, slug } = await params
try {
const { html, frontmatter } = await getMDXContent(category, slug)
// ...
<time dateTime={frontmatter.date.toISOString()} />
} catch {
notFound()
}
}
That pattern is reasonable for a missing slug, but it also swallows content bugs. Two kinds hit us:
A stray code fence above the frontmatter. Five files started with a line of three backticks before the opening ---. To gray-matter, that means the file has no frontmatter at all. frontmatter.date was undefined, .toISOString() threw, and the reader got a 404. This often happens when content is pasted from an AI chat or a Markdown preview that wraps everything in a code block.
Invalid YAML. One file defined lastModified twice. js-yaml, which gray-matter uses, rejects duplicate keys, so the parse threw and the page went to 404.
Neither problem showed up in a normal next build, because the build logged the error and moved on. Catch them with a script that parses every file the same way your site does:
// scripts/check-frontmatter.mjs
import fs from 'fs'
import path from 'path'
import matter from 'gray-matter'
const walk = (d) =>
fs.readdirSync(d, { withFileTypes: true }).flatMap((e) =>
e.isDirectory() ? walk(path.join(d, e.name)) : [path.join(d, e.name)]
)
let failed = 0
for (const file of walk('content').filter((f) => f.endsWith('.mdx'))) {
const firstLine = fs.readFileSync(file, 'utf8').split(/\r?\n/)[0]
if (firstLine.trim() !== '---') {
console.log(`NO FRONTMATTER ${file} (first line: ${firstLine})`)
failed++
continue
}
try {
const { data } = matter(fs.readFileSync(file, 'utf8'))
if (!data.title || !data.date) {
console.log(`MISSING FIELDS ${file}`)
failed++
}
} catch (e) {
console.log(`INVALID YAML ${file} ${e.message.split('\n')[0]}`)
failed++
}
}
process.exit(failed ? 1 : 0)
Run it in CI before next build so a broken file fails the deploy instead of quietly shipping a 404.
Step 4: One Article, Two URLs
This one is subtle and much more common. Our content loader looked up a post by its frontmatter slug, and if that failed, fell back to the filename:
export function findMDXFileBySlug(directory: string, slug: string) {
const files = fs.readdirSync(directory).filter((f) => f.endsWith('.mdx'))
// 1. Match frontmatter slug
for (const file of files) {
const fm = getMDXFrontmatter(path.join(directory, file))
if (fm?.slug === slug) return path.join(directory, file)
}
// 2. Fallback: match filename
const fallback = path.join(directory, `${slug}.mdx`)
return fs.existsSync(fallback) ? fallback : null
}
The fallback is convenient. But a file named getting-started.mdx with slug: vue-js-3-getting-started now answers at both URLs:
/learn/vuejs/getting-started/learn/vuejs/vue-js-3-getting-started
Worse, the canonical tag was built from the route params, so each URL declared itself canonical:
// ❌ Canonical follows whatever URL was requested
canonical: `/learn/${category}/${slug}`
From Google's side, that's two pages with identical content, each claiming to be the original. Google picks one, ignores your preference, and reports the other as Duplicate, Google chose different canonical. We had 31 files where the slug and the filename differed.
The fix: redirect to one URL
Keep the fallback so old links still work, but redirect to the canonical slug instead of serving the page twice. One catch in the App Router: permanentRedirect() works by throwing, so if you call it inside a try/catch that calls notFound(), the redirect is caught and becomes a 404. Do the check before the try:
import { notFound, permanentRedirect } from 'next/navigation'
export default async function DocPage({ params }: PageProps) {
const { category, slug } = await params
const post = await getMDXContent(category, slug).catch(() => null)
if (!post) notFound()
// Filename URLs still resolve, but send them to the one true URL
if (post.frontmatter.slug && post.frontmatter.slug !== slug) {
permanentRedirect(`/learn/${category}/${post.frontmatter.slug}`)
}
const { html, frontmatter } = post
// ...render
}
Make the canonical tag use the frontmatter slug as well, so it never depends on which URL was requested:
// ✅ Canonical is stable regardless of the requested URL
canonical: `/learn/${category}/${frontmatter.slug || slug}`
permanentRedirect sends a 308, which Google treats like a 301: ranking signals move to the target URL and the old URL drops out of the index.
Step 5: Stop Linking to Redirects
A redirect fixes the duplicate, but every internal link that still points at the old URL makes Googlebot take an extra hop, and Search Console fills up with Page with redirect entries. Internal links are also one of the strongest signals for which URL you consider canonical, so they should agree with your canonical tags.
Build a map of old URL → canonical URL from your content, then rewrite links across content and components:
// Build alias map: filename URL -> frontmatter slug URL
const alias = {}
for (const [category, file, data] of allContent()) {
const fileSlug = file.replace('.mdx', '')
if (data.slug && data.slug !== fileSlug) {
alias[`/learn/${category}/${fileSlug}`] = `/learn/${category}/${data.slug}`
}
}
// Rewrite links in .mdx / .ts / .tsx files
const out = source.replace(
/(\/learn\/[a-z0-9]+\/[a-z0-9-]+)(?=[)"'`#\s?])/g,
(url) => alias[url] ?? url
)
The lookahead keeps the regex from matching partial slugs. On our site, this rewrote 53 links in 23 files, including hard-coded "related tutorial" links in a tools catalog that nobody would have thought to check by hand.
Step 6: Make the Sitemap Tell the Truth
The App Router's app/sitemap.ts is the right place for the sitemap, because it's generated from the same content functions that render your pages. Three rules keep it honest:
Only list URLs that return 200. Generate entries from the same loader your pages use, never from a hard-coded list that can drift. If a post is filtered out (a future date, a draft), it must be filtered out of the sitemap too.
Include every page you want indexed. Our sitemap had every article but no category index pages like /learn/react. Those hub pages link to every tutorial in a category, which makes them some of the most useful pages for crawling:
const allLearnPosts = getAllPosts()
const categoryPages = getAllCategories()
.filter((c) => allLearnPosts.some((p) => p.category === c))
.map((c) => ({
url: `${siteUrl}/learn/${c}`,
lastModified: new Date().toISOString(),
changeFrequency: 'weekly' as const,
priority: 0.85,
}))
Have exactly one source. If you moved from next-sitemap to app/sitemap.ts, check public/ for a leftover sitemap.xml or robots.txt. Next.js throws "A conflicting public file and page file was found" in development, and depending on your setup, a stale static file can win in production. Keep one source of truth and delete the other.
Step 7: Re-Verify, Then Tell Google
After deploying, run the sitemap status check from Step 2 again. Every line should be a 200. Then check that a redirected URL and its target behave correctly:
curl -s -o /dev/null -w "%{http_code} -> %{redirect_url}\n" \
https://example.com/learn/vuejs/getting-started
# 308 -> https://example.com/learn/vuejs/vue-js-3-getting-started
curl -s https://example.com/learn/vuejs/vue-js-3-getting-started \
| grep -o '<link rel="canonical"[^>]*>'
# <link rel="canonical" href="https://example.com/learn/vuejs/vue-js-3-getting-started"/>
Then in Search Console:
- Sitemaps → resubmit
sitemap.xml. - URL Inspection → run Test live URL on each page you fixed, then Request indexing.
- On the Why pages aren't indexed report, open each fixed reason and click Validate fix.
Validation takes days to a few weeks. Google re-crawls the affected URLs on its own schedule, and the grey bar shrinks as it does.
What Code Can't Fix
Once the technical issues are gone, the remaining Crawled – currently not indexed and Discovered – currently not indexed pages are about Google's opinion of your site, not your configuration. These help:
- Internal links from strong pages. Link new articles from your homepage, category hubs, and older posts that already rank. An orphan page with no internal links is the first thing Google skips.
- Unique value per page. Thin pages that rephrase documentation compete with the documentation and lose. Real code, real numbers from your own projects, and specific trade-offs are what get pages indexed.
- Fast, stable pages. Crawl budget is partly a function of how quickly your server responds. If you haven't already, look at prefetching with the Speculation Rules API and the Next.js 16 upgrade guide.
- Patience. New domains get crawled slowly. Consistent publishing and a few genuine backlinks change that faster than any meta tag.
The Checklist
| Check | Command or location | Target |
|---|---|---|
| Canonical host redirects | curl -sI on www / http |
Single 301/308 to HTTPS apex |
| No accidental noindex | Page source, X-Robots-Tag header |
None on public pages |
| Sitemap URLs all 200 | Status loop over <loc> entries |
Zero non-200 lines |
| Frontmatter valid | check-frontmatter.mjs in CI |
Build fails on bad files |
| One URL per article | Slug vs. filename audit | Aliases 308 to the slug URL |
| Canonical tag | Built from frontmatter slug | Never from request params |
| Internal links | Alias rewrite script | No links to redirecting URLs |
| Single sitemap/robots source | public/ vs. app/ |
Only one exists |
None of these fixes is clever, and that's the point. Google indexes sites that send consistent signals: every listed URL works, every article has one address, and every link points to it. Get that right before worrying about anything else in your SEO setup.