I deployed this publication before naming it. The name it currently carries is a placeholder, chosen so I could build the thing, and it will be replaced once I’ve written enough to know what this actually is.
That creates a specific problem. A deployed site with a sitemap and clean metadata is exactly what a crawler wants. If Google indexes the placeholder, it starts associating my URLs with a brand I’m about to delete, and every one of those URLs changes when the real domain arrives. I’d be cleaning up redirects for a name that never mattered.
So indexing is off, and it’s off behind a single switch.
One flag, read in one place
allowIndexing: process.env.NEXT_PUBLIC_ALLOW_INDEXING === "true",The comparison against the string "true" is deliberate. Environment variables are always strings, so an unset variable, an empty value, and a typo all evaluate to false. The only way to get indexed is to say so exactly.
I wanted this as a config value rather than scattered process.env reads because two separate files need the same answer, and those two files disagreeing is the failure mode I was trying to design out.
The two layers do different jobs
The first is robots.txt:
export default function robots(): MetadataRoute.Robots {
if (!site.allowIndexing) {
return {
rules: { userAgent: "*", disallow: "/" },
};
}
return {
rules: { userAgent: "*", allow: "/" },
sitemap: absoluteUrl("/sitemap.xml"),
};
}Note that the sitemap reference disappears along with the permission. Advertising a sitemap while disallowing the whole site is a mixed signal, and there’s no reason to send it.
The second is the robots field in the root layout’s metadata, which renders a noindex directive into every page:
robots: {
index: site.allowIndexing,
follow: site.allowIndexing,
},Why both, when they overlap
They don’t actually overlap, and the difference matters if you ever need to remove something.
robots.txt controls crawling. It asks a crawler not to fetch your pages. It does not, on its own, guarantee a URL stays out of the index — a URL that’s linked from elsewhere can still show up, listed without content, because the crawler was told not to look.
noindex controls indexing, and it lives in the page itself. Which means a crawler has to fetch the page to see it. If you block crawling, your noindex is never read.
That sounds like the two tags contradict each other, and in a strict sense they do. I’m accepting that tradeoff knowingly: right now nothing links to this site, so blocking the crawl is enough, and the noindex is there for the case where something does link to it and a crawler ignores the first signal. What I’d avoid is relying on robots.txt alone to pull down a page that has already been indexed. For that, you have to let the crawler in so it can read the noindex.
The part that isn't technical
The reason I did this at all is that a name is not a label you swap out later for free. Once search results, links, and other people’s bookmarks point at a brand, changing it costs real traffic.
Deciding to stay unindexed for a while also removed a temptation. There’s no point optimising a publication for search before it has three articles, and with indexing off I can’t pretend otherwise. The work in front of me is writing, not fiddling with metadata.
When the real name is chosen and the domain is attached, one variable flips to true, the sitemap reappears in robots.txt, and the site becomes crawlable on the correct domain with its final brand. That’s the whole migration.
The name is Hiba Builds now. The flag is still the switch. Google already has impressions on a handful of URLs. I turn indexing on in production with NEXT_PUBLIC_ALLOW_INDEXING=true, not by publishing a hundred extra posts.