Fixing a Robots.txt That Accidentally Blocks Search Engines from an Entire Site After a Migration

The Problem

Organic search traffic drops sharply and suddenly, usually within days of a site migration, redesign, or a redeploy that touched infrastructure. Google Search Console's coverage report starts showing pages as "Blocked by robots.txt" or "Indexed, though blocked by robots.txt" — pages that used to rank and that render completely normally for any human visiting them directly. Nothing about the site itself is broken; visitors browsing it notice nothing wrong at all.

Why It Happens

A staging environment's robots.txt is a common, easy thing to accidentally ship to production

Staging and development environments frequently use a robots.txt containing a blanket Disallow: / specifically so search engines never index unfinished or duplicate content. That file is often just a static asset in the build output, and during a migration, a redesign, or a deploy pipeline change, it's entirely possible for the staging version to get built into or copied onto the production deployment without anyone specifically checking it per environment — nothing about the deploy itself fails or errors when this happens.

A CDN, proxy, or per-subdomain config can serve a different robots.txt than the one in source control

On a multi-subdomain setup, the file actually reachable at /robots.txt on each live domain isn't guaranteed to be the exact file a developer has open in their editor — a CDN rule, a reverse proxy configuration, or an environment-specific override can serve something different for one subdomain than for another, which is exactly the kind of divergence that only shows up by actually fetching the live URL, not by reading source.

A blocking robots.txt is invisible to normal testing and produces no error

The site renders, loads, and functions completely normally for a human visitor and for the overwhelming majority of manual or automated smoke testing — a restrictive robots.txt doesn't produce a 500, a broken page, or any visible symptom at all. It only becomes visible to someone who specifically fetches /robots.txt and reads it, or who checks Search Console's coverage report after the fact, which is why it commonly goes unnoticed until organic traffic has already measurably dropped.

Search engines respect a disallow rule immediately and without exception

There's no grace period or partial enforcement — once a crawler fetches a robots.txt with a blanket disallow, it stops crawling and, on recrawl, existing indexed pages can be deindexed as they're reevaluated against the new rule. A large, well-established site can lose meaningful search visibility within days of a blocking file going live, not gradually.

The Fix

1. Fetch and read the actual live robots.txt on every domain as a mandatory post-deploy step

curl -s https://corejsc.com/robots.txt
curl -s https://us.corejsc.com/robots.txt
curl -s https://in.corejsc.com/robots.txt
curl -s https://eg.corejsc.com/robots.txt

Verify the file actually served in production, on every subdomain independently, rather than assuming it matches what's committed in source control — this closes the gap where a CDN or environment-specific config serves something different from what a developer expects.

2. Add an automated check that fails the deploy if production robots.txt contains a blanket disallow

# in a post-deploy CI step
if curl -s https://corejsc.com/robots.txt | grep -qE "^Disallow: /$"; then
  echo "ERROR: production robots.txt blocks the entire site"
  exit 1
fi

A scripted check that fails the pipeline outright removes reliance on someone remembering to manually verify this after every migration or redeploy — the exact kind of check that's cheap to add and catches a mistake with outsized consequences.

3. Keep staging's robots.txt explicitly separate from production's build artifact

Rather than relying on a single static file that could be accidentally promoted between environments, serve robots.txt dynamically based on the environment (a hostname check that returns a blocking file only for staging domains), or keep genuinely separate build artifacts per environment so a staging config has no path to reaching production at all.

4. If a blocking file did ship, fix it and then explicitly request recrawling

Correcting the robots.txt file alone doesn't force an immediate recrawl of already-deindexed pages — use Search Console's URL Inspection tool to explicitly request reindexing for key pages after the fix ships, since recovery can otherwise lag behind the fix by longer than necessary.

Why This Works

Each fix targets the specific gap that let this happen invisibly: checking the live file directly (not source control) catches CDN or environment-level divergence; an automated pipeline check removes reliance on someone remembering a manual step; explicitly separating staging and production configs removes the actual mechanism by which a blocking file gets promoted in the first place; and requesting recrawl after a fix closes the gap between "the file is correct again" and "search engines have actually noticed."

Conclusion

A sudden organic traffic drop after a migration or redeploy that coincides with Search Console reporting pages blocked by robots.txt is very rarely a ranking or algorithm problem — it's almost always a staging or environment-specific robots.txt that shipped to production by accident, invisible to normal testing because the site otherwise works completely fine. Fetch and verify the live robots.txt on every domain after every deploy, add an automated check that fails the pipeline on a blanket disallow reaching production, keep staging and production robots.txt handling explicitly separate, and request recrawling via Search Console once a blocking file is fixed rather than waiting for it to happen on its own.