Blocked by robots.txt in Search Console: How to Fix

What Does “Blocked by robots.txt” Mean in Search Console?

Quick answer: Blocked by robots.txt means your robots.txt file tells Google not to crawl that URL. It’s fine for private or low-value pages, but a problem for pages you want in search. Fix it by editing the matching Disallow rule, testing the URL and validating the fix in Search Console.

Many site owners see this status in the Page indexing report and worry. First decide whether you meant to block the page.

What the status means

robots.txt controls crawling, not indexing directly. If Google can’t crawl a page, it can’t read the content. The URL may still appear in results without a description if other pages link to it, which is why you might also see “Indexed, though blocked by robots.txt.”

When it’s fine

• Admin, login, cart and internal search pages
• Filter and parameter URLs
• Staging or test pages

When it’s a problem

• Key service, product or blog pages
• CSS or JavaScript files Google needs to render your page
• A leftover “Disallow: /” from a development site

How to fix it

1. Open Search Console → Page indexing and click “Blocked by robots.txt” to see affected URLs.
2. Open your robots.txt at yourdomain.com/robots.txt and find the rule matching the URL.
3. Edit or remove the Disallow rule. In WordPress, check SEO plugins and the “discourage search engines” setting.
4. Test the URL with Search Console’s robots.txt report and URL Inspection.
5. Save and upload the corrected file.
6. Click Validate fix in the report, then be patient.

To keep a page out of Google

Don’t rely on robots.txt alone. Use a noindex tag on a page Google can crawl, or password protection for private content. If robots.txt blocks crawling, Google can’t see the noindex tag.

After the fix

Once crawlable, pages still need to be indexed. See crawled currently not indexed and discovered currently not indexed. If traffic fell after a site change, read why impressions drop in Search Console.

AI crawlers

Some sites also block AI crawlers in robots.txt. That’s a business decision. See what GEO is before you decide.

Our Search Engine Optimisation (SEO) team can audit your robots.txt and indexing.

FAQs

What does blocked by robots.txt mean?
Google was told not to crawl the URL, so it can’t read the page.

Does robots.txt remove a page from Google?
Not reliably. Use noindex or remove the page properly.

Is blocked by robots.txt a penalty?
No. It’s a crawl instruction you or your platform set.

How long does validation take?
It can take days to weeks.

Should I block my CSS and JS files?
No. Google needs them to render pages properly.

Conclusion

Blocked by robots.txt is only a problem when it hits pages you want to rank. Check your rules, fix mistaken blocks, test and validate. Find more Search Console guides on the Biznex blog.