Can robots.txt block my page from Google?

Yes. If a URL is blocked by robots.txt, Google usually cannot crawl the page. If Google cannot crawl a page, it cannot fully index or rank it.

What does it mean if a URL is allowed by robots.txt?

Allowed means Google is permitted to crawl the URL. It does not guarantee indexing or rankings. Content quality, relevance, and internal links still matter.

Why is my page allowed by robots.txt but not indexed?

If a page is allowed but not indexed, the issue is usually not robots.txt. Common reasons include thin content, duplication, weak internal linking, or low search demand.

What is the longest match rule in robots.txt?

When multiple rules match a URL, Google applies the most specific rule. This is usually the rule with the longest matching path. If rules are equally specific, Allow takes precedence.

Does robots.txt remove pages from Google?

No. robots.txt only controls crawling. To remove pages from Google, you need a noindex directive, Search Console removals, or proper authentication.

Should I use robots.txt to hide private pages?

No. robots.txt is public and does not secure content. Private pages should be protected with authentication or server-side access controls.

Yerman Tools

Fast checks • No account required

Robots.txt Tester

Q: What does robots.txt do?

robots.txt tells search engines which URLs they are allowed or not allowed to crawl. It does not control rankings directly and does not remove pages from Google by itself.

Enter any URL to check whether it is Allowed or Blocked by that site’s robots.txt for Googlebot (or a custom user-agent). We show the matched rule and the robots.txt source.

robots.txt tester allowed or blocked googlebot rules seo debugging

Test a URL against robots.txt

Google-style longest match • Tie = Allow

SEO Debug Tools

Build your stack

Redirect Chain Visualiser See every hop + final URL + status codes.
Sitemap Diff Checker Find URLs in sitemap that 404/redirect/noindex.
Canonical Resolver Compare final URL vs canonical tag quickly.
Indexability Checker Check noindex, X-Robots-Tag, canonicals, robots.
GSC Regex Tester Test Search Console regex filters against query lists.

Internal linking: Each tool links to the next so Google can crawl the whole tool cluster.

Privacy: We fetch only the public /robots.txt file and test your URL against its rules. We do not store your input.

What robots.txt actually does (plain English)

Beginner-friendly

A robots.txt file tells search engines which parts of a website they are allowed or not allowed to crawl. It does not control rankings directly — it only controls access.

Important: If a URL is blocked by robots.txt, Google usually cannot crawl or understand the page’s content.

This tool checks whether a specific URL is allowed or blocked, and shows the exact rule that matches.

Common robots.txt rules explained

Examples

User-agent: *

Applies to all bots, including Googlebot.

Disallow: /

Blocks everything on the site. Often used temporarily during development.

Disallow: /admin/

Blocks only URLs that start with /admin/.

Allow: /

Explicitly allows crawling. Usually unnecessary unless overriding a broader block.

Allow: /images/

Allows a subfolder that would otherwise be blocked.

Why a page might be blocked unexpectedly

Very common

A broad Disallow: / rule left over from staging
Folder-level blocks affecting important pages
Overlapping rules where the longest match wins
Blocking parameters or filtered URLs by accident

Tip: This tool uses a Google-style rule match. The longest matching rule wins, and ties favour Allow.

Allowed vs indexed (they are not the same)

Common confusion

A URL being allowed by robots.txt does not guarantee it will be indexed. It only means Google can crawl it.

Allowed

Google is permitted to crawl the URL.

Indexed

Google has crawled the page and decided to include it in search results.

If a page is allowed but not indexed, the issue is usually content quality, internal links, or duplicate signals.

What robots.txt does NOT do

Myths

It does not remove pages from Google
It does not act as a noindex directive
It does not improve rankings
It does not secure private content

To remove indexed pages, you need noindex, removals in Search Console, or proper authentication.

What to check if a URL is blocked

Next steps

Confirm the matched rule shown above
Check whether the block is intentional
Fix overly broad patterns
Re-test after updating robots.txt

After fixing robots.txt, Google may need time to re-crawl the site. You usually don’t need to request indexing immediately.

What to check next

Recommended tools

🔁 Redirects before the final page → Redirect Chain Visualiser
🧭 Canonical conflicts → Canonical Resolver
🗂️ URLs blocked or redirected in sitemaps → Sitemap Diff Checker

These tools are linked in a logical order to help diagnose indexing problems step by step.

Final takeaway

Summary

robots.txt is a powerful file, but small mistakes can block entire sections of a website without anyone noticing.

If Google can’t crawl a page, it can’t rank it — no matter how good the content is.