Website + SEO

Read and test the site's robots.txt.

Fetch the root robots file, see declared sitemap URLs, and test whether a selected path is allowed for a chosen crawler token.

Scope

Host, protocol and port matter

robots.txt rules apply to the host, protocol and port where the file is served. A rule on one hostname does not automatically govern another.

Index

Disallow is not a noindex directive

Blocking crawling can prevent a crawler from seeing page-level indexing directives. Use the right control for the actual indexing goal.

Map

Sitemap declarations are absolute URLs

Major crawlers support Sitemap fields in robots.txt. The declared location may be on another verified host.

Match

Longest matching path wins

The path tester uses common Google-style longest-match behavior, including Allow winning equal-length ties. Other crawlers may support different extensions.