CrawlerCheck

CrawlerCheck is a tool that solves one specific problem: it tells you if you're accidentally blocking search engines like Google from seeing your website. It's easy to make a mistake in your site's settings that makes a page invisible to bots. Instead of you having to look in three different places, CrawlerCheck checks them all at once. It reads your robots.txt file, finds any robots meta tags in the page's HTML, and inspects the X-Robots-Tag sent by your server. It then shows you exactly which rules apply and if bots are allowed or blocked.