Paste your robots.txt content and a page URL — the tool determines whether the chosen bot may crawl it and shows the rule that decided the outcome.
Runs in your browser — data is not sent to a serverMatching works like Google and Yandex: the group for the bot (or *) is selected, and among the matching rules the longest one wins; on equal length, Allow beats Disallow. * and $ are supported in paths. An empty Disallow means "everything is allowed".
Character and pixel counts with a live Google snippet preview.
Word Frequency CounterTop words, bigrams, and trigrams in a text — with a stop-word filter.
Robots.txt GeneratorBuild a robots.txt: disallow rules, ready presets, and a sitemap link.
XML Sitemap GeneratorPaste your URLs — get a ready-to-upload sitemap.xml file.
We'll collect and tidy up website data: prices, products, contacts, reviews — in the format and at the frequency you need.