JarvelBot is the web crawler behind Jarvel Search, the search feature of the Jarvel suite (jarvel.com). It visits public web pages and indexes their text so Jarvel users can find them.
• It identifies itself with the User-Agent "JarvelBot/1.0 (+https://jarvel.com/bot)". • It fetches and honors your robots.txt, including Disallow rules and Crawl-delay. • It respects <meta name="robots"> noindex and nofollow directives. • It waits at least two seconds between requests to the same site, and only fetches HTML pages.
To keep JarvelBot off your site, add this to your robots.txt: User-agent: JarvelBot Disallow: / If you have questions or want pages removed from the Jarvel index, contact us via jarvel.com/support.
Want your site in Jarvel Search? Submit it at jarvel.com/submit-site.