The Omni-Crawler

If you have found this page because of the Omni-crawler, and are wondering what it's doing, the Omni-Crawler is crawling the web and providing a feed to partner sites building search engines and services on that data.

For example, if your site has any car, job, vacation rental, or other classified listings, then those listings are being pointed to from Vast.com and its partners, thanks to the Omni-Crawler.

If you would prefer to provide your listings data directly to Vast.com in the form of a feed then please fill out our feed submission form. A feed will ensure that your data is always current while eliminating the need to crawl your site.

If you would prefer that your site not be included at all, you can exclude our crawler.

Please note that blocking this crawler may prevent your results from showing up on various search engines, including Vast.com.

Simply insert the following into your robots.txt file (if you don't know what one is, see the Robotstxt.org site).

The Omni-Explorer agent is: OmniExplorer_Bot/[version]. To prevent it from crawling your site, please put the following in your robots.txt file:

User-Agent: OmniExplorer_Bot
Disallow: /
We will also obey the delay directive (in seconds) for how long to wait between page views on your site:
Crawl-delay: 2

Please feel free to send us feedback. And if you feel that the crawlers are not matching the behavior stated on this page, please include the OmniExplorer_Bot version, HTTP log file lines and your site URL so we can verify the issue. Thank you.

Page last modified 7/19/06