TombaPublicWebCrawler Web Robot

Fecha de revisión : Sep 4, 2022

TombaPublicWebCrawler

Version: 3.0

Obeys Robots.txt: Yes

User-Agent String: Mozilla/5.0 (compatible; TombaPublicWebCrawler/3.0; +https://tombascraper.com)

What is Tomba robot?

TombaPublicWebCrawler is an indexing robot for a web search engine, similar to Google. Created by Tomba technology web service LLC.(Tomba.io), TombaPublicWebCrawler’s patented technology continually scans millions of corporate websites, press releases, electronic news services, SEC filings and other online sources. Using advanced natural language processing algorithms, Tomba has created a next generation search engine focused on finding pages with information about businesses and business professionals.

What is Tomba robot doing on your website?

The robot only analyzes public web pages .

Does it respect robots.txt file?

Does it respect robots.txt file? Yes. Absolutely. We strictly respect robots.txt, both disallow and allow rules.

How to control TombaPublicWebCrawler on your website?

As mentioned above, TombaPublicWebCrawler strictly follows the robots.txt file on your website. So you can fully control it on your website if you need.

To change the frequency of TombaPublicWebCrawler visiting your site, you can specify the minimum acceptable delay between two consecutive requests from our bot in your robots.txt file:

User-agent: TombaPublicWebCrawler

Crawl-Delay: [value]

Where Crawl-Delay value is time in seconds. If for some reason you want to prevent TombaPublicWebCrawler from visiting your site, put the two following lines into the robots.txt file on your server:

User-agent: TombaPublicWebCrawler

Disallow: /

Please note that TombaPublicWebCrawler may need some time to pick the changes in your robots.txt file. This will be made prior to each next scheduled crawl.

Please also note that if your robots.txt contains errors and TombaPublicWebCrawler won’t be able to recognize your commands it will continue crawling your website the way it did before.