URIBOT

What is URIBOT?

URIBOT is a Web Crawler that powers the 650 million link database for URIRANK online marketing toolset. It constantly crawls web to fill our database with new links and check the status of the previously found ones to provide the most comprehensive and up-to-the-minute data to our users.

Link data collected by URIBOT from the web is used by thousands of digital marketers around the world to plan, execute, and monitor their online marketing campaigns.

This data has immense value for SEO community as it helps marketing professionals to better understand the fundamental algorithms of the world's largest search engines so that they could optimize websites accordingly. Every 24 hours our crawler visits over 10 million web pages and updates URIRANK’ index every 2 days.

What is URIBOT doing on your website?

URIBOT is crawling your website making notes of outbound links and adding them to our database. It will periodically re-crawl your website to check the current status of previously found links.

Our crawler does not collect or store any other information about your website. It does not trigger ads on your website (if any) and won’t add numbers to your Google Analytics traffic.

Does it respect robots.txt file?

Yes. Absolutely.

We strictly respect robots.txt, both disallow and allow rules.

How to control URIBOT on your website?

As mentioned above, AhrefsBot strictly follows the robots.txt file on your website. So you can fully control it on your website if you need.

To change the frequency of URIBOT visiting your site, you can specify the minimum acceptable delay between two consecutive requests from our bot in your robots.txt file:

Where Crawl-Delay value is time in seconds.

If for some reason you want to prevent URIBOT from visiting your site, put the two following lines into the robots.txt file on your server:

Please note that URIBOT may need some time to pick the changes in your robots.txt file. This will be made prior to each next scheduled crawl.

Please also note that if your robots.txt contains errors and URIBOT won’t be able to recognize your commands it will continue crawling your website the way it did before.

Read more about robots.txt at www.robotstxt.org.

If you think that URIBOT is someway misbehaving on your website or if you have any questions about it, please don’t hesitate to contact our support team [email protected].