Web Crawler Access Detection - DotBot

Info Nessus Network Monitor Plugin ID 9188

Synopsis

Requests observed to this server indicate it was accessed by the Moz DotBot automated web crawler.

Description

Moz' 'DotBot' recently discovered and indexed web content available on this server. Crawling is the process by which various search engine bots discover new and updated pages to be added to their search results. These automated scans are conducted regularly (or periodically) to peruse the entire public IP space (including domain listings) in order to update their search results.

Solution

Determine if the affected service or content should be externally accessible. In the event this server cannot be removed from the public realm, consider implementing 'robots.txt' to help limit automated bots from scanning sensitive zones within the site.

See Also

http://www.opensiteexplorer.org/dotbot

https://en.wikipedia.org/wiki/Web_crawler

Plugin Details

Severity: Info

ID: 9188

Family: Generic

Published: 2016/04/06

Modified: 2018/05/18

Dependencies: 9179

Risk Information

Risk Factor: Info