[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
- Subject: Bing/MSN/Microsoft contact
- From: jayfar at jayfar.com (Jay Farrell)
- Date: Tue, 12 Mar 2013 13:29:40 -0400
- In-reply-to: <[email protected]om>
- References: <[email protected]om>
This might help:
How do I Report an Issue With Bingbot?
Bingbot is the name of the crawler used by Bing to crawl or ?spider? the
web. It is Bingbot's job to find new and updated pages on websites across
the Internet, so that they can be processed for indexation. When crawling a
website, Bingbot looks at robots.txt for special instructions from the
website owner. Bingbot honors robots.txt directives, including the *
crawl-delay:* setting, and, in the absence of a crawl-delay, respects the
input from Webmasters in the Crawl Control Feature.
Generally, Bingbot does a good job at determining how frequent it should
visit pages on your site, taking robots.txt and Crawl Control rules and
hints into consideration. We call this ?Crawl Politeness?. However, there
still may be cases were you feel Bingbot is not polite enough and is
visiting your pages more than works for you (overcrawling).
Here?s the steps you can follow if you think Bingbot is overcrawling your
site or not observing robots.txt rules:
1. Verify that the bot traffic you are seeing is in fact from a valid
Bingbot server. You can do this by not only looking at the User-Agent
string (which can be easily spoofed by anyone) but also at the IP address
and using the *Verify Bingbot
* to get a verdict
2. Once you have verified that this involves genuine Bingbot traffic,
you can reduce crawler traffic as follows (if you haven?t done so already):
3. Lower the speed of crawl during busy hours using the Crawl Control
4. If that?s not sufficient, add a crawl-delay: directive to your
robots.txt: Bing supports whole number values ranging from 1 ? 30. Each
number maps to the number of seconds we need to wait between requests, so 1
means a maximum of 1 request per 1 second ? which is slow, but still
adequate for smaller sites. 30 is extremely slow and means we are allowed
only 1 request per 30 seconds.
5. If you have followed step 1 and 2 and the issue is still present, you
can contact *Bing Webmaster
*. Fill out the required fields and in the ?*What type of problem do you
have?? *dropdown, select ?*Under-Crawling or Over-Crawling inquiry*? and
describe the problem you are seeing. You can expect a reply within 24-48
hours. When you report over-crawling issues, the support team will ask you
to provide server log samples that show the Bingbot activity over a certain
period of time in a next step, so make sure to have those ready.
On Tue, Mar 12, 2013 at 11:57 AM, Blair Trosper <blair.trosper at gmail.com>wrote:
> If possible, I need someone from Microsoft/Bing (a la the MSN and Bing
> crawler bots) to contact me off list.
> Several IPs going back to AS8075 (with the user agent MSN and "bingbot")
> are basically attacking several IPs on my network with hundreds of requests
> per second.