second: you can see what time the spider to our site, crawl those pages of the web site, there is no successful crawling, and included. Whether meet dead links, such as crawling back into code is 404, then this link is dead link; the return code is 200, success is crawling; 20028691156 of this code that is successful and included crawling; 20000 is successful but cannot explain a crawling included; if the return code is 304, then there is no new the contents and so on.
see a lot of friends often ask me in the station for more than 10 days or a month, has not been included, what is the reason, my answer is every day to the number of spiders is how much, the answer is not known, visible IIS logs is easily ignored, but often easy to be overlooked is the most important thing, we in the heavy margin of Shanghai Longfeng: site barometer IIS log (1) which discussed how to get the IIS log, here is a look at IIS log information that can tell us about the site.
: the first through the IIS log we can see those spiders come to our website every day, and the number of times to know whether successful, crawling. As long as the number of spiders see our website every day care, you can analyze whether the spider love our website. To be included, and put out is only a matter of time.
third: spider crawling we can know in the website, the website server is stable, if there is a large number of the 500 code, the server problems, beginning with 5 code back to the operation of our information is the information in the IIS server log code.
fourth: when our website, we will find that the number of spiders to will become less, then our first reaction is to check our previous operation or give us a link to sites whether meet the. Combined with the site snapshot update, chain data collected, do a relatively accurate judgment, being punished ago in response, we will minimize the loss of the website. So when we operate every day or to others in exchange link records, is a Shanghai dragon er must develop good working habits.
fifth: if we found 404 of the code in the IIS log, you can use robots to screen out these links, don’t let the spider to crawl; if you don’t see IIS log habit, even death link and you don’t know, let alone go to pick up the shield these links. The spider on the emergence of a large number of dead links is not what good, do not give the weight of this kind of website is very high. Some of my friends say I can check the dead links with webmaster tools, such as the dead link Chinaz detection, the site administrator in the noble baby crawl errors and so on these tools, but don’t forget the wrong tool also has a day.