MSNBot Spider and Yahoo! Slurp Spider

[zeuscane]

Dear administrator,
I have discovered, with who's Online menu that on the forum as guests enter kind MSNBot Spider and Yahoo! Slurp Spider.
MSNBot Spider is automatic web robot that follows links throughout the internet internally though websites and from site to site.
The information it gathers makes a 'map' of the net for the search engines and also determines how high a website will be ranked... so
the MSNbot will be gathering info for the MSN search engine and GoogleBot and YahooSpider will be gathiering info for Google and Yahoo, respectively.
What us we can do for stopping them?
Regards
zeuscane

[blaster949]

Hi Zuescane
I believe you can add a text file in the root directory of the hosted site, so the bot will read it and STOP going further. It is called ROBOTS.txt.

Please check this
http://www.antezeta.com/blog/avoid-search-engine-indexing