Thanks Thanks:  0
Showing results 1 to 2 of 2

Thread: MSNBot Spider and Yahoo! Slurp Spider

  1. 2010-04-17, 03:58 AM #1
    zeuscane
    zeuscane is offline
    Super Moderator Reputation: 8541 zeuscane's Avatar
    Join Date
    2008-09-06
    Location
    In the world and sometimes on the moon
    Posts
    2,917

    Default MSNBot Spider and Yahoo! Slurp Spider

    Dear administrator,
    I have discovered, with who's Online menu that on the forum as guests enter kind MSNBot Spider and Yahoo! Slurp Spider.
    MSNBot Spider is automatic web robot that follows links throughout the internet internally though websites and from site to site.
    The information it gathers makes a 'map' of the net for the search engines and also determines how high a website will be ranked... so
    the MSNbot will be gathering info for the MSN search engine and GoogleBot and YahooSpider will be gathiering info for Google and Yahoo, respectively.
    What us we can do for stopping them?
    Regards
    zeuscane
    "Educate yourselves because we'll need all your intelligence.
    Stir yourselves because we'll need all your enthusiasm.
    Organize yourselves because we'll need all your strength."
  2. 2010-04-17 03:58 AM # ADS
    Finetopix
    Circuit advertisement
    Join Date
    Always
    Location
    Advertising world
    Posts
    Many
     
  3. 2010-04-17, 04:31 AM #2
    blaster949
    blaster949 is offline
    Senior Member Reputation: 740
    Join Date
    2009-11-20
    Posts
    341

    Default

    Hi Zuescane
    I believe you can add a text file in the root directory of the hosted site, so the bot will read it and STOP going further. It is called ROBOTS.txt.

    Please check this
    http://www.antezeta.com/blog/avoid-s...ngine-indexing
« Previous Thread | Next Thread »

Bookmarks

Bookmarks

Posting Rules

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

Forum Rules