Robots

There has been a discussion elsewhere on the ring about search engines spidering blogs.

For instructions on how to block search engine bots This Page covers reasonably clearly how to do it.

But if you simply want to block all bots all you need to do is create a text file containing the following two lines:

User-agent: *
Disallow: /

Save it with the name robots.txt

Upload it to the root of your site so that it sits alongside your home page.

This entry was written by Tim, posted on Thu Aug 26 2004 at 18:03, filed under Uncategorized. Bookmark the permalink. Follow any comments here with the RSS feed for this post. Both comments and trackbacks are currently closed.

11 Comments

  1. Merry
    Posted Thu Aug 26 2004 at 18:19 | Permalink

    Gosh, you’re good. :~)

  2. jax
    Posted Thu Aug 26 2004 at 18:37 | Permalink

    Harumph, using my blog to show off! Doesn’t he know that’s my job here? ;-)

  3. Barbara
    Posted Thu Aug 26 2004 at 19:01 | Permalink

    Totally didn’t understand your comment, Jax, because I assumed you had posted this post. I would have just kept quite ;-)

  4. Merry
    Posted Thu Aug 26 2004 at 19:15 | Permalink

    kept quite what? ;~)

  5. Barbara
    Posted Thu Aug 26 2004 at 19:21 | Permalink

    :-P~~~~~~~~~~~~~~~~~~~

  6. Jonathan
    Posted Thu Aug 26 2004 at 20:05 | Permalink

    What about just blocking the blog section of Barbara’s site. An eg for that would be good too :)

  7. Tim
    Posted Thu Aug 26 2004 at 20:10 | Permalink

    I supplied the url, so if people want more complicated rules, they can find out how to do them easily enoug, by reading the examples there. And they can’t blame me if they end up blocking off all their traffic. :-)

  8. Merry
    Posted Thu Aug 26 2004 at 20:40 | Permalink

    rofl - you always say those sites are straight forward and when i go look they always appear to be in ancient greek!

  9. Sarah, Roo and Kids
    Posted Thu Aug 26 2004 at 21:12 | Permalink

    surely that should be ancient GEEK?

  10. Tim
    Posted Fri Aug 27 2004 at 0:21 | Permalink

    B***dy charming. Who are you calling ancient!!!

  11. Merry
    Posted Sat Aug 28 2004 at 10:21 | Permalink

    giggles away loopily!