« Cheap Domain
What is the best method of marketing Web Hosting Packages? »

How to prevent direct viewing of the robots.txt file?

Typing any www.domainname.com/robots.txt shows the contents of this file. How can I prevent people from viewing this while still allowing search engines access? I know htaccess and htpasswd files can’t be viewed in plain site like this.

You can’t. Not properly anyway.

You can try blocking individual WEB BROWSERS based on the agent string, but it’s not foolproof:

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} ^First_agent [OR]
RewriteCond %{HTTP_USER_AGENT} ^Another_agent
RewriteRule ^(.*)$ http://yahoo.com

Here’s a list of all agent strings (search engines, browsers, bots):
http://www.useragentstring.com/pages/useragentstring.php

Blocking with htaccess:
http://www.clockwatchers.com/robots_bad.html

This entry was posted on Sunday, September 13th, 2009 at 3:27 pm and is filed under domainname. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

One Response to “How to prevent direct viewing of the robots.txt file?”

  1. Jim J Says:
    September 13th, 2009 at 8:53 pm

    You can’t. Not properly anyway.

    You can try blocking individual WEB BROWSERS based on the agent string, but it’s not foolproof:

    RewriteEngine on
    RewriteCond %{HTTP_USER_AGENT} ^First_agent [OR]
    RewriteCond %{HTTP_USER_AGENT} ^Another_agent
    RewriteRule ^(.*)$ http://yahoo.com

    Here’s a list of all agent strings (search engines, browsers, bots):
    http://www.useragentstring.com/pages/useragentstring.php

    Blocking with htaccess:
    http://www.clockwatchers.com/robots_bad.html
    References :

Leave a Reply