22 users online (0 members and 22 guests)  

Thread: robots.txt


  Results 1 to 3 of 3

Related

  1. Help With Robots.txt File    Forum: HTML Forum
    Replies: 1
  2. robots.txt    Forum: Search Engine Optimization - SEO - Forum
    Replies: 9
  3. robots.txt    Forum: HTML Forum
    Replies: 0
  1. #1
    veronica's Avatar
    New User

    Status
    Offline
    Join Date
    Sep 2004
    Posts
    2

    robots.txt

    I have a site that is redirected to several domain names whereas no searcher can track them down as existant. They told me I should make a robots.txt file but all the explanations I find are to make it to tell the searcher which pages to crawl nothing to do with several domain names.
    Any help on this?

  2. #2
    dlongnecker's Avatar
    Tired Student...(Z_z)

    Status
    Offline
    Join Date
    Aug 2004
    Location
    Wichita, KS
    Posts
    86
    Veronica-

    The Web Robots Page at http://www.robotstxt.org/wc/robots.html can provide you with the information you seek. To successfully filter multiple domains, you'll need to place the robots.txt at the root of the domain for each site.

    The file would include two simple lines.

    HTML Code:
    User-agent: *
    Disallow: /
    Basically, that is saying, for * (all) user agents, disallow everything from the / (root) and below.

    You can also use META tags on your pages, etc. These examples are covered in the site I posted above.

    Hope this helps!

    -David

  3. #3
    Experticles's Avatar
    New User

    Status
    Offline
    Join Date
    Nov 2004
    Location
    California
    Posts
    3
    If I am understanding your problem correctly, you can also use the .htaccess file with 301 redirects to handle this.


    [Admin: Link Removed]
    Last edited by QuietDean; 11-27-2004 at 04:16 PM. Reason: Remove link



Tags for this Thread