Robots.txt ?????

[gvardy]

Hi guys,

A topic for discussion Robots.txt is this something we need to have to enable a spider to crawl our site.

If so how do we insert it into our ezimerchant site?

Gary

http://www.hobbytools.com.au/

[dferguson]

Hi Gary

The DEFAULT behaviour for search engines is to INDEX a site fully if no robots.txt file is found. A robots.txt file is only required IF you want to omit somthing from the search engine index.

[gvardy]

Hi Gary

The DEFAULT behaviour for search engines is to INDEX a site fully if no robots.txt file is found. A robots.txt file is only required IF you want to omit somthing from the search engine index.
Thanks david. Just on that though when i check my stats I see that the robots.txt comes up as a code 404 - It seems that the robot or spider was looking for this. Certains spiders looked for this didn't find it and then left the site without spidering.

[origem]

Hi

Before a robot crawls your site it looks for the file robots.txt to check for instructions, if you do not whant to recieve the fault code 404 you need to upload a blank file named robots.txt along with the rest of your files.

Regards David

[gvardy]

Hi

Before a robot crawls your site it looks for the file robots.txt to check for instructions, if you do not whant to recieve the fault code 404 you need to upload a blank file named robots.txt along with the rest of your files.

Regards David
Thankyou, this may seem a stupid question but how do I do that?

[dferguson]

in your buildxxxx folder, create a normal text file.. then rename it to robots.txt

next time ezimerchant uploads your site, it will upload the file.

[gvardy]

in your buildxxxx folder, create a normal text file.. then rename it to robots.txt

next time ezimerchant uploads your site, it will upload the file.
Thanks David, you're a champion.