Everyone seems to think that everything you put online is assessable by anyone at anytime via a simple search query on a search engine like Google or Bing. This is scary when it comes to personal or private professional information. So what do you do to keep something online ‘off limits’ to the unwanted? Ever hear of a robots.txt file? No? Well here is a bit of information on this amazing little innovation.

What Is It?

A robots.txt file is a very useful file that keeps the grubby fingers of search engines from getting a hold of specific information on your domain. When a search engine ‘spiders’ (methodically searches the internet for up-to-date information), a well placed robot.txt file will prevent that pesky little internet arachnid from seeing anything you don’t want it to see.

Consider it a spider proof “web”.

What Are Some Examples?

Here are some simple examples of ways to insert robot.txt files into your domain.

>If you’d like to keep those pesky spiders from even entering your domain you would write the information below into the robot.txt file:

User-agent: *

Disallow:  /

If you want to keep the search engine spiders out of a certain area or directory on your domain you would enter the following information into the robot.txt file:

User-agent:  *

Disallow:  /aboutme/ (or whichever directory you would like to exclude)

If you’d like to add more directories to the list, you would just add another line of:

Disallow:  /stats/

Disallow:  /account/

Etc…

All in all, it’s pretty simple and very useful.

What Are The Benefits?

If your business runs on creating spam files, emails and other unsolicited communications, creating a robot.txt file for that particular area of your domain will keep that information from spreading. Businesses all over the globe run using methods that aren’t always appreciated by the general public, and if the general public were to learn these classified things, these businesses might very well go under. Who wants that for their business?

A robot.txt file will keep the search engine spiders from crawling all over your business’ sensitive information.

What Are The Warnings?

The effectiveness of the robot.txt file depends solely on whether or not that particular search engine recognizes the ‘authority’ of the file. Basically, the robot.txt file is only as good as the search engine doing the crawling. Not every search engine in operation will heed the ‘off-limits” status present in your domain. Keep that in mind.

Overall, robot.txt files are incredibly useful in today’s world of wide open information. Keep what you want kept secret and away from the prying multi-faceted eyes of the search engine spider.

–

This post was written by Danica Sorber, owner of DSorber Media Works.