Tagged: search engine optimization

Nov 04

Thank You Mr. Roboto: The Robots.txt File

Everyone seems to think that everything you put online is assessable by anyone at anytime via a simple search query on a search engine like Google or Bing. This is scary when it comes to personal or private professional information. So what do you do to keep something online ‘off limits’ to the unwanted? Ever hear of a robots.txt file? No? Well here is a bit of information on this amazing little innovation.

What Is It?

A robots.txt file is a very useful file that keeps the grubby fingers of search engines from getting a hold of specific information on your domain. When a search engine ‘spiders’ (methodically searches the internet for up-to-date information), a well placed robot.txt file will prevent that pesky little internet arachnid from seeing anything you don’t want it to see.

Consider it a spider proof “web”.

What Are Some Examples?

Here are some simple examples of ways to insert robot.txt files into your domain.

>If you’d like to keep those pesky spiders from even entering your domain you would write the information below into the robot.txt file:

User-agent: *

Disallow:  /

If you want to keep the search engine spiders out of a certain area or directory on your domain you would enter the following information into the robot.txt file:

User-agent:  *

Disallow:  /aboutme/ (or whichever directory you would like to exclude)

If you’d like to add more directories to the list, you would just add another line of:

Disallow:  /stats/

Disallow:  /account/

Etc…

All in all, it’s pretty simple and very useful.

What Are The Benefits?

If your business runs on creating spam files, emails and other unsolicited communications, creating a robot.txt file for that particular area of your domain will keep that information from spreading. Businesses all over the globe run using methods that aren’t always appreciated by the general public, and if the general public were to learn these classified things, these businesses might very well go under. Who wants that for their business?

A robot.txt file will keep the search engine spiders from crawling all over your business’ sensitive information.

What Are The Warnings?

The effectiveness of the robot.txt file depends solely on whether or not that particular search engine recognizes the ‘authority’ of the file. Basically, the robot.txt file is only as good as the search engine doing the crawling. Not every search engine in operation will heed the ‘off-limits” status present in your domain. Keep that in mind.

Overall, robot.txt files are incredibly useful in today’s world of wide open information. Keep what you want kept secret and away from the prying multi-faceted eyes of the search engine spider.

–

This post was written by Danica Sorber, owner of DSorber Media Works.

1
comments

Posted in Search Engines, SEO

Oct 04

Blog for Your Readers, Not For Google

A lot of Search Engine Marketing “Experts” will say that a blog is perfect for getting those much-needed keywords onto your website that will help you rank higher and better in search engine organic results. The problem with that is, search engines aren’t actually “reading” your content. They are simply scanning it. And eventually, when you have an actual real person that visits your website, they will read your “content” and realize what a hack you are. They will not pass your content on nor will they visit your website again. This means that all the effort and time you spent writing blog entires stuffed with keywords and anchor text that had nothing important to say may have gotten visitors to your website, but it didn’t keep them there. It didn’t help push people to spread what you have to say via Twitter, Facebook, or email.

When it comes down to it, writing thoughtful, well-written blog entries are the best way to attract steady, relevant users to your website. Sure, they may not have all the “desired” search terms or keywords that you’re hoping to rank for, but it WILL show visitors and potential customers your expertise and make them interested in what you have to say.

In many ways, it’s harder to write for the search engines than it is for real people. Trying to fit in the word “costume” when only “costumes” seems to make sense can be a real challenge and lead to awkward sentences that will repel readers. Writing out your natural thoughts about a subject and teaching others what you have to say seems to roll off the tongue (or off the fingers, in computer-speak) much easier than any well-structured search engine blog post ever could.

–

Image courtesy of Caitlinator on Flickr.

2
comments

Posted in Et cetera, Featured, Search Engines