Friday, November 14, 2008

Getting Found By Spiders, Bots and Crawlers

"Hey, I'm back here - on page 63!"

You've got a good site, you're selling a good product or providing an important service and you still show up at the bottom of page 63 of the SERP (search engine results page). You've got a problem - a recognition problem. Your best, potential customers or consumers don't even know your cyber hut exists. This is not a good problem, but a problem that can be solved, or at least mitigated.

You need to be found, or your site does, actually. So how, exactly, do search engines find your little bit of digital real estate. Well, there are lots of ways but let's start with the most common, and coincidentally, the least expensive - as in free!

Search Engines - Powered Up For Success
There are hundreds (yes, hundreds) of search engines on the www. Some, like Google and Alta Vista are free. You submit the appropriate information and your site will be listed for free. Of course, your site will be buried on page 63 of the SERP, but everyone has to start somewhere.

Then, there are PPC sites - pay-per-click sites that nick you for every click they generate leading a potential customer to your cyber store. Yahoo is the best known of the PPC sites. Yahoo (in its usual fashion) has set off on its own path to search engine development, breaking off a once-promising partnership between Google and Yahoo - a good thing for us. The more competition among SEs, the better the quality of search results.

Directories - Get Proactive; Get Noticed
As a newcomer to cyber town, you should also consider registering your site in directories. There are open directories, like the aptly named Open Directory Project (DMOZ), in which volunteer editors search the www for interesting, helpful sites and then catalog each site for inclusion under one of many subject headings.

Then, there are smaller, more targeted directories where you can list your related site. For example, there are directories for charitable giving sites, sites of legal firms and so on. In fact, there are directory sites for just about any business or service you can imagine.

There are also association sites with outbound links, community directories, interest groups, political groups - many specialized directories that will list you for free - as long as your site is somehow related to the directory. If you're selling the better mousetrap, it's a sure bet your site won't be accepted on a national jewelers' directory. There has to be some connection between you and the subject of these topic-driven directories.

When you register with any of these SEs (free or PPC) and directories, in addition to the usual contact information, you'll also provide a list of key words - words and phrases that a potential customer might enter in conducting a search for a site just like yours. Your selection of key words is critical to the success of your e-biz. Enter inappropriate or off-target key words and you're not going to get the visitor traffic you'd get if every key word were dead on the money. In fact, there are a variety of assessment tools top site design firms employ to help you develop a tight, targeted key word list for submission to search engines and directories.

Key words often have to be adapted to specific SEs. You might use one set of key words for a search engine like Google, which covers the entire universe, and a completely different set of key words for an industry directory where most searchers already know the technical terms, model numbers, etc. of what they're looking for on a site like yours.

Let's say it one more time: It is essential to the success of your on-line business to develop a list of on-the-money key words to make sure your site is, well, on the money.

Spiders and Creepy Crawlers
Submitting your site to search engines and directories is a great way to start building site visibility. Moreover, they're all proactive steps - steps you can take on your own.

However, there's another way to increase your site presence on the www - and that's to get spidered by a crawler, which sounds pretty bad but, in fact, is pretty good. A search
engine 'crawler' - software that crawls (and 'reads') pages of various sites, stores the content data on your site and reports it back to the mother ship - the search engine index. Once this happens, restrained congratulations are in order. You've been discovered - maybe.

You see, a crawler (or spider) has one mission - to gather the content data on millions of sites and send that info back to the SEs index- a massive amount of stored data. Google maintains a +1 billion-page index and it grows every day. So, now you type in a search word at Google, Google scans the 1 billion pages in its database and delivers the SERP for your review. All pretty straightforward, right?

Wrong! Each search engine uses its own, 'Eyes Only' formula to assign weight to various search criteria, which in turn, determines your placement within the SERP. And if you aren't on the first page or two, you might as well be working out of Mongolia. Typical users don't look beyond the first two pages. It's a busy world.

That's where spidering comes in. You see, spiders don't just crawl the web randomly like, well, like spiders. No, this is smart software we're talking about here. Spiders follow links from one site to the next, based on the assumption that the links are, somehow relevant to the site from whence the link originated. If you're selling snowshoes, for example, and you have an outbound link to a local ski resort, the spider will follow the link and crawl that site.

That's why links - quality links - are so important. Why? Well, if you're selling snowshoes and the spider follows a link to Mom and Pop's Ye Olde On-Line Candy Shoppe, oops, that's a garbage link - a link that isn't helpful to your snowshoe customers. The result? You lose points in that search engines PR calculations.

There are sites, affectionately referred to as 'link farms' that are nothing more than sites providing in- and out-bound links. Spiders hate link farms because they diminish the quality of search results. The newest search engine algorithms place a much higher value on quality links - especially quality links pointing to your site.

If people in your area of commerce believe that your site would be helpful to their customers, the SE is going to rank you higher, based on the assumption that quality sites link to other quality sites.

An SE spider can do wonders for your PR (PageRank), moving you up in the SERP rankings from page 63 all the way up to page 6. Of course, getting spidered in and identified as an 'expert page" (one other sites send their visitors to) will do wonders for your ranking.

But you can't stop there. It's a dog-eat-dog cyber world in which we exist. That's why so many other factors enter into getting found and moving on up. Search engine optimized (SEO) text that's spider-friendly plays a key roll. The quality of content is another important factor. In fact, there's a checklist of do's and don'ts that form the basis of search engine marketing (SEM). So, before you build a single pixel, or reserve a single gig of space on your host's server, put together a business plan that includes SEM in the budget.

Why? Because if you spend all of your initial capital on building a good looking, visitor-friendly site, you won't have the necessary cash to hire an SEM professional to deliver the traffic that will make your site profitable.

No comments: