Struggling with Search Engines

While getting registered with a search engine is fairly simple, getting good placement in the search engine is another matter. The more specific a search word or phase is to your web site, the better the chance it will appear close to the top. The more general the term, the less likely your page will be listed. This is why, when using your specific company name in quotes (a very specific phrase), your listing will usually appear at the top of the search results list, whereas general phrases like "design services" or (in our case) "web developer" aren't likely to list our site. There are just too many pages out there that we're competing with for top placement.

So how do the search engines determine which sites go to the top, and which sites are buried hundreds or thousands of pages down? The goal of any search engine is to provide the most relevant search results based on search terms. The more relevant the results, the more useful the search engine is as a tool and, by extension, the more people will use the search engine (and the more money they will make through advertising). Search engines use many different formulas for determining the relevance of any given site in their database for specific queries. Matching a keyword on a page with a keyword in a query is one of the most basic ways of determining relevance, but there are many others as well. The number of times the word is found on the page, whether the word is in the title tag as well as the body copy is another. The size in which the word is displayed is another variable in determining relevance, for example if the word is big, like in a heading tag, then the search engine would assume that it must be an important word and therefore give it more weight in its calculation of relevance.

There are things you can control on your site that will help maximize the relevance of your pages to a search engine's results. However, there are also things that are out of your control.

You can control the text on your web pages, which is a major factor in determining relevance. There are three main areas of a web page that need to be considered for appropriate text: the title tag, meta tags, and the actual page content.

Title tags - The title tag is the copy that defines the title of the browser window that displays the page. Whenever you look at a web page, the very top bar of the browser (the bar where you would click to open or close the window) displays the page's title. The title tag is often overlooked because it doesn't display on the page itself, but rather in the top bar. For this reason, some sites forget to put any content in their title tags. But as you can imagine, the title of a page is weighted fairly heavily in terms of relevance. Placing appropriate keywords or phrases in your title tags can help improve search engine placement.

Meta tags - Meta tags are special tags that allow you to add keywords and descriptions to a page that do not get displayed directly to a visitor. Again, because they are not visible, they are often overlooked when producing a web site. You can check to see if you have meta tags by viewing the source on your web page. They tags will look something like this...

<META name="description" content="Newfangled is a company...">

<META name="keywords" content="CMS, web design, developers, website, newfangled,>

Because these tags do not display on a site, it is easy for some sites to "spam" their meta tags. In other words they might put their keywords in the meta tags one hundred times in an attempt to increase their relevance. Or they might use popular words that do not have anything to do with their site. Because of this some search engines ignore meta tags altogether. However, for the search engines that do consider meta tags, they can provide another opportunity to increase the relevance of your site in the search results. A good rule of thumb for meta tags is to be conservative and strategic in terms of the number if keywords you add. There is no physical limit, but the more words you add, the less each one is weighted. The more you add, the more they "water down" the effectiveness of all of the keywords. The description meta tag should be about 25 words long and simply state a general description of the page.

Website copy - The most important keywords on your site are the words on the page itself, your site's content. While we wouldn't recommend writing the copy on your site for the purposes of getting a better position in a search engine, you might consider using a few less pronouns in cases where their use is not necessary. For example, we might write, "Newfangled Web Factory uses grayscreen prototyping and the NewfangledCMS," instead of, "we use grayscreen prototyping and a CMS," as long as it doesn't make the copy too redundant or awkward. While good writing and appropriate grammar should always determine the style of your copy, keeping in mind how search engines work can help improve your positioning.

Placing the appropriate information in each of the areas of your web pages will help you achieve better relevancy in search engine results. Unfortunately, even with all of the appropriate keywords in the right places, there are still other factors that determine relevancy. Many of these factors are out of your control. For example, some search engines will look at how many other sites link to yours as a factor in determining relevancy. The more sites that link to you, the more "popular" your site is considered, and therefore its relevancy is increased. While requesting links on other sites is always a good idea, but you cannot control whether or not another site will link to you.

There are other factors in relevancy such as how frequently site content changes, how far down the page a keyword is found, and others. Keeping track of how each search engine works can be a monumental task. This is why we generally recommend simply making sure that your site includes all the appropriate elements and that you register it with all the search engines. Becoming more aggressive with placement requires the help of a search engine optimization specialist.

Indexes - There are three main types of search engines, Indexes, Directories, and Pay for View. Indexes are the most typical kind of search engine and most of this newsletter has been focused on placement within indexes. Indexing search engines use software to scour the web for links and content and "index" all of the found words and their associated web locations, assigning them rankings according to relevancy. They are very automated and simply input data into their vast databases and use the relevancy rules they create to determine how to list results from search queries. "Google," and "AltaVista," are examples of search engine indexes.

Directories - There are also some "search engines" that are not actually search engines at all. In fact, one of the most popular search engines, "Yahoo!" started out as a directory. A directory does not automatically add links to a database. Instead it relies on human editors to look at sites and determine where they belong in the directory and place links to the sites in their database. In the past, when using Yahoo! to search the web, you actually only searched the various categories, sub categories and sites that Yahoo! had determined to be appropriate for cataloging. While today's directories like dmoz.org as still helpful, they are most helpful when searching within niche interests.

Pay for view - Some search engines require companies to pay for each visitor that they send to their site, allowing them to purchase the top placement for a specific keyword. For example, a company might bid 25 cents per click to appear at the top of the results listing for the keyword "cars." However, if someone else comes along and bids 26 cents per click for the same keyword, then they will be listed above the first company. We haven't ever used these search engines for our clients, but they can drive qualified traffic to a site. Often these search engines can generate excellent search results because they filter out inappropriate sites by requiring payment for placement. Overture.com (now part of Yahoo!) was an example of a pay for view site. A search on the site, listed only companies that paid to be there - they even displayed the amount of money that companies were paying to have you click through to their site. Today, sponsored links like Google AdWords have replaced most pay for view sites.

If you would like more information about how search engines work we highly recommend the site www.searchenginewatch.com. There is a ton of free information on this site and even more detailed information if you subscribe.

The NewfangledCMS is a database driven system with "search engine friendly" URLs. In addition to this we have set up Meta tabs in each page's article definition to allow you to easily add and maintain Meta keywords and description on each site page. This helps to keep your keywords targeted and specific to your content and improve your listings and position in search engines. We also have advanced search engine optimization tools and tracking reports for the NewfangledCMS.