Search engine optimization

From Wikipedia, the free encyclopedia

"SEO" redirects here. For other uses, see SEO (disambiguation).

Search engine optimization (SEO) is a subset of search engine marketing, and deals with improving the number and/or quality of visitors to a web site from "natural" (aka "organic" or "algorithmic" search engine) listings. In effect, SEO is marketing to the machines to increase search engine relevance and ultimately web traffic, similar to foot traffic in retail advertising. The term SEO can also refer to "search engine optimizers", an industry of consultants who carry out optimization projects on behalf of clients.

Search engines display different kinds of listings on a search engine results page (SERP), including paid advertising in the form of pay per click advertisements and paid inclusion listings, as well as unpaid organic search results and keywords specific listings, such as news stories, definitions, map locations, and images. SEO is concerned with improving the number and position of a site's listings in the organic search results.

SEO strategies vary widely, in accordance with the specific site. Broadly speaking, SEO may be geared towards increasing either, or both, the total number and quality of visitors from Search Engines. The quality of a visitor can be measured by how often a visitor using a specific keyword leads to a desired conversion action, such as making a purchase or requesting further information.

Search engine optimization is available as a stand-alone service or as a part of a larger marketing campaign. Because SEO often requires making changes to the source code of a site, it is often most effective when incorporated into the initial development and design of a site, leading to the use of the term "Search Engine Friendly" to describe designs, menus, Content management systems and shopping carts that can be optimized easily and effectively.

A range of strategies and techniques are employed in SEO, including changes to a site's code (referred to as "on page factors") and getting links from other sites (referred to as "off page factors"). These techniques include two broad categories: techniques that search engines recommend as part of good design, and those techniques that search engines do not approve of and attempt to minimize the effect of, referred to as spamdexing. Some industry commentators classify these methods, and the practitioners who utilize them, as either "white hat SEO", or "black hat SEO".^[1] Other SEOs reject the black and white hat dichotomy as an over-simplification.

SEO, as a marketing strategy, can often generate a good return. However, as the search engines are not paid for the traffic they send from organic search, the algorithms used can and do change, and there are many problems that can cause Search Engine problems when crawling or ranking a site's pages, there are no guarantees of success, either in the short or long term. Due to this lack of guarantees and certainty, SEO is often compared to traditional Public Relations (PR), with PPC advertising closer to traditional advertising.

1 History
- 1.1 Early search engines
- 1.2 Development of more sophisticated ranking algorithms
2 The relationship between SEO and the search engines
3 Getting into search engines' databases
4 Blocking robots
5 "White hat" methods
6 "Black hat" methods
7 SEO and marketing
8 Legal issues
9 References
10 See also
11 External link

[edit] History

[edit] Early search engines

Webmasters and mobile content providers began optimizing sites for search engines in the mid-1990s, as the first search engines were cataloging the early Web.

Initially, all a webmaster needed to do was submit a site to the various engines which would run spiders, programs that "crawls" a page and stores the collected data in a database.

The process involves a search engine root or spider downloading a page and storing it on the search engines own server, where a second program, known as an indexer, extracts various information about the page, such as the words it contains and where these are located, as well as any weight for specific words, as well as any and all links the page contains, which are then placed into a scheduler for crawling at a later date.

At first, search engines were guided by the webmasters themselves. Early versions of search algorithms relied on webmaster-provided information such as the keyword meta tag, or index files in engines like ALIWEB. Meta-tags provided a guide to each page's content.

Inevitably, webmasters began to abuse meta tags, causing their pages to rank for irrelevant searches. To combat this, Search engines developed more complex ranking algorithms, taking into account additional factors including:

Text within the title tag
Domain name
URL directories and file names
HTML tags: headings, emphasized (<em>) and strongly emphasized (<strong>) text
Term frequency, both in the document and globally, often misunderstood and mistakenly referred to as Keyword density
Keyword proximity
Keyword adjacency
Keyword sequence
Alt attributes for images
Text within NOFRAMES tags
Content development

A number of attributes within the HTML source of a page were often manipulated by web content providers attempting to rank well in search engines.^[2]

By relying extensively on factors that were still within the webmasters' exclusive control, search engines continued to suffer from abuse and ranking manipulation. In order to provide better results to their users, search engines had to adapt to ensure their SERPs showed the most relevant search results, rather than useless pages stuffed with numerous keywords by unscrupulous webmasters using a bait-and-switch lure to display unrelated web pages. This led to the rise of a new kind of search engine.

[edit] Development of more sophisticated ranking algorithms

Google was started by two PhD students at Stanford University, Sergey Brin and Larry Page, and brought a new concept to evaluating web pages. This concept, called PageRank, has been important to the Google algorithm from the start.^[3] PageRank is an algorithm that weights a page's importance based upon the incoming links. PageRank estimates the likelihood that a given page will be reached by a web user who randomly surfed the web, and followed links from one page to another. In effect, this means that some links are more valuable than others, as a higher PageRank page is more likely to be reached by the random surfer.

The PageRank algorithm proved very effective, and Google began to be perceived as serving the most relevant search results. On the back of strong word of mouth from programmers, Google quickly became the most popular and successful search engine. PageRank measured an off-site factor, Google felt it would be more difficult to manipulate than on-page factors.

Despite being difficult to game, webmasters had already developed link building tools and schemes to influence the Inktomi search engine, and these methods proved similarly applicable to gaming PageRank. Many sites focused on exchanging, buying, and selling links, often on a massive scale. This has spawned an online industry, that survives to this day, focused upon selling links designed to improve PageRank and link popularity, and not to drive human site visitors, with links from higher PageRank pages selling for the most money.

A proxy for the PageRank metric is still displayed in the Google Toolbar, though the displayed value is rounded to be an integer, and the toolbar is believed to be updated less frequently and independently of the value used internally by Google. These days, Google claim that PageRank is only one of more than 100 "signals" considered in ranking pages, and many experienced SEOs recommend ignoring the displayed PageRank^[4].

Google — and other search engines — have, over the years, developed a wider range of off-site factors they use in their algorithms. The Internet was reaching a vast population of non-technical users who were often unable to use advanced querying techniques to reach the information they were seeking and the sheer volume and complexity of the indexed data was vastly different from that of the early days. Combined with increases in processing power, search engines have begun to develop predictive, semantic, linguistic and heuristic algorithms. Around the same time as the work that led to Google, IBM had begun work on the Clever Project ^[5], and Jon Kleinberg was developing the HITS algorithm.

As a search engine may use hundreds of factors in ranking the listings on its SERPs; the factors themselves and the weight each carries can change continually, and algorithms can differ widely, with a web page that ranks #1 in a particular search engine possibly ranking #200 in another search engine, or even on the same search engine a few days later.

Google, Yahoo, Microsoft and Ask.com do not disclose the algorithms they use to rank pages. Some SEOs have carried out controlled experiments to gauge the effects of different approaches to search optimization. Based on these experiments, often shared through online forums and blogs, professional SEOs attempt to form a consensus on what methods work best, although consensus is rarely, if ever, actually reached.

SEOs widely agree that the signals that influence a page's rankings include:^[6]^[7]

Keywords in the title tag.
Keywords in links pointing to the page.
Keywords appearing in visible text.
Link popularity (PageRank for Google) of the page.

There are many other signals that may affect a page's ranking, indicated in a number of patents held by various search engines, such as historical data^[8].

[edit] The relationship between SEO and the search engines

The first mentions of Search Engine Optimization do not appear on Usenet until 1997, a few years after the launch of the first Internet search engines. The operators of search engines recognized quickly that some people from the webmaster community were making efforts to rank well in their search engines, and even manipulating the page rankings in search results. In some early search engines, such as Infoseek, ranking first was as easy as grabbing the source code of the top-ranked page, placing it on your website, and submitting a URL to instantly index and rank that page.

Due to the high value and targeting of search results, there is potential for an adversarial relationship between search engines and SEOs. In 2005, an annual conference named AirWeb^[9] was created to discuss bridging the gap and minimizing the sometimes damaging effects of aggressive web content providers.

Some more aggressive site owners and SEOs generate automated sites or employ techniques that eventually get domains banned from the search engines. Many search engine optimization companies, which sell services, employ long-term, low-risk strategies, and most SEO firms that do employ high-risk strategies do so on their own affiliate, lead-generation, or content sites, instead of risking client websites.

Some SEO companies employ aggressive techniques that get their client websites banned from the search results. The Wall Street Journal profiled a company that allegedly used high-risk techniques and failed to disclose those risks to its clients.^[10] Wired reported the same company sued a blogger for mentioning that they were banned.^[11] Google's Matt Cutts later confirmed that Google did in fact ban Traffic Power and some of its clients.^[12]

Some search engines have also reached out to the SEO industry, and are frequent sponsors and guests at SEO conferences and seminars. In fact, with the advent of paid inclusion, some search engines now have a vested interest in the health of the optimization community. All of the main search engines provide information/guidelines to help with site optimization: Google's, Yahoo!'s, MSN's and Ask.com's. Google has a Sitemaps program ^[13] to help webmasters learn if Google is having any problems indexing their website and also provides data on Google traffic to the website. Yahoo! has Site Explorer that provides a way to submit your URLs for free (like MSN/Google), determine how many pages are in the Yahoo! index and drill down on inlinks to deep pages. Yahoo! has an Ambassador Program^[14] and Google has a program for qualifying Google Advertising Professionals^[15].

[edit] Getting into search engines' databases

Today's major search engines, by and large, do not require any extra effort to submit to, as they are capable of finding pages via links on other sites.

However, Google and Yahoo offer submission programs, such as Google Sitemaps, for which an XML type feed can be created and submitted. Generally, however, a simple link from a site already indexed will get the search engines to visit a new site and begin spidering its contents. It can take a few days or even weeks from the acquisition of a link from such a site for all the main search engine spiders to begin indexing a new site, and there is usually not much than can be done to speed up this process.

Once the search engine finds a new site, it uses a crawler program to retrieve and index the pages on the site. Pages can only be found when linked to with visible hyperlinks. For instance, some search engines do not read links created by Flash or Javascript.

Search engine crawlers may look at a number of different factors when crawling a site, and many pages from a site may not be indexed by the search engines until they gain more PageRank, links or traffic. Distance of pages from the root directory of a site may also be a factor in whether or not pages get crawled, as well as other importance metrics. Cho et al.^[16] described some standards for those decisions as to which pages are visited and sent by a crawler to be included in a search engine's index.

A few search engines, such a Yahoo!, operate paid submission services that guarantee crawling for either a set fee or CPC. Such programs usually guarantee inclusion in the database, but does not guarantee specific ranking within the search results.

[edit] Blocking robots

Webmasters can instruct spiders not to crawl certain files or directories through the standard robots.txt file in the root directory of the domain. Additionally, a page can be explicitly excluded from a search engine's database by using a robots meta tag.

When a search engine visits a site, the robots.txt located in the root folder is the first file crawled. The robots.txt file is then parsed, and only pages not disallowed will be crawled. As a search engine crawler may keep a cached copy of this file, it may on occasion crawl pages a webmaster does not wished crawled.

Pages typically prevented from being crawled include login specific pages such as shopping carts and user-specific content such as search results from internal searches.

[edit] "White hat" methods

There are a range of ways a website can be coded, or the server they reside on set up, that do not allow search engines access to either the pages on the site or the content they contain. White hat SEOs attempt to discover and correct these mistakes, such as JavaScript content delivery and navigation, URL sessions and content hidden in flash, as well as increase and improve the content on the site.

An SEO tactic, technique or method is considered "White hat" if it conforms to the search engines' guidelines and/or involves no deception. As the search engine guidelines^[17]^[18]^[19]^[20]^[21] are not written as a series of rules or commandments, this is an important distinction to note. White Hat SEO is not just about following guidelines, but is about ensuring that the content a search engine indexes and subsequently ranks is the same content a user will see.

White Hat advice is generally summed up as creating content for users, not for search engines, and then make that content easily accessible to their spiders, rather than game the system. In many ways, white hat SEO is very similar to web development that promotes accessibility^[22], although the two are not identical.

Some SEO methods generally accepted by search engines: ^[23]

Using unique and relevant titles (<title></title>) for each page.
Editing web page content to replace vague wording with specific terminology relevant to the subject of the page.
Providing unique, quality content to address visitor interests.
Using an accurate description meta tag to make search listings more informative.
Ensuring that all pages are accessible via anchor tag hyperlinks.
Allowing search engine spiders to crawl pages without session IDs, cookies, or logging in.
Developing "link bait" strategies. High quality websites that offer interesting content or novel features tend to accumulate large numbers of back links.
Writing useful, informational articles in exchange for attribution to the author by hyperlink.

[edit] "Black hat" methods

Main article: Spamdexing

"Black hat" SEO are methods to try to improve rankings that are disapproved of by the search engines and/or involve deception. This can range from text that is "hidden", either as text colored similar to the background or in an invisible or left of visible div, or by redirecting users from a page that is built for search engines to one that is more human friendly. As a general rule, a method that sends a user to a page that was different to the page the search engined ranked is Black hat. One well known example is Cloaking, the practice of serving one version of a page to search engine spiders/bots and another version to human visitors.

Search engines can and do penalize sites they discover using black hat methods, either by reducing their rankings or eliminating their listings from their databases altogether. Such penalties can be applied either automatically by the search engines' algorithms, or by a manual review of a site.

One infamous example was the February 2006 Google removal of both BMW Germany and Ricoh Germany for use of deceptive practices.^[24]

[edit] SEO and marketing

There is a considerable sized body of practitioners of SEO who see search engines as just another visitor to a site, and try to make the site as accessible to those visitors as to any other who would come to the pages. They often see the white hat/black hat dichotomy mentioned above as a false dilemma. The focus of their work is not primarily to rank the highest for certain terms in search engines, but rather to help site owners fulfill the business objectives of their sites. Indeed, ranking well for a few terms among the many possibilities does not guarantee more sales. A successful Internet marketing campaign may drive organic search results to pages, but it also may involve the use of paid advertising on search engines and other pages, building high quality web pages to engage and persuade, addressing technical issues that may keep search engines from crawling and indexing those sites, setting up analytics programs to enable site owners to measure their successes, and making sites accessible and usable.

SEOs may work in-house for an organization, or as consultants, and search engine optimization may be only part of their daily functions. Often their education of how search engines function comes from interacting and discussing the topics on forums, through blogs, at popular conferences and seminars, and by experimentation on their own sites. There are few college courses that cover online marketing from an ecommerce perspective that can keep up with the changes that the web sees on a daily basis.

While endeavoring to meet the guidelines posted by search engines can help build a solid foundation for success on the web, such efforts are only a start. Many see search engine marketing as a larger umbrella under which search engine optimization fits, but it's possible that many who focused primarily on SEO in the past are incorporating more and more marketing ideas into their efforts, including public relations strategy and implementation, online display media buying, web site transition SEO, web trends data analysis, HTML E-mail campaigns, and business blog consulting making SEO firms more like an ad agency.

[edit] Legal issues

In 2002, SearchKing filed suit in an Oklahoma court against the search engine Google. SearchKing's claim was that Google's tactics to prevent spamdexing constituted an unfair business practice. This may be compared to lawsuits that email spammers have filed against spam-fighters, as in various cases against MAPS and other DNSBLs. In January of 2003, the court pronounced a summary judgment in Google's favor. ^[25] In March 2006, KinderStart.com, LLC filed a first amended complaint against Google and also attempted to include potential members of the class of plaintiffs in a class action. ^[26] The plaintiff's web site was removed from Google's index prior to the lawsuit and the amount of traffic to the site plummeted.

[edit] References

^ Goodman, Andrew, SearchEngineWatch, Search Engine Showdown: Black Hats vs. White Hats at SES
^ What is a tall poppy among web pages?, Proceedings of the seventh conference on World Wide Web, Brisbane, Australia, 1998, written by Pringle, G., Allison, L., and Dowe, D.
^ Brin, Sergey and Page, Larry, The Anatomy of a Large-Scale Hypertextual Web Search Engine, Proceedings of the seventh international conference on World Wide Web 7, 1998, Pages: 107-117
^ WebmasterWorld.com- search engine optimization forum
^ The Clever Project, May 4, 2006
^ Search Engine Watch - Search Engine News and Forums. Organizer of SES (Search Engine Strategies) Conferences.
^ Highrankings Forum - forum for experienced SEOs and those new to the field
^ Information Retrieval Based on Historical Data, Google Patent Application, October 10, 2005
^ AirWeb Adversarial Information Retrieval on the Web, annual conference and workshop for researchers and professionals
^ Startup Journal (Wall Street Journal), 'Optimize' Rankings At Your Own Risk by David Kesmodel at The Wall Street Journal Online, September 9 2005
^ Wired Magazine, Legal Showdown in Search Fracas, Sep, 08, 2005, written by Adam L. Penenberg
^ Cutts, Matt, Confirming a penalty, published on 2006-02-02 at Matt Cuts Blog
^ Google Web Master Central, formerly known as Google Sitemaps
^ Ambassador Program by Yahoo! Search Marketing
^ Google Advertising Professionals, a Program by Google AdWords, Google's Pay-Per-Click Advertising Service
^ Efficient crawling through URL ordering by Cho, J., Garcia-Molina, H. , 1998, published at "Proceedings of the seventh conference on World Wide Web", Brisbane, Australia
^ Ask.com Editorial Guidelines
^ Google's Guidelines on SEOs
^ Google's Guidelines on Site Design
^ MSN Search Guidelines for successful indexing
^ Yahoo! Search Content Quality Guidelines
^ Andy Hagans, A List Apart, High Accessibility Is Effective Search Engine Optimization
^ Whalen, Jill, HighRankings Forum, Ten Tips to the Top of the Search Engines
^ Ramping up on international webspam by Matt Cutts, published February 4, 2006, at Matt Cutts Blog
^ [1] Google replies to SearchKing lawsuit, James Grimmelmann at LawMeme (research.yale.edu), January 09. 2006
^ [2] (PDF) KinderStart.com, LLC, et al v. Google, Inc., C 06-2057 RS, filed March 17, 2006 in the Northern District of California, San Jose Division.