2/12/2008

MSN Beta's Search Results vs. Yahoo! & Google

Microsoft's beta release of their new search engine prompted several fast reviews on the web by diverse organizations, from the BBC Onlineexternal link, to Search Engine Journalexternal link. The quick glances at the search engine unfortunately do not take a detailed, comprehensive look at the differences between the results returned by the engines and the quality of the content.

The analysis I have completed below indicates that the search wars are wide open. Google currently provides the highest quality content and best matches to the search query in its results, but is only slightly ahead of Yahoo!'s search engine. MSN's beta release trails Yahoo! by a small amount in the numbers, but an individual glance at the results reveals much work is needed to play catchup. From the results, it also appears that both Yahoo! and MSN are sorely lacking the index size that Google has. This competitive advantage may be the difference in Google's ability to provide more relevant results.

A look at the PageRank & measurable backlinks to the sites reveals similar trends between the 3, but Google's massively larger index and ability to crawl more web pages more frequently may be the key to their current success. Yahoo! and MSN still have only a fraction of the pages in cache that Google does. A rough guess based on the results would be that Yahoo!'s index is approximately 1/2 the size of Google's, while Microsoft's is probably less than 1/4 Google's size.

The measures of content quality and match quality were made as objectively as possible and show that while Google on average provides the best results, there is still a very long way to go towards providing an excellent search engine for the web. The search term "HDTV Comparison", for example, produced not 1 result among 30 (the top 10 results at each search engine) that I considered excellent. Searches for "George Plimpton" & "Order Digital Prints Online" produced very satisfactory results at Yahoo! & Google, with Yahoo! actually providing slightly better results overall. Google, meanwhile, was very dominant in results for "US National Holidays", providing many excellent resources that the competition did not have.

Spamming each of the search engines is still surprisingly easy. Many results at all 3 search engines were clearly little more than traffic grabbing pages with no real, unique content. The search engine that deals with this issue most effectively will gain a significant advantage for the future, as sites & pages like these multiply on the web into the millions.

I was initially disappointed in MSN's poor results, but it appears that they are not far off the results of Yahoo! & Google. While their beta release can not be called a superlative effort, it clearly shows the potential to provide competition to the other industry players. The key to short-term success for both Yahoo! & MSN will be finding a way to build their index fast enough to compete with Google's giant head-start.

I expect to add further analysis that reflects the discussion on SEO forums about these results.

SEO Analysis

For an SEO professional, the results below indicate that Google's trend of favoring "authority" sites as defined by the hilltop algorithm continues unabated. With the exception of several odd standouts, Google, Yahoo! & MSN all give great credence to sites that have many thousands of backlinks, a high PageRank & a large number of pages on the site. The era of small, niche sites is falling to the wayside as the search engines look for the dominance of major web presences like news organizations, government websites & large commercial endeavors.

For MSN specifically, the results are very hard to pick apart. It appears that MSN may be using some additional pieces of ranking technology in their algorithm, possibly further questioning the value of non-relevant links. However, the current size of their index in comparison with Yahoo! & Google makes it exceptionally difficult to determine which pieces they consider important and which pieces are simply left out because they have not yet been spidered. The only piece which stuck out in my analysis, although it was not included in my statistics, is that MSN does not appear to have Google's marked preference for older sites.

For additional information see Google vs. Yahoo! Results.

I welcome additional analysis from anyone who wishes to use the results. Please reference this page if you copy or use the information contained herein.

Survey Data & Key

This survey comprises four relatively common searches at the three major search engines, including MSN's new Beta search engine release. Each search was conducted with the default settings at each of the following URLs:

  • http://beta.search.msn.com external link
  • http://www.google.com external link
  • http://search.yahoo.com external link

The tables contain five measurements for each result in the SERPs.

  1. PR - This is the Google PageRank according to the toolbar for the homepage of the site
  2. YLinkD - This is the number of results according to a query run at Yahoo! for the number of links to the top level domain (linkdomain:url.com -site:url.com).
  3. MSNLink - This is the number of results according to a query run at MSN's Beta Search engine for the number of links to the top level domain (link:url.com).
  4. Size - This value is taken from a search at Google for the number of pages in the index (site:url.com).
  5. Qual - This is a subjective measure of the quality of the page's content. The score is X/10 based on the terms listed in the quality of content criteria chart.
  6. Match - This is a subjective measure of the closeness of the match in the site's content to the search terms entered in the query. The score is X/10 based on the terms listed in the match criteria chart.
PR
YLinkD
MSNLink
Size
Qual
Match
MSN Beta Average for 4 Searches
5.0
907,704
110,017
58,997
5.4
4.2
US National Holidays - 1303 searches in October*
Average
5.4
32,868
6532
46,803
5.7
3.3

1. http://www.villas4all.com/american-holidays.php

5
1200
1924

754

5/10
6/10

2. http://www.discoverfrance.net/France/DF_holidays.shtml

6
2600
3484
760
7/10
2/10

3. http://usinfo.state.gov/usa/infousa/facts/factover/holidays.htm

8
245,000
52,055
231,000
7/10
6/10

4. http://falcon.jmu.edu/~ramseyil/holidays.htm

6
76,600
159
158,000
7/10
4/10

5. http://www.prttravel.com/National_Holidays.htm

3
334
765
24
5/10
4/10

6. http://www.asiatraveltips.com/NationalHolidays.shtml

6
1,170
1887
25,500
3/10
1/10

7. http://www.nzstay.com/new-zealand-holidays.htm

5
686
962
370
6/10
2/10

8. http://www.allhotelsitaly.com/italy/holidays.htm

5
172
1668
171
4/10
2/10

9. http://www.us-parks.com/

5
898
2285
51,200
7/10
1/10

10. http://www.granisle.com/holidays.html

5
24
138
256
6/10
5/10
HDTV Comparison - 619 searches in October*

Average
3.8
3039
1604
91,948
4.2
3.7

1. http://www.bigpicture-hdtv.com/compare.html

5
1560
963
14
6/10
4/10

2. http://www.about-hdtv.com/hdtv-comparison.htm

0
5
271
83
2/10
4/10

3. http://www.about-hdtv.com/

0
5
271
83
2/10
1/10

4. http://www.ee.washington.edu/conselec/CE/kuhn/hdtv/95x5.htm

7
26,600
2672
916,000
9/10
5/10

5. http://www.twcsc.com/hdtv.htm

5
380
2244
293
4/10
4/10

6. http://www.eegent.com/hfeature.htm

6
57
124
87
4/10
4/10

7. http://www.twckc.com/services/hdtv/compare.asp

5
380
1359
905
4/10
4/10

8. http://satellitesense.com/

3
15
1515
556
4/10
1/10

9. http://www.plasmatvbuyingguide.com/...html

5
1310
6627
611
7/10
7/10

10. http://www.onlinemarketingmall.com/lcd-tv/hdtv-comparison.htm

2
86
2
851
0/10
3/10
George Plimpton - 593 searches in October*
Average
5.7
281,787
36,040
48,420
5.6
3.4

1. http://www.siteway.com/illustrations_georgeplimpton.php

5
783
285
533
6/10
5/10

2. http://www.parisreview.com/audio/audio.htm

7
5140
1542
84
6/10
4/10

3. http://www.gadflyonline.com/archive-plimpton.html

4
579
120
519
6/10
3/10

4. http://time.com/time/community/...plimptontime100.html

8
848,000
105,158
190,000
6/10
3/10

5. http://www.pagitica.com/extras/plimptondialog.html

4
123
186
85
6/10
3/10

6. http://www.historybound.com/...Code=BIOHISTMEM

4
29
3
255
4/10
2/10

7. http://www.lectures.org/plimpton.html

6
253
457
198
6/10
5/10

8. http://www.salon.com/audio/2000/10/04/johnson_plimpton/

8
1.93mil
238,795
279,000
6/10
3/10

9. http://www.milkmag.org/plimpton.htm

5
465
364
227
4/10
4/10

10. http://www.museumofhoaxes.com/siddfinch.html

6
32,500
13,494
13,300
6/10
2/10
Order Digital Prints Online - 184 searches in October*
Average
5.1
3.31mil
1891
48,817
6
6.5

1. http://www.chatsworthsalesphoto.com/

2
9
88
102
7/10
8/10

2. http://www.chatsworthsalesphoto.com/digital

2
9
88
102
7/10
6/10

3. http://www.freewebs.com/digital-photo-prints/

7
1.62mil
7754
196,000
2/10
6/10

4. http://www.picturetrail.com/webpages/makeprints.shtml

6
311,000
7780
53,400
6/10
8/10

5. http://www.adobe.com/digitalimag/tips/phsalonlsvc/main.html

10
15.6mil
1.97mil
119,000
7/10
7/10

6. http://www.adobe.com/digitalimag/photoshop_services.html

10
15.6mil
1.97mil
119,000
6/10
5/10

7. http://www.edmonsonweddings.com/Shutterfly-Order-Prints.php

3
89
2274
351
6/10
5/10

8. http://www.greatnationalcamera.com/

4
69
866
160
6/10
6/10

9. http://www.photocounter.ca/digital/online.html

3
6
3
10
6/10
7/10

10. http://www.cbopimaging.com/orderonline.html

4
20
60
47
7/10
7/10
PR
YLinkD
MSNLink
Size
Qual
Match
Google Average for 4 Searches
6.9
1.94mil
176,191
474,746
5.8
6.2
US National Holidays - 1303 searches in October*
Average
6.4
83,673
12,469
43,825
6.2
6.7

1. www.villas4all.com/american-holidays.php

5
1200
1924
754
5/10
6/10

2. aa.usno.navy.mil/faq/docs/holidays.html

8
356,000
15,395
1.09mil
7/10
8/10

3. usinfo.state.gov/usa/infousa/facts/factover/holidays.htm

8
245,000
52,055
233,000
8/10
8/10

4. www.vpcalendar.net/Holiday_Dates/2000_2005.html

7
1410
1525
80
7/10
8/10

5. www.madmanmike.com/us_events_dates.html

5
196
307
149
8/10
8/10

6. www.opm.gov/fedhol/

9
178,000
43,824
184,000
7/10
9/10

7. www.smart.net/~mmontes/ushols.html

7
27,500
523
981
8/10
8/10

8. www.usflag.org/flag.holidays.html

6
25,700
7867
92
7/10
6/10

9. www.maladyspoetry.com/poetry1.htm

4
174
349
199
5/10
6/10

10. www.global7network.com/.../russian_holidays.asp

5
1550
927
19,000
0/10
0/10
HDTV Comparison - 619 searches in October*
Average
7
3.06mil
61,832
96,721
5.9
5.3

1. www.plasmatvbuyingguide.com/dtv-hdtv-comparison19.html

5
1310
6627
611
7/10
7/10

2. www.bigpicture-hdtv.com/compare.html

5
1560
942
14
6/10
4/10

3. www.mysimon.com/HDTV_Sets/4007-6485_8-0.html

9
8.88mil
110,124
386,000
6/10
5/10

4. www.mysimon.com/...4007-6487_8-0.html

9
8.88mil
110,124
386,000
6/10
4/10

5. www.proactiveelectronics.com/...Hdtv

5
1470
573
7090
3/10
3/10

6. www.epinions.com/Flat_Panel_Televisions-Plasma-HDTV

7
1.81mil
43,626
4.79mil
6/10
7/10

7. www.epinions.com/...hdtv_ready

7
1.81mil
43,626
4.79mil
6/10
7/10

8. www.howstuffworks.com/hdtv.htm

8
517,000
211,041
174,000
6/10
5/10

9. shopper.cnet.com/...orderby%3D90%26sort%3D

8
8.7mil
77,081
1.97
6/10
7/10

10. www.directv.com/DTVAPP/learn/FAQ_DTVAdvanced_HDTV.jsp

7
61,500
14,556
13,500
7/10
4/10
George Plimpton - 593 searches in October*
Average
7.9
2.13mil
273,972
274,501
5.5
5.1

1. www.georgeplimpton.com/

4
113
138
11
0/10
3/10

2. www.imdb.com/name/nm0687321/

9
4.93mil
433,689
1.79mil
7/10
5/10

3. www.nytimes.com/2003/09/26/obituaries/26CND-PLIM.html

9
4.79mil
700,428
520,000
8/10
7/10

4. www.randomhouse.com/boldtype/1297/plimpton/

7
273,000
28,144
270,000
6/10
4/10

5. en.wikipedia.org/wiki/George_Plimpton

8
2.88mil
147,695
1.62mil
6/10
8/10

6. www.time.com/...plimptontime100.html

8
848,000
103,092
190,000
6/10
3/10

7. espn.go.com/page2/s/thompson/030929.html

9
1.76mil
176,402
946,000
6/10
5/10

8. www.npr.org/display_pages/features/feature_1447605.html

8
845,000
247,031
187,000
5/10
5/10

9. www.npr.org/display_pages/features/feature_1406696.html

8
845,000
247,031
187,000
5/10
4/10

10. www.usatoday.com/.../2003-09-26-plimpton-obit_x.htm

9
4.17mil
656,078
445,000
6/10
7/10
Order Digital Prints Online - 184 searches in October*
Average
6.3
2491990
45,488
32,935
5.7
7.7

1. www.shutterfly.com/learn/order_digital_prints.jsp

7
180,000
165,681
23,200
6/10
8/10

2. www.shutterfly.com/learn/digital_prints.jsp

7
180,000
165,681
23,200
6/10
8/10

3. www.microsoft.com/.../orderprints.mspx

10
24mil
3.11mil
1.52mil
6/10
8/10

4. www.colormailer.com/

5
29,500
1161
735
5/10
8/10

5. www.icdphotos.com/

5
186
547
5340
5/10
7/10

6. www.chebucto.ns.ca/~rakerman/digiphoto.html

6
77,900
5025
54,200
9/10
9/10

7. www.bonusprint.co.uk/pages/digital_prints.htm?OnpageA=1

5
192
156
190
5/10
7/10

8. photos.wanadoo.co.uk/

6
136,000
9032
40,100
5/10
7/10

9. www.mcbaincamera.com/digitalprints.htm

4
117
144
388
4/10
7/10

10. www.kodak.com/eknec/PageQuerier.jhtml?...

8
316,000
107,457
182,000
6/10
8/10
PR
YLinkD
MSNLink
Size
Qual
Match
Yahoo! Average for 4 Searches
6.5
5.6mil
198,358
807,658
5.3
5.3
US National Holidays - 1303 searches in October*
Average
6.1
3.07mil
20,586
27,205
5
4.6

1. www.villas4all.com/american-holidays.php

5
1200
1898
754
5/10
6/10

2. www.usemb.se/Holidays/celebrate

6
17,200
333
3520
5/10
6/10

3. usinfo.state.gov/usa/infousa/facts/factover/holidays.htm

8
245,000
52,162
233,000
8/10
8/10

4. www.patriotism.org/page2.html

6
879
123
35
5/10
5/10

5. www.asiatraveltips.com/NationalHolidays.shtml

6
1170
1929
25,400
3/10
1/10

6. patriotic.kith.us/holidays.html

3
139
7
243
5/10
4/10

7. www.timechange.com/us_holidays.htm

6
702
308
8910
5/10
3/10

8. www.geocities.com/geraljo1/us_nationalholidays.htm

8
28.6mil
129,672
2.45mil
4/10
2/10

9. www.h1bresources.com/html/nationalholidays.shtml

5
524
433
189
5/10
6/10

10. dir.yahoo.com/Society_and_Culture/Holidays_and_Observances

8
1.86mil
19,000
3.15mil
5/10
5/10
HDTV Comparison - 619 searches in October*
Average
5.8
1.61mil
10,464
95,057
4.6
4

1. http://www.nextag.com/...1400000

7
67,200
11,074
1.2mil
6/10
5/10

2. www.shopping.com/xGS-hdtv_comparison_brands

8
4.07mil
31,414
949,000
3/10
3/10

3. www.plasmatvbuyingguide.com/dtv-hdtv-comparison19.html

5
1310
6627
611
7/10
7/10

4. www.bigpicture-hdtv.com/compare.html

5
1560
942
14
6/10
4/10

5. www.digitalmegamall.com/hdtv.htm

3
36
4
21
4/10
5/10

6. http://www.proactivewm.com/...&subcat=HDTV

4
697
112
34
3/10
4/10

7. amdzone.pricegrabber.com

7
5.45mil
17,438
3.28mil
5/10
1/10

8. www.twckc.com/services/hdtv/compare.asp

5
583
1328
807
4/10
4/10

9. www.dealtime.com/xGS-hdtv~FD-0~DL-0~SK-0~CLT-DNL01

8
6.55mil
35,575
1.43mil
4/10
3/10

10. www.eegent.com/hfeature.htm

6
57
128
87
4/10
4/10
George Plimpton - 593 searches in October*
Average
8
13.7mil
412,880
176,250
6.7
5.7

1. www.nytimes.com/2003/09/27/obituaries/27PLIM.html

9
4.91mil
700,428
520,000
8/10
7/10

2. us.imdb.com/Name?Plimpton,+George

8
2.17mil
433,689
1.79mil
7/10
5/10

3. www.newyorker.com/talk/content? 031006ta_talk_remnick

7
334,000
37,614
24,500
8/10
7/10

4. www.randomhouse.com/boldtype/1297/plimpton

7
274,000
28,144
270,000
6/10
4/10

5. slate.msn.com/id/2089012

8
956,000
87,998
145,000
9/10
7/10

6. www.npr.org/display_pages/features/feature_1447605.html

8
853,000
247,031
187,000
5/10
5/10

7. http://www.amazon.com/...2801400

8
60.2mil
823,927
5.81mil
6/10
5/10

8. www.amazon.com/exec/obidos/ASIN/0871135035

8
60.2mil
823,927
5.81mil
6/10
4/10

9. www.usatoday.com/.../2003-09-26-plimpton-obit_x.htm

9
4.25mil
656,078
445,000
6/10
7/10

10. www.msnbc.com/news/972260.asp?0cv=CB20

8
2.99mil
289,973
171,000
6/10
6/10
Order Digital Prints Online - 184 searches in October*
Average
6.2
4.03mil
38,499
11,118
5
6.7

1. www.ofoto.com

6
141,000
11,749
52,600
5/10
7/10

2. www.shutterfly.com/about/com_overview.jsp

7
177,000
165,681
23,200
6/10
7/10

3. http://www.microsoft.com/.../orderprints.mspx

10
23.8mil
3.11mil
1.52mil
6/10
8/10

4. www.shutterfly.com

7
177,000
177,000
23,200
6/10
7/10

5. www.photoaccess.com

6
16,800
1774
1050
4/10
4/10

6. www.filmworks.com

7
1370
4675
485
5/10
7/10

7. online-photo-albums.bousperlavila.com

X
701
5
X
1/10
7/10

8. www.snapfish.com

7
112,000
11,814
10,600
5/10
7/10

9. www.cbopimaging.com/orderonline.html

4
18
60
47
7/10
7/10

10. store.yahoo.com/lindenhillsphoto/ordiprandhog.html

8
15.9mil
12,241
2.77mil
5/10
6/10
* # of Searches was taken from Overture's Search Term Suggestion Tool

Quality of Content Criteria Chart

The quality of content criteria judges the usability, writing, images, files & overall quality of information on the page. It does not factor in how particularly relevant the information was to the query.

  • 0/10 - Did not have any valuable/unique content
  • 1/10 - Entirely duplicate/spam content
  • 2/10 - Mostly duplicate/spam content
  • 3/10 - Very low quality content
  • 4/10 - Poor content with some helpful information
  • 5/10 - Mediocre content
  • 6/10 - Effectively covered the subject
  • 7/10 - Very effectively the subject, provided links to other resources, images, files, etc.
  • 8/10 - In-depth coverage, very high quality information
  • 9/10 - Excellent resource on the subject, exceptional information
  • 10/10 - Authoritative resource worthy of an award

Match Criteria Chart

The match criteria judges how effectively the content provided on the page matches the query that was entered. For queries with multiple potential meanings (i.e. George Plimpton, US National Holidays) the highest match criteria scores will be for those that tackle the subject multiple potential aspects of the search.

  • 0/10 - Had nothing to do with the search
  • 1/10 - Almost no relation to the search
  • 2/10 - Very slight match
  • 3/10 - Some information matched the query
  • 4/10 - Nearly matched the search
  • 5/10 - Mediocre match for the search
  • 6/10 - Acceptable match
  • 7/10 - Above average match, covered at least one aspect very well
  • 8/10 - Good match, covered one aspect very well or multiple aspects
  • 9/10 - Near perfect match, covered multiple aspects in-depth
  • 10/10 - Flawless match, perfect coverage of every possible aspect of the query

Interesting Numbers

PageRank was clearly most important to Google, with the average PR for a result from Google at 6.9, compared to 6.5 for Yahoo! & 5.0 for MSN.

Massive numbers of backlinks, however, seemed most important to Yahoo!, whose average # of bls was 5.6mil compared to 1.9mil for Google & 900K for MSN.

The highest match criteria results for a single search was for "Order Digital Prints Online" at Google - a 7.7, while the highest quality criteria results came from the Yahoo! search for "George Plimpton".

MSN's average quality criteria results actually slightly edged out Yahoo!'s average - 5.4 to 5.3.

Yahoo! gave the most results with large sites - an average of 807K pages per site to Google's 474K and MSN's 58K.

Results for averages were often widely skewed by sites with exceptional numbers of links & pages. Normalized results will appear here once these are complete to allow for a more balanced analysis.

SeoMoz

No comments:

Live Page Popularity