search
top

Fact-Testing on the Britannica Blog: Analyzing Internet Search Skills

google wisely!

Few wouldn’t argue that the way in which we collect and interpret information – increasingly on the web as opposed to traditional off-line media – is at the foundation of almost everything we do in education. About 6 weeks ago, Britannica Blog contributor Gregory McNamee ran a piece called “10 Ways to Test Facts” that recommended approaches to finding and filtering information.

I came to this piece by way of Joanne Jacobs’ blog. In her post, “Are you sure?” Joanne highlights McNamee’s first tenet of effective, skeptical search:

1. Trust not the first answer the search engine turns up. In the spirit of the tyranny of the majority, it will usually be wrong or, if not outright wrong, not the answer you really need. A while back, Inside Higher Ed reported that, even though most teachers take it as a matter of faith, rhetorically if nothing else, that finding and filtering information are important skills, too few students know even to go beyond the first couple of hits that come back from a Google search. Less than 1 percent move to page 2 and beyond of the search results. Be one of that exalted few.

I took issue with this and commented:

This is bad advice that sneaks silently toward the ridiculous.

Greg McNamee has created a mutated bastard child of “question authority.” Reading comprehension and analysis are important skills – there’s no question about that. But what is the value in discarding a source because it’s on the front page of Google? Search engine placement is nearly irrelevant – it depends on many factors that have little to do with authority.

Placement matters more on some topics than others. For example, technology-related search placement is more likely to be skewed based on expert search engine optimization/SEO tactics [for obvious reasons]. A Google search for on a basic topic such as “George Washington,” however, yields adequate results. I continued:

I have to take issue with his suggestion to go beyond page 1 of Google results just for the sake of it. Most of the time I find well-referenced answers to my questions within the first 5 results. I simply don’t need a second page. Then again, I might be yet another one of the American “sheeple” who believes everything he reads and is entirely unable to identify truth.

Pardon the snark at the end; I did – and still do – believe that McNamee regards we first-pagers with scorn. Remember the George Washington search – I’ll get to that a little later.

I suggested that encouraging users to spend more time on a search is more important than forcing one to go through 10 or 20 additional pages. The average user spends about 20 seconds on a website [there are too many surveys/studies to cite here and the data changes frequently, so I'm using a general figure that is an aggregate of current analysis]. That simply isn’t long enough to evaluate information with any attention to detail, regardless of whether it’s Google result #1 or #253. I ended my comment:

I’m embarrassed for McNamee and Britannica.

I was and still am. McNamee wasn’t thrilled:

Mr. Tabor, I wonder if you can understand how shallow your objection is. Please be embarrassed on your own account.

I gave him a quick explanation for my judgment:

As for your categorization of my objection as “shallow,” I find that my explanation of why time spent on analysis matters more than a volume of sources provides more depth than mistrusting a page for the sake of it. I found it odd that you mention the “sea of information,” then outline how information consumers should assume everything is a man-eating shark until proven otherwise, and then push balanced skepticism in point #7.

I wasn’t about to hijack a thread on Joanne’s site, and by the time this discussion got going, the appropriate thread on the Britannica blog was largely dead. I e-mailed McNamee offering to give a full analysis in a point/counterpoint style that could be published on the Britannica blog [I don't have the message because it was sent via a contact form, not regular e-mail]. McNamee declined, citing the endless futility of the debate and other topics on the docket.

That “some of us” I referenced was a group of three: one academic, one education theorist and blogger, one former educator. I wrote a reply that day which never got sent – I wanted to think on the topic a bit more before concluding. I’ll demystify McNamee and others publicly.

I respect McNamee’s decision not to pursue the debate further – he has his reasons and the editorial direction of the Britannica blog is obviously unknown to me. Even so, I find few matters more pressing than how we think about using the internet. Common sense would suggest that the content online is irrelevant if we aren’t approaching it in a profitable way.

I suggested positioning our arguments as real content on the Britannica blog because my experience with Google Analytics tells me that comments are largely unread in comparison. As I mentioned in my original note to McNamee, I simply can’t afford to comment as often as I’d like because of the opportunity cost. Comments, when read and replied to, are too susceptible to inappropriate interjections, bizarre logical splices and a host of other problems. While I occasionally leave comments on some of my favorite blogs – like Joanne’s – I am largely forced to ignore the feature out of practicality.

I wanted to make sure that both sides of this issue received exposure.

In McNamee’s suggestion to go past page 1, he said:

“1. Trust not the first answer the search engine turns up. In the spirit of the tyranny of the majority, it will usually be wrong or, if not outright wrong, not the answer you really need.”

Suggesting that page 1 results will “usually be wrong” instructs readers to look past top results. Though he didn’t say explicitly that all page 1 results are invalid, the logical, common sense extension of tip #1 is that page 1 results are inferior. I would be very surprised if the average reader, when asked to summarize tip #1 in their own words, said otherwise. [I am ignoring the misuse of Tocqueville's principle here - it simply doesn't apply well to Google pagerank.]

A corollary of this stance blames Google or websites for not delivering the answer one needs. I am sure we all would agree that failing to find the proper answer is more commonly a function of searching poorly than the quality of search results – it is the classic ‘garbage in, garbage out.’

Difficulty with finding answers is also a function of asking more specific questions; guys like McNamee and myself are more apt to look for the obscure, while a 10-year old Googling “George Washington” is well served by nearly all of the page 1 results [see Google results for George Washington]. Nearly all the page 1 results in this example deliver solid, accurate and, consequently, trusted content.

These are terribly important points to stress, especially on a blog that focuses on information. My gripes in my comment were because of these omissions.

McNamee gave a tacit endorsement of current web viewing trends – that is a quick F-pattern gloss-over and 15 to 20 seconds on a page – by not addressing them as a major obstacle to better fact-testing. Simply put, those seeking information, whether it’s coming from page 1 or page 100, need to spend more time on a site and read more carefully. It is a serious mistake to assume that even readers of the Britannica blogs – certainly a more literate readership than most sites – deviate significantly from the mean.

Thankfully, we don’t have to guess about web usage now, and McNamee won’t have to guess at statistics [which were unverifiable, as a citation in his entry didn't appear] next time.

Consider iProspect’s white paper on Search Engine User Behavior, first published in April of 2006. It’s designed for marketers to better utilize their websites, but much of the information is wholly transferable to our discussion here. Some of the highlights:

  • “Key among the findings related to the current search engine user community is that 62% of search engine users click on a search result within the first page of results, and a full 90% of search engine users click on a result within the first three pages of search results.”
  • “The data indicates that more search engine users are clicking on the first page now (62%) than in 2003 (60%) than in 2002 (48%). Inversely, fewer search engine users are willing to click on results past the third page now (10%) than in 2004 (13%) than in 2002 (19%).
  • “In a second, related finding, 41% of search engine users who continue their search when not finding what they seek, report changing their search term and/or search engine if they do not find what they are looking for on the first page of search results. A full 88% do so if they do not find what they seek in the first three pages.”

A quick interpretation:

  • Most users click page 1 results. There are lots of reasons for this ranging from indiscriminate web searching to adequate search results.
  • Page 1 has remained – and will remain – popular for those same reasons.
  • A user’s initial search terms are often inadequate for the question they’re looking to answer.

Let’s look at some graphical representations of this data:

The majority don’t go beyond page 1, but 38% do so regularly. This doesn’t take into account facility with search terms [ie, a skilled searcher could yield adequate page 1 results more frequently by virtue of his skills and vice versa]. It’s not quite as hopeless as it seems – not everyone’s a page 1-er.

4 out of 5 searchers need to refine their keywords to find the answer they need. This demonstrates two things: 1) a major problem with the average user’s ability to target information accurately and quickly; 2) internet users generally don’t give up on the search.

That 41% have to switch queries is, I’d guess, a function of determining quickly that a given search was poor. This distribution shows that users give a bit more attention to more than a few search results than McNamee seems to think they do.

And do users really trust blindly the page 1 results? No, says Jupiter/Ipsos; consumers, when asked whether results translate to company prominence [transferable here as search placement relating to authority], don’t see a correlation.

About a third do, a slightly fatter third don’t care and the balance flat out disagree. As a company, I’d want to be at the top of page 1 to reach the guys who make up that red block, but that’s a different discussion.

In short, we’re better internet users than McNamee thinks. We generally don’t need to be told not to trust the first result. We seem to go through results until we find a trusted answer.

McNamee’s 6th tip is not only good advice, but it applies here:

6. Rigorously practice the principle of symmetrical skepticism. Assume goodwill, but also assume that everything people tell you is wrong until you have looked it up for yourself, no matter how much you may agree with your source of information politically, religiously, culturally, or otherwise.

Noted. I just I wish I didn’t have to be so skeptical of the Britannica blog.

UPDATE at 10.34pm, 08.12.07,

Edited to remove quotation from e-mail. 

4 Responses to “Fact-Testing on the Britannica Blog: Analyzing Internet Search Skills”

  1. David Mackey says:

    It always amazes me – the statistics that is – I generally read pages I am actually interested in entirely.

  2. John Kaiser says:

    Your dispute over rule one doesn’t seem shallow at all. Although the rest of the list looks quite well founded and worthwhile.

  3. Matthew says:

    I think that most of the list is overly paranoid, but erring on the side of caution is usually better. I don’t subscribe to the tenets of ‘question all authority’ or ‘everything you think you know is wrong.’ I have more respect for myself than that. As I said on Joanne’s blog, I found the approach very Zinn-like.

    I found the weak political bit at the end especially distasteful:

    “And think of how the world might be if we all had that question before us at all times. Are you sure about those WMDs? Are you sure about that yellow cake uranium from Niger? Are you sure there’s no such thing as global warming? The list goes on, and on . . .”

    Had I a blog themed differently, I’d have touched on that.

    Also, I’m with David – I generally read pages in full, too. I’m glad that the usage statistics for this site don’t boil down to 20-second visits. At the same time, there’s quite a bit of garbage on the internet that wouldn’t warrant more than a second, so maybe 20 seconds is generous for some sites.

  4. Benny says:

    I think the admin of thyis web site is truly working hard for his
    web site, as here every data is quality based information.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>

top