Website Ranking Differences: 3 Little Reasons Why

March 11, 2009
by Rob Watts

If I had a pound for every story I’d read around confusion in the rankings I’d be a relatively rich man. Not everyone knows that different search engine URL’s return different sets of results and that these are often influenced by a range of other factors brimming beneath the hood. Personalisation, User Agents, Cookies, Time of day, Geo Location, all stack up to present different result sets to different people for the same search queries.

So with this in mind, I thought it might be useful to take a simplified look at what these are and perhaps try to demystify some of these things in the process.

1- User Agent Driven Search Results

People access the web from a multitude of different platforms. For the non techie heads amongst you, each device, be it a Web browser, a PDA or a Phone delivers a “user-agent” to the web server that it accesses. This information is then used by the requesting web server to return best fit data sets in formats that the device will recognise.

An example of this is for mobile phones that rely on compliant XHTML (extensible hyper text mark up language) to output human readable content.

If you search Google from a HTC620 smart phone you’ll see that it interprets the device and serves up an XHTML specific set of results. You can see this in action by looking at the url here: http://www.google.com/m?mrestrict=xhtmlonly&eosr=on&ct=fsh&q=distance+calculator

Normal Search Results

and compare it with the results here http://www.google.com/search?q=distance+calculator

Mobile Search Results

Two things happen here. Firstly, Google delivers a web page that validates to XHTML and secondly it outputs completely different result sets.

This means that the results it returns will favour web pages that have been tagged with a document type (doctype) that specifies that the content is of XHTML format. The thinking is that there is little point in serving up a result set that people will be unable to read, or render if they click through.

2- Geographic Location Influenced Search Results

Different people in different areas often get different results based upon the IP address being used. So a user in Japan using Google.com might see a different result set from a person in say England or France using the same.

You can test this using a Japan based proxy server and witness the behaviour for yourself for the query of ‘Hotels’ (you’ll need to be an advanced user to do this as Google will attempt to redirect you elsewhere).

If you managed to get there great, you’ll see that whilst on the whole the results have a degree of commonality, they are nonetheless slightly different. Another point to note is how the Google AdWords results are affected too. Japanese users are served with Japanese based PPC results, whereas UK users receive the results based on the UK market.

It doesn’t stop there either; as touched upon previously Google will often try to redirect users based upon their IP and deliver them to the country specific search results. Google do this for all sorts of reasons but principally to deliver country specific results for the person using the product.

Country specific search results are determined by a complex set of factors related to TLD (top level domain) extension, language employed on page, hosting location and link graph. When you think about it, this makes a lot of sense. After all, a search for a generic key phrase like ‘Plumber’ from a user perspective is pretty location specific. By forcing the user to use a country specific search engine TLD, the search engine can best deliver the result set that best suits that user.

3- Personalised Search Results and Tracking

Personalisation is another tool in the armoury of search engines. Putting to one side any privacy or tracking implications, your search behaviours and history are often tracked via cookies dropped and stored on the computer you use to search the web. Your searches and behaviours may be tracked on a number of levels in a variety of ways.

Whilst the full reasons for this are not entirely proper to a document like this (it’s a big animal!) what we can reasonably surmise is that access to and collection of user behaviour data may be used to inform on things like click through rates, types of sites visited, ad clicks, bounce rates and a whole host of other metrics useful to data mining purposes.

Did you know for example that every single click from a Google search result is measured via an OnMouseDown mouse event? The moment your finger performs the click action, data of that event is then stored and recorded somewhere. They’ll grab your IP location, your browser agent, your IP address, the type of operating system used, how long it took you from load to click, how many times you clicked back and more and that’s just for people without accounts.

Where a user has a Google account say Gmail for example, then these behaviours can be directly attributed to the login and stored in ‘the cloud’ to inform subsequent searches from different computers or locations. Again this info is then crunched and interpreted to deliver a modified set of results based upon the types of sites you visit and frequent.

What this means of course is that for the unsuspecting user they are often blissfully unaware that what they are being shown may not necessarily be the same as their Aunt Fanny or Uncle Mick. Which of course, makes the job of reporting on search rankings doubly difficult to the point where in truth in 2009, search rankings should by and large, form just a fraction of the overall picture of a websites performance to the point where the things that really matter like traffic, conversions, and the tools that analyse them come into their own.

Seeing the raw results

So, what do you do? How can you see what the bog standard user will see? What if you don’t want to be redirected or tracked or insert any other reason pertinent to you? To be honest, it’s difficult to do so, and our modern day tools make it a bit of a pain even for advanced users, but if you really wanted to then:

• Ensure that you delete cookies after every visit

• Uninstall any toolbar you may have installed

• Surf the web using an Anonymous Proxy

• Use a smart browser like Firefox and Install any number of the privacy extras that exist
Of course to say all that might suggest that Google is in some way evil and has ill intentions around what it intends to do with your behaviours, which we all know is abundantly true untrue!

The fact remains that search engines like Google are immensely useful offering massive value to the world and the businesses that use them, their search results and the way in which they are outputted should be of significant interest to anyone looking to get ahead on the web, I hope you gained a little more insight into how some of all this works!