Interesting email exchange tonight about rankings.
On Mon, Oct 29, 2012, at 08:37 PM, Graeme Ackland wrote:
> Hi Neil,
>
> I'm about to run a Masters student project on Orienteering Rankings. I
> was going to look at variants on the BOF "course average and spread"
> method, and how reliable the results are as a measurement. I saw your
> AP post about "paired comparisons" which looks like an interesting
> alternative. How do you imagine it should work? You could build a
> (sparse) matrix with entries for pairwise comparisons (win/lose, or
> somehow include how much you win by?) but then how do you create an
> ordered list?
>
> A problem might be that the age group structure makes the matrix quite
> blocky?
>
> Graeme
>
> Prof. Graeme Ackland FRSE
> Head of Institute for Condensed Matter and Complex Systems
> School of Physics
>
http://homepages.ed.ac.uk/graeme
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
Hi there
Some background first so you can understand why I am thinking differently.
I'm a research psychologist by training so my statistical approach bears the stamp of psychometrics, and scaling is one of the recurrent issues. (Having worked with psychometricians, econometricians and biometricians, my rule of thumb is that the first think in terms of scaling and factor analysis, the second think time series and regression and the third think ANOVA designs. I won't claim any appreciation of how physicists think... ;-) )
Many years ago I ran an informal experiment on taste preference for peach varieties. This was beset with two problems:
1. Human taste preferences is very easily influenced by framing effects. The only reliable method of measuring preference was by paired comparison. A simple comparison task for two varieties at a time was the most defensible. When repeated with many observers, this produces a ratio measure of preference.
2. The problem with peaches is that the fruit availability extends across a four month season, but each variety is generally each only available for two weeks. This means that any paired comparison matrix will have lots of blank spaces, although with seasonal overlap and enough observations, the matrix is soluable with the right approach. The approach I found was in a classic of the scaling literature...
Torgerson WS (1958) Theory and Methods of Scaling, Chapters 9 and 10.
http://books.google.com.au/books/about/Theory_and_...
http://www.amazon.com/Theory-Methods-Scaling-Warre...
Chapter 10 deals with the case of incomplete matrices. From memory I used Gulliksen's Least Squares solution as described in that chapter.
http://libra.msra.cn/Publication/34576171/a-least-...
As you can see by the date, its nothing new.
Many years later I fell into the world of orienteering and was intrigued by the very different approach adopted by orienteers in their attempts to build ranking methods. I think these methodologies reflected the disciplinary backgrounds in which the builders lived. The methods seemed alien and inelegant to me coming from a background of psychometrics, but I knew I wasn't going to offer to take on the task. I could foresee a number of interesting questions which would need research and on which you have already commented...
1. How do you create a paired comparison metric. Is it a ratio of wins over losses? Or does the metric use information on race margins?
2. How do you decide which races to include? The more that are included, the quicker you get a soluable matrix. But how many races will a runner try and peak for? Do you weight certain races according to importance to overcome this?
3 How do you explain the process to the competitors who might grasp the impact of their runs in the current system, but would perhaps find the matrix resolution process opaque.
I think this would make an interesting research project. It excites me. Hope this helps.
Neil