Discussion: Rankings
in: Orienteering; General
Apr 16, 2010 6:24 AM
#
Shep:
ev started a discussion about the aus ranking system on his trog which got me thinking about my super ideal awesome ranking scheme again... the world/australian, american and swedish ranking systems have their pros and cons, but not so many aussies know how the swedish system works so i did a quick dodgy translation (see below). i think feet gave me a good explanation of the american system once, i'll see if i can find it...
The Swedish ranking system aims to have the best runner in the country score zero points, and slower runners score more points. In each race they work out a “base time”, which is the time the best runner would have run, and of course it is possible to run faster than that time, in which case you would score negative points. The system is self-calibrating in a way, as they regularly set the highest ranked runners score (which is the average number of points in their best 6 races) to zero, every other runner has the same constant added/subtracted from their score. The system works as below, I haven't thought about it in enough detail as to what effect the numbers 60 and 75 have, the Swedes are a little conservative in that they don't like giving out ranking points for sprints and think that the longer distances are the only genuine races, hence the 60 minutes for girls and 75 minutes for boys in there... I think it's only going to scale the scores and set the range, eg whether a slow time scores 100 points or 50 points etc.
1.Average time (Tm) of the first 3 ranked runners in the race, to prevent unrealistically low points for the winner. Tm is a maximum of winners time + 10%.
2.Average points (Pm) for the 3 best ranked (lowest ranking scores) runners who started the race. Races with less than 3 ranked runners are not counted.
3.Correction factor (Kk) is calculated as Kk=(60+Pm)/Tm for the women and Kk=(75+Pm)/Tm for the men. This sets the relationship between the “correction time” and average time/points.
4.Base time (Tb) is the (theoretical) time that Australia best runner would have run on the course, Tb = Tm Pm/Kk
5.Ranking points (P) calculated for each runner as P = (Runners_Time Tb) x Kk
6.It can be simplified by first calculating Kk then P = Time x Kk 60 for women and P = Time x Kk 75 for men.
Example (Men)
Times for the first three: 84.86, 87.14, 87.93
Bruce Arthurs Time: 94.75
Three lowest ranking scores in the runners that started: 0.38, 1.08, 1.11
Average Time (Tm) = (84.86 + 87.14 + 87.93)/3 = 86.64
Average Points (Pm) = ( 0.38 + 1.08+1.11)/3 = 0.86
Correction Quotient (Kk) = (75 + 0.86)/86.64 = 0.8756
Base Time (Tb) = 86.64 - 0.86/0.88 = 85.66
Bruce Arthurs Score = (94.75 - 85.66) x 0.88 = 7.96
Winners Score = (84.86 - 85.66) x 0.88 = -0.70
Every week the highest ranked runner has their score (average of their best 6 races) set to zero with a correction, every runner has the same correction added/subtracted from their (average) score.
Some races are weighted to reflect the importance of that race. Eg subtract 2 points for NOL, no change for Badge events/State Champs etc, add 2 points for club races.
check out ev's log if you're interested. anyone have anything to add? there must be some clever statisticians in the orienteering attackpoint world!
The Swedish ranking system aims to have the best runner in the country score zero points, and slower runners score more points. In each race they work out a “base time”, which is the time the best runner would have run, and of course it is possible to run faster than that time, in which case you would score negative points. The system is self-calibrating in a way, as they regularly set the highest ranked runners score (which is the average number of points in their best 6 races) to zero, every other runner has the same constant added/subtracted from their score. The system works as below, I haven't thought about it in enough detail as to what effect the numbers 60 and 75 have, the Swedes are a little conservative in that they don't like giving out ranking points for sprints and think that the longer distances are the only genuine races, hence the 60 minutes for girls and 75 minutes for boys in there... I think it's only going to scale the scores and set the range, eg whether a slow time scores 100 points or 50 points etc.
1.Average time (Tm) of the first 3 ranked runners in the race, to prevent unrealistically low points for the winner. Tm is a maximum of winners time + 10%.
2.Average points (Pm) for the 3 best ranked (lowest ranking scores) runners who started the race. Races with less than 3 ranked runners are not counted.
3.Correction factor (Kk) is calculated as Kk=(60+Pm)/Tm for the women and Kk=(75+Pm)/Tm for the men. This sets the relationship between the “correction time” and average time/points.
4.Base time (Tb) is the (theoretical) time that Australia best runner would have run on the course, Tb = Tm Pm/Kk
5.Ranking points (P) calculated for each runner as P = (Runners_Time Tb) x Kk
6.It can be simplified by first calculating Kk then P = Time x Kk 60 for women and P = Time x Kk 75 for men.
Example (Men)
Times for the first three: 84.86, 87.14, 87.93
Bruce Arthurs Time: 94.75
Three lowest ranking scores in the runners that started: 0.38, 1.08, 1.11
Average Time (Tm) = (84.86 + 87.14 + 87.93)/3 = 86.64
Average Points (Pm) = ( 0.38 + 1.08+1.11)/3 = 0.86
Correction Quotient (Kk) = (75 + 0.86)/86.64 = 0.8756
Base Time (Tb) = 86.64 - 0.86/0.88 = 85.66
Bruce Arthurs Score = (94.75 - 85.66) x 0.88 = 7.96
Winners Score = (84.86 - 85.66) x 0.88 = -0.70
Every week the highest ranked runner has their score (average of their best 6 races) set to zero with a correction, every runner has the same correction added/subtracted from their (average) score.
Some races are weighted to reflect the importance of that race. Eg subtract 2 points for NOL, no change for Badge events/State Champs etc, add 2 points for club races.
check out ev's log if you're interested. anyone have anything to add? there must be some clever statisticians in the orienteering attackpoint world!
Apr 16, 2010 6:34 AM
#
Shep:
ok feet's comments are here. on a previous discussion about ranking scores ;)
Apr 16, 2010 10:47 AM
#
ev:
sheppa, theres a case of your choice in it for you if your super ideal awesome ranking scheme can get me up into the top 15 without me having to run another race
Apr 16, 2010 11:37 AM
#
I vaguely remember being here before on AP.
A very different approach can be found in this:
Torgerson, W. (1965) Theory and Methods of Scaling, Wiley, Oxford.
Chapter 9 The Law of Comparative Judgment.
You use results to build a matrix of relative performance between pairs of runners over all races in the season. The result will be an incomplete matrix of comparisons, but with enough comparisons one can use a number of methods to resolve the matrix into a scale of performance. One can have a good debate about which races to include, whether to weight the races and which metric to use to build the relative performance measure.
For example, do you want the ranking system to encourage consistency or occasional great results? The current Australian ranking system uses the best 5 results for each runner. In a season with 20+ races, once a mistake is made, there is no ranking incentive to finish the race and minimise the damage. If all races count in the metric calculation then once you start there is a strong incentive to minimise the damage and finish the race well.
A very different approach can be found in this:
Torgerson, W. (1965) Theory and Methods of Scaling, Wiley, Oxford.
Chapter 9 The Law of Comparative Judgment.
You use results to build a matrix of relative performance between pairs of runners over all races in the season. The result will be an incomplete matrix of comparisons, but with enough comparisons one can use a number of methods to resolve the matrix into a scale of performance. One can have a good debate about which races to include, whether to weight the races and which metric to use to build the relative performance measure.
For example, do you want the ranking system to encourage consistency or occasional great results? The current Australian ranking system uses the best 5 results for each runner. In a season with 20+ races, once a mistake is made, there is no ranking incentive to finish the race and minimise the damage. If all races count in the metric calculation then once you start there is a strong incentive to minimise the damage and finish the race well.
Apr 17, 2010 1:01 AM
#
ev:
I think that the system has held up really well over the past decade or so and bruce has done a great job with it,
but this year, for some reasons, an update may be needed.
heres a brief summary of some of the issues which have been raised in the discussion for those who cant be arsed reading it
1. M21E points over the NOL rounds this year have been pretty low and havnt really had an impact on the rankings. not sure why this is but jules thrashing us all may have had some effect
2. M20E points seem to be pretty high and the M20E ranking points pool also seems also seems to be feeding itself and increasing a bit too rapidly over time as dave alluded to
ie. there is a ~400point disparity between the schools champs winner in 2002 compared to th schools champ winner in 2009
3. Hard classics and middles do not score very well. a disproportionate number of the high ranking scores at the top end of the field come from urban sprints
4. Races with small fields rank highly. surely the scores which make up the top rankings should come from national level events
Some possible solutions (noting that im not a statistician, so this is just to get the discussion going):
1. a field-size factor, ie the more people in the field the more points, the less people, the less the points
2. in races where M20s and M21s run together, rank them together,
3.a swedish style "gnarliness" factor to bump up the points on offer in the sprint
but this year, for some reasons, an update may be needed.
heres a brief summary of some of the issues which have been raised in the discussion for those who cant be arsed reading it
1. M21E points over the NOL rounds this year have been pretty low and havnt really had an impact on the rankings. not sure why this is but jules thrashing us all may have had some effect
2. M20E points seem to be pretty high and the M20E ranking points pool also seems also seems to be feeding itself and increasing a bit too rapidly over time as dave alluded to
ie. there is a ~400point disparity between the schools champs winner in 2002 compared to th schools champ winner in 2009
3. Hard classics and middles do not score very well. a disproportionate number of the high ranking scores at the top end of the field come from urban sprints
4. Races with small fields rank highly. surely the scores which make up the top rankings should come from national level events
Some possible solutions (noting that im not a statistician, so this is just to get the discussion going):
1. a field-size factor, ie the more people in the field the more points, the less people, the less the points
2. in races where M20s and M21s run together, rank them together,
3.a swedish style "gnarliness" factor to bump up the points on offer in the sprint
Apr 17, 2010 1:35 AM
#
correction quotient, Kk=(60+Pm)/Tm standard deviation, mean, performance matrices, statistical anomolies, Law of Comparative Judgment, wank, wank, wank, wank, wank, wank, wank.
What do you need a ranking system for anyway? The basic reason for ranking systems in sports like tennis and golf is to determine who can enter tournaments with limited places. (OK, who is ranked first, or in the top ten is used for publicity and hype but that is not the primary reason for the system.)
The same applies for orienteering in Sweden where elite classes are restricted. But that is not a problem in Aus and not likely to be anytime soon. So the ranking system really doesn't matter - it's just for fun. (Please don't tell me ranking is a selection criteria for WOC, JWOC, etc.)
but jules thrashing us all may have had some effect
Perhaps it would be better to spend a bit more time figuring out how to close the gap to Jules and less time worrying about an irrelevant statistical game. Or, if your ranking really is so important to you, play the game - choose the right races to run to get your score up.
Elite sport is not about rankings - it is about performance in the races that matter.
Perhaps there is a reason why "ranking" rhymes with "wanking". I feel a limerick coming on.
What do you need a ranking system for anyway? The basic reason for ranking systems in sports like tennis and golf is to determine who can enter tournaments with limited places. (OK, who is ranked first, or in the top ten is used for publicity and hype but that is not the primary reason for the system.)
The same applies for orienteering in Sweden where elite classes are restricted. But that is not a problem in Aus and not likely to be anytime soon. So the ranking system really doesn't matter - it's just for fun. (Please don't tell me ranking is a selection criteria for WOC, JWOC, etc.)
but jules thrashing us all may have had some effect
Perhaps it would be better to spend a bit more time figuring out how to close the gap to Jules and less time worrying about an irrelevant statistical game. Or, if your ranking really is so important to you, play the game - choose the right races to run to get your score up.
Elite sport is not about rankings - it is about performance in the races that matter.
Perhaps there is a reason why "ranking" rhymes with "wanking". I feel a limerick coming on.
Apr 17, 2010 2:01 AM
#
there was a young man of good ranking
who found that his ranking was tanking
he neglected to train
and fucked up his brain
with too much statistical wanking
who found that his ranking was tanking
he neglected to train
and fucked up his brain
with too much statistical wanking
Apr 17, 2010 3:28 AM
#
In general, I agree with Rob's sentiments (at least the "it's not really that important" part) but just an uninformed question... (and I admit I'm a bit out of touch here) are rankings used to determine seeded start orders for some events? If so, then they have *some* relevance (or at least affect the odds of who might be able to catch a ride when a top ranked runner comes past... and hence get a potentially better result (although obviously not a win)
For those that do care about rankings, I agree with the points raised by ev above (particularly the self-fuelling upwards drift in junior scores - i can be a grumpy old man now I'm the upwards side of 30...)
For those that do care about rankings, I agree with the points raised by ev above (particularly the self-fuelling upwards drift in junior scores - i can be a grumpy old man now I'm the upwards side of 30...)
Apr 17, 2010 8:42 AM
#
I am not the others would agree with you Jules.
...just make sure you keep on thrashing 'em.
...just make sure you keep on thrashing 'em.
Apr 18, 2010 1:32 AM
#
Shep:
rob plowright's still stuck in japan
with nothin to do, but now he can
get the list of our ranking
spend 10 minutes wanking
to make himself feel more like a man
it's still an interesting problem rob! need somethin else to think about during the long days at work...
with nothin to do, but now he can
get the list of our ranking
spend 10 minutes wanking
to make himself feel more like a man
it's still an interesting problem rob! need somethin else to think about during the long days at work...
Apr 18, 2010 4:53 AM
#
Um...I was going to comment on this, thinking that it's a serious thread, but now I'm not so sure. (There must be other rhymes for ranking but all I can come up with right now is spanking).
It doesn't make sense to me that the majority of my points, such as they are, come from sprints, and the rest from middles. Okay, so it's easier to be closer, percentage wise, to the winner in a sprint, and as far as I can tell, the available points are higher in a race where Hanny's present than where she's not, but I get the impression that shorter races are effectively weighted more heavily, even if that's not the intention?
It doesn't make sense to me that the majority of my points, such as they are, come from sprints, and the rest from middles. Okay, so it's easier to be closer, percentage wise, to the winner in a sprint, and as far as I can tell, the available points are higher in a race where Hanny's present than where she's not, but I get the impression that shorter races are effectively weighted more heavily, even if that's not the intention?
Apr 18, 2010 7:52 AM
#
To run a good race
yet see no ranking impact
disappoints young men
yet see no ranking impact
disappoints young men
Apr 20, 2010 10:49 PM
#
I think my whole point Neil is that you shouldn't be looking to the rankings for motivation, or be disappointed if the rankings don't reflect what you think was a good performance.
No one seems to have addressed my question of what the ranking system is actually for - except for Shep sayng that it is an "interesting problem".
Forget about the rankings. There is more than enough information to figure out where you stand. You all got thrashed by Jules over Easter and that might feel depressing. On the other hand his results last weekend in Sweden show that he is right up there internationally. And if Jules can do it, then other Australians can too. Forget about your ranking and focus on that.
Shep - if you are bored at work today and need an "interesting problem" to occupy your mind, you might want to spend some time studying the rules of meter for limericks.
No one seems to have addressed my question of what the ranking system is actually for - except for Shep sayng that it is an "interesting problem".
Forget about the rankings. There is more than enough information to figure out where you stand. You all got thrashed by Jules over Easter and that might feel depressing. On the other hand his results last weekend in Sweden show that he is right up there internationally. And if Jules can do it, then other Australians can too. Forget about your ranking and focus on that.
Shep - if you are bored at work today and need an "interesting problem" to occupy your mind, you might want to spend some time studying the rules of meter for limericks.
Apr 21, 2010 1:07 AM
#
A true poet, as I'm sure Shep considers himself, hardly has to worry about such suffocating concepts as "rules" and "meter". It's a limerick, it rhymes, and it's dirty. I see no problem with it.
Apr 21, 2010 5:06 AM
#
Shep:
yeah you and your rules plowright, you've been stuck in oppressive japan for too long ;) i'm disappointed you wrote all that and not another limerick... i'd like to see you write one about your man-crushes on australia's number one orienteers. eg
when winning, rob plowright, he loves you
(eat white rice, rob plowright, he can't poo)
if you drop your ranking
rob plowright starts wanking
the guy that pushed you to number two
how's the meter?
when winning, rob plowright, he loves you
(eat white rice, rob plowright, he can't poo)
if you drop your ranking
rob plowright starts wanking
the guy that pushed you to number two
how's the meter?
Apr 21, 2010 5:16 AM
#
Shep:
and robplow, i'm gonna answer your "question of what the ranking system is actually for" with another question... why publish results at all? in fact, why keep times? aren't we all winners, just for getting out and having a go? i see the ranking list as a bit of a result board for the year. obviously it can't be done like the Easter weekend with a cumulative time cause not everyone can run every race. so the results in each race need to be normalised in some way to give us our winners for the year. that normalisation is the interesting part from the mathematics point of view.
to be honest i personally don't care a great deal about my ranking, but i think that when i was younger it was a real motivation. sure, i wanted to win each race, but i also wanted to be consistent and get 5 decent runs on the board and improve my ranking. the list may have shown i was getting smashed by TDH, but jumping ahead of hoggster, blair, ecmo etc gave me huge boosts in confidence. it means nothing really, and as you say we don't need to use rankings to restrict entry to races etc. so, you're absolutely right, but it doesn't mean you're not wrong ;)
to be honest i personally don't care a great deal about my ranking, but i think that when i was younger it was a real motivation. sure, i wanted to win each race, but i also wanted to be consistent and get 5 decent runs on the board and improve my ranking. the list may have shown i was getting smashed by TDH, but jumping ahead of hoggster, blair, ecmo etc gave me huge boosts in confidence. it means nothing really, and as you say we don't need to use rankings to restrict entry to races etc. so, you're absolutely right, but it doesn't mean you're not wrong ;)
Apr 21, 2010 6:36 AM
#
that was a limerick - it just had more than 5 lines and didn't rhyme.
I don't care about suffocating concepts like that - I'm a true poet.
You have a secret admirer Shep - and she likes it when you talk dirty!
I don't care about suffocating concepts like that - I'm a true poet.
You have a secret admirer Shep - and she likes it when you talk dirty!
Apr 21, 2010 6:41 AM
#
My interest is that it is an interesting technical problem. I can't help my training in scaling back in my days as a psychology student. I really would like to know if anyone else is thinking about pair to pair comparisons for rankings.
I agree with you Rob. Australian rankings are not consequentially important. For that we should be thankful. Have a look at the knots the septics are tying themselves in given they have tied their rankings to selection.
http://www.attackpoint.org/discussionthread.jsp/me...
I also disagree with you Rob. For some individuals teh rankings are a personal motivation, and if that helps tip the balance to attending an event in a distant state, its a good thing.
PS to Shep. This time its the spelling.
I agree with you Rob. Australian rankings are not consequentially important. For that we should be thankful. Have a look at the knots the septics are tying themselves in given they have tied their rankings to selection.
http://www.attackpoint.org/discussionthread.jsp/me...
I also disagree with you Rob. For some individuals teh rankings are a personal motivation, and if that helps tip the balance to attending an event in a distant state, its a good thing.
PS to Shep. This time its the spelling.
Apr 21, 2010 7:04 AM
#
Shep:
oops, classic mistake! thanks for the tip. now it's a work of art ;)
yeah barrbarr i've been thinking about the pair-to-pair comparisons matrix as i mentioned in the thread on ev's log. we need some way of weighing up all the individual comparisons... you beat runner A by 5% but their ranking says you should beat them by 8%, so that says you ran worse than "usual", but runner B beat you by 2% while the ranking say you should be 6% behind them so that would suggest you ran better than "usual". add up all those comparisons for all the runners and solve to see whether your ranking should improve, stay the same, or go down, and by how much... simple yeah? i don't think the size of our fields means the order(n*(n-1)) complexity is a worry...
yeah barrbarr i've been thinking about the pair-to-pair comparisons matrix as i mentioned in the thread on ev's log. we need some way of weighing up all the individual comparisons... you beat runner A by 5% but their ranking says you should beat them by 8%, so that says you ran worse than "usual", but runner B beat you by 2% while the ranking say you should be 6% behind them so that would suggest you ran better than "usual". add up all those comparisons for all the runners and solve to see whether your ranking should improve, stay the same, or go down, and by how much... simple yeah? i don't think the size of our fields means the order(n*(n-1)) complexity is a worry...
Apr 21, 2010 7:13 AM
#
Don't add em up. use some matrix algebra to find the best fitting ranking. The one that minimises the errors. That part of it shouldn't be that hard. its the conversion of the results into a matrix that would need some programming. And some thinking about the form of the comparison measure. If you use the best 5 runs for each runner, they will probably be quite different for each of the runners in a pair.
Apr 23, 2010 9:27 AM
#
Robplow it seems likes a wank
Jennycas is up for a spank
If you're Oz number 1
It sure sounds like fun
For some it could drive you to drink.
Jennycas is up for a spank
If you're Oz number 1
It sure sounds like fun
For some it could drive you to drink.
This discussion thread is closed.