in: Swampfox; Swampfox > 2008-05-27;
| # Posted 2008-05-28 09:02:48 | |
| jjcote: | (By "results better", I assume you're referring to the sum of the times for the sprint, middle, and long, which isn't how this is calculated since it's felt that it overemphasizes the long and underemphasizes the sprint.)
The answer, of course, is that a) the fourth item in the selection criteria is the previous year's ranking, and b) since two people turned in dominating times in the Middle Distance, the points for that event got devalued so much that, except for those two people, hardly anyone* had the Middle Distance score being one of their top three. Except for people who didn't have a ranking at all. Like Leif. Which isn't to say that the system isn't flawed, it may be. The overall method of using numbers rather than opinions has not failed, but one can ask whether the actual scoring formula has shortcomings. The Rules do not specify how the Team Trials is to be scored, and the inclusion of the rankings in the mix was added after the current rules were adopted. *Specifically, three other ranked people had the Middle count in their scores: one who tanked the Long, one who arguably has a ranking lower than he deserves, and one who has a preposterously low ranking, presumably due to an insufficient number of ranked finishes. |
| # Posted 2008-05-29 00:56:10 | |
| Swampfox: | You can add up total times. You can use the total of the scores from each race. You can do head-to-head comparisons from each race. You can add up placings from each race, cross country style. Right there you have four different, completely objective, reasonable selection approaches that would have resulted in Leif being selected.
Sum that with the selection process used, and ask critically what it is about the current system that is better than any of the other four individually, or collectively as a group. Why is it? You can hand the raw results from the trials to 10 random people and tell them to make selections, and I have to think the odds are *really* good that Leif still gets selected. Nobody has explained--nor, in my opinion, can they explain-- how the calculation is improved by throwing in a line of data (the rankings) where even the "good" numbers are extremely noisy at very best, and where, in the case of this particular trials, for four of the top 10 team trialers, the numbers are objectively complete garbage. My challenge would be to run the selection process by a statistician and see what they have to say about the inclusion of the rankings as input of equally valid statistical significance in comparison to the team trials race data. Someone like, say, Ernst Lindner. |
| # Posted 2008-05-29 05:37:25 | |
| jjcote: | Nobody has explained--nor, in my opinion, can they explain-- how the calculation is improved by throwing in a line of data (the rankings)
Nor would I attempt to do so. I don't remember who added the rankings into the mix, or why. They definitely didn't used to be there. |
| # Posted 2008-05-29 08:19:50 | |
| jjcote: | A thought experiment: suppose instead of the rankings, the Team Trials had a fourth race (maybe something that would simulate some aspect of a relay). And suppose Leif wound up with a zero score for that race, because he DNFed. How would that affect your viewpoint? Next, suppose he DNSed. Then suppose he, as well as three others of the top 10 DNSed (perhaps because the race was on a Thursday, and they couldn't get there in time due to work schedules). One could argue that if they were serious about trying to make the team, then getting to the race would be a priority. On the other hand, one could say the same thing about getting to at least four days of A-meet competition during the year preceding the Trials, in order to have a ranking. I offer no opinion one way or the other as to what the preferred outcome should be under these scenarios. |
| # Posted 2008-05-29 09:30:57 | |
| Swampfox: | JJ, we could go through various thought experiments and it would be interesting and all that, but it doesn't address the real, already existing problem with the selection system, which is the problem of feeding garbage into a black box process. You know as well as I do the problem with that.
When you have a series of Team Trials races, and someone has been left off the team whom has clearly performed better in those races than people who were selected to the team, then you are effectively saying that the rankings number is more significant than the head-to-head race results at the Trials. That is obviously bogus! |
| # Posted 2008-05-29 09:34:45 | |
| jjcote: | Like I said, I won't defend the use of rankings for team selection. |
| # Posted 2008-05-29 09:37:20 | |
| Swampfox: | I understand. |
| # Posted 2008-05-30 07:15:56 | |
| coach: | Look at the woman's scores, and figure what the team would be without the rankings. In fact, figure what the team would have been for the past few years without the rankings........
You also might ask Leif what he thinks. |
| # Posted 2008-05-30 09:02:15 | |
| cedarcreek: | I've got no horse in this race, so I'm not throwing darts here...
What would you recommend? (I notice a lot of threatened manager types hide behind this exact question, so I'd add that not having a recommendation, i.e., just saying there is a problem, is a reasonable thing.) |
| # Posted 2008-05-30 09:04:31 | |
| Swampfox: | I don't know what your point is, coach. I do know that you're not explaining how a selection system is improved by using bogus data. |
| # Posted 2008-05-30 15:05:36 | |
| BorisGr: | Using the rankings vs not using the ranking is a totally different question from trusting the rankings vs not trusting the rankings. If we don't trust the rankings, then why do we have them at all? Presumably, they reflect some sort of level of performance the runners have shown over the previous year. Nothing was preventing Leif from getting a ranking, especially since the team selection process has been known for months in advance. Even without the ranking, he was very close to making the team, but got beaten out by Wyatt and Clem, who are both recovering from injuries, but who both have shown better results over the course of the past year, which is reflected in their rankings scores. I don't see any problem with the selection process, though obviously it would be improved if we had more "elite" runners racing head-to-head throughout the year, thus improving the robustness of the ranking score. Furthermore, I hope you would agree that Eddie deserves to make the team this year. However, he skipped over a control in the long distance, running 40 meters by it. He would have won the race by 4 minutes, but obviously, he got a DSQ instead. If the ranking score did not count, Eddie would be sitting at home during WOC. However, the fact that he made the effort to get out to some A-meets during the year and get a reasonable ranking was his insurance policy for just such an occurrence as the bad day he had during the long. I think that is one of the major reasons we added the ranking to the team selection process, and I think this was an excellent example of it in action. |
| # Posted 2008-05-31 00:55:42 | |
| Swampfox: | Cedarcreek, I don't think there is a stated mission statement or goal for the team at WOC (at least that I can find, and that's not a criticism here; I'm just pointing it out as preparation for my assumption), and so my working assumption is that the overarching goal of the selection process is to select the team that will produce the best possible results at WOC. There may be other goals too, but I assume that trying to get the best possible WOC results in the coming WOC is what drives things.
It used to be easier, back in the days when there were just two disciplines--relay and classic--and when the courses for those disciplines were basically about the same in terms of technical and physical demand. It meant you didn't have to think at all about trying to match athletes with particular disciplines. So, some recognition here that it has become (at least potentially) a harder task today. In designing a selection system, or even re-confirming a current process, you probably want to begin with some philosophical underpinnings and take consideration of the prevailing, broader circumstances of where US orienteering is (a system that might work very well for, say, Switzerland, might not work well at all here, and versa.) I think it's important to recognize there is never going to be a 100% perfect system. Any reasonable system is going to have strengths, but it will also have one or more shortcomings. There will always be tradeoffs in terms of things like ease of use, expense, objectivity, flexibility, and so on. What is not acceptable is a system with one or more crippling flaws. It doesn't matter if the flaws only manifest themselves occasionally or not. The current system is too badly flawed to produce acceptable results and there is no workaround that uses unreliable or outright wrong data (rankings number), which are fed into a black box system to output selections. The resulting selections may occasionally, or often, or even usually make it appear like everything is working as it should, but it's quite analagous to a broken clock, which may appear to be OK, depending on when you look at it. The fact that the time on the clock agrees with what you think it should be doesn't mean the clock doesn't need to be fixed. The process must be robust enough to produce good results not only when the circumstances are normal and expected, but also when ususual events crop up. The process must also not yield chaotic, large differences subject to small differences in the data. For example, the seemingly arbitrary decision by the Trials organizers of the Team Trials having the Trials being on one given weekend versus one weekend earlier or later, should not make it possible for a different team being selected (because of ensuing differences in the rolling rankings.) One piece of philosophy that has prevailed for a long time in US Orienteering--and, as far as I know, still prevails today--is that a competitor ought to be able to show up at the Team Trials and win, through performance over those several days, a spot outright. I've never heard of even one person in the community who disagrees with that concept (though of course there could be some.) There have always been several spots reserved for automatic selection. To me personally, as a once athlete, that was always a strength of the way went about things. But note that today, showing up at the Team Trials and winning *all three* days is no longer any guarantee of automatic selection, because of the pernicious effect of throwing rankings into the mix. It doesn't matter if you have a ranking or not--you can win all three races, and still not be selected. To me, philosophically, that just seems wrong. Especially when the team trials rules themselves explicitly state that the makeup of the team will be based primarily on the results of a team selection competition. What could possibly be more objective, relevant, and decisive than showing up at the Team Trials and winning all three races outright? I suspect it is not generally understood that someone can win all three races and not end up being selected. Another philosophical matter is whether or not you allow discretion in the selection of the team, and if so, how, and how much. Here, too, for as long as I have been involved in O', prevailing sentiment has been there should be some amount of discretion, and, as far as I know, there always has been at least some. In recent years, the amount of discretion in selections has been relatively tightly proscribed. My opinion is that having some discretion built into the system is a good thing, and another strong point of the way we have usually done things. It is a really good thing to have some latitude to adjust for things that you know will crop up from time to time, like illnesses, injuries, family emergencies, and so on. And it is just as important to have some latitude for weird things that will happen from time to time but which are utterly unpredictable. My opinion is further that the present process does not allow for enough discretion, particularly in the petitioning process. It makes little sense to have exceptions, for example, for athletes living overseas, but not, for example, for domestic based runners who for any reason whatsoever do not meet the US Championships results and rankings criteria needed to petition. When you unnecessarily restrict who is allowed discretionary availability to a team spot, then you are actively undercutting the premise that the driving goal is to maximize team results. Basically, if someone wants to petition, their petition ought to be taken. It's not as if the selectors have even been deluged with dozens of petitions, nor as if it need take more than a couple of minutes to separate out the one or several petitions that would have real merit and deserve real consideration. Adequate discretion is especially important in the circumstances where you have a thin talent pool and where the range of talent within that pool drops off rapidly. That's how you would describe US orienteering today. You really need to have some safeguards so that when freaky things happen, you still have some means for making sure at least your top one or two athletes end up on the team. A concrete example might serve the for kind of thing I'm getting at in terms of discretion. Suppose you're in charge of making the selections, and you have complete discretion to choose however you want, and on Day 3 of the Team Trials, who should pop up but a very fit looking Brian May, and he walks over to you and he tells you he's been training again, and he really wants to race at WOC. Then he goes out and absolutely demolishes the field in the Classic race. Well, I don't know what you would do, but I know what I would do. Brian is a very serious individual, with very impressive athletic bonafides. He's a real winner type. If he told me he was as fit as he was ever been, he had all his old enthusiasm back and more, and he just smoked everyone at Classic distance, that's all I would need to know. For sure I would want him on the team. You can't plan for a Brian May to suddenly reappear out of the blue. But you can build in some "give" in the system to allow for it. The current system has no sort of an exception like the circumstance I just described. So, some kind of more concrete recommendation? Well, if you say the US was going to send teams of 5 and 5, and you were happy with a "generalist" approach (not worrying about selecting for particular disciplines", then taking the top 3 men and women from the Team Trials races, and allowing the other 2 spots to be discretionary is an easy sort of default choice. Where the calls are close, be guided by the results from the Team Trials. Where an obvious top athlete is being left off and there are extenuating circumstance, give strong consideration to naming them to a discretionary spot. Trust in your selectors to do a good job just as you trust in your athletes to prepare well and to do a good job when they step up to their WOC Start. If you want to shape the selection process somehow to take into account the demands of the particular disciplines, then it gets trickier. Given the unusual degree to which US Team athletes are themselves involved in helping shape fundamental decisions about how the Team is run and so on, then it might well be best to discuss it at a Team meeting and see if any kind of overall consensus could reached, since there are so many ways you could go, and since it could affect choices about team size. |
| # Posted 2008-05-31 01:53:19 | |
| jjcote: | I suspect it is not generally understood that someone can win all three races and not end up being selected.
Particularly because, unless I've missed something, that's flatly not true. Under the current system, winning all three races, even with a zero ranking, would give you a score over 300. Assuming that two berths were given to people who submitted petitions, leaving only three spots, you'd need to have three people scoring higher than you in order to not get a berth. At most two people can having ranking scores > 100. That means that the third person would have to make up the deficit in races where you beat him. The exact scoring details get complicated, since each race is normalized against the average of the top three finishers, but unless I've made a mistake in my reasoning, there's a game of musical chairs going on, and even with pathological numbers, there's not enough room for three people to be ahead of you on the scoring list. You might not get the first spot, because, for example, if another competitor has a ranking over 100, and he comes in second place in all three races by one second, then his score will be slightly higher. But I believe it's impossible for you to be any lower than third place on the scoring list. Can you provide a scenario where this would happen? Where an obvious top athlete is being left off and there are extenuating circumstance, give strong consideration to naming them to a discretionary spot. I can name at least one time, back in the era when there were three automatic spots and two discretionaries, that a situation occurred that was very much like what you are decrying here. A competitor who was in top shape did well enough at the Team Trials that you'd think, from looking at the results, that he should have been on the team. But the selectors put someone in ahead of him, someone who he had beaten in both selection races, for unclear reasons. Complaints about this sort of thing is what led to the current system (there were calls for the Team Trials results to be used directly, with no provision for petitions at all). If we don't trust the rankings, then why do we have them at all? So we can send out patches? If the ranking score did not count, Eddie would be sitting at home during WOC. Not necessarily. If the scoring was the best two out of three Team Trials races, Eddie would still have been in third place on the scoring list. (Doing that, incidentally, would give the following order: Eric, Ross, Eddie, Erin, Leif, (Tom), Wyatt, Clem, Joe, Gerald.) |
| # Posted 2008-05-31 02:05:39 | |
| Swampfox: | Naah, J-J you're right. You could win two races and be very close behind on the 3rd and not get selected, but not if you win all three. I goofed on that.
No excuse for my mistake; I think I was thinking about how Wyatt won the Team Trials last year on a total time basis--which in most Team Trials years would have been an automatic slot--but wasn't selected. And I understand it was complaints about one or more situations like you described (controversial discretionary choices) that led to the evolution of the present system. As I stated, there are going to be tradeoffs with any system. In this case, the flexibility to be able to add top athletes has to be balanced with the risk that the selectors might make a bad decision. It's a judgment call as to where you think the balance lies over the course of many WOC selections. It would be fully legitimate to judge the risk of serious dissension within the team was not worth the possible improvement in WOC results. Cedarcreek asked what I would recommend, and as part of a comprehensive answer my recommendation would be some more discretionary choice be added back in. Whether to have discretion and how much, or whether to have any at all, of course has nothing to do with my primary, top line point that the existing system is badly flawed by cranking rankings numbers into the selection algorithm. Boris, I'm not ignoring you. |
| # Posted 2008-05-31 05:06:30 | |
| Spike: | A few observations...
...my working assumption is that the overarching goal of the selection process is to select the team that will produce the best possible results at WOC. There may be other goals too, but I assume that trying to get the best possible WOC results in the coming WOC is what drives things. I don't think your working assumption accurately reflects how selection processes have been designed. It would be interesting to get an idea of how well team trial results predict WOC performance. How often does the person who finishes 1st in the trials (by whatever method was used at the time) have the best WOC perforance? How often do they have the 2nd, 3rd, 4th, etc. best? One way of thinking about a selection process is to look at the very good orienteers who don't go to the WOC or who don't even go to the trials. Swampfox clearly would have helped the WOC results if he'd gone to the WOC at least some of the years between 1993 and 2004. But, he didn't run a WOC during that period. While that's not a flaw of the particular scoring system, it seems like it is a flaw of some kind (if the goal is the best WOC performance possible). Look at the US blue rankings through 5/27 and notice that 2 of the top 5 ranked US WOC-team eligible runners didn't even run the trials. Again, that's not a flaw of the scoring system, but it is some sort of flaw. |
| # Posted 2008-05-31 05:37:21 | |
| PG: | So many words, so much to disagree with.... :-)
I'll just point out a couple of things. If you want to shape the selection process somehow to take into account the demands of the particular disciplines, then it gets trickier. Given the unusual degree to which US Team athletes are themselves involved in helping shape fundamental decisions about how the Team is run and so on, then it might well be best to discuss it at a Team meeting and see if any kind of overall consensus could reached, since there are so many ways you could go, and since it could affect choices about team size. Guess what, that's exactly what we do. There was a lot of complaining from some folks -- though not from team members -- about the selection process in 2007, to which I responded that in the fall, after WOC was over, we would evaluate the process and see if any changes were called for. Which we did, first preliminary discussion via the team e-mail list (which is available to anyone), then with a serious discussion at the Team meeting at the US Champs at Quantico, and then via more e-mail discussion. We debated whether to introduce some provision to automatically select event winners at the Trials, but the very strong consensus was that we were not yet at the point where we thought that was a better system. We debated changing the rules to add more discretion, but there was a very strong consensus not to do that. We debated trying to incorporate a method to allow overseas based athletes to use their WRE ranking instead of a non-existent USOF ranking, but we were unable to come up with a satisfactory algorithm. We debated changing the rules for determining start orders, and this we changed to make the start orders for each event randomly drawn (which was an improvement, I believe). and to use a 3-minute interval for the middle distance as well as the long. We debated changing the time requirement for overseas residence needed to allow an athlete to petition, and reduced this to 6 months. And that was that. And most important, by the time we were done, as far as I can tell, all the athletes were ok with the system, and all understood exactly how it worked. There is no black magic. You count the 3 best of 4 scores. If you want a ranking score, you need to get to at least 4 days of A meets over the preceding 12 months. Something that anyone who is serious about running, and running well, at WOC should have no trouble doing. And, I repeat, everyone knew this. We also have very specific rules about how we determine how many men and how many women to take, and these get reviewed each year too. So we are doing exactly what your bottom line suggestion is. You just may not be aware of it. And I expect that we will have a similar process this year after WOC to take another look at the selection rules. Cedarcreek, I don't think there is a stated mission statement or goal for the team at WOC (at least that I can find, and that's not a criticism here; I'm just pointing it out as preparation for my assumption), and so my working assumption is that the overarching goal of the selection process is to select the team that will produce the best possible results at WOC. There may be other goals too, but I assume that trying to get the best possible WOC results in the coming WOC is what drives things. Here are our goals for the Team at WOC. !. Have as many athletes qualify for as many A finals as possible. 2. Beat Canada in the relay. 3. For athletes who do not qualify for the A final in an event, get as close to 15th as possible (i.e. 16th is better than 20th is better than 25th.... We don't yet have goals for performance in the A finals because at this point we feel that meeting the goal of making an A final should not then be devalued by not meeting some additional goal in the A final. As we get better, this will change. We also don't yet have specific placing goals for the relay, though that also is coming. All of this has been discussed and accepted at team meetings. All of this affects how the selections are made as to which individuals run which races at WOC. It also affects how we allocated funds in our budget. Just because you are not aware of things, do not assume that things are not happening. (You might be interested to know that we have other performance goals too, dealing with World Cup events and also the North Americans. And also other goals not related to specific events. But that is a separate subject.) ... on Day 3 of the Team Trials, who should pop up but a very fit looking Brian May.... The problem with this example is the premise is faulty. If Brian May, fine fellow, fine athlete, was training again and wanted to make the WOC team, then there is no doubt that Brian would know the rules, that he would show up to run all three days of the Trials, and that he would probably, since he was training seriously and wanted to make the WOC team, also have been to enough A meets to get a ranking score. What you describe just wouldn't happen. We spend a good bit of time trying to make the system work in a way the athletes will accept it. The rules are known by all and as far as I can tell, accepted by all. Overall, I think our process works very well for our circumstances. I am baffled by the trashing it takes, but it's a free country. |
| # Posted 2008-05-31 12:31:42 | |
| Swampfox: | Boris, the rankings have been around a lot longer than their use in team selections, and they weren't designed with that intent in mind. The rankings are neat to have and I imagine a lot of folks have fun seeing how the rankings change as updates are made with races rolling on and off and so on. Probably some folks use them as a motivating tool to train (to try to get a better ranking). But I like J-J's answer best (we have the rankings so we know who to send the patches to.)
I wrote a bit about the usefulness of the ranking number last year in the discussion following the 2007 Team Trials http://www.attackpoint.org/discussionthread.jsp/me... and basically I would reiterate the same types of things. When you look at a rankings number, you can presume it says something about how a runner has performed and make some inference about how they might perform in the future. And, most of the time for most people, you probably won't be too far off. It does correspond roughly to some kind of ability to perform. But for some people some of the time the rankings number may not even roughly reflect their ability level. There's no way to just look at the raw number and know if it's a good reflection of someone's ability or a poor reflection. You can't tell how many races went into that number. You can't tell if the races were all from last fall or all from the last few months, or a mixture. You can't tell if someone prepared and rested carefully for each race or if they just took them in stride as part of their training and did nothing special at all to get ready. You can't tell if someone raced injured or sick. You can't tell if someone only raced in races in terrain types they were good at and felt comfortable in or if they raced in unfamiliar terrain types or in terrains where they knew they weren't good. You can't tell if someone was a very erratic racer with results that were either good or bad and nothing in between, or if their results tended to be very steady and even. You can't tell if their race results were trending better or worse or if there was any trend at all. There's all kinds of stuff you can't tell, and so you don't really know what that number means unless you look at the numbers (the individual race results included) behind the number. For rankings numbers you do think are pretty good, how good do you think they are? Can you be sure they're within, say, even 1% of a person's "true" performance/ability level? I don't know. That looks like a pretty tough assumption to make. Would you tolerate a timing system that was only accurate to plus/minus 1%? For comparison, at last year's Team Trials, the separation between 4th and 5th place--being selected or not being selected--was less than 0.3%. Given the current selection system, of course I agree that anyone who seriously wants to make the team would be well advised to not only go to enough races to get a ranking, but to additionally do everything they can do within reason to get the highest ranking number possible to take to the trials. But once everyone has shown up at the Team Trials and has done their best at those trials, I think asking: "Well, if so-and-so really wanted to be selected, why didn't they take the trouble to get a ranking?" is the wrong question. There could be any number of reasons--good and bad--why someone doesn't have a ranking. I think the better question at that point is: "With the information we now have, which runners do we think will do best at the WOC?" In looking at the results from the Trials and observing that Wyatt and Clem were injured and giving some consideration to their prior results, and doing the same thing with Eddie's skipping a control, you're adding in some subjectivity. You're thinking exactly like a selector might and that's fine. It also looks like you're subjectively looking at the outcomes and from that concluding the selection process is sound. I disagree with that line of backwards looking analysis, if that's what it is. Do you see no problem with a selection system that selects Wyatt when Wyatt is injured (and has been for quite a while now, right?) and when he finished fourth on the scoring list after rankings results are taken out, but which did not select Wyatt last year with the rankings included--when he won the Classic race by a good margin and the overall Trials (total time)? Hopefully Wyatt gets completely better very quickly, and will be 100% when WOC rolls up. However, when you consider it all, it looks like he was a much better selection last year than this year. Would you disagree? |
| # Posted 2008-05-31 16:41:53 | |
| PG: | So many more words, and still faulty reasoning.
For rankings numbers you do think are pretty good, how good do you think they are? Can you be sure they're within, say, even 1% of a person's "true" performance/ability level? I don't know. That looks like a pretty tough assumption to make. Probably not. But continue that line of inquiry. How good a reflection of a person's "true" performance/ability level are the results of each of the three selection races? Within 1%? You've got to be kidding me. People's performances vary. Assuming that one sprint perfectly reflects their sprinting ability, and one middle their middle ability, and one long their long ability, just isn't true. Do they do a better or worse job at predicting than the rankings? I think it could be either way. But it think it is faulty reasoning to put the three individual results up on a pedestal and relegate the rankings to a trash heap. Do you see no problem with a selection system that selects Wyatt when Wyatt is injured (and has been for quite a while now, right?) and when he finished fourth on the scoring list after rankings results are taken out, but which did not select Wyatt last year with the rankings included--when he won the Classic race by a good margin and the overall Trials (total time)? More faulty reasoning. First, total time doesn't mean squat. It's three separate races at vastly different distances. Second, people's performances reflect not only their ability, but also the circumstances. Sometimes you play it safe, sometimes you go for broke. Wyatt needed a terrific run on the last day in 2007. To his credit, he did it. But Eddie and Eric and John just needed solid runs. Which they did. But to draw the conclusion from that one race that Wyatt was our best at the long distance is just not good reasoning. But once everyone has shown up at the Team Trials and has done their best at those trials, I think asking: "Well, if so-and-so really wanted to be selected, why didn't they take the trouble to get a ranking?" is the wrong question. There could be any number of reasons--good and bad--why someone doesn't have a ranking. I think the better question at that point is: "With the information we now have, which runners do we think will do best at the WOC?" And that's a question that can't be answered in its entirety. Meaning, you can be pretty sure about one or two or maybe even three of the runners, but picking the exact five that will do the best at WOC is just a crap shoot. You know that and I know that. If you ignore the rankings in 2007, Wyatt gets in but John Fredrickson gets dropped because of a bad run in the middle. Is that a better team? If you ignore the rankings in 2008, then Leif gets in ahead of a temporarily injured Wyatt and Clem? Is that a better team? Leif may have potential, but none of his runs at the Trials were anything special? Would he do better at WOC than Wyatt or Clem in better condition? Who knows. And so at the end of all this you have to consider a factor that you seem to ignore, and that is the fact that we are talking about a "Team" here, and there is a huge benefit if people think like a mutually supporting team rather than just individuals. And having a selection process that has team members coming away from it thinking that it is fair is very important. And that is what we have. |
| # Posted 2008-05-31 22:53:45 | |
| Swampfox: | Hi PG (or, for certain seasons of the year, Mr. Taxman),
Why aren't you off rogaining training instead of reading this? Very dubious priorities! ; ) I was not implying or meaning to imply that team members are not actively part of helping make decisions about the selections process in particular or team matters in general. In replying to cedarcreek about what I might recommend, I was saying in so many words that right now we have a generalist type approach, that I would be inclined to stick to that, but that there could be merits to a disciplanry approach too (choosing by discipline) and that, not having any particular format for that in mind, I would start by talking it over with the Team. In saying there are merits in having more discretionary flexibility in the system, I am not arguing that the way things are being done now are either wrong or unfair. Or that my recommendation would necessarily be better. I would do it a little different, and I gave some reasons for that. I imagine if other people were asked, they might well have other, different recommendations. The Team meeting you mentioned was great--I was there--and you did a great job of leading it, just like you always do. Anyone who wanted had a chance to speak up and weigh in, and I agree fully that it went just the way you say it did, and that at the end the athletes there had no problems with the selection process. Before responding to cedarcreek, I took a quick look around to check on US Team goals. I wondered what someone might find if they were curious and had no idea. I figured they might start by looking at Team stuff at the USOF website. When I checked the Team page there and followed several of the most likely links, I didn't see anything about WOC goals. If I were re-writing that intro, I would write instead that I had done a quick search for the goals, didn't find any, and then had gone on to my working assumption (that the main goal was best possible WOC results.) My wording "I don't think there is a stated mission statement or goal for the team at WOC" was wrong and I apologize for that. My point about Brian May was not that it was something he would do, but that it was something that *could* happen, as an example of a scenario the selection system would be confounded by. Of course it's an extremely low probablility type event. Even though this one scenario has very low probability of happening, there are many, many other low probability events that *could* happen, and that it becomes almost inevitable that one or more of them will happen, considering the possibilities collectively. In making some of these recommendations (really in a fairly hypothetical way in response to cedarcreek) expending, as you point out, so many words, and writing about a number of different things that may or may not be related to my central point, it would not surprise me if someone trying to follow what I was arguing was confused about it all. For the record, I want to make it clear the one thing I am being highly critical of is including the rankings in the selection algorithm. Maybe it is statistically sound, but I don't believe it is, and nobody has yet shown that it is. I was critical last year after I looked at the Team Trials results and thought and wrote about it some then, and then went a long time without thinking about it at all. Now, after looking at this year's Team Trials results, again it strikes me that there is something flawed about using the rankings number, so I've thought and wrote about it some more. I didn't begin by talking to anyone at last year's Team Trials or this year's about what they thought about the selections, I don't know what any of the athletes--chosen or not--thought about the selections, nobody wrote me on the side to point out anything--nothing like that. I just looked at the numbers. And, like cedarcreek, I don't have a horse in this race. I don't care which of the particular athletes are selected to WOC. If someone had handed me the raw data from the Trials along with the selections made, but identified the athletes by number instead of name, I would have had the same reaction. Maybe I'm wrong. But I can think of many ways in which those numbers can be flawed, and already we have had back-to-back years where the selection process has served up some interesting results, when compared with the raw Team Trials data. As for the Team itself, the leadership, the athletes, the conduct of Team affairs, the transparency in the way things are done, the striving for fairness, the way the whole thing has made progress over the years, and so on, I have nothing but praise. The highest praise. The Team is unmistakably better off today than it was 5 years ago, 10 years ago, whatever, and it's unmistakably moving in the right direction. I sure hope nobody has confused my criticism of the selection process with criticism of the Team itself. The above is for your first post. For your second post, I agree in general that you make good points for each issue you address. I still assert that there is a higher level of accuracy implicit in clock timing at selection races than there is in any ranking number, and I will suggest it is at least an order of magnitude more accurate. For the issue of a selection system that selects an injured Wyatt who did OK at the Trials--but not great--but which did not select a healthy Wyatt who did have great race results at last year's Trials, with the reason being because of the inclusion of ranking, you didn't say whether you saw any problem or not. Do you really maintain that there's nothing odd about that? Would you really argue that Wyatt this year is a better WOC selection than last year. Of course you are right when you say in the current selection process total time means squat. Would you not agree, however, that it is a really good overall fitness indicator (here, "fitness" in terms of overall ability to perform in competition, meaning the combination of conditioning, technique, mental attributes, and so on.)? If a really good fitness indicator is flashing green while the selection process flashes red, I think it's at least worth thinking about. Don't you? I do not argue that the Team is better when you substitute Wyatt for John, or put in Leif ahead of Clem and Wyatt. That's all very subjective, and I have no more idea than anyone else whether that would be better or worse. I only argue that using the rankings number as an input--in these cases, decisive input--is flawed. I absolutely am not overlooking or discounting your final point (the group being a "Team" with huge benefits from thinking like a mutually supporting group, with team members thinking that the processes are fair) and I agree with you 100%. I will leave this with one other oddity for you or others to consider, if you choose. I believe it's the case (someone would need to double check the dates and the calculation) that if I had shown up at last year's Team Trials, and if the Team Trials had been moved up by about 2 weeks, then if I had walked around the courses all three days, then a different team would have been selected (at least one runner would have been different.) Why? Because of the inclusion of the rankings number. My number was high enough that it would have changed the scoring values for the rankings numbers, with the effects rippling through the selection algorithm. |
| # Posted 2008-06-01 04:40:01 | |
| feet: | This last point almost reared its head this year in a different form. Tom Hollowell placed in the middle distance and this had the net effect of reducing the scores for all other runners by around 3-4 points. It could easily have been crucial in selection.
However, as one who's just missed out on the AUS team, I feel that those on the margins of selections have no real right to complain anyway, and should just get themselves into a position where the vagaries of selection scoring algorithms aren't going to worry them. Since the last selections are essentially crapshoots anyway (switching the fifth best runner with the sixth best doesn't affect much since they are essentially as good as each other anyway), who cares? I'd add that I like that the ranking score is included, in light of this, not because it picks a better WOC team but because it encourages serious competition at A meets. Which is good for orienteering in the US. |
| # Posted 2008-06-01 06:49:24 | |
| j-man: | This thread is even better than the water one!
I don't have anything to say at this point, aside from the fact that this process is objective. We've used subjective ones in the past. I recall 1999 when I think I finished 7th in the TT results but the 4th and 5th athletes selected for the WOC team finished 9th and 10th in the results at the TTs. That didn't make much sense to me. But, it was permissable under the system at that time. Incidentally, Mr. Gagarin finished 5th at those trials. Should PG have gone to WOC? Should I have? I surely don't know. I will say that feet's (succint) comments, as usual, are worth noting. |
| # Posted 2008-06-01 07:34:36 | |
| j-man: | The more I think about it, the more this discussion seems like Alice in Wonderland rabbit hole stuff.
I guess I could expend a lot of words on this, to argue either side of things. But not from a position of moral or intellectual superiority. Maybe these words are being generated for entertainment value, which is ok. As a normative or prescriptive exercise, it is untenable. Last year is an interesting example. Then, according to my calculations, without rankings, Wyatt would have finished 4th in the trials, I would have been 2nd. Wyatt had a great run in the classic, but still Johnnie beat him. Johnnie, however, had the lowest combined score of the three trials disciplines. Do you stick Wyatt right ahead of me, or ahead of Johnnie? Somewhere else? What is the point of all of this? It seems like number games to me. Idle buzzing. Chatter. One could argue for multiple permutations. One can complain about inequities. Or, paraphrasing feet, you come to the trials and lay the smack down and leave no doubt about your wherewithal and merit. |
| # Posted 2008-06-18 19:38:12 | |
| ndobbs: | thanks boris for the link to this :)
a couple of points from the outside... 1) In Ireland event winners get selected. This is a good thing. Anybody capable of beating everyone else in the country is good enough for the team. Also there is a huge difference between sprint and classic. 2) The fifth selected runner probably won´t be challengin seriously for A-finals these days (I´d be happy to be proven wrong), and won´t be much different in standard to the 6th or th placed person. If your goal is A-final, win the trials... 3) I would be in favour of alowing some discretion for many of the reasons given above, but at least with the automatic system, wars won´t start. |
You must be logged in to add a message