Discussion: SI-Box Time Drift

I've been running events with SI controls for 13 years and never even thought about syncing them except after the semi annual time changes. However at our last event Boris G had a negative 8 second leg, and in the process of correcting it I discovered the 20 controls, that were synced to UTC about 10 weeks ago, ranged from 12 to 68 seconds fast. It was fortunate (or unfortunate) that a large and small error conspired to give Boris the -8 second time that made me aware of this. I posted an article with the time errors for all the controls on the BOK website.

This is not atypical. My club synchronizes controls before every event. Over a period of a week or two, drift is usually insignificant. If nothing else, the start and finish units should certainly be sequenced for each event, especially for larger events which might use multiple start or finish units.

Boris should Just Slow Down...

We sync our controls before every event to reduce time drift but do not adjust our control clocks for the time change. Is this necessary?

I rely on the sync master. 5 minutes to sync all the controls a day or two prior to the event and then there are not issues. I do not recommend using a computer to sync through re-programing.

I use the sync master to sync all the controls the day before events I host.

Before every event and off the SI Master which is synced off the computer which is getting its time from GMT -5.

I recommend syncing after time change so if units are read out for some reason, the time is not an hour off. Bobby was at the 10th control at 10:09. No, he was there at 9:09, or was it 11:09?

Could this be Boris hacking into the hardware?

I read in NYTimes Russians hacked our water, sewage, and electric systems, they can even shut power plants off at will.

MNOC sees drift of up to 8 seconds per month (both fast and slow) . We use the SI Timemaster to sync the units about once per month and always synch them the day before when we have a) sprint or other possible very short legs b) rogaines ( where we often have sprint to finish within the allotted time (and a heavy penalty for being late by even part of a minute) and c) very important events. We use GPS time to synch to.

Good luck if your setters read the article! We ask setters (via the setting manual, which most don't read) to synch bricks before each event but most remain blissfully unaware of the need to do so, which can cause headaches (especially uploading results to AP because it won't accept negative splits).

I once ran all four courses at a sprint event with two finish bricks and tested the bricks by using each one twice. The result was one brick twice came out two seconds ahead of my own recorded time, the other two seconds slower. This would have been unacceptable at a championship event (we fixed the problem for future events by telling setters to use just one finish brick, which seemed easier than getting them to synch).

@ Sherlock - up to you whether you worry about whether the units are synced to local time (before or after seasonal changes). The important thing is that all control units are synced from the same master at the same time (masters can drift too). If you're using controls from multiple clubs or syncing some at different times, then there must be coordination...

Two (or more) finish bricks is asking for trouble. We had a case not too long ago where they set up two finish bricks for a mass start event (a relay) and they were ~3 seconds out. So many finish placings were reversed, particularly obvious when a person passes someone in the finish chute but gets placed after them in the results.

Reminds me I did ride in a Aus Champs MTBO event once where I caught the guy who started two minutes ahead of me (timed start, no brick). He proceeded to follow me most of the way around the course. At the finish there were two bricks that we punched at the same time but his result was only 1:58 behind mine. I did point it out to the organiser at the time because I picked up on it immediately but nothing was done.

Synching bricks can also be a problem if there is no start brick and the start clock time is taken from somewhere other than the master brick. Even if bricks are synched, the start clock can vary significantly from the stations.

To emphasize what has been stated: If controls are not synced before each event you are putting the results of your event at risk. This should be a primary task for all course setters.

Why are these units so poor at keeping accurate time? I mean you can buy digital watches that keep accurate time for years for a couple of bucks.

I don't know this as a fact, but I strongly suspect that the crystal oscillator in the SI unit is uncompensated. That would certainly match what I have observed working with SI timing, a drift of about 15 sec per month. Many cheaper consumer wristwatches use something called inhibition compensation to improve on this.

https://en.wikipedia.org/wiki/Quartz_clock#Accurac...

Also keep in mind that when electronic punching was first developed it was not intended to time the race itself. It was indented to remove needle punching. That it could provide splits between controls was a bonus, but that didn't need high precision timing. I still remember that the first couple of years after Emit punching was introduced in the district I lived in Sweden, the timing of the race itself was still done like before. Still today the timing at the highest level events (WOC) is done with a separate timing system.

But even if SI used compensation to get better accuracy you'd still want to sync the units somewhat regularly anyway. Certainly if you had multiple boxes on the finish line you'd have to. Even with inhibition compensation a watch will drift a few seconds per year.

That's interesting Rob. I've been wondering why my Garmin watch (time comes from a satellite I would think) is several seconds away from the time pips on the radio.So first - who's correct? I browsed https://greenwichmeantime.com/time-zone/pacific/ne... just now and think my watch is a clear 1sec ahead. Could there be a 1sec lag in a website delivering page refreshes to my computer?

But anyway the radio is rather worse than that, maybe someone just pushes the time-pip button when they are about ready to read the news?

Well, a few sec diff between, for example, satellite, internet and radio time is a different question entirely - so long as neither drifts they would make a perfectly fine reference time for an event.

The question is the considerable drift in the si units. Sure, historically split times might have been seen as just an added bonus but these days they are as much a part of competitors' expectations as the finish time. It might be reasonable to expect to have to sync the start and finish units for each event to ensure the absolute accuracy of the overall times but I would have thought you could expect normal control units to keep time well enough to be able to provide split times accurate to a second or two without having to resync every month or so.

The reason Emit events had to have a separate timing system was because the original system was designed solely as a punching system. The clocks in the emit system were in the runners' "brick" or stick not the punching unit - so effectively every runner had a different clock (which is why you can't use it for overall time) . I am not sure how it works now - presumably they have developed start and finish units that have their own clock which is set to race time.

Having a clock in the runners' sticks which is used for split times does mean you don't have to worry about unequal drift in the control units - even if there is drift in the runner's sticks it is not going to be noticeable over a single course.

To get better accuracy than the 0.5 s/day you get from uncompensated oscillation, SI would have to implement inhibition compensation logic, plus do a factory calibration for each unit in production. Given that units used to time the race anyways must be synced to ensure 1 sec timing it might be hard to get SI to do this just to get more accurate splits? Instead they opted for the time master solution that allows you to sync all units for an event in a few minutes.

But I certainly agree that a better clock would be nice. If you do need multiple start/finish boxes, which many larger events in Scandinavia need, then you really have to sync the moment before the race starts. One or two days before the event is not good enough, then units can be a few seconds off already.

I am not sure how it works now

It works the same way as before. We usually have online controls at finish line, wires to computer (or tablet). Computer's clock is used for finish time. Splits are taken from brick, calculated backwards from finish punch. No start punching, you start when you hear beep. Organizers need to set two clocks right, start clock (these days often app in a tablet) and finish clock (computer). Punching equipment never needs any syncing or programming.

Wristwatches also have the advantage of being strapped to an object that has an active temperature regulation system (a person), and even when they are not, they're often kept indoors. SI boxes are sitting out in the open where the temperature can vary considerably, which affects the crystal frequency.

Punching equipment never needs any syncing or programming.

Yes I always thought that was the great thing about EMIT - the simplicity of the system for the organizer/course planner.

So Jagge, what are the products you are using? I'm aware of radio stations for Sportident, but what are 'online controls'? Do you mean radio stations? In Australia (which doesn't use Emit), the radio controls are really only used at major events, certainly no-one goes to the trouble of setting them up for a minor event, and probably even most States' Championship events would not have them. In my state we would only use them if we have a Carnival including the National Championships. For our other events we once used two finish stations (especially at sprint events), but after complaints we reverted to a single finish station. Now, with Covid measures we are returning to two finish stations.

So how costly - in $$, time, and knowhow - is it to have the online finish you describe.

Online unit is a sort of regular unit but with wires and a connector plug. Serial port I think. So you can connect it to something that transfers the data over radio or web. Or conect it directly to your laptop at finish. I think it's twice the price of a regular unit. Making it work as intermediate conrol over radio may require some knowhow, using it at finnish line is standard procedure, you connect it to yout computer (I think serial to usb adapter is needed it it really still has serial plug) and I think you need to select the correct com port in result app or something.

There is "joker" units available you can program as any code. And those cost only 5 euros or so more than regular unit. But those are rare around here. After all, the code in unit should really be just internal code anyway and not the code in CD. So why all the work with programming if you can really place any unit there as long as you know the code. You just convert the codes to CD ones when you diplay/print out split times.

I havent been working with sresult systems for quite some times, tehre must be people her who know better. Just correct me if I got something all wrong here.

This guy is an expert: http://o-lynx.com/

Now, with Covid measures we are returning to two finish stations.

Wut?

@pi There are temperature compensated crystal oscillators which cost a few dollars per unit, and have accuracy less than 5ppm (which is < 0.5s per day). DS3231 is popular among the Arduino/mbed crowd, and it's a 2ppm device.
But I'm not sure if the SI stations really need it - synchronize SI units a day ahead unless there are really short legs, only the start/finish has to be done really close to the race, with the same time master station.

0.5 sec/day is still 15 sec/month. Worst case, if you have two stations drifting in opposite directions would be twice that.

one big reason for using such plugged Emit unit as last control is the announcer gets to see the result right away, not minute later after download.

Yes so that they can incorrectly announce the winner, who after download is discovered to have MPed (this happened to me at a MTBO in Czech last year - I was in 1st and a 'new winner' was announced, which made me sad, but he subsequently downloaded and MPed).

@simmo - Jagge is quite definitely talking about Emit, and probably "classic" Emit (the bricks that never fit without doing the Emit dance at each control). My understanding is there are some limited radio solutions available for Emit classic, but for Jukola etc they go back to RS485 or current-loop protocols over two-wire connections running multiple kilometres put up by Finnlands largest fitness studio (as the advertising for the Finnish military proudly states). Because the clock is in the brick, there were always stories of brick-clocks running away, especially with slowly dying batteries, sometimes even running at only half real time, therefore at local meets with timing by brick some superhuman running times happen - this is why timing by brick is not allowed for any "real" event.
(And to possibly jog Jagges memory or for info for anybody interested, the many serial lines are very often connected in a "star" topology in the forest using special "concentrator" boxes, for example the legendary "Sunjo Samlingsboxen", see http://hem.bredband.net/sunjo/ for details).

@gruver - note that GPS time is offset from UTC time by 18 seconds because the satellites don't know about the leap seconds introduced since GPS launch in 1980 - see http://leapsecond.com/java/gpsclock.htm for details. I imagine (read: don't know) that Garmin firmware compensates for leap seconds known at time of release, but older Garmins might be missing most recent leaps (last leap was 2016) and therefore be off compared to your radio.

@various - regarding SI quartz. They are, lets say, partially compensated. The microprocessor (MCP430 series) includes an (uncalibrated) temperature reading. This reading is used by the SI firmware to look up a compensation value in a predefined table. That is applied to the (uncalibrated) quartz timing. Adding in that the one same firmware is running on BSx7 stations from 2004 and 2020 and quartzes do have some age derating and some stations do have quite a hard life (especially the infamous series of BSx7 with black housing in sunshine), you will certainly get some drift over weeks. Normally, drift during competition for stations synced on the day before should be absolutely insignificant, if you're reeeeally nervous during long TioMila relay you might timemaster the finish stations sometime during the long boring hours of the night.

@tRicky - a good commentator will only announce a potential new leader, keeping the suspense up until the download (and the mention of this will possibly encourage the download crew to rip the SIAC off the panting or collapsed athlete to download it faster), but yeah, mistakes happen.

Time drift in the BS11 units until the latest firmware was huge... I always time synced units on the day of a gravity enduro event. That I felt is best practice.

@Toph, that is correct, the BS11-firmware 3.xx is a complete rewrite compared to the BS7/8 series and did not implement the compensation lookup table I described above. With the newer 7.xx that was added.

Not always possible for events (particularly bush) on the day. Day prior is most likely outcome or for bigger carnivals, possibly two or more days before.

Ouch, this is a discussion I should probably avoid for a few days, but accurate timing and clocks is a big interest of mine. :-)

I've been a member of the "NTP Hackers" team for 20+ years, we develop and maintain the Network Time Protocol which is how _all_ computers keep their clocks synchronized. I've had specialized GPS timing receivers just as long, they provide time ticks every second with 10-35 ns RMS errors.

Any GPS with non-broken firmware has to pick up the current time with approximately 10 ns precision, since 10 ns is the same as 10' or 3 meters.

The offset between the monotonic TAI clock used by GPS and UTC is given in the full list of messages transmitted by every satellite approximately 4 times/hour, but there used to be an issue every 20 years (actually every 1024 weeks) when the 10-bit week counter in the basic navigation message would wrap around, but modern chipsets have fixed this, i.e. by using firmware build time, TAI/UTC offset tables, saving last startup/shutdown time in local flash memory etc. At the same time the GPS system have also added 3 more bits in the extended (every 13+ min) message so now wraparound will only happen after 160 years.

The key to wrist watches working as well as they do is (as noted above) is the fact that they spend almost all the time on the wrist of the user, so a very stable temperature environment.

EMIT is far better than SI for an organizer, but otherwise both do well enough as long as you have (as required by the IOF and their Senior Event Advisors) a separate timing system for start and finish in world cup/woc level events.

Terje, does that make you a Time Lord?

Terje, I am sure you didn't mean to be as ambiguous as I read your last sentence - nit-picky as I am I first read you saying IOF mandates separate clocks at start and finish :) I'm sure you mean IOF requires a timing system separate from the punching chip. Which, at least in terms of SI AIR+, is fulfilled by the clock being in the (BS11-loop) SI-station rather than the SIAC. And interestingly, while in past years the IOF required a timing system to record the start time at WOC sprint, even that requirement has been dropped in most recent years. So it's minimum two independent timing systems at finish only, at the moment.

@jSh: We have to look at the spirit of the rules here, and that is that the system used to time all participants should provide a fair record of how fast they ran, so this is why timing start-to-finish has to be separate from the punch control, right?

Personally I would be perfectly happy with multiple start and/or finish gates, as long as the organizers can provide a full log of NTP (or directly attached GPS) measurements between those systems, covering the competition time period. I.e. this would effectively be a single logical clock even if there are multiple hardware clocks: When they are all synchronized with each other and with UTC to the 10 us or better level, then they are for all intents and purposes _one_ clock system.

I am always happy when I get to the start area of an event and can see and hear that the start beeps are in fact perfectly synchronized with my Garmin watch, when the organizers can do this right they are probably doing the rest OK as well. :-)

@origamiguy: Yes, of course!

@Terje, I fully agree timing should (actually must!) be a fair record for all participants. While I am no NTP guru, I do appreciate organizers who take the timing aspect of orienteering races seriously. However, in the nitty gritty details of the IOF rules (and certainly in their implementation at various top-level events in all three "fast moving" IOF disciplines) I have in the past seen some slack.

For example, the current IOF foot-o rules for the start (22.x) don't require any form of recording the exact start time of each athlete, even just for the purpose of (dis)proving an early start. In sprint races, a second may be decisive. In ski-o, camera recording of mass- and chasing-start races is mandated, but for individuals, the good old hand-on-shoulder practice is still permitted - FIS rules have abolished that for me-too-reasons and also a hand can possibly be biased by the federation of the athlete it is resting on.

The finish rules (23.x) also have some loop-holes. 23.6 says the timing "shall measure times of competitors in the same class, relative to each other" - strictly speaking this doesn't mean the clocks must be synced to UTC, the measured time could differ from real time, it just needs to be correct relative to each athlete. That is fair, but in the age of live-streaming and GPS-tracking it looks unprofessional. And 23.5 opens up no less than 4 different options for the exact timing procedure. The top athletes have all learnt the "superman" finish with outstretched arm - well, apart from Tove doing her usual collapse on the line (or, WOC 2018, managing to do a limbo slide under the light-beam). At least up to World Cups, timing by SIAC crossing the loop antenna as primary timing is accepted practice.

And to get back to topic, syncing that finish loop antenna to a precise real time source on the (each) competition day is definitely recommended.

We always change the timemaster to local time when the time changes for daylight saving time. This avoids big offsets for routegadget and other similar programs that rely on the runner's GPS watch matching the SI units. Some programs handle the offset well, and others don't. I haven't been able to get the clocks set better than about 1 second, but that seems to work pretty well.

The thread has moved on but I have to apologise for my criticism of radio time pips. I have traced the noticeable discrepancy with my watch, to taking my radio feed through my computer. When I listen to my radio thru an actual radio there's no lag. Sorry RNZ.

@gruver: All moderne transmission mediums (coax/cable/internet/DAB radio/whatever) which all employ some form of compression, will automatically also introduce some latency, i.e. delay.

Here in Norway our TV channels are 6-10 seconds (typically 7) behind real time, which means that the onscreen clock which counts down to the 1900 daily news is off by the same amount.

Back before common view GPS measurements you would use a full analog TV channel for a week or so just to compare two atomic clocks some distance apart, but that only works with analog, and only when you control pretty much everything.

The only timing source I really trust these days are the PPS (pulse per second) output from my Stratum 0 NTP reference clocks, but for Orienteering a GPS watch which is currently synced to GPS, i.e. doing timing, or has done so shortly before, is good enough.

Or the so-called "atomic clocks" that synchronize at night from a radio signal tied to an official time reference. Radio propagation delay where I live is probably about 3 ms, which is more than good enough.

@jjcote: A fixed 3ms radio delay is easy to correct for, it is harder to handle variation in radio paths between day and night.

And _no_, 3 ms is not "more than good enough", it is just more or less OK.

In the NTP world infinity is defined as 128 ms, this is the point where the protocol throws up its hands and say: "I'm outta here! Reset the local clock and all clock drift parameters and try to start from scratch."

We also have a built-in limit for the maximum time drift we're prepared to adjust for, this used to be 100ppm, but had to be increased to 500ppm to handle badly broken clock hardware in many systems.

500ppm is +/- 43.2 seconds/day, many PC-class motherboards have clock crystals that are significantly worse than wrist watch crystals, so we expect and handle easily anything up to half a minute or so per day, with spare room left over for the hybrid frequency/phase offset control loop.

What I'm referring to is that 3 ms is more than good enough for a situation where we're timing people to the nearest second. The likelihood that it would result in two people ending up in the wrong order in the results is much smaller than the many other random factors in orienteering (assuming something like two finish units that are out of sync by that much).

@Terje: Are you aware of this little $40 (US) GPS module from Adafruit?

It has a one-pulse per second (1 PPS) output. On the latest version (Ver 3) with the "H" rev module, the datasheet claims 10ns (nanosecond!) accuracy on the PPS output with synced satellites.

I've got one of the older ones. Obviously the modules are cheaper, but they're surface mount, and this breakout board makes it easy to use.

I've also got several old GPSs (from the early 1990s) with 1PPS outputs.