Training Log Archive: edwarddesIn the 1 days ending Aug 29, 2014: |
| « | » |
| » now | ||||
| Fr |
Friday Aug 29, 2014 #
Note
Analysis of ROM (Rogaine Manager) from Tak-Soft when reading SI-6 cards.
ROM is the event management half of the software from Tak-Soft used for romaine events. The other half is the SI-Readout program that reads from the download station and sends back to a central computer running ROM. This in theory is a decent design, lots of dumb download stations with one central database where all data is reviewed and fixed as needed. Unfortunately, this failed at the recent romaine champs.
Various weird data corruption issues were encountered. I have been able to clearly document one issue with SI-6 sticks, and will go no further in investigating the other issues. Having established that the software is broken is good enough for me. I did this analysis to be able to be confident that it was not an SI hardware problem. While I believe this issue is one of the ones that caused errors at the WRC, it does not explain everything. I believe that possibly different versions of SI-Readout were used on different computers each with their own bugs, and that other bugs also exist when dealing with over versions of SI cards in all versions of the software. After looking at the data recovered from the stations in the woods, and reading back a few competitors sticks into SI-Config after the event, I am very confident that the configuration and programming of the stations was done correctly, and was not a contributing factor to the corruption in ROM.
First a little background on the SI protocol, and why it is error prone. The download station when programed to read out SI cards, does little more than just pass through data from the computer to the stick that is inserted. It sends the computer messages on stick insertion and removal, and responds to requests to read a block of memory from the stick. There are options in SI-Config that effect this though. Auto send means that it will send a card inserted message, and then without being asked will read out all the blocks of memory from the stick and send them to the computer. This sounds good, but doesn't give the program a lot of control over timing and error handling. All of the sticks after the 5 are internally arranged as blocks of 128 bytes of data. Each different SI family {5, (6,6*),(8,9),(10,11,SIAC)} have a completely different internal memory layout, and even within a family, 8 vs 9 for example, where in the memory the punch data is stored is different. It is vital the the downloading program correctly understands the memory layout and reads the correct blocks of data to get punch data. This is why all the programs must be updated every time a new family of sticks come out. On a side note, if the download station is not programmed in extended mode, then the firmware in the station fakes anything newer than an SI-6 and returns an SI-6 memory layout. No, I've never written code that was not using extended mode, have no idea what it does with lots of punches, and think this is a horrible hack that isn't really documented well.
The SI-6 is what we are dealing with here, so lets review its memory structure. The card has 8 blocks of 128 bytes. The first two blocks contain overall card data, and personal info. This includes start, finish, check and clear times, card number, user name, and lots of other fields that no one uses and waste an entire 128 byte (32 punch) data block. Blocks 6 and 7 each hold 32 punches making for the 64 punch capacity of a regular SI6. The SI6* holds 192 punches, and starts storing punch data in blocks 6 and 7, and once those are filled will fill blocks 2,3,4 and 5. The option to program stations as SI6 192 mode forces the firmware to use the extra blocks on all SI6 cards, making them all SI6*s. This has to be programmed into every download station and every station in the woods. It has been verified that this was done, and was not the cause of any errors. Note that one common error on the software side is to ask for all the blocks from the download station, expecting to get back blocks 0,6,7 but an SI-6 was used, or the station is in 192 mode, and you get back blocks 0,2,3,4,5,6,7 and use the punch data from blocks 2,3 as 1-64 instead of 64-128 like it should be.
When an SI6 is inserted, a command is sent from the station to the computer to let it know that a new stick is inserted, for a 6 and 6* this is the E6 command, and contains the stick number. There is a corresponding E7 command for removal of any SI stick. I have analyzed how SI-Config, OE, ROM and my own software read SI6 sticks, and all properly deal with insertion and removal of sticks. To read data from the stick, the proper command is E1, along with which block you want to read. You can either request a specific block (0-7) or request block 8 which looks up in the station memory a field that lists which blocks are required for that card. When in normal mode the station will send back blocks 0,6,7 to that request, and in 192 mode it will read back back all of the blocks. SI-Config and OE2010 both use the mode where block 08 is requested, and allow the station to determine (via the CardBlocks byte) which blocks to send back. SI-Readout requests each block manually.
At this point everything sounds fine. SI-Readout requests the blocks differently, but that should work out ok. Where it falls apart is when it has to read block 7 from the card. After card insertion I properly see an 0xE1 command to read block 0, then a 0xE1 command to read block 6. If there are more punches that fit in block 6, another 0xE1 command should be issued to read block 7. Instead of 0xE1, the command sent is 0x69, which is in the block of the old non extended mode commands, and even in that sketchy non official documentation I can't figure out what 0x69 would do, its just an error. At this point the download station properly responds with a NAK. The Tak-Soft software just keeps starting over and trying to download the stick again, never succeeding. If there are 32 or less controls on the stick though, it works fine, since it only has to read blocks 0 and 6 to get all the data it needs.
This test was done with the newest version of the software available online. Using an older version from 2013 does issue the correct commands and downloads a 6 fine. This is why I suspect that different versions of the readout software may have been on different computers resulting in some people downloading fine, and others having issues. I did not investigate the SI10/11 issues where again different blocks of 32 punches would be downloaded or not, or not in the right order, but can imagine that a similar bug could cause them also.
Overall, to me, this shows that the major problems were in the form of software bugs that should have been easily caught had anyone, the organizers, or the programers tested the software through all of its functionality. I can envision a testing scenario where only a few controls were used and therefor the downloads with an SI6 worked fine. To trigger the bug, more than 32 controls must be punched.
In regards to the backup memory issues with the download stations, it is my understanding that only blocks of data that are requested by the downloading software will be written to the backup memory. If a block is not requested to be read out, it will not be in the memory. The memory is also arranged as just a dump of blocks, and it is up to the reading software to figure out what kind of stick each block belongs too. There are patterns that you have to detect to properly interoperate the data. If say only blocks 0 and 6 were read successfully, this may confuse any software that is trying to interoperate the backup memory dump as it would expect there to be an additional block for that stick since block 0 says there are say 60 punches. It would then interoperate the first block of the next stick downloaded as the third block of the first stick messing everything up. I have no seen theses stations and have not investigated exactly what is in their memory, so this is only a guess.
ROM is the event management half of the software from Tak-Soft used for romaine events. The other half is the SI-Readout program that reads from the download station and sends back to a central computer running ROM. This in theory is a decent design, lots of dumb download stations with one central database where all data is reviewed and fixed as needed. Unfortunately, this failed at the recent romaine champs.
Various weird data corruption issues were encountered. I have been able to clearly document one issue with SI-6 sticks, and will go no further in investigating the other issues. Having established that the software is broken is good enough for me. I did this analysis to be able to be confident that it was not an SI hardware problem. While I believe this issue is one of the ones that caused errors at the WRC, it does not explain everything. I believe that possibly different versions of SI-Readout were used on different computers each with their own bugs, and that other bugs also exist when dealing with over versions of SI cards in all versions of the software. After looking at the data recovered from the stations in the woods, and reading back a few competitors sticks into SI-Config after the event, I am very confident that the configuration and programming of the stations was done correctly, and was not a contributing factor to the corruption in ROM.
First a little background on the SI protocol, and why it is error prone. The download station when programed to read out SI cards, does little more than just pass through data from the computer to the stick that is inserted. It sends the computer messages on stick insertion and removal, and responds to requests to read a block of memory from the stick. There are options in SI-Config that effect this though. Auto send means that it will send a card inserted message, and then without being asked will read out all the blocks of memory from the stick and send them to the computer. This sounds good, but doesn't give the program a lot of control over timing and error handling. All of the sticks after the 5 are internally arranged as blocks of 128 bytes of data. Each different SI family {5, (6,6*),(8,9),(10,11,SIAC)} have a completely different internal memory layout, and even within a family, 8 vs 9 for example, where in the memory the punch data is stored is different. It is vital the the downloading program correctly understands the memory layout and reads the correct blocks of data to get punch data. This is why all the programs must be updated every time a new family of sticks come out. On a side note, if the download station is not programmed in extended mode, then the firmware in the station fakes anything newer than an SI-6 and returns an SI-6 memory layout. No, I've never written code that was not using extended mode, have no idea what it does with lots of punches, and think this is a horrible hack that isn't really documented well.
The SI-6 is what we are dealing with here, so lets review its memory structure. The card has 8 blocks of 128 bytes. The first two blocks contain overall card data, and personal info. This includes start, finish, check and clear times, card number, user name, and lots of other fields that no one uses and waste an entire 128 byte (32 punch) data block. Blocks 6 and 7 each hold 32 punches making for the 64 punch capacity of a regular SI6. The SI6* holds 192 punches, and starts storing punch data in blocks 6 and 7, and once those are filled will fill blocks 2,3,4 and 5. The option to program stations as SI6 192 mode forces the firmware to use the extra blocks on all SI6 cards, making them all SI6*s. This has to be programmed into every download station and every station in the woods. It has been verified that this was done, and was not the cause of any errors. Note that one common error on the software side is to ask for all the blocks from the download station, expecting to get back blocks 0,6,7 but an SI-6 was used, or the station is in 192 mode, and you get back blocks 0,2,3,4,5,6,7 and use the punch data from blocks 2,3 as 1-64 instead of 64-128 like it should be.
When an SI6 is inserted, a command is sent from the station to the computer to let it know that a new stick is inserted, for a 6 and 6* this is the E6 command, and contains the stick number. There is a corresponding E7 command for removal of any SI stick. I have analyzed how SI-Config, OE, ROM and my own software read SI6 sticks, and all properly deal with insertion and removal of sticks. To read data from the stick, the proper command is E1, along with which block you want to read. You can either request a specific block (0-7) or request block 8 which looks up in the station memory a field that lists which blocks are required for that card. When in normal mode the station will send back blocks 0,6,7 to that request, and in 192 mode it will read back back all of the blocks. SI-Config and OE2010 both use the mode where block 08 is requested, and allow the station to determine (via the CardBlocks byte) which blocks to send back. SI-Readout requests each block manually.
At this point everything sounds fine. SI-Readout requests the blocks differently, but that should work out ok. Where it falls apart is when it has to read block 7 from the card. After card insertion I properly see an 0xE1 command to read block 0, then a 0xE1 command to read block 6. If there are more punches that fit in block 6, another 0xE1 command should be issued to read block 7. Instead of 0xE1, the command sent is 0x69, which is in the block of the old non extended mode commands, and even in that sketchy non official documentation I can't figure out what 0x69 would do, its just an error. At this point the download station properly responds with a NAK. The Tak-Soft software just keeps starting over and trying to download the stick again, never succeeding. If there are 32 or less controls on the stick though, it works fine, since it only has to read blocks 0 and 6 to get all the data it needs.
This test was done with the newest version of the software available online. Using an older version from 2013 does issue the correct commands and downloads a 6 fine. This is why I suspect that different versions of the readout software may have been on different computers resulting in some people downloading fine, and others having issues. I did not investigate the SI10/11 issues where again different blocks of 32 punches would be downloaded or not, or not in the right order, but can imagine that a similar bug could cause them also.
Overall, to me, this shows that the major problems were in the form of software bugs that should have been easily caught had anyone, the organizers, or the programers tested the software through all of its functionality. I can envision a testing scenario where only a few controls were used and therefor the downloads with an SI6 worked fine. To trigger the bug, more than 32 controls must be punched.
In regards to the backup memory issues with the download stations, it is my understanding that only blocks of data that are requested by the downloading software will be written to the backup memory. If a block is not requested to be read out, it will not be in the memory. The memory is also arranged as just a dump of blocks, and it is up to the reading software to figure out what kind of stick each block belongs too. There are patterns that you have to detect to properly interoperate the data. If say only blocks 0 and 6 were read successfully, this may confuse any software that is trying to interoperate the backup memory dump as it would expect there to be an additional block for that stick since block 0 says there are say 60 punches. It would then interoperate the first block of the next stick downloaded as the third block of the first stick messing everything up. I have no seen theses stations and have not investigated exactly what is in their memory, so this is only a guess.