How accurate is "timed" racing?

[derbyspeed]

I would add in staging as a possibility of the variance in times. If the boys staged their own cars, then it was fair, but may have not been as focused each time they staged. Although I still believe the lanes caused most of the variance.

At our council race my son was started on the same lane everytime except once. And by reaction of most (if not all) cars it was the slow lane. We knew if he won in that lane than he had the entire heat won. Each heat is run as a best 2 out of 3 switching lanes for each run. If both cars win one race then they are put in the middle two lanes and then a tie breaker is run to determine the winner of the heat.

The car that started in the faster lane always had an advantage because if he won in that lane and then lost in the slow lane he would be put in the middle by the faster lane which was also faster. By that I mean both left lanes were faster than the two right.

I know I'm getting off the subject a little here, but I think it's almost impossible to make it completely fair in any situation and if you are using times than each car should run in every lane against every car at least two times against that car? Shouldn't it?

[Stan Pope]

BTW, the times make it look much more like a three-lane track with very distinct lane differences!

7 cars, 6 races, raced each competitor only once, Car B 6-0, Car A 5-1, could be a 3 lane track, but only if they use 2 lanes at a time!

Argh! Of course ... on a 3-lane track they would have raced each other twice! and the knowledgable Derby Chair would have chosen a CPN chart, so that would have met each other the secont time wiht lanes reversed!

[3 Cub Dad]

Or these times could be the result of a manually pulled starting gate. In which case the races would be both fair and accurate since the racers were using points in head-to-head competition, and not using cumulative times to determine placing.
The variability in times would be partially a measure of who was pulling the gate, and how quickly and consistently it was pulled. In my opinion, a big strike against using cumulative times in determining a winner.

Joe,

Great thought, I hadn't considered that! Hey RMoose? A little help?

[Stan Pope]

At our council race my son was started on the same lane everytime except once. And by reaction of most (if not all) cars it was the slow lane. We knew if he won in that lane than he had the entire heat won. Each heat is run as a best 2 out of 3 switching lanes for each run. If both cars win one race then they are put in the middle two lanes and then a tie breaker is run to determine the winner of the heat.

My first few reactions to their method is that it is a tremendous waste of time and needlessly limits the variety of opponents without improving the accuracy. More thorough analysis of that scheme is one of the reasons that I am collecting population data ... see other thread about getting race data! So, before I blast away at its apparent inanity, I'd like have "my ducks aligned and quacking in unison", i.e. analysis and/or simulation based on real population data that shows the value and cost of the method. So far, I have about 6 populations to work from ... need more! Especially, I need more from selective district or council racing (i.e. racing in which competitors need qualify by racing well in other races.)

I have been disturbed at the extent to which details of the process you describe are hidden, e.g. sometimes it matters which car starts in which lane (most of the time it doesn't matter), so, potentially with every pairing it does matter, but how is the lane assignment decision reached? I suspect that the bracket in the computer leads each pairing to have a higher and lower position on the screen, and that the higher position starts in a particular lane. If so, it would have the effect of reducing many random decisions into a single (hopefully) random decision and reducing the possibilities for random processes to average out the impacts.

Any "glassy-eyed stares" yet?

[RMoose]

Hey RMoose? A little help?

Let me see if I can answer the questions raised.

First off, track is a homemade, wooden 2-lane track, similar to those at the Association and State races.
Start gate is manually pulled open, kids stage their own cars with same adult helper operating gate throughout event.
Format of race was 3 runs in each lane, points awarded by finish order, fewest points wins. Opponent determined for each race randomly by computer program. This year's groups were fairly small in number, 5 to 9 each with this group in the middle at 7. Top three in each group got awards and advanced to grand finals to determine overall fastest car.

As far as the number of runs/lane, this was selected mainly due to time constraints for race event length.
I would like to say points scoring was chosen after careful consideration of the inaccuracies of timed scoring with a manual start gate; but the truth is it was because our kids seem to prefer beating their friends rather than racing the clock and are smart enough to know the difference. Something they've picked up in Cub Scouts I think.

Times by Lane for Car A went like this:

Code: Select all

Race 1. Lane 2 - 3.1182  Race 2. Lane 1 - 3.0294  Race 3. Lane 2 - 3.0256  Race 4. Lane 1 - 3.0668  Race 5. Lane 1 - 3.1443  Race 6. Lane 2 - 3.0602  

Times by Lane for Car B were as follows:

Code: Select all

Race 1. Lane 1 - 3.0680  Race 2. Lane 2 - 3.0774  Race 3. Lane 1 - 3.0885  Race 4. Lane 2 - 3.0756  Race 5. Lane 1 - 3.1173  Race 6. Lane 2 - 3.1140 

I think that's about it, hope these answers help!

[Stan Pope]

Wow! The "lane affinities" that I thought I saw in the data were illusory!

Code: Select all

     	Car A	       	Car B 	
     	Lane 1	Lane 2	Lane 1	Lane 2
     	3.0294	3.1182	3.0680	3.0774
     	3.0668	3.0256	3.0885	3.0756
     	3.1443	3.0602	3.1173	3.1140
				
Avg	  3.0802	3.0680	3.0913	3.0890
StDev	0.0479	0.0382	0.0202	0.0177

Expressed in inches the standard deviations are:
StDev	   7.2	   5.7	   3.0	   2.7

The variances seem to be extremely high, even for Car B in lane 2. This suggests either that the track is really rough, increasing the random component in each car's runs, or that the starter was operating in a speed range in which such variance is typical. (If he is "fast enough" his variance disappears from the times.)

A comment on the definition of accuracy. Every method that I've looked at contains some random components which, if rerun with the same population, would cause some variation in the results. So, I've used the term accuracy to be the percent of time that the objectively fastest cars finish in their deserved place. Evaluation of a method's accuracy requires either an intense analytic exercise or an extensive simulation. A byproduct of this study is recognition that no single example proves or disproves the accuracy of the method.

[RMoose]

Track is fairly rough at the joints and requires tweaking before each race to smooth them out. We thought it was set up about as good as it has ever been this year. The start gate doesn't open that smoothly, so it is difficult to open quick on a consistent basis.

Nevertheless, reviewing the race data afterwards the times and the points results were tracking together nicely until I came to these two cars. Since the difference would have affected 1st and 2nd place for our Crusader group I was concerned.

Now I feel better that our decision to judge the outcome by points was a good call - even though we made that decision with no regard to our equipment but based on racer preference. Hopefully this little exercise will be helpful to others in planning out their races in the future.

Both Car A and Car B advanced to our grand finals and their results there were similar:

Code: Select all

       Car A                       Car B
         Lane   Time              Lane   Time
Race 1    (2)   3.0502             (1)   3.0363
Race 2    (1)   3.0728             (2)   3.0658
Race 3    (1)   3.0979             (1)   3.1071
Race 4    (2)   3.0696             (2)   3.0876
Race 5    (1)   3.1245             (1)   3.1308
Race 6    (2)   3.0734             (2)   3.1160
Race 7    (1)   3.1090             (2)   3.0997
Race 8    (2)   3.0995             (1)   3.1139

Car A when 6-2 and got 3rd overall and Car B went 7-1 and got 2nd overall.

Thanks for the analysis, it has been most helpful.

[Stan Pope]

Here's some analytical results using lane and population characteristics from the two district race data that I received (Clay and Oshman). Comparison is based on accurately awarding 1st through 5th place trophies with populations of 40 and 60 cars. Methods are "Total Time, one heat per lane on 4-lane track" and "Quitutple Elimination on 3-lane track." (Note: Oshman-Fnl is the population characteristics for just the approx. dozen finalists from the Oshman race. When 40 to 60 racers are assumed to have that same distribution, the racers are "much more similar" and random influences increase. Oshman-Fnl is, therefore, not a realistic comparison, but is included to emphasize the effect when racing "better cars." In other words, what works well for pack racing may not work as well for district and council racing!)

Analysis uses Cory Young's Simulation programs with Trials of 10,000 for "Timed" and 5000 for "Elim" for each case.

Code: Select all

                                  40 Cars	                            60 Cars		
Method:	         Clay         Oshman     Oshman-Fnl       Clay       Oshman      Oshman-Fnl
Timed 1 Ht/Ln       68.0%        44.4%      38.3%            65.2%      41.8%       35.6%
QE-3:               61.1%        41.1%      37.0%            59.3%      38.1%       35.0%

Thoughts:
Timed is marginally more acurate. The surprising part is that it is only marginally more accurate!

The cost of the marginal increase in accuracy is additional equipment cost and complexity and loss of "trasnparency" in the process.

A side cost is that 88% to 90% of the QE-3 racers finish the day with at least one heat win versus some much lower number for the timed race participants.

[Stan Pope]

Here's some results from the 2008 Wotamalo District Wolf Track, timed racing, four runs, drop worst time:

For those who like to use Cory Young's DerbySim program, here are the critical numbers for the Wolf population:

Car Speed StDev ................ 0.0981 sec (14.7 inches at 150"/sec)
Random Element StDev ....... 0.0364 sec (5.5 inches)
Lane Speed StDev .............. 0.0110 sec (1.7 inches)
No Lubricant degradation.
Num Racers ....................... 73
Trials ................................. 10,000

Race as run (4 heats per racer): 5-Trophy Accuracy: 44%

What this means is that more than half of the 5 place trophies were probably awarded to someone other than the person who deserved it. Note, other stats provided by the simulation indicated that most of the time, a person deserving a trophy did receive one, just not the right one, and that accuracy in higher place trophies (first or second place) was better than in lower place (4th or 5th place).

Increasing the racing to 8 and 12 heats per racer would improve the accuracy to 54% and 59%.

The question that seems to jump out at me is, "How, when times are recorded to 4 decimal places and averaged over 3 or 4 runs, can the results have such low accuracy?" The answer seems to be that when there are a lot of racers and their average times are bunched, then even relatively small run-to-run variances in individual run times can muck up the works!

Thus, the timed racing scheme that works well for a pack of 30 Cub Scouts with widely varying building skills may be less effective when applied to a district or council race in which each pack is sending a select group of best racers!

---------

The racing that produced these parameters was done on an aluminum Piantedosi, nominal 32' (28' start to finish) Racing used the middle four of six available lanes.

[Stan Pope]

What this means is that more than half of the 5 place trophies were probably awarded to someone other than the person who deserved it.

The question is raised, "Isn't the racer who finishes first according to the scoring rules deserving of the first place trophy?" My answer is, "Yes, according to the scoring rules!"

The purpose of this study is to evaluate the scoring rules! The method of these simulations is to stipulate times (and, therefore, rankings) for each competitor in the trial according to the population description and stipulate lane differences according to the parameters. Then apply the racing method, in simulation utilizing the prescribed "random element", to those competitors to see how well the racing method results compare to the stipulations. For 5-Trophy Accuracy (of interest to me since my district awards 1st through 5th place trophies), each trial is scored 0 through 5 according to how many resulting rankings in the top 5 matched the stipulated rankings! After repeating the trials a few thousand times, random variations average out and the accuracy of the method (the scoring rules) can be seen!

There is a secondary measure that is also informative. It is the percentage of racers who should finish in the top 5 (according to their stipulated rankings) ended in the top 5 in the simulation results.

[GravityRacer]

Stan, you are obviously far more conversant with statistics than you want to admit!

I want to point out that until the number of variables in question are properly quantified, the statistics are going to have an unacceptable variance in the most stringent analysis. My first race in a "real" competition- with all adult builders- I had the fastest car, until after the race, the second place finisher beat me by .008 seconds. I beat him by .011 seconds for the trophy. But, when I consider the number of variables involved, I don't consider that loss to be any different from just being at "the next race". I haven't spent a lot of time on the statistical end of racing, but instead focused on the mechanics of the car and track. Bottom line is, you just have to make your car the best you can, in the hope of overcoming the vagaries that are going to creep in.

Example: You get lane choice in Top Fuel based on ET. The track usually isn't "fair"! Even when $ millions are involved, that can't be fixed! By the same token, the Pro Stock guys can find that same "slow" lane for Top Fuel to be their "fast" lane.

The number of variable available, even on a PWD track, are sufficient (and sometimes incredibly subtle, like someone putting his car partway up the starting gate), that I'd hate to try to predict which car is going to be first.

If you can assure that the car is placed exactly in the same place in the lane, with:

a. Wheels the same distance from the axle
b. Cambered the same way as last race
c. Toed the same as the last race
d. Has the same graphite "load"
e. Pointed the same way so it's drift is repeatable
f....well, you all get the idea.

I just had this conversation with a metrology engineer at work today. We have some people who were complaining at the level of detail he had in his procedure. The truth is, until ALL variables are properly quantified, and even then, sometimes, you will not be able to ensure a win. The difference is, we do not have objective scientific evidence for the cars' performance, we don't have the tools. Has anyone here done a test to find the best lubricant, other than generally trying "different" graphites and/or liquid lubes and combinations thereof? The pressure placed on the axle/wheel interface is so low as to be ridiculous- far below the kinds of pressures at which these lubricants would fail. My take is this is still pretty much in the "voodoo" stage of quantification of just this one variable!

Then, who would be able to get some kid to replicate the easy steps above to stage his car, let alone the number that really exist?

No offense intended...but imho there is insufficient quantification of the variables in question to apply a rigid statistical analysis to this system.

btw, I would expect that we would need a timer with another order of magnitude precision by the time we were done.

...and I wouldn't even worry about consistency until we can quantify all the variables. Maybe we could start a bunch of threads, one for each variable. But, does Macy's tell Gimbal's? This is, after all, a competition, and if I walked into the competitor's pits at a race, I'd get kicked out, and vice-versa. I have seen deliberate dissimulation, I won't say where. But, it really is a competition, and I wouldn't expect anything less!

[Stan Pope]

Stan, you are obviously far more conversant with statistics than you want to admit!
...
I haven't spent a lot of time on the statistical end of racing, but instead focused on the mechanics of the car and track. Bottom line is, you just have to make your car the best you can, in the hope of overcoming the vagaries that are going to creep in
....
No offense intended...but imho there is insufficient quantification of the variables in question to apply a rigid statistical analysis to this system.

btw, I would expect that we would need a timer with another order of magnitude precision by the time we were done.
...
and I wouldn't even worry about consistency until we can quantify all the variables.

While I think that what you say is true, I think that you are talking about something other than evaluating the competition method. For purposes of evaluating the competition method, the variances between cars, between separate runs of the same car, and between runs of the car on different lanes are "givens" that affect the accuracy of the competition method and which make our decisions on competition method more important.

To some extent most folks who run "timed" recognize the need to reduce the effect of randomness in runs. Otherwise, they would just have "one timed run for all the marbles." Instead, they typically have at least three or four runs and average the times, correctly applying the idea that "more random samples causes the average to more accurately reflect the population."

I found one curious fact during the latest stats run: Randy Lisano kindly extracted the detail from two of my track's competitions. He gave me both the "CSV" format and the "PDF" format. For whatever reason, the PDF format had times to 4 decimal places, but the CSV format (that I could plug directly into a spreadsheet) had times rounded to 2 decimal places. In my sleep-deprived state, I did not recognize the rounded state of the data until after the data had been entered, processed and distribution numbers calculated and dutifully recorded.

After discovering my error, I repeated the exercise using the times extracted from the PDF format. Only one of the six distribution St. Dev. numbers changed and that was by one in the least significant digit!


Stripped of the "make it fun" aspects (which would not be a good idea in practice), a Pinewood Derby is a "grand experiment" to compare the building and racing skills of an array of competitors. The validity of the comparison depends on how well that experiment is designed and executed. (The accuracy analysis above assumes that the "execution" aspect was done exactly correctly and that there were no undetected equipment malfunctions.)

[GravityRacer]

I think you're on the right track. I would like to see something a little different at a kid's race, though: photo finish. It's obvious who is first over the line, usually, and the camera would only be needed on the "by a nose" cases.

A bunch of us engineering nerd types were discussing this topic at lunch today, and I was having a devil of a time explaining how a car can win with a "slower" time. I finally got the one guy to understand, but it's still tough.

I personally would only use a timer to get quantitative data for engineering purposes.

We often use software that has .csv files as the data format, then transferred to EXCEL. We get data at 20 kHz, or 50 microsecond intervals, and it reads out to 6 decimal places. Like EXCEL, the number format can be set to two decimal places. I hope you didn't type in all those numbers from the .pdf file. Were the .csv files truncated by the software, ala EXCEL? This should have been the native file, and if it reads and reports four decimal places during the race, it should be stored that way.

[Stan Pope]

I would like to see something a little different at a kid's race, though: photo finish. It's obvious who is first over the line, usually, and the camera would only be needed on the "by a nose" cases.

A bunch of us engineering nerd types were discussing this topic at lunch today, and I was having a devil of a time explaining how a car can win with a "slower" time. I finally got the one guy to understand, but it's still tough.

This idea was discussed a few weeks ago on this forum. I'd like to see your "engineering nerd types" tackle it. The idea was to utilize a "line scan camera" (I think I recall the term correctly) to record just the pixels at the finish line. The resulting picture has one axis across the track and the other axis is time. Each scan line represents so many microseconds. The actual finish order of all cars is shown. There is a possibility of ambiguity depending on the scan direction if two cars first appear in the same or adjacent lines.

It is an alternative to a timer or an adjunct to the timer. In any case, the pic can be shown immediately upon heat completion! And, each scan line can be labeled with a specific time range.

The downside is that because one axis is time, the "length" of the cars in the pic depends on their respective speeds during that interval. Faster cars are shorter (show in fewer scan lines) than slower cars! Their "length" dimension is skewed!

I'd love to see someone attack this and make something that would feed a USB2 input... just about like a very fast flatbed scanner!

I personally would only use a timer to get quantitative data for engineering purposes.

I agree ... dunno if for the same reasons. Timed racing hides too much for my taste.

"Time" is marginally more accurate than QE for 5 trophies, and is many fewer heats. The timed heats are slower to run. QE for large groups can run on multiple tracks off one queue of racers to speed up the process greatly. QE decisions (which car won the heat) can be observed by the audience and they can make informed decisions on whether the race is being "called" correctly or not. Long term equipment stability is needed for Timed to be useful. Can't tell if the car just "ran funny" or the timer malfunctioned or somebody "fat fingered" something (although some real-life cases in our recent experience were almost certainly equipment malfunctions or "fat fingers".)

On the intangible side (which is where many of us end up making our judgements, I think) QE on 3 lanes results in between 87.5% and 90% of the entrants winning at least one heat against, typically, two other racers. By comparison, by actual count, less than 50 percent of our 73 Wolf Scouts won a timed heat and barely over 50 percent of our 29 Webelos 2 Scouts won a timed heat. (I'm not counting the Wolf Scout who won one heat all day because his time was recorded as 1.52something seconds. His other other finishes were 3rd or 4th and he had left before his average time was reported as placing him in 1st place! Including data such as that in the analysis would make the results of the simulation even less favorable, but I tossed 'em cuz 1.5 seconds on a 28' track "just ain't gonna happen" under gravity power!)

We often use software that has .csv files as the data format, then transferred to EXCEL. We get data at 20 kHz, or 50 microsecond intervals, and it reads out to 6 decimal places. Like EXCEL, the number format can be set to two decimal places. I hope you didn't type in all those numbers from the .pdf file. Were the .csv files truncated by the software, ala EXCEL? This should have been the native file, and if it reads and reports four decimal places during the race, it should be stored that way.

Actually, it was rather easy to extract the data from the pdf ...
Copy from the Acrobat display (Ctrl-A, Ctrl-C),
Paste into Notepad (Ctrl-V),
Delete extraneous lines,
Replace-All "2 spaces" to "1 space",
Replace-All "one space" to "comma",
Replace-All "two commas" to "one comma",
Copy from Notepad (Ctrl-A, Ctrl-C),
Paste into spreadsheet (Ctrl-V).
Check columns for correct alignment in case some names were "funny."

I didn't even have to retype the formulas ... just get the data into the same order and cut/paste from one spreadsheet to the other!

When I'm happy with the data reduction process, I'll document it, along with the logic behind it, for others to use.

[Stan Pope]

FWIW, and as a postscript to this older thread, I compiled a page of "considerations for timed racing" and posted it at http://www.stanpope.net/pwtimed.html. It includes some aspects of topics discussed in this thread plus others that seemed appropriate.

The web page is still kinda rough, so I'd like direct feedback on questionable items, missing topics, and insufficiently expressed topics.