How accurate is "timed" racing?

[Stan Pope]

How accurate is "timed" racing? Sounds like a silly question on the face of it, which, perhaps why I had not asked it before. Because when everything is working, the times read out accurately to 0.0002 seconds. So, lets assume that it all works as advertised! The times are correctly measured and stored in the management software, correctly tallied and averaged and sorted and the lowest is the one that the software says is #1! What could be better?

In real life all those assumptions don't always hold, as at our district races last Saturday, where the "winner" had gone home before the results were announced because he had lost all of his races, some by substantial margins. Or so he thought. But, I digress.

We as race participants look at the racing environment and plan our car so that it will do as well as it can in that race environment. If consistency is rewarded, we try to be consistent. If "one hot run" is rewarded, we try to have at least one really hot run! And so on. As participants, the racing environment is what it is.

But we as race organizers have a different responsibility. Part of that responsiblity is to provide the participants with an undeniably fair racing environment, an environment that is fun for the participants and for those cheering for them, and, finally, as accurate a determination of "the fastest car" as we can within the constraints imposed.

The "accurate determination" is the point of this thread.

In a simplistic sense, "the fastest car" is the car that the racing environment identifies as the fastest car! If the environment includes a method that runs the cars once on each of four lanes and, for each, tosses the low time and averages the rest, then the car with the lowest average is, by definition, "the fastest car."

But, of course, this is too simple to let stand without examination! The first observation is whether it is reasonable to "toss the low time". Let's assume that it is reasonable, for the sake of this discussion. (I'm certainly open to revisiting the assumption.)

My second observation is that cars never seem to run exactly the same times on a lane. It almost always seems that if a car runs 5 times on one lane, its times tend to be similar, but not identical. A statistician might describe this phenomena as variance of the samples around a mean. In this respect, a PWD Race is rather like a grand experiment in which we sample each car's performance (time) and conclude from that sample what its performance is. The more times we sample each car's performance, the more likely that the variance in the samples will cause the average of the samples to more closely match the car's intrinsic performance. (If I had studied statistics, maybe I'd know the technical jargon that a statistician would apply to this. If you did, please apply 'em.)

To try to get a better handle on this variance, I acquired the heat times for a recent race in which good builders participated and in which each had more than one run in each lane. With more than one run in each lane, I could better see the inherent run to run variance without the difference between lanes messing up the averages! The variances were not big, but were big enough. Each racer's average and variance created a "tolerance band" for the measurement. At modest confidence level, the various racers' ranges didn't overlap much. As higher confidence levels the bands became larger and there was a lot more overlap between racers. The overlap indicates that true performance of the various cars might not sort in the same order as the samples sorted out!

Had Cub Scouts been doing the building, I should expect to see run-to-run variances several times that great and a lot more "overlap" and a lot more misrepresentation of the participants' true performance because of limited sampling.

With this in mind, how many times should we have each car race if we want to be, say, 90% sure that we awarded the 5 place trophies to the 5 most deserving? (The answer, of course, is that I have provided insufficient data. I've asked that the data be preserved and forwarded to me for this kind of analysis. Maybe in the future I can supply it.)

From your point of view, how sure do you want to be that you award your trophies to the most deserving? And how many times should you have each car in your races run in order to accomplish it?

(I wonder if I should start writing at this hour ... it seems that I get a bit long winded, if not obtuse.)

[gpraceman]

In a simplistic sense, "the fastest car" is the car that the racing environment identifies as the fastest car!

Plain and simple, this it what it boils down to.

You can try to change that environment, if you don't like how the outcome is being determined. However, time that it takes to run the event is usually a big consideration. Races generally don't have enough time to run the cars enough to get a good statistical performance measure on each car.

You could always run each car down a single lane say 10 times (or however many times to get enough data make to be statistically significant). Crunch some numbers and then declare a winner, but how exciting will that be?

In real life all those assumptions don't always hold, as at our district races last Saturday, where the "winner" had gone home before the results were announced because he had lost all of his races, some by substantial margins. Or so he thought. But, I digress.

So, you feel the times were not accurate? That would seem to be the impetus for this thread. Were the other top cars just more inconsistent than this boy's car and ended up with a lower average? Or do you suspect some timing system problem?

[Stan Pope]

Well, yes and no, Randy. There were major glitches in our timed race that did not relate to sampling error. Rather they arose from clearly bad times (e.g. 1.500 seconds on a 28' X 4' track) getting past newbie track staff. (Those happened to another age group, not to Alex' age group.)

What I'm trying to dig into is "Is 4 runs enough (in a district race) to keep the effect of sampling error down low enough so that the resulting trophy awards bear a strong resemblance to what would show had the runs been enough to have statistical significance." My intuition says probably not, but I don't have the statistical knowhow to resolve the question to my satisfaction. And I certainly don't have the statistical knowhow to convince others.

The other aspect is that I just don't think 4 runs is enough payback to the kids, but that is a different analysis!

BTW, I have no complaints about "my personal stake in the races" ... Alex won his age group (on tie break) and was very close to a very good car in the ROC!

[Stan Pope]

You could always run each car down a single lane say 10 times (or however many times to get enough data make to be statistically significant). Crunch some numbers and then declare a winner, but how exciting will that be?

How about, "Almost as exciting as our actual races!"

[SlartyBartFast]

After a hiatus, I see that we're still discussing minutiae:
"effect of sampling error"

Well, it is a race Stan. It's never the best driver and best car that win in auto racing. It's the best car/driver combo THAT day that wins.

If you want to eliminate statistical error, you'll need to run your event with all the excitement and drama of a science lab. A dozen runs per lane before you average times out...

Races determine winners. Not fairness.

The fairness in a competition is that the rules are well known at the beginning and that they are followed and enforced.

By the time you approach differences of .001 of a second, there are a million little chance effects that can be affecting the outcome.

[Stan Pope]

I hear ya, SlartyBF!

Somewhere between "One run for all the Marbles" and 20 runs (isn't that a "magic number" in sample size among statisticians for getting significance?) there must be a "happy medium." Some number of runs (sample size) that balances significance and enough racing so that the kids think building the car was worthwhile and short enough to keep the parents from complaining. Is it 3 runs on a 3-lane track? 4 runs on a 4-lane track? 6 runs on a 3-lane track? 6 runs on a 6-lane track? 8 runs on a 4-lane track? ... What is your preference? and how do you weight the factors?

[3 Cub Dad]

My personal preference is 8 runs on a 4 lane track, maximizing opponents, score by points so the kids can tell who won right away!

Steve

[joe]

The fairness in a competition is that the rules are well known at the beginning and that they are followed and enforced.

I bet it would have taken me 3 paragraphs to make this point and Slarty said it in one sentence!

[Stan Pope]

The fairness in a competition is that the rules are well known at the beginning and that they are followed and enforced.

I bet it would have taken me 3 paragraphs to make this point and Slarty said it in one sentence!

It is an excellent and valid point, but it is not germane to the issue that I raised in the first post!

That issue is accuracy in identifying the objectively fastest car! Folks have told me how incredibly accurate racing by time is, so I looked. What I found is that, as done in many of our PWD venues, it isn't that accurate. The reason is the run-to-run variance by even the best cars. Worse, the more closely matched the competitors, the worse the accuracy!

[SlartyBartFast]

and how do you weight the factors?

My position would be: "Why bother."

Once again, at 0.001, what's the variance of the cars run to run and track to track?

At some point all the top competitors are just going to start randomly moving up and down in the standings based on the numerous factors involved in the race time.

Chose a system, stick to it. At the bottom end, it won't matter. The really slow cars and middle cars will probably be classified fairly easily by any system.

At the top end of competition though, chance is a huge factor.

[SlartyBartFast]

Looking back through my psots, I re-read this thread and had to comment on the following:

The reason is the run-to-run variance by even the best cars. Worse, the more closely matched the competitors, the worse the accuracy!

Stan, you're a smart guy. How do you equate run-to-run variance with innaccuracy?

Nature varies. Cars will NEVER have exactly the same times run to run.

I think you're getting hung up on the math/engineering of finding the "best" car. No sport is about finding the absolute theoretical best. It's all about finding the best under the circumstances.

[Stan Pope]

Concern for method accuracy varies inversely with the extent to which one is content to give the 1st place trophy to the 2nd fastest car just because in some small number of races the run-to-run variances compounded to let the slower car edge out the faster. If I were content that such occurred 50% of the time OR if I just didn't car about the question, I probably should be organizing raffles instead of races! I would rather aim for less frequent occurrence, for instance, awarding the wrong trophy less than 5 to 15% of the time.

Here is some thoughts behind the original question:

If I have a certain amount of time to run the races and a certain array of equipment, I still have choices in terms of how to use that equipment to run the races. If I choose a method that is less sensitive to run-to-run variances in car performance, then the method will probably rank the intrinsicly faster cars above their lesser competitors more consistently than a lesser method, i.e. it will be more accurate.

If I plan racing for a group in which the run-to-run variance of the competitors is a lot higher than the variance between the competitors, then the rankings afforded by the method will vary in accuracy according to the sample size, i.e. the number of heats over which the car's average is computed. For instance, even in timed racing it is common to pick the fastest cars (cars with the lowest average heat time) from preliminary heats to race against each other in finals. The primary goal it to refine the accuracy of the ranking. Often, cars in the finals start over and race about the same number of times as they did in the prliminaries. But if the The accuracy of the finals is no greater than the accuracy of the preliminaries! For Finals to add accuracy, the times accrued during the preliminary racing must be included or the duration of the finals must be extended. (A curious artifact of 8 heat per car prelims and 8 heat per car finals is that the average times of all the finalists in finals will (almost always) be slower than the average for those finalists during the priliminaries! Run-to-run variance is the cause. I'll let the reader figure out why this is so.)

[SlartyBartFast]

But Stan, what accounts for run to run variance? Let's face it, when you're timing to .001s it's dumb luck and a combination of factors outside the control of the carbuilder.

And by what definition of sporting competition does fairness mean that the theoretical engineering calculations of which car SHOULD be fastest should determine the winner?

Why isn't the gold medal in the Olympics handed to the runner that last posted a world record? Why isn't the Stanly cup just handed to the team with the best season performance? Why not just measure a soccer team for speed and kicking accuracy and give the prizes without playing an actual game?

[pack529holycross]

I will not dilute the conversation back into the old points vs. times discussion, but ina very real sense, I still hold to the strong belief that you have to compare pinewood derby to drag racing:

-standing start
-timed runs over a fixed distance
-head to head competition
-SOME type of elimination "process" whether it be sinble/double/triple/quad elimination, points, etc... to determine the winners

If you were to run a Derby as close to the formulation of drag racing, ET is used to determine the "class" in which you will compete.

From there, there is a bracket that is seeded based on ET performances in time trials. ( Time trials are also used within NASCAR, for example, to determine starting position within the "grid" ) From there, individual races determine winners ( those that advance ), and non winners ( possibly into a losers bracket ).

Lane choice goes ( typically ) goes to competitor within the heat who has the best average ET thus far, so in that sense low ET's are a good source of gaining a performance advantage within the competition.



I believe that ET is incendental to the outcome of the race within PWD, and that the accuracy of "timed" racing is more important ONLY because the method lacks any other criteria for performance tracking. Lane variances, track variances, starting gate variances, finish line location variances, and all other environmental factors slice into the reliability of "timed" racing ( in my opinion ).

Nicholas

[SlartyBartFast]

Lane variances, track variances, starting gate variances, finish line location variances, and all other environmental factors slice into the reliability of "timed" racing ( in my opinion ).

And no one, IMO, has yet given any reason why these variances have to be accounted for to create fairness.

Why run on a track if "fair" is the best built car should win?

Test the wheels for friction and inertia, test the car for balance and CG, throw the numbers into the Derby Calculator. Fastest theoretical car wins!

"Best" car/runner/team does not make you a winner in competition.

But if the The accuracy of the finals is no greater than the accuracy of the preliminaries! For Finals to add accuracy, the times accrued during the preliminary racing must be included or the duration of the finals must be extended.

So the world series, Stanley cup, super bowl, etc, should all include regular season statistics in order to correctly determine the outcome and the present systems are "unfair" and "inaccurate"? Player/equipment performance variance should be accounted for?