Help with statistical analysis

General race coordinator discussions.
Post Reply
tomormatt
Apprentice
Apprentice
Posts: 16
Joined: Mon Mar 03, 2014 5:22 pm
Location: Calabasas, CA

Help with statistical analysis

Post by tomormatt » Sun Apr 06, 2014 3:09 pm

We had our Pinewood Derby last week and it went as well as it could have. Per my earlier posts, I inherited a totally unplanned derby with four weeks to go. I made the decision to bite the bullet and purchase a Best Track and Champ timer and it made for a successful event.

Our pack is not an ultra competitive pack as far as the Pinewood Derby goes; however, we do have a couple dads that are a bit out of control. We outsourced the event for a couple years, but last year I was not on the cub committee and they made the decision to use an old wooden track and try to determine the winner by having parents judge which car crossed first in each heat. One of the dads that is the source of trouble was very unhappy about the judging and that one of the lanes was obviously slower than the others. The individuals running the event did not know how to adjust or run heats to eliminate that and the event ended with this dad yelling and driving some of the den leaders to tears.

Fortunately we eliminated the potential for those problems with the track setup this year. This particular dad was overheard telling another parent that his son never actually touched the car they used this year. He said his son just wanted to win so he did not care that his dad built the car in its entirety. I would have disqualified the car had I heard that before the event ended. In addition to the sad fact that this individual seems to place some importance on his competing against 8-11 year old boys, the dad actually jumped up and fist pumped when I revealed the results with his car winning.

The format of our race was using the Perfect-N schedule and having each car race in each lane twice (six total races for each boy as we had a three lane track). I did a short intermission at the half-way point (each car had raced in each lane once) to let people lube their cars. The next time this particular car raced, it set a track record and was substantially lower than its previous three races. I am including the times below and asking is anyone thinks that the car was potentially altered to increase its speed. Adding weight would seem to be the only real option and in retrospect I should have done a post-race insepction and weighing of the top three cars. I would not have even thought this, but for the fact that the dad is so obsessed with winning.

Here are the times for the car in question for the six races:




Lane #1 3.25010
Lane #2 3.23020
Lane #3 3.23780
Lane #1 3.17870
Lane #2 3.24310
Lane #3 3.19470


I expect some pick up in speed if the car has not been "broken in", but I am sure this individual knows enough to run the wheels enough to break the car in before the race. The time in the following race is back to the same range the pre-intermission race times. The last time is considerably faster than the average also.

Our process was to have the boys remove the cars from the track and hold them until their next race at which point they place them at the starting line. Since we move pretty quickly, altering a car would be difficult during the racing since you might be called up to race at any point.

My quick analysis is that it is the fastest time is not quite two standard deviations from the mean, but it would fall in the 94th percentile of a normal standard distribution. The fact that it came right after the intermission is what makes me think something is amiss.

Ultimately, the event is done and I learned quite a bit that we will do differently next year. Curious as to how many people do a post-race inspection on cars. I am loathe to pull apart a car and potentially damage it, but a simple visual inspection and weighing does not seem to be too much.

Looking forward to any thoughts and want to say thanks for helping with my previous posts. Not sure we would have had the event we did without the information on this site.



User avatar
FatSebastian
Pine Head Legend
Pine Head Legend
Posts: 2646
Joined: Wed Jun 17, 2009 2:49 pm
Location: Boogerton, PA

Re: Help with statistical analysis

Post by FatSebastian » Sun Apr 06, 2014 8:30 pm

tomormatt wrote:The fact that it came right after the intermission is what makes me think something is amiss.
The most relevant statistic would test the equivalence of two populations. This is usually done by comparing the sample mean before the suspected change-point with the sample mean afterward. Unfortunately, this results in two populations of size 3. At 5% significance (95% confidence) using Student's-t test, the sample means appear to be statistically equivalent (no detectable change of performance after intermission).



User avatar
Stan Pope
Pine Head Legend
Pine Head Legend
Posts: 6888
Joined: Sat Jul 05, 2003 7:01 pm
Location: Morton, Illinois
Contact:

Re: Help with statistical analysis

Post by Stan Pope » Mon Apr 07, 2014 12:12 am

Here are some answers for questions that you didn't ask:

1. Relube after 3 heats? Lubrication should be good for 10 heats or more... there is no need for re-lube during such a short race!

2. Boys are staging and racing their own cars ... that's great! While the boys are not racing, they should return their cars to an "impound area", e.g. a table with a thick cloth (such as towels) or boards that will prevent the cars from rolling. This avoids accidental damage to the cars between their heats and avoids most questions of unauthorized mods!


Stan
"If it's not for the boys, it's for the birds!"

User avatar
FatSebastian
Pine Head Legend
Pine Head Legend
Posts: 2646
Joined: Wed Jun 17, 2009 2:49 pm
Location: Boogerton, PA

Re: Help with statistical analysis

Post by FatSebastian » Mon Apr 07, 2014 7:28 am

tomormatt wrote:...in retrospect I should have done a post-race insepction and weighing of the top three cars. […] Curious as to how many people do a post-race inspection on cars.
FWIW, the BSA appears to discourage post-race inspections.



User avatar
Stan Pope
Pine Head Legend
Pine Head Legend
Posts: 6888
Joined: Sat Jul 05, 2003 7:01 pm
Location: Morton, Illinois
Contact:

Re: Help with statistical analysis

Post by Stan Pope » Mon Apr 07, 2014 10:06 am

FatSebastian wrote: FWIW, the BSA appears to discourage post-race inspections.
:goodpost:


Stan
"If it's not for the boys, it's for the birds!"

Johnscooling
Journeyman
Journeyman
Posts: 31
Joined: Wed Feb 29, 2012 8:54 pm
Location: Philadelphia, Pennsylvania

Re: Help with statistical analysis

Post by Johnscooling » Mon Apr 07, 2014 1:35 pm

I'm not sure of the reason. But this past derby my olders son car went faster as the races went on.



tomormatt
Apprentice
Apprentice
Posts: 16
Joined: Mon Mar 03, 2014 5:22 pm
Location: Calabasas, CA

Re: Help with statistical analysis

Post by tomormatt » Tue Apr 15, 2014 1:02 pm

I am pretty sure that many cars will go faster as races progress as I have found that my cars often need a breaking in period. I will usually spin the wheels a significant amount of time before racing as I have often seen a pretty substantial impact the more a car races.

I learned lots from this race and will note that we will not be doing any intermission or allowing additional lubricant to be added during the race. I should have had the cars impounded between races, but decided to have the boys hold their cars to speed up the racing. As full background, I deliberated between having the boys do one or two races in each lane. With 20-25 boys, I thought that three races each was a bit light and they would not feel like they had enough races. I knew that doubling it was the only way to keep the number of races in each lane equal, but that would take a little more time than I wanted. Ultimately I decided that more races was better and was looking for ways to speed things up. Even so, some of the boys / parents who had cars that were obviously slower than everyone left before the last couple races were run.

I was hoping that the random nature of the heats and the short time between each heat would prevent anyone from trying to modify their car. The thought of getting called to race in the middle of secretly adding weight would seem scary to me, but I will be impounding cars in the future between heats.

Thanks for helping. I am frustrated by this one parent (and even the scout) as they both are content to have the dad do all the work and make no apologies (from what I have heard). This is the same dad who was pretty upset at my son and I after we won a few years in a row against his older boy. The younger boy is in for a couple more years so I am going to ask each boy to attest that they worked on their car when they present the car for inspection. I cannot control if the boy is willing to say he did even if he did not, but I am not going to announce this in advance in hopes that any boy that did not do any work will be forthcoming if they are not "prepped" by their dad in advance.

The upside is that I have had several parents and cub scouts comment that they had a great time and this was their best derby. Will look to learn and improve next year, but happy with the outcome overall.



User avatar
Stan Pope
Pine Head Legend
Pine Head Legend
Posts: 6888
Joined: Sat Jul 05, 2003 7:01 pm
Location: Morton, Illinois
Contact:

Re: Help with statistical analysis

Post by Stan Pope » Tue Apr 15, 2014 2:39 pm

tomormatt wrote: I am frustrated by this one parent (and even the scout) as they both are content to have the dad do all the work and make no apologies (from what I have heard). This is the same dad who was pretty upset at my son and I after we won a few years in a row against his older boy.
Consider offering to help 'em! They are probably just missing something important. :)


Stan
"If it's not for the boys, it's for the birds!"

User avatar
Noskills
Master Pine Head
Master Pine Head
Posts: 787
Joined: Tue Jan 31, 2012 1:37 pm
Location: Bellevue, WA

Re: Help with statistical analysis

Post by Noskills » Wed Apr 16, 2014 3:59 pm

Let me offer a bit of perspective. His best and worst times are 0.072 apart and on the same lane which is nice (removes one variable). So the question I have is how commonly would one see a delta of .07 or more for a typical pack. I just looked at my pack and 17 out of 59 cars 29% have a max - min of >= 0.07.

That's the whole pack. When I look at the 10 fastest cars none had a delta larger than 0.06 sec. Of the top 20 fastest cars only one had a delta of 0.07 more.

So the difference itself is not so atypical for the average car in the average pack but for a competitive car its unusual. I also think 0.07 is a pretty big jump and makes me suspicious.

Noskill


"Nunchuk skills... bowhunting skills... pinewood derby skills... Girls only want boyfriends who have great skills!"
Napoleon Dynamite

User avatar
Noskills
Master Pine Head
Master Pine Head
Posts: 787
Joined: Tue Jan 31, 2012 1:37 pm
Location: Bellevue, WA

Re: Help with statistical analysis

Post by Noskills » Wed Apr 16, 2014 4:11 pm

I am going to ask each boy to attest that they worked on their car
Come up with some sort of pledge that the boys signs at inspection. "A scout is Honest. I swear that I build with car with the help and guidance of my adult partner..."

I share you frustration on this!

Noskills


"Nunchuk skills... bowhunting skills... pinewood derby skills... Girls only want boyfriends who have great skills!"
Napoleon Dynamite

User avatar
FatSebastian
Pine Head Legend
Pine Head Legend
Posts: 2646
Joined: Wed Jun 17, 2009 2:49 pm
Location: Boogerton, PA

Re: Help with statistical analysis

Post by FatSebastian » Wed Apr 16, 2014 4:37 pm

tomormatt wrote:The thought of getting called to race in the middle of secretly adding weight would seem scary to me...
I would hesitate to attribute to malice what might have happened naturally. We know that performance in general, and alignment in particular, can be affected noticeably by ordinary wear and tear during a race. I think it is reasonable to consider the possibility that the alignment became slightly tweaked. Perhaps the DFW ended up with slightly less toe-in following the 3rd heat, which caused both faster times (per the 4th and 6th heats) and more variability (per the 5th heat)?



User avatar
FatSebastian
Pine Head Legend
Pine Head Legend
Posts: 2646
Joined: Wed Jun 17, 2009 2:49 pm
Location: Boogerton, PA

Re: Help with statistical analysis

Post by FatSebastian » Thu Apr 17, 2014 6:56 am

Noskills wrote:I also think 0.07 is a pretty big jump and makes me suspicious.
One must consider that differences between independent readings have twice the population variance of the individual readings. That is to say, if the standard deviation of a set of independent readings is ~0.029s, then the standard deviation of their differences should be ~0.041s. Assuming normality, a 0.072s difference still remains less than "two sigma" and would not be considered statistically significant.
Noskills wrote:...the boys sign at inspection. "A scout is Honest. I swear..."
:scratching: Really?!



Post Reply