National Test Discussion

[ScottMaurer19]

5. MIT ecology was hands down the most brutal test I've ever taken. Golden Gate also had a much more difficult test than nationals, with more critical thinking involved. We won both. The national test came nowhere close. We finished most stations and double-checked with time to spare.

.

I'm glad you liked my MIT test. It was supposed to be brutal and make sure to separate the good teams from the weak ones. I remember you won first by a significant margin, so congrats!

I'm fortunate enough to have never taken any of your tests during competition

[chalker]

Chalker, is there ever any discussion of removing or refining events if they consistently show a poor correlation with team scores? If WIDI is so variable and a poor predictor of team outcome, why has its grading/scope not been better standardized or controlled?

No there isn't. There are bigger factors that come into play, like corporate sponsorships, ease of running / participating, and committee foci. We can't standardize / control too much, or else we'll cause all kinda of work and problems at the ~400 tournaments that occur each year around the country.

WIDI is a signature event, that goes back almost to the beginning of SO. We've tweaked it a little bit over the years, but it's so integral to the organization I doubt we'll change it significantly anytime soon.

As a side note (and I readily admit this is a bit of a humblebrag), I can personally attest to the fact that it's possible to perform consistently in WIDI. Way back when I was a competitor, I got 1st place in the event at Nationals in 1992 and then again in 1993. In 1994 I got 5th place.

[maxxxxx]

In general correlation ONLY tells you things move together, NOT why.

I think this is very important to remember, however in the case of Ecology we have our cause(the sketchy test), and the correlation affirms that the event had a weak relationship between individual and overall placement, so that should be enough to warrant a discussion(which I know you aren't advocating against, I'm just piggybacking off of your post) and some further action from the National office. At the very least I hope they at least take a look at the test and look into a way to raise standards for tests in the future, although I am well aware of some of the various challenges it would involve.

If I'm going to take this in another direction, I would suggest that more Event Supervisors and test writers reach out to alumni and ask for input or feedback in the same way that they do for Astronomy. I know that at the heart of it, SO is a volunteer organization, but I'm sure there are plenty of alumni who would be willing to help out in any event if they were more aware of that possibility. Since this would all come down to the specific ES's decisions on whether or not to reach out, I would ask for SOINC to start giving a small nudge or recommendation so that more would start to follow.

[ScottMaurer19]

Chalker, is there ever any discussion of removing or refining events if they consistently show a poor correlation with team scores? If WIDI is so variable and a poor predictor of team outcome, why has its grading/scope not been better standardized or controlled?

No there isn't. There are bigger factors that come into play, like corporate sponsorships, ease of running / participating, and committee foci. We can't standardize / control too much, or else we'll cause all kinda of work and problems at the ~400 tournaments that occur each year around the country.

WIDI is a signature event, that goes back almost to the beginning of SO. We've tweaked it a little bit over the years, but it's so integral to the organization I doubt we'll change it significantly anytime soon.

As a side note (and I readily admit this is a bit of a humblebrag), I can personally attest to the fact that it's possible to perform consistently in WIDI. Way back when I was a competitor, I got 1st place in the event at Nationals in 1992 and then again in 1993. In 1994 I got 5th place.

I think part of the question is WHY is it always an event? Almost all other SO events are put on some sort of rotation or topic rotation.

I guess what I am asking is what specific role does this one event play that is so integral that significant changes cannot be made to allow for a more representative placement of teams?
I'm not saying that a reason does not exist but I personally do not see one from my perspective.

[nicholasmaurer]

In general correlation ONLY tells you things move together, NOT why.

I think this is very important to remember, however in the case of Ecology we have our cause(the sketchy test), and the correlation affirms that the event had a weak relationship between individual and overall placement, so that should be enough to warrant a discussion(which I know you aren't advocating against, I'm just piggybacking off of your post) and some further action from the National office. At the very least I hope they at least take a look at the test and look into a way to raise standards for tests in the future, although I am well aware of some of the various challenges it would involve.

If I'm going to take this in another direction, I would suggest that more Event Supervisors and test writers reach out to alumni and ask for input or feedback in the same way that they do for Astronomy. I know that at the heart of it, SO is a volunteer organization, but I'm sure there are plenty of alumni who would be willing to help out in any event if they were more aware of that possibility. Since this would all come down to the specific ES's decisions on whether or not to reach out, I would ask for SOINC to start giving a small nudge or recommendation so that more would start to follow.

I would agree that getting alumni input would significantly improve the quality and appropriate difficulty of many national tests. There are certainly many alumni who would be willing to provide this input. Some events have been consistently high quality over the years (e.g. disease, most mechanicals) while others have varied widely based upon ES.

[Unome]

Keep in mind, absolute correlation between individual events and the overall team ranking isn't really desirable. That would say there is no way for individuals on a team to outperform the overall team.

^^^
This exact topic came up on the forums some years ago (2012 or 2013 I believe).

From what I have gathered, the common thread for Anatomy is that ES Patty Palmietto writes tests that are generally too easy and fail to separate out strong teams from weak teams. I know she was the Div B ES at nationals this year and I have taken tests from her in the past. They were generally focused on very basic anatomy or process skills, which therefore means in-depth knowledge of Anatomy is of limited benefit. The only area she tends to go into detail on is disease identification, often based off of images.

I've generally heard this as well, although people don't usually say outright that the test is bad, just not as good as it could be.

As a side note (and I readily admit this is a bit of a humblebrag), I can personally attest to the fact that it's possible to perform consistently in WIDI. Way back when I was a competitor, I got 1st place in the event at Nationals in 1992 and then again in 1993. In 1994 I got 5th place.

Also for those not aware this was in the time before most events had consistent event supervisors at the national level (usually ~60% of the ESes were local) so this is over multiple different styles.

Although I'd agree that WIDI is hard to be consistently good at (I definitely am not ), there are teams who can do it. For example my former middle school team has consistently won WIDI at almost every tournament over the last two years - I believe they've recorded 2-3 losses over 13 tournaments, including winning Nationals the last two years - with several different writers (though mostly the same doer). Clearly they've figured out a way to become consistent at it.

Almost all other SO events are put on some sort of rotation or topic rotation.

Also, 4 of the 7 are in the Inquiry committee. It's always seemed to me that Inquiry is a bit different than the others (in a generally negative way) - this could just be a consequence of them not having any well-defined discipline like the others though.

[bernard]

Here's data for correlations between event placements. I suggest not drawing any serious conclusions from this.

[Skink]

The assumption that the "best" teams should perform well in everything is a dangerous, myopic one. An event isn't de-legitimized just because the "right" teams didn't win, and it's been pointed out several times on this site (in a data-driven manner multiple times) that WIDI performance is not terribly indicative of overall team score. It's for this reason that it's probably the most important event in the rotation. It is a performance-based event assessing skill sets that the best teams have consistently not mastered!

[EastStroudsburg13]

Just wanted to add my two cents in here.

Using correlations should not be used as a single indicator about an event's worth; there are many other factors at play. However, a below average correlation, combined with evidence that a test was short, easy, and/or non-comprehensive, could be an indication that certain teams were hurt by (or benefitted from) what could have been a high margin of error between places.

Now, I don't think any single event at nationals had what could be considered an alarmingly low correlation constant. Even WIDI indicates that there was still somewhat of a positive correlation between the event and the full results. And I do think there is truth in the idea that some teams put a lot of effort into WIDI, where some traditionally good teams may not put as much effort in favor of other events, due to a perceived random nature. There is, of course, a high level of subjectivity in the event, and I think adding more guidance in the rules (i.e. teams should be ranked on a rubric of 50 points or higher) would help to lower the effect of randomness, especially at competitions that are not nationals. However, I definitely think the event has a place in Science Olympiad, and there is evidence that teams can do consistently well in it, should they put in the effort required.

Finally, in terms of the correlations, I definitely don't think that a high correlation necessarily means the event was the best. However, the good teams tend to do reasonably well in most events. If there is a particular event where lower-ranked teams do better because of effort, then that's great; but if they ranked above other teams because there wasn't enough to separate them, that's not quite as ideal. This is why I think correlation coefficients bare much more notice at the low end of the spectrum, because while an average correlation indicates that an expected number of teams placed higher than their overall rank indicated, a below-average correlation indicates that this happened more than usual. Perhaps, like WIDI, this was a case of certain teams putting in more effort than others. However, if accompanied by evidence of a test that maybe wasn't up to par, it may have more importance.

[rfscoach]

Chalker, is there ever any discussion of removing or refining events if they consistently show a poor correlation with team scores? If WIDI is so variable and a poor predictor of team outcome, why has its grading/scope not been better standardized or controlled?

No there isn't. There are bigger factors that come into play, like corporate sponsorships, ease of running / participating, and committee foci. We can't standardize / control too much, or else we'll cause all kinda of work and problems at the ~400 tournaments that occur each year around the country.

WIDI is a signature event, that goes back almost to the beginning of SO. We've tweaked it a little bit over the years, but it's so integral to the organization I doubt we'll change it significantly anytime soon.

As a side note (and I readily admit this is a bit of a humblebrag), I can personally attest to the fact that it's possible to perform consistently in WIDI. Way back when I was a competitor, I got 1st place in the event at Nationals in 1992 and then again in 1993. In 1994 I got 5th place.

Our WIDI team got first place this year and last year at Nationals Same Doer both years, different writers. Our amazing WIDI coach has developed a lot of strategies over the years to "take the random out" of the event.