Last week, I volunteered to judge the Greater San Diego Science and Engineering Fair. I found the experience interesting last year, so I participated again this year. Judges have about 3 hours to visit 12 projects and discuss the contents of the posterboard and research journal with the student who did the work. These conversations allow the students to tell us what they did, but they also give them practice at answering questions they may not have anticipated, and the conversations give judges the opportunity to gently teach and discuss things that could have made the project better. Last year, I judged middle school behavioral science projects – unique questions that 6th, 7th, and 8th graders came up with about human behavior, and often equally unique ways of testing those questions. The students put in a lot of effort and did great work, but judging the projects also made me realize that science is really hard. They still had so much progress to make before these projects could even be considered sound, let alone innovative or informative.
But these kids were on their way! I felt that if I could see their science fair projects a few years down the line, my mind would be blown by their progress. This year, I was assigned to the high school division – exactly the opportunity to see the progression of scientific thinking. I knew these students had been working on the projects since the beginning of the year, and many had dedicated class time for guidance. But after visiting just a few projects, I was let down. I wasn’t let down by the students – they were all so earnest, pleasant, and proud, and it was clear that they had put a lot of work into their projects. Instead, I was disappointed with their teachers. So much of the scientific process needs to be explicitly taught, and for some reason, these kids weren’t taught it. Either the students weren’t getting the guidance they needed, or they were actually being misguided.
Imagine that someone eats eggs for breakfast on every weekend day, and never on a weekday. That person tells you, “every time I eat eggs, I have a great day. I guess having eggs for breakfast must cause my day to be good.” You’d probably quickly object to this conclusion – what if the fact that it’s a weekend causes your day to be good? In fact, this explanation seems likely. One variable in your “experiment” is whether the day is a weekend or weekday. The other variable is whether you ate eggs, which happens to vary with the weekend vs. weekday variable. In this case, if you want to know the effect of eggs on the quality of your day, the type of day (weekend vs. weekday) is a confound. It makes it impossible to attribute the results you saw to the variable you want to attribute it to. I saw lots of these egg/weekend confounds in the students’ problems, which is alarming because they invalidate the results the students tried to convey.
For example, one student tested the effect of different font colors on reading speed. She pulled three equal-length passages from a book. One passage she left in black font, another she turned orange, and the third she made multicolored. She then had all her participants read the three passages while she timed them. She found that they were fastest to read the passage written in black ink. But wait – what if that passage just happened to be an easier passage to read? Wouldn’t that account for her results, without taking ink color into account? She thought about this, and then agreed. Together, we worked through the solution that would have avoided the confound – if some people had read passage A in black ink, others had read the same passage in orange, and still others had read that same passage in multicolor, and then we did the same with passages B and C. This way, everyone would have read each passage once and experienced each ink color once, but that the passage-ink pairings would have differed for everyone. This is counterbalancing. Counterbalancing is done specifically to avoid confounds.
These kids of errors were evident in many projects. Another pair of students presented people with songs once to see how much of the chorus they could remember. Oddly enough, they used two songs with the same lyrics in the chorus, but extremely different melodies. They presented Song A first for everyone, and had them recall the lyrics. Then they presented Song B, and had them do the same. Perhaps not surprisingly, people recalled more lyrics for Song B. The students told me this was because Song B was a more familiar genre to their participants. While that’s a possible explanation, it’s not a scientifically valid one. Their participants all had more practice by the time they got to Song B, which had the same chorus as Song A. They should have been better at the latter simply because practice improves performance.
Luckily, as I gently explained these confounds to the students, something seemed to click – they could see the logical problems in their methods and conclusions. A few mentioned to me that they didn’t counterbalance important variables because their teachers told them to keep as much constant as possible. Normally, this is true – you want to keep as much constant as possible when testing different conditions so that variations don’t make your results noisy. Noise in data makes it harder to detect real effects. But the teachers forgot to impart an important caveat of the keep-everything-constant rule: You can’t keep things constant when the constancy could explain your results – when it could become a confound!
This opens an important question for me – were the teachers not able to give guidance on these fundamental logical ideas for doing science? I realize that they have many students to oversee. Or do the teachers lack an understanding of how experiments should be designed, implemented, and interpreted? My intuition is that it might be some of both, but it seems to be pretty problematic, regardless of the source. Allowing students to carry out months-long projects that violate important rules of scientific logic seems like a very bad way for them to learn how things should be done.
But then I started to wonder, do these students actually need to understand how experiments are conducted? Do they have to know why confounds are to be avoided at all costs and how to do so? Many will pursue non-scientific fields. Others will pursue science to the extent that they might not ever need to conduct research, and will get by learning the things that other scientists have found and trusting those scientists’ conclusions. And the students that do pursue scientific research can learn from their future mentors how to conduct science (and how not to). Maybe this is all true, and maybe I can chill, but I’m still thinking about these questions almost a week after the fair, so there must be something to my concern. Shouldn’t educated citizens be able to understand the scientific process, so they can understand why scientists make claims about global warning or about how innocuous (and important!) vaccines are? I’m not sure, but these are some of the questions I’m trying to work out.