Musings on Methodology 1: Curricular Selection Bias

Many of our Curriculum Corner posts have surveyed academic studies on effective teaching. These cover a small, though representative, sample of the articles we have been reading to prepare for camp. However, our choice to focus so much on academic research could be challenged. I have made the choice to prioritize data over my own experiences with debate camp in crafting our curriculum. But it is not obvious that this is the correct methodological choice

After all, these studies will not perfectly apply to debate camps. Most of these studies deal with a standard cross section of students, and debaters do not form a standard cross section.  Debaters, especially those attending camp, tend to be above average in academic drive, internal motivation, intelligence, and wealth. Additionally, debate camps are different from schools. It is not obvious that studies on how to improve learning in school for students generally applies to improving camp for debaters.

Beyond that, our instructors are all extremely experienced with debate camp. They have attended camps numerous times, both as students and instructors. They are experienced both with teaching and being taught debate. Our instructors were successful debaters, they seem to know how to learn debate well.

So, why spend so much time reading data-driven studies on education? In this post, I want to mention one reason for that methodological decision.

It is important to be data driven because it helps counteract the strong selection bias that disposes camps towards maintaining the status quo. To clarify the selection bias it will be useful to compare debate camps with colleges.

The primary people who shape college curriculum are professors. Now, when professors try to decide how they want to teach, what are they going to think about? One obvious answer is that they will think about how they learnt best while they were a student. What sort of lectures did they find most interesting? What sort of assignments did they find the most engaging? Professors infer that these were effective teaching techniques and so replicate those techniques in their own classrooms.

However, this leads to a problem. The experience that most professors had in school was different from the experience of other students. In general, people who go on to teach in college were the people who really loved their college classes. They were the people who learnt the most and performed the best.

Suppose that 99% of students hate how physics is taught and 1% love it. From which group will college professors be drawn?  They will be drawn disproportionately from the 1%. Thus, if physics professors use their own experience to decide how best to teach, they will be biased in favor of the way they were taught.

When professors use their own experience to decide how teach it tends to reinforce the status quo. I think a similar problem occurs at debate camps.

Who tends to want to work at debate camps, those who, as a student, enjoyed camp or those who did not?

Whom do debate camps tend to hire, those who were successful or those who were not? And who tends to be successful, those who learnt a lot at debate camp or those who did not?

Lots of factors compound this selection bias. For example, camps tend, somewhat, to hire students who attended their camp multiple times. Who tends to return to the same camp, those who got a lot from the experience, or those who did not?

Debate camps have a general mold. And most camps repeat this mold will little change year after year. This is unsurprising, because as institutions camps will have a strong selection bias in their instructor experiences which favors the status quo. The reason innovation is so slow in debate camp curriculum (most camps looks the way they looked when I was a student), is not because debate camps have figured out the ideal structure, but because they are biased towards the status quo.

This is one reason I feel it’s important that our curriculum is data-driven. Robust academic studies help counteract the selection bias present in staff experience. Academic studies look at the students who do poorly, as well as those who do well. While camps tend to be composed only of instructors who excelled.