A Little Experiment (Part 2 of 2)

Happy Valentines?

A failed hypothesis is better than no hypothesis at all.

---

Snow!

This week has finally provided a chance to analyse the results of the survey that I set up two weeks ago. All the assignments have been completed, and even a snow day couldn’t stop the momentum (albeit with a brief break to go make snowmen outside the Winspear Opera House). Yesterday was the first day in about a month where the majority of it wasn’t spent working on assignments or internships. It was good to have a little downtime.

Anyway, after two weeks, the results are in. There were 40 people who submitted entries to the survey, which was pretty good (though us data geeks always want bigger sample sizes).

So what was I testing in the first place? Well, it had nothing to do with kittens or puppies. Sorry to lead you on like that. Basically, I wanted to test a hypothesis that had been bugging me since my heady days as an undergraduate back at the Australian National University. It was called the Play School theory of personality.

---

Choose a window.

The Play School Theory of Personality

In Australia there’s a popular children’s show called Play School, that most kids grow up watching. In most of the shows, there is a sequence in which the presenters will refer to a set of three windows (arched, circle or square) and ask the audience to guess which window they will look through that day. They will then choose a window, and they will zoom in on it and fade to some video of kittens or puppies or something.

Probably over one too many drinks at a party, a group of us got to discussing which of the windows was our favourite, that we always wanted the hosts of the show to pick. What I noticed amongst this small group was that the selection of window appeared to fit with what degree the student was studying. For the more creative degrees (e.g. liberal and fine arts), the circle window; for more practical, job oriented degrees (e.g. accounting or business), the square window ; and for the degrees requiring a bit of both (e.g. science or engineering), the arched window.

My theory was that there was an association between the two, some affinity with the structure of the window shape, be it the flowing curve of the circle, the “normal” shape of the square window, or the inherent complexity of the arched window that fused the circle and the square.

Over the years, I would informally bring it up in conversation (again, over a few drinks), and the theory would seem to more or less hold. But doing science by drinking wine and talking about children's television is hardly rigorous (though very very fun). So I thought I would try and improve the rigour (slightly) by conducting a small survey. 

---

The Results

Well, without getting too much into the gritty statistical analysis and dredging up the repressed memories of my fellow MBAs of statistics class last semester, the results of the survey were aggregated and analysed using a Chi-Squared test of independence. The idea of this test is to establish whether a set of categorical responses are independent or not. In this case, we wanted to determine simply whether there was a relationship between window choice and degree. If there was, we could then look deeper into the data and try and determine the nature of that relationship.

So, what were the results of the analysis?

Failure.

Putting the raw data through a Chi-Squared analysis could only establish a relationship between window and degree to a confidence of 83%. Which from a scientific standpoint is pretty crap (90% or 95% is considered to be the lower limit in most disciplines; in physics it is more like 99.999% or higher). Putting the various degrees into three separate groups that I felt reflected the theory made the confidence level even worse, merely 62%.

In layman’s terms, this means that we cannot reject the idea that these results were just random fluctuation, and that there is no relationship between window choice and degree. It’s still possible my idea might be true, but it’s unlikely.

The Joy of Failure.

This result isn’t actually upsetting. In science, failure is a good thing. It stops us from going on wild goose chases, forces us to reevaluate our position, and can lead to creative insights that can solve problems. Rejection of the window theory is a fantastic result, since we have learned that there’s probably no relationship between window and degree choice.

We can choose, on the basis of this result whether to investigate the design of the survey to find whether there were factors that may have affected the results (like poor wording, biases in the visual design, small sample size etc.), or try to develop new hypotheses to test with new surveys.

This is how science is meant to work. You get an idea, you test it, and you determine whether your idea works. To quote Adam Savage, “Failure is always an option”.

Sadly, in the cutthroat world of research science, failure isn’t glamourous. Funding is usually tied to research success, and failed experiments run the risk of being swept under the carpet. This behaviour undermines the scientific process itself. However, the fear of (and refusal to recognise) failure is also an inherent human quality, and reconciling the need to be open about results with our own foibles as people will be an ongoing struggle.

But perhaps it starts by learning that a failed hypothesis is a successful result in itself. Scientific education would do well to reinforce this point.

---

What else did we learn?

Beyond establishing the likely failure of the main hypothesis, we did learn things from this survey. After all, the great thing about data is that you can answer lots of questions with it if you are willing to do the analysis. So what else did we learn?

You like the arched window.

Out of the 40 submissions, 23 of you preferred the arched window over either the square or the circle. That’s nearly 58%.

---

Men like puppies over kittens.

Females are roughly evenly split over the issue, but men seem to favour the canine over the feline 14 to 5. Sounds like there might be a ring of truth in the saying “man’s best friend” (maybe an experiment is in order…).

---

Play School Theory #2: Rocket Clock is awesome.

Now the sample size is a bit too small to imbue these results with a real sense of confidence, but they are nevertheless interesting.

As time becomes available I might try and do a bit more analysis of the data. I’m especially interested in looking for possible relationships between colour choice and degree choice as it ties into the Humm-Wadsworth psychometric model.

But that will have to wait for another day…

---