Catcalling and Bad Methodology

I've had occasion to post something about the infamous "catcalling" video set in New York (and some of its spin-offs and derivatives). I never really thought of it as a particularly well-done experiment (if it can be called that). The experiences of a single fairly attractive white woman wearing slightly provocative (but not overtly so) clothing strolling through a few streets in a single city and then sifting the footage and pairing it down to 2 minutes sounds more like selective anecdotes than actual data, even for sociology.

And so here is a sociologist explaining why that is. Some choice samples of the piece:

In effect, this was a research project, and it had an implicit research question (“Do conventionally attractive white women get verbally harassed in New York?”) and produced an answer: the video. However, in doing so without any reflection on its own method, it amply demonstrates the crucial substantive and political importance of research methods.... 

The Hollaback video also shows why “data” without theory can be so misleading—and how the same data can fit multiple theories. Since all data collection involves some form of data selection (even the biggest dataset has selection going into what gets included, from what source), and since data selection is always a research method, there is always a need for understanding methods.... 

Removing the means of implicit biases can be eye-opening. For example, after decades of lack of women in major orchestras, some big symphonies started doing their interviews blind — musicians played their instruments behind a curtain. Lo and behold, women, previously greatly lagging in professional employment in symphonies, started being evaluated as performing much better.

Data without theory is without context--but beware the biases in the theory!