News broke this morning that the Journal of Politics—a leading Political Science outlet—will begin to require experiments they will review to be pre-registered. Almost immediately, Political Science twitter was abuzz with discussions about the new policy—what it means, how it would be applied, whether it would advantage some researchers over others, and more. Continue reading
Author Archives: Timothy Ryan
Fraudulent responses on Amazon Mechanical Turk: A Fresh Cautionary Tale
A bit more than two years ago, I wrote a post laying out some of the evidence I had seen showing that people were submitting fraudulent responses on MTurk. Since that time, at least two research teams have conducted far more systematic investigations than what I originally did. This paper has been published at Political Science Research and Methods. I suspect that this one is not far behind.
COVID-19 Data Resources
This post is primarily for my Poli381 students. It is a collection of resources I have come across wherein people are using data analytical techniques to understand, communicate about, and fight the COVID-19 outbreak.
- Very impressive visualization to illustrate spread of the virus, by country. The underlying visualizations here are done in R, and the author is making all the files available here.
- Similar thing here. Also done in R.
- An experiment on how to slow down the spread of misinformation about the disease.
- Nicely formatted data on infections and deaths, updated daily. (This is what I was going to use for the final project, if we had gone that route.)
- This is a contrarian take that kinda argues that we’re overreacting. Ioannidis is a prominent statistician and has a good reputation. However, a lot of people pushed back on this piece really forcefully, such as here and here and here. Having read this stuff, I think Ioannidis is right that there is a lot of uncertainty about how things will turn out. Our estimates have wide confidence intervals, if you will. And he does a pretty good job laying out some reasons why, using some concepts related to class. BUT he makes a really big leap (too big) to then call the thing a possible fiasco and suggest we should be doing a lot less. See the criticism.
- A poll outlining partisan biases in interpretation of pandemic news.
- Brian Schaffner is examining partisanship in response to the crisis by seeing if Republican-leaning areas are less likely to do Google searches for “hand sanitizer” than Democratic-leaning areas. Here are his figures, procedures, and code.
- Nice rundown of a lot of stuff, including some of the above, from the NY Times.
- This is just a fantastic visualization of the trends–best one I’ve seen. Helpfully, it focuses on deaths, rather than diagnoses (wherein the trends are hugely confounded by the availability of tests). And it uses a logarithmic scale and has useful reference lines.
- County-level data from the NY Times.
- Paper attempting to estimate the effects of temperature and humidity on transmission rates.
- Facebook releases data resources to help researchers assess the effectiveness of social distancing measures.
How a 1982 SCOTUS Case Let Trump Circumvent Congress and Fund his Wall
I’m teaching Introduction to Government in the United States this semester. I love following the news while teaching this course. Invariably, events unfold in the news that are difficult to reconcile with the basic principles I’m teaching my students. Sorting out what’s going on often deepens my understanding of government, and points to interesting unanswered questions. In this case, I found out that Donald Trump’s surprisingly broad capacities as concern emergency powers and border wall funding trace their origins to a 1983 Supreme Court case that seems ripe for reassessment. Continue reading
Comments on MTurk Data integrity.
Please feel free to comment on my previous post about data contamination on MTurk below.
Data contamination on MTurk
Update 1: I meant to have a place for people to leave comments on all of this, but messed up and can’t change it. You can make longer-than-tweet comments here if you like.
Update 2: I mistook TurkPrime to be an official Amazon-run blog. But I was corrected on this. In fact, it’s a third-party. (I was confused because the post says things like “In the coming days, we will launch a Free feature that allows researchers to block suspicious geolocations.”) Editing below to reflect this.
As some people have been talking about on Twitter and elsewhere (Kurt Gray; Max Hui Bai; Jennifer Wolak; me), some evidence is coming to light that survey data collected on MTurk is being compromised by problematic responses. Whether these are bots, inattentive responders, or something else is open for discussion. TurkPrme, a third party, recently discussed the problem, though for reasons I discuss below, I think they understate the extent of the issue. There is no official response from Amazon that I am aware of. Continue reading