The Political Consequences of Personality Biases in Online Panel Surveys

Abstract

Online surveys now represent a large segment of the academic and commercial opinion research markets due to their low cost and flexibility. A growing literature examines the implications of online surveys for data quality, most commonly by comparing demographic and political characteristics of different samples. Psychological differences between samples collected online and via more traditional modes have received less attention. In this paper, we explore the possibility that personality may affect participation rates online versus face-to-face. Specifically, we hypothesize that online surveys, compared to face-to-face surveys, might be less attractive to respondents who are high in openness to experience. As openness to experience has been linked to liberal policy positions, such differences have the potential to produce dissimilar estimates of public opinion between face-to-face and online surveys. Using data from the 2012 and 2016 dual-mode American National Election Studies, we compare responses to identical survey questions across the parallel face-to-face and online samples. We find that respondents in the online samples were, on average, less open to experience and more politically conservative on a variety of issues compared to their face-to-face counterparts. This was true especially in 2012, when online respondents were drawn from a large panel of experienced survey takers. The implications of these findings for using relatively inexpensive online samples for estimating mass policy opinion are discussed.