I Don't Care Anymore: Identifying the Onset of Careless Responding
share
Questionnaires in the behavioral and organizational sciences tend to be lengthy: survey measures comprising hundreds of items are the norm rather than the exception. However, recent literature suggests that the longer a questionnaire takes, the higher the probability that participants lose interest and start responding carelessly. Consequently, in long surveys a large number of participants may engage in careless responding, posing a major threat to internal validity. We propose a novel method to identify the onset of careless responding (or an absence thereof) for each participant. Specifically, our method is based on combined measurements of up to three dimensions in which carelessness may manifest (inconsistency, invariability, fast responding). Since a structural break in either dimension is potentially indicative of carelessness, our method searches for evidence for changepoints along the three dimensions. Our method is highly flexible, based on machine learning, and provides statistical guarantees on its performance. In simulation experiments, we find that it achieves high reliability in correctly identifying carelessness onset, discriminates well between careless and attentive respondents, and can capture a wide variety of careless response styles, even in datasets with an overwhelming presence of carelessness. In addition, we empirically validate our method on a Big 5 measurement. Furthermore, we provide freely available software in R to enhance accessibility and adoption by empirical researchers.
READ FULL TEXT