A research paper by CESS Director Raymond Duch, Denise Laroze, Thomas Robinson and Pablo Beramendi titled “Multi-modes for Detecting Experimental Measurement Error” was published in the Political Analysis journal, April 2020 issue.
In this paper the authors explore optimal replication strategies to test the robustness of experimental findings. They focus in particular on detecting mode-related measurement error within experiments. Their findings demonstrate that researchers should replicate identical experiments across multiple modes rather than within a single-mode. When researchers have even weak priors about the noisiness of modes, multi-mode replication is more likely to reduce measurement error. Empirically, they demonstrate how researchers can use machine-learning techniques like Bayesian Additive Regression Trees (BART) to predict counterfactual experimental observations, and recover conditional average treatment effects that aid the diagnosis of measurement error. They then provide concrete guidance on embedding measurement diagnostics within experiments, and separately show how researchers can deliberately manipulate potential measurement error to verify their beliefs about the noisiness of different modes.