flâneur

Experimentology - 16 Meta-analysis

experimentology.io · 7,369 words · saved by 1 readers

Throughout this book, we have focused on how to design individual experiments that maximize measurement precision and minimize bias. But even when we do our best to get a precise, unbiased estimate in an individual experiment, one study can never be definitive. Variability in participant demographics, stimuli, and experimental methods may limit the generalizability of our findings. Additionally, even well-powered individual studies have some amount of statistical error, limiting their precision. Synthesizing evidence across studies is critical for developing a balanced and appropriately evolving view of the overall evidence on an effect of interest and for understanding sources of variation in the effect. Synthesizing evidence rigorously takes more than putting a search term into Google Scholar, downloading articles that look topical or interesting, and qualitatively summarizing your impressions of those studies. While this ad-hoc method can be an essential first step in performing a l

16 Meta-analysis – Experimentology Note learning goals Discuss the benefits of synthesizing evidence across studies Conduct a simple fixed-effects or random-effects meta-analysis Reason about the role of within-study and across-study biases in meta-analysis Throughout this book, we have focused on how to design individual experiments that maximize measurement precision and minimize bias. But even when we do our best to get a precise, unbiased estimate in an individual experiment, one study can never be definitive. Variability in participant demographics, stimuli, and experimental methods may l

related reading