Treatment Format and External Validity in International Relations Experiments

Political scientists use short vignettes or mock news stories to embed treatments in experimental survey designs. We investigate whether the choice of medium entails a trade-off between internal and external validity. On the one hand, short vignettes may improve internal validity by isolating key variables without overloading respondents with information, thereby leading to satisficing and degrading data quality. On the other hand, mock news stories may improve external validity by approximating the circumstances under which individuals consume political information, which may enhance the credibility of the information. We find no evidence, however, that short vignettes mitigate satisficing more than mock news stories. Nor do we find that mock news stories enhance individual perceptions of the credibility of the information. Instead, we find that short vignettes are susceptible to confounding, which mock news story mitigate. These findings have important implications for debates about the use of survey experiments in political science.政治科学研究者会借助短情境 vignette 与模拟新闻报道，在实验性调查设计中嵌入实验处理（treatment）。本研究旨在探讨媒介选择是否会在内部效度与外部效度之间形成权衡取舍。一方面，短情境 vignette 可通过聚焦核心变量且避免给受访者造成信息过载，从而提升内部效度；但该方式可能引发满意化作答（satisficing），进而损害数据质量。另一方面，模拟新闻报道可通过还原个体接触政治信息的真实场景，提升外部效度，进而增强信息的可信度。然而，本研究并未发现短情境 vignette 比模拟新闻报道更能缓解满意化作答。同时，本研究也未发现模拟新闻报道能够提升个体对信息可信度的感知。反倒发现短情境 vignette 更容易受到混杂偏倚（confounding）的影响，而模拟新闻报道则可缓解此类混杂偏倚。上述研究结论对政治科学领域围绕调查实验（survey experiments）应用的相关学术讨论具有重要启示意义。