Automated Narrative Analysis

The accuracy of four Machine Learning methods in predicting narrative macrostructure scores was compared to scores obtained by human raters utilizing a criterion-referenced progress monitoring rubric. The machine learning methods that were explored covered methods that utilized hand-engineered features, as well as those that learn directly from the raw text. The predictive models were trained on a corpus of 414 narratives from a normative sample of school-aged children (5;0-9;11) who were given a standardized measure of narrative proficiency. Performance was measured using Quadratic Weighted Kappa, a metric of inter-rater reliability. The results indicated that one model, BERT, not only achieved significantly higher scoring accuracy than the other methods, but was consistent with scores obtained by human raters using a valid and reliable rubric. The findings from this study suggest that a machine learning method, specifically, BERT, shows promise as a way to automate the scoring of narrative macrostructure for potential use in clinical practice.本研究比较了四种机器学习方法在预测叙事宏观结构得分时的准确性，并将其与人类评分者采用标准参照进展评分量规（criterion-referenced progress monitoring rubric）所得的评分进行了对比。本次探究的机器学习方法涵盖了基于人工构造特征的方法，以及直接从原始文本中学习的方法。预测模型的训练语料来自参与标准化叙事能力测评的学龄儿童（5岁0个月至9岁11个月）常模样本，共计414篇叙事文本。模型性能以二次加权kappa（Quadratic Weighted Kappa）——一种评分者间信度指标——进行评估。结果显示，BERT模型不仅显著优于其余三种方法，其预测得分还与人类评分者使用有效且可靠的量规所得的评分高度一致。本研究结果表明，机器学习方法（具体为BERT）具备实现叙事宏观结构自动化评分的潜力，可应用于临床实践场景。