Charts - Evaluation of mAuth

The soundness of the conceptual framework of IQ measures and the ranking model are validated by means of a user study performed by using a web application called mAuth - Mining Authoritativeness in Art History (http://purl.org/emmedi/mauth/search). The application allows users to input the URL of a cataloguing record describing an artwork and to browse the sorted list of attributions fetched in the web of data. We designed a task-based evaluation. Users performed three tasks remotely and filled in an evaluation form (https://goo.gl/forms/xDLwvCCaEFWm4D5h2). Tasks are designed so as to reproduce three common scenarios in connoisseurship, namely:Gather information on an artwork whose attribution is unanimously accepted.Gather information on an artwork whose authorship attribution is debated and that is not sufficiently documented.Gather information on an artwork whose authorship attribution is debated and that is well-documented.For all of the three scenarios we measured a number of parameters. For the sake of brevity we discuss here only three measures for assessing the User Satisfaction, namely: the User Satisfaction (US) measure, the Rank Satisfaction Score (RSS) measure, and the Perception of Authoritativeness Score (PAS) measure. The US measure measures whether retrieved information is useful and sufficient to assess the goodness of an authorship attribution. Users were asked to answer the question “Was it easy to find sufficient information for validating the most authoritative authorship attribution?”. The RSS measures user’s satisfaction with respect to the order of results and the score associated to each information source. To evaluate the RSS measure, users were asked to answer the question “Do you agree with the ranking of results (i.e. the score attributed to each provided attribution and the order in the list)?”. The PAS measure is based on the Net Promoter Score (Reichheld and Markey 2011) that measures whether a user would prefer and suggest the most rated attribution as the most authoritative one. To evaluate the PAS measure, users answered the question “Do you agree with the suggested attribution?”. Participants provided the US, the RSS and the PAS measure by using a Likert scale from 1 to 5 (Strongly disagree to Strongly agree). For all of the three measures we calculated the inter-raters agreement by means of the Fleiss Kappa measure (Fleiss 1971). Lastly, we collected users’ feedbacks for improving the ranking model. Users were asked to select one or more dimensions that affect the ranking.We collected feedbacks from 31 users.As expected, the US is high in the first and third scenario (84% of user either agree or strongly agree), since the first artwork is unanimously ascribed to the same artist and the third presents plenty of evidences supporting an attribution rather than others. In the second scenario the US is significantly lower (58%) since attributions are less documented, there are only two sources and both are supported by scholars’ opinions, and there is no agreement. When evaluating RSS, we see that in the first scenario 74% participants either agree or strongly agree; in the second scenario only 38,7% either agree or strongly agree, while 35,5% neither agree or disagree, and 25,8% disagree; in the third scenario 81% either agree or strongly agree. The kappa measure is, indicating a fair agreement between raters.When evaluating PAS, in the first scenario we see that 84% either agree or strongly agree; only 42% either agree or strongly agree in the second scenario, while 51,6% neither agree or disagree; 71% either agree or strongly agree in the third scenario. The Fleiss kappa measure (Fleiss 1971) is calculated for the 31 raters that evaluated the three cases according to the five categories of the Likert scale: kappa is 33% when evaluating the US measure, 34% for the RSS measure, and 36% for the PAS measure, indicating a fair agreement between raters.<br><br>

本研究通过一款名为**mAuth——艺术史权威性挖掘（Mining Authoritativeness in Art History）**的Web应用开展用户实验，以此验证IQ指标与排序模型的概念框架的合理性（应用链接：http://purl.org/emmedi/mauth/search）。该应用支持用户输入描述艺术品的编目记录URL，并浏览从数据网中抓取的经排序的作者归属列表。
本研究设计了基于任务的评估方案：用户远程完成三项任务并填写评估问卷（链接：https://goo.gl/forms/xDLwvCCaEFWm4D5h2）。任务的设计旨在复刻艺术品鉴赏考据中的三类常见场景，具体如下：
1. 收集关于作者归属已获得一致认可的艺术品的相关信息；
2. 收集关于作者归属存在争议且相关文献记载不足的艺术品的相关信息；
3. 收集关于作者归属存在争议但文献记载充分的艺术品的相关信息。
针对三类场景，本研究测量了多项参数。为简洁起见，本文仅讨论三项用于评估用户满意度的指标，即**用户满意度（User Satisfaction, US）**指标、**排序满意度评分（Rank Satisfaction Score, RSS）**指标与**权威性感知评分（Perception of Authoritativeness Score, PAS）**指标。
US指标用于衡量所获取的信息是否足够且有效，以评估某一作者归属的合理性。调研中要求用户回答问题："是否易于找到足够信息，以验证最具权威性的作者归属？"。
RSS指标用于衡量用户对搜索结果排序顺序及各信息源对应评分的满意程度。为评估RSS指标，要求用户回答："您是否认可本次搜索结果的排序（即各候选归属所对应的评分及列表顺序）？"。
PAS指标基于**净推荐值（Net Promoter Score, NPS）**（Reichheld与Markey，2011）构建，用于衡量用户是否会优先选择并推荐评分最高的归属作为最具权威性的结果。为评估PAS指标，用户需回答问题："您是否认可本次推荐的作者归属？"。
参与调研的用户通过1至5分的**李克特量表（Likert scale）**对US、RSS与PAS指标进行评分（1代表"非常不同意"，5代表"非常同意"）。针对这三项指标，本研究均通过**弗莱西斯kappa系数（Fleiss Kappa measure, Fleiss 1971）**计算评分者间一致性。
此外，为优化排序模型，本研究还收集了用户的反馈意见：要求用户勾选影响排序结果的一项或多项维度。本次调研共回收31份有效用户反馈。
如预期所示，US指标在第一、第三类场景中得分较高：第一类场景中84%的用户表示同意或非常同意，这是因为第一件艺术品的作者归属已获得一致认可，而第三类场景则存在大量可支撑某一归属的佐证材料。第二类场景的US指标得分则显著偏低（仅58%），原因在于该场景下的作者归属相关文献记载不足，仅存在两个信息源且均仅得到学者观点的支撑，未达成共识。
在RSS指标评估中，第一类场景有74%的用户表示同意或非常同意；第二类场景仅38.7%的用户表示同意或非常同意，另有35.5%的用户表示中立，25.8%的用户表示不同意；第三类场景则有81%的用户表示同意或非常同意。
在PAS指标评估中，第一类场景有84%的用户表示同意或非常同意；第二类场景仅42%的用户表示同意或非常同意，另有51.6%的用户表示中立；第三类场景则有71%的用户表示同意或非常同意。本研究针对参与三类场景评估的31名评分者，基于李克特量表的五个评分维度计算了弗莱西斯kappa系数（Fleiss 1971）：US指标的kappa值为33%，RSS指标为34%，PAS指标为36%，上述结果均表明评分者间存在中等一致性。

搜集汇总

背景与挑战

背景概述

该数据集用于评估mAuth（艺术史权威性挖掘应用）的用户研究，包含31名用户对三种艺术鉴赏场景的反馈数据，通过用户满意度、排名满意度和权威性感知得分等指标，结合Fleiss Kappa分析评分者一致性，旨在验证信息质量度量和排名模型在艺术史领域的有效性。数据集聚焦于艺术、数字遗产和信息检索领域，采用开放许可。