Anishinaabemowin AI Translation Tool Evaluation Dataset

This dataset contains comprehensive evaluation results from systematic testing of five commercially available AI translation tools for Anishinaabemowin (Ojibwe language). The research addresses critical questions about AI reliability for Indigenous language education, particularly for second-language learners in off-reserve locations with limited access to fluent speakers.

Phase 1: Comparative Tool Analysis

5 AI translation tools tested (AnythingTranslate, Musely.ai, Ojibwechat, ChatGPT Indigenous Language Supporter, Claude.ai Pro)

17 Anishinaabemowin words plus 2 complex sentences

Structured documentation with validation against authoritative print and online dictionaries

Systematic consistency testing through repeated queries to demonstrate all models have errors

Data files: 4 CSV files (2025-08-12-14 Ojibwe Translation Tests)

Phase 2: Guardrail Development

JSON-based cultural protocol guardrails for Indigenous language AI

Conversational testing with ChatGPT, Claude.ai, and Perplexity

Evaluation of bilingual response formats (Anishinaabemowin first, English in brackets), gentle correction methodologies, and community authority deference

Assessment of educational suitability for language learning contexts

Data files: AI TEST 2025-08-17 JSON File.md (guardrail specifications) + 4 conversation test files

Phase 3: Open Conversation Testing

"Cold start" conversations without JSON guardrails

Testing of Perplexity, ChatGPT Indigenous Language Bot, and Claude.ai

Mixed Anishinaabemowin-English conversation evaluation

Error pattern documentation and learning risk assessment

Data files: 4 conversation test files

KEY FINDINGS

Tool Performance:

AnythingTranslate and Musely.ai demonstrated complete unreliability with different translations each query

ChatGPT, Claude.ai, and Perplexity showed best overall performance with grammatical understanding but persistent word stem errors

Ojibwechat demonstrated some understanding but produced grammatical errors and defaults to translation

All systems exhibited errors requiring fluent speaker verification

Critical Translation Error Example: English: "stop walking" AI Translation (ChatGPT): "gego bimosen" ❌ (means "don't walk") Correct Translation: "boon bimosen" ✓ (means "stop walking")

This demonstrates English-based reasoning (negation vs. cessation) that poses significant learning risks.

JSON Guardrail Results:

Successfully enforced culturally appropriate behaviors (bilingual formats, gentle correction methods)

Underlying accuracy issues persist despite behavioral constraints

Demonstrates potential for Indigenous pedagogy integration with critical limitations

Risk Assessment:

Highest risk: Beginner learners without verification access (cannot identify errors)

Moderate risk: Intermediate learners with occasional fluent speaker access

Lowest risk: Advanced learners with regular community speaker access

Critical concern for off-reserve second-language learners (one-third of Indigenous language speakers per 2021 Census)

RESEARCH CONTEXT

This research responds to the 4% decline in Indigenous language speakers from 1991-2021 (Statistics Canada) and the increasing proportion of second-language learners. With Indigenous communities seeking technological tools to support language revitalization, this dataset provides critical evidence for evaluating AI tool safety and developing community-controlled alternatives.

The data supports development of culturally-grounded AI evaluation frameworks—specifically contributing to the proposed "Anishinaabe Turing Test"—that measure AI systems against Indigenous knowledge protocols rather than European cognitive patterns.

DATASET CONTENTS

Phase 1 Data Files:

2025-08-12-14 Ojibwe Translation Tests [4 CSV files] - Comparative tool analysis results with 17 words and 2 complex sentences

Phase 2 Data Files:

AI TEST 2025-08-17 JSON File.md - Cultural protocol guardrail specifications in JSON format

AI TEST 2025-08-17 JSON [4 MD files] - JSON guardrail conversation testing results (Claude, ChatGPT, Perplexity)

Phase 3 Data Files:

AI TEST [4 MD files] - Open conversation testing results without guardrails

Supporting Materials:

methodology.md - Detailed testing protocols and validation procedures

README.md - Complete documentation and usage guidelines

LICENSE - CC BY 4.0 license text

METHODOLOGY

Testing employed standardized protocols including:

Consistent word/phrase lists across all tools (17 words + 2 complex sentences)

Multiple researcher cross-validation

Authoritative dictionary verification against Anishinaabemowin print and online sources

Systematic documentation for reproducibility

Initial consistency tracking to demonstrate all models have errors

Iterative testing with multiple queries to assess reliability

Cultural Protocols: Research conducted with respect for Indigenous knowledge systems and in support of language revitalization efforts. Testing focused exclusively on publicly available AI tools to assess community safety. No traditional knowledge was used or shared with the AI systems. Researchers worked exclusively with language data the LLMs already possessed, ensuring protection of cultural knowledge.

IMPLICATIONS

For Researchers:

Framework for evaluating AI tools for Indigenous languages

Methodology for culturally-grounded AI assessment

Evidence of systematic errors in commercial translation systems

For Educators:

Evidence-based assessment of tool safety for classroom use

Documentation of error patterns for post-use correction protocols

Verification protocols requiring fluent speaker oversight

For Indigenous Communities:

Data to inform AI tool adoption decisions before integration into language programs

Support for community-controlled AI development initiatives

Validation of data sovereignty principles in language technology

Protection against potential harm from unreliable AI systems

EDUCATIONAL IMPACT

AI systems pose high risk for isolated learners who may internalize errors as correct language patterns. Current tools require intensive community oversight and systematic verification protocols. Without fluent speaker access, beginners cannot distinguish between correct and incorrect outputs, leading to potential long-term language acquisition problems.

TECHNICAL SPECIFICATIONS

Data Formats: CSV (comma-separated values), Markdown (.md) Character Encoding: UTF-8 Testing Period: August 12-17, 2025 Number of AI Systems Tested: 5 translation tools + 3 conversational LLMs Research Context: PhD research, York University Digital Media program Funding Support: IBET (Indigenous and Black Engineering and Technology), Abundant Intelligences Tkaronto Pod

CITATION

When using this dataset, please cite:

McConnell, A., & Ly, J. (2025). Anishinaabemowin AI Translation Tool Evaluation Dataset (Version 1.0.0) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.17274098

Associated Publication: McConnell, A., & Ly, J. (2025). "Anishinaabemowin Aabajichigan Gaawiin Nibwaakaasii (Efficacy of AI for Ojibwe Language Education)." Connected Minds 2025, Toronto, ON, October 6-8, 2025.

BibTeX: @dataset{mcconnell_anishinaabemowin_2025, author = {McConnell, Andrew and Ly, Jasmine}, title = {Anishinaabemowin AI Translation Tool Evaluation Dataset}, year = {2025}, publisher = {Zenodo}, version = {1.0.0}, doi = {10.5281/zenodo.17274098}, url = {https://doi.org/10.5281/zenodo.17274098} }

KEYWORDS

Indigenous languages, Anishinaabemowin, Ojibwe language, artificial intelligence, machine translation, language revitalization, language education, AI evaluation, Indigenous data sovereignty, cultural appropriateness, language learning, second language acquisition, Indigenous pedagogy, AI ethics, language technology, ChatGPT, Claude, Perplexity, translation tools, JSON guardrails, Indigenous futurities, off-reserve language learning

LICENSE

Creative Commons Attribution 4.0 International (CC BY 4.0) https://creativecommons.org/licenses/by/4.0/

ACKNOWLEDGMENTS

This work is supported by IBET (Indigenous and Black Engineering and Technology) and Abundant Intelligences, Tkaronto Pod. Research conducted as part of PhD studies in Digital Media at York University.

GITHUB REPOSITORY

Complete dataset and documentation available at: https://github.com/giigdo/AI_Bot_Tests_Q3_2025

本数据集包含针对阿尼希纳贝姆语（Anishinaabemowin，又称奥吉布瓦语）的5款商用AI翻译工具所开展的系统性测试的完整评估结果。本研究聚焦原住民语言教育领域中AI可靠性的关键问题，尤其针对保留地外、难以接触流利母语使用者的第二语言学习者群体。

第一阶段：工具对比分析

本次测试覆盖5款AI翻译工具：AnythingTranslate、Musely.ai、Ojibwechat、ChatGPT原住民语言助手（ChatGPT Indigenous Language Supporter）、Claude.ai Pro。
测试样本包含17个阿尼希纳贝姆语词汇与2个复合语句。
所有测试均通过权威印刷版与在线词典进行校验，并形成结构化文档。
通过重复发起查询开展系统性一致性测试，以证明所有模型均存在翻译误差。
数据文件：4个CSV格式文件（命名为2025-08-12-14 Ojibwe Translation Tests）。

第二阶段：防护框架开发

面向原住民语言AI的JSON格式文化规范防护框架。
针对ChatGPT、Claude.ai与Perplexity开展对话测试。
评估双语回复格式（阿尼希纳贝姆语前置，英语置于括号内）、温和纠错方法以及对社区权威的尊重机制。
评估工具在语言学习场景中的教育适用性。
数据文件：AI TEST 2025-08-17 JSON File.md（防护框架规范文档） + 4份对话测试文件。

第三阶段：开放式对话测试

无JSON防护框架的“冷启动”对话测试。
针对Perplexity、ChatGPT原住民语言助手与Claude.ai开展测试。
混合阿尼希纳贝姆语-英语的对话评估。
误差模式记录与学习风险评估。
数据文件：4份对话测试文件。

核心研究发现

工具性能表现

AnythingTranslate与Musely.ai表现出完全不可靠性：每次查询均生成不同的翻译结果。
ChatGPT、Claude.ai与Perplexity整体性能最优，可理解语法结构，但始终存在词干翻译误差。
Ojibwechat具备一定理解能力，但会生成语法错误，且默认直接输出翻译结果。
所有系统均存在误差，需经流利母语使用者校验后方可使用。

典型翻译误差示例：英文指令“stop walking”经ChatGPT翻译后为“gego bimosen”（❌，实际意为“不要走路”），正确翻译应为“boon bimosen”（✓，意为“停止行走”）。
该示例体现了基于英语的推理逻辑（否定式与终止式混淆），会带来严重的学习风险。

JSON防护框架测试结果

成功实现符合文化规范的行为约束（双语回复格式、温和纠错方法）。
尽管实现了行为约束，底层的翻译准确性问题依然存在。
证明了原住民教学法与AI结合的可行性，但存在显著局限性。

风险评估

最高风险等级：无校验渠道的初学者（无法识别翻译误差）。
中等风险等级：可偶尔接触流利母语使用者的中级学习者。
最低风险等级：可定期接触社区母语使用者的高级学习者。
保留地外的第二语言学习者群体面临的风险尤为严峻（根据2021年加拿大统计局数据，该群体占原住民语言使用者的三分之一）。

研究背景

本研究针对1991年至2021年间原住民语言使用者减少4%（加拿大统计局数据）以及第二语言学习者占比持续上升的现状展开。随着原住民社区寻求技术工具以支持语言振兴，本数据集为评估AI工具安全性、开发社区自主可控的替代方案提供了关键依据。

本数据集可为构建以文化为基础的AI评估框架提供支撑，特别是为已提出的“阿尼希纳贝姆图灵测试”（Anishinaabe Turing Test）提供助力——该测试以原住民知识规范而非欧洲认知模式为标准来评估AI系统。

数据集内容

第一阶段数据文件

2025-08-12-14 Ojibwe Translation Tests [共4个CSV文件]：包含17个词汇与2个复合语句的工具对比分析结果。

第二阶段数据文件

AI TEST 2025-08-17 JSON File.md：JSON格式的文化规范防护框架文档。
AI TEST 2025-08-17 JSON [共4个MD文件]：JSON防护框架的对话测试结果（覆盖Claude、ChatGPT、Perplexity）。

第三阶段数据文件

AI TEST [共4个MD文件]：无防护框架的开放式对话测试结果。

辅助材料

methodology.md：详细的测试流程与校验步骤文档。
README.md：完整的数据集说明与使用指南。
LICENSE：CC BY 4.0许可证文本。

测试方法

本次测试采用标准化流程，具体包括：
- 所有测试工具使用统一的词汇/语句列表（17个词汇+2个复合语句）
- 多名研究人员开展交叉校验
- 对照阿尼希纳贝姆语印刷版与在线权威词典进行校验
- 形成系统性文档以确保实验可复现
- 开展初始一致性追踪，以证明所有模型均存在误差
- 通过多次重复查询开展迭代测试，以评估工具可靠性

文化规范说明：本研究尊重原住民知识体系，旨在支持语言振兴工作。测试仅针对公开可用的AI工具开展，以评估社区使用安全。研究未使用或向AI系统共享传统知识，研究人员仅使用大语言模型（Large Language Model）已获取的语言数据，确保文化知识的安全性。

研究启示

面向研究人员

- 面向原住民语言的AI工具评估框架
- 以文化为基础的AI评估方法
- 商用翻译系统存在系统性误差的实证依据

面向教育工作者

- 面向课堂使用的工具安全性循证评估
- 误差模式记录，可用于课后纠错流程
- 需经流利母语使用者校验的流程规范

面向原住民社区

- 为社区在语言项目中集成AI工具前的采用决策提供依据
- 为社区自主可控的AI开发项目提供支持
- 验证语言技术领域的数据主权原则
- 防范不可靠AI系统可能带来的危害

教育影响

AI系统对孤立学习者存在较高风险，这类学习者可能会将错误翻译当作正确的语言模式内化。当前工具需要高强度的社区监督与系统性校验流程。若无法接触流利母语使用者，初学者无法区分正确与错误的输出结果，可能引发长期的语言习得问题。

技术规格

数据格式：CSV（逗号分隔值）、Markdown（.md）；字符编码：UTF-8；测试周期：2025年8月12日至17日；测试AI系统数量：5款翻译工具+3款对话式大语言模型；研究背景：约克大学数字媒体专业博士研究；资助方：IBET（Indigenous and Black Engineering and Technology，原住民与黑人工程技术组织）、Abundant Intelligences多伦多分部（Abundant Intelligences Tkaronto Pod）。

引用规范

使用本数据集时，请按照以下格式引用：
McConnell, A., & Ly, J. (2025). Anishinaabemowin AI Translation Tool Evaluation Dataset (Version 1.0.0) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.17274098

关联出版物：McConnell, A., & Ly, J. (2025). "Anishinaabemowin Aabajichigan Gaawiin Nibwaakaasii (Efficacy of AI for Ojibwe Language Education)." Connected Minds 2025, Toronto, ON, October 6-8, 2025.

BibTeX引用格式：
@dataset{mcconnell_anishinaabemowin_2025, author = {McConnell, Andrew and Ly, Jasmine}, title = {Anishinaabemowin AI Translation Tool Evaluation Dataset}, year = {2025}, publisher = {Zenodo}, version = {1.0.0}, doi = {10.5281/zenodo.17274098}, url = {https://doi.org/10.5281/zenodo.17274098} }

关键词
原住民语言、阿尼希纳贝姆语、奥吉布瓦语、人工智能、机器翻译、语言振兴、语言教育、AI评估、原住民数据主权、文化适配性、语言学习、第二语言习得、原住民教学法、AI伦理、语言技术、ChatGPT、Claude、Perplexity、翻译工具、JSON防护框架、原住民未来主义、保留地外语言学习

许可证
知识共享署名4.0国际许可协议（CC BY 4.0）https://creativecommons.org/licenses/by/4.0/

致谢
本研究得到IBET（原住民与黑人工程技术组织）与Abundant Intelligences多伦多分部的资助，为约克大学数字媒体专业博士研究的一部分。

GitHub仓库
完整数据集与文档可通过以下链接获取：https://github.com/giigdo/AI_Bot_Tests_Q3_2025