guychuk/code-duplicates-across-languages

该数据集包含两个特征：code和label。code是字符串类型，表示代码片段；label是分类标签，包含了从0到490的类别名称，每个类别名称对应一个具体的任务或主题，如Ascending primes、Array concatenation等。该数据集包含两个特征：code和label。code是字符串类型，表示代码片段；label是分类标签，包含了从0到490的类别名称，每个类别名称对应一个具体的任务或主题，如Ascending primes、Array concatenation等。

原始信息汇总

数据集概述

许可证

• Apache 2.0

数据集信息

• 特征
  • code: 数据类型为字符串（string）。
  • label: 数据类型为分类标签（class_label），包含以下类别：
    • 0: Ascending primes
    • 1: Array concatenation
    • 2: ASCII art diagram converter
    • 3: Array length
    • 4: Arithmetic/Integer
    • 5: Arithmetic numbers
    • 6: Arithmetic-geometric mean/Calculate Pi
    • 7: Arena storage pool
    • 8: Arithmetic/Rational
    • 9: Arithmetic/Complex
    • 10: Arithmetic-geometric mean
    • 11: Arithmetic evaluation
    • 12: Archimedean spiral
    • 13: Arrays
    • 14: Zero to the zero power
    • 15: Zhang-Suen thinning algorithm
    • 16: Zeckendorf number representation
    • 17: Zeckendorf arithmetic
    • 18: Arbitrary-precision integers (included)
    • 19: Zumkeller numbers
    • 20: Yellowstone sequence
    • 21: 100 doors
    • 22: Yahoo! search interface
    • 23: XML/XPath
    • 24: Zebra puzzle
    • 25: Y combinator
    • 26: Yin and yang
    • 27: Zig-zag matrix
    • 28: Write to Windows event log
    • 29: Write entire file
    • 30: Write float arrays to a text file
    • 31: XML/DOM serialization
    • 32: World Cup group stage
    • 33: Xiaolin Wus line algorithm
    • 34: XML/Output
    • 35: XML/Input
    • 36: Wordiff
    • 37: Write language name in 3D ASCII
    • 38: Word wheel
    • 39: Word ladder
    • 40: Window management
    • 41: Word search
    • 42: Window creation/X11
    • 43: Wilson primes of order n
    • 44: Word wrap
    • 45: Wieferich primes
    • 46: Window creation
    • 47: Weird numbers
    • 48: Wireworld
    • 49: Web scraping
    • 50: Word frequency
    • 51: War card game
    • 52: Vogels approximation method
    • 53: Walk a directory/Non-recursively
    • 54: Voronoi diagram
    • 55: Vigenère cipher/Cryptanalysis
    • 56: Water collected between towers
    • 57: Visualize a tree
    • 58: Walk a directory/Recursively
    • 59: Verify distribution uniformity/Naive
    • 60: Verhoeff algorithm
    • 61: Vigenère cipher
    • 62: Verify distribution uniformity/Chi-squared test
    • 63: Video display modes
    • 64: Variable-length quantity
    • 65: Variable size/Set
    • 66: Vector
    • 67: Variadic function
    • 68: Variable size/Get
    • 69: Variable declaration reset
    • 70: Vampire number
    • 71: Validate International Securities Identification Number
    • 72: Vector products
    • 73: Van der Corput sequence
    • 74: Van Eck sequence
    • 75: Use another language to call a function
    • 76: Variables
    • 77: URL parser
    • 78: URL encoding
    • 79: UTF-8 encode and decode
    • 80: URL decoding
    • 81: Update a configuration file
    • 82: User input/Graphical
    • 83: User input/Text
    • 84: UPC
    • 85: Undefined values
    • 86: Unix/ls
    • 87: Unicode variable names
    • 88: Unprimeable numbers
    • 89: Ultra useful primes
    • 90: Unbias a random generator
    • 91: Untouchable numbers
    • 92: Unicode strings
    • 93: Ukkonen’s suffix tree construction
    • 94: Two bullet roulette
    • 95: Twin primes
    • 96: Truncate a file
    • 97: Ulam spiral (for primes)
    • 98: Truth table
    • 99: Tree datastructures
    • 100: Twelve statements
    • 101: Tree from nesting levels
    • 102: Truncatable primes
    • 103: Universal Turing machine
    • 104: Topswops
    • 105: Topic variable
    • 106: Trabb Pardo–Knuth algorithm
    • 107: Total circles area
    • 108: Totient function
    • 109: Tonelli-Shanks algorithm
    • 110: Topological sort
    • 111: Trigonometric functions
    • 112: Tokenize a string with escaping
    • 113: Tree traversal
    • 114: Thue-Morse
    • 115: Top rank per group
    • 116: Tokenize a string
    • 117: Tic-tac-toe
    • 118: Time a function
    • 119: Towers of Hanoi
    • 120: The sieve of Sundaram
    • 121: Thieles interpolation formula
    • 122: The Name Game
    • 123: Textonyms
    • 124: Text processing/2
    • 125: Text processing/Max licenses in use
    • 126: Terminal control/Unicode output
    • 127: Terminal control/Ringing the terminal bell
    • 128: The ISAAC Cipher
    • 129: Test integerness
    • 130: Test a function
    • 131: Terminal control/Preserve screen
    • 132: Terminal control/Positional read
    • 133: Terminal control/Hiding the cursor
    • 134: Terminal control/Inverse video
    • 135: Terminal control/Display an extended character
    • 136: Text processing/1
    • 137: Terminal control/Dimensions
    • 138: Ternary logic
    • 139: The Twelve Days of Christmas
    • 140: Terminal control/Cursor positioning
    • 141: Terminal control/Cursor movement
    • 142: Terminal control/Coloured text
    • 143: Teacup rim text
    • 144: Taxicab numbers
    • 145: Tau number
    • 146: Tarjan
    • 147: Tau function
    • 148: Table creation/Postal addresses
    • 149: Synchronous concurrency
    • 150: Take notes on the command line
    • 151: Terminal control/Clear the screen
    • 152: Superpermutation minimisation
    • 153: Super-d numbers
    • 154: Sutherland-Hodgman polygon clipping
    • 155: Sylvesters sequence
    • 156: Superellipse
    • 157: Temperature conversion
    • 158: Summarize primes
    • 159: Symmetric difference
    • 160: Summarize and say sequence
    • 161: System time
    • 162: Sum of elements below main diagonal of matrix
    • 163: Sum and product puzzle
    • 164: Suffixation of decimal numbers
    • 165: Sum to 100
    • 166: Sum of squares
    • 167: Sum multiples of 3 and 5
    • 168: Substitution cipher
    • 169: Sum digits of an integer
    • 170: Sum and product of an array
    • 171: Sum of a series
    • 172: Subtractive generator
    • 173: Successive prime differences
    • 174: Strong and weak primes
    • 175: Substring/Top and tail
    • 176: Sudoku
    • 177: Strip block comments
    • 178: Subleq
    • 179: String prepend
    • 180: Strip control codes and extended characters from a string
    • 181: Strip whitespace from a string/Top and tail
    • 182: Strip comments from a string
    • 183: Substring
    • 184: Strip a set of characters from a string
    • 185: String concatenation
    • 186: String interpolation (included)
    • 187: String comparison
    • 188: String matching
    • 189: String case
    • 190: String length
    • 191: Stirling numbers of the second kind
    • 192: Strassens algorithm
    • 193: String append
    • 194: Straddling checkerboard
    • 195: Stirling numbers of the first kind
    • 196: Stream merge
    • 197: Statistics/Normal distribution
    • 198: Stem-and-leaf plot
    • 199: State name puzzle
    • 200: Start from a main routine
    • 201: Stern-Brocot sequence
    • 202: Stack traces
    • 203: Stair-climbing puzzle
    • 204: Square form factorization
    • 205: Square-free integers
    • 206: Statistics/Basic
    • 207: SQL-based authentication
    • 208: Square but not cube
    • 209: Speech synthesis
    • 210: Split a character string based on change of character
    • 211: Spinning rod animation/Text
    • 212: Stack
    • 213: Spelling of ordinal numbers
    • 214: Stable marriage problem
    • 215: Special variables
    • 216: Sorting algorithms/Strand sort
    • 217: Sparkline in unicode
    • 218: Spiral matrix
    • 219: Special characters
    • 220: Sorting algorithms/Sleep sort
    • 221: Sorting algorithms/Stooge sort
    • 222: Sorting algorithms/Permutation sort
    • 223: Sorting algorithms/Shell sort
    • 224: Soundex
    • 225: Sorting algorithms/Radix sort
    • 226: Sorting algorithms/Selection sort
    • 227: Sorting algorithms/Patience sort
    • 228: Sorting algorithms/Pancake sort
    • 229: Sorting algorithms/Insertion sort
    • 230: Sorting algorithms/Quicksort
    • 231: Sorting algorithms/Heapsort
    • 232: Sorting algorithms/Merge sort
    • 233: Sorting algorithms/Cocktail sort with shifting bounds
    • 234: Sorting Algorithms/Circle Sort
    • 235: Sorting algorithms/Comb sort
    • 236: Sorting algorithms/Bogosort
    • 237: Sorting algorithms/Counting sort
    • 238: Sorting algorithms/Gnome sort
    • 239: Sorting algorithms/Bead sort
    • 240: Sorting algorithms/Cocktail sort
    • 241: Sort an outline at every level
    • 242: Sort three variables
    • 243: Sort using a custom comparator
    • 244: Sort numbers lexicographically
    • 245: Sort stability
    • 246: Sorting algorithms/Bubble sort
    • 247: Sort disjoint sublist
    • 248: Sort a list of object identifiers
    • 249: Solve a Numbrix puzzle
    • 250: Solve a Hopido puzzle
    • 251: Solve the no connection puzzle
    • 252: Sort an integer array
    • 253: SOAP
    • 254: Smallest number k such that k+2^m is composite for all m less than k
    • 255: Sort an array of composite structures
    • 256: Solve a Holy Knights tour
    • 257: Sokoban
    • 258: Sockets
    • 259: Smarandache prime-digital sequence
    • 260: Sleeping Beauty problem
    • 261: Solve a Hidato puzzle
    • 262: Snake
    • 263: Smith numbers
    • 264: Singleton
    • 265: Simulate input/Mouse
    • 266: Simulate input/Keyboard
    • 267: Simple turtle graphics
    • 268: Singly-linked list/Element definition
    • 269: Singly-linked list/Traversal
    • 270: Singly-linked list/Element insertion
    • 271: Sleep
    • 272: Simple windowed application
    • 273: Sierpinski square curve
    • 274: Sierpinski arrowhead curve
    • 275: Sierpinski triangle/Graphical
    • 276: Sierpinski pentagon
    • 277: Show the epoch
    • 278: Sierpinski triangle
    • 279: Simple database
    • 280: Shortest common supersequence
    • 281: Shoelace formula for polygonal area
    • 282: Shell one-liner
    • 283: Sierpinski carpet
    • 284: Short-circuit evaluation
    • 285: SHA-256 Merkle tree
    • 286: Sexy primes
    • 287: Set right-adjacent bits
    • 288: Show ASCII table
    • 289: SHA-1
    • 290: SHA-256
    • 291: Set puzzle
    • 292: Seven-sided dice from five-sided dice
    • 293: Sequence: smallest number greater than previous term with exactly n divisors
    • 294: Sequence: nth number with exactly n divisors
    • 295: Set consolidation
    • 296: Sequence: smallest number with exactly n divisors
    • 297: Sequence of primorial primes
    • 298: Set of real numbers
    • 299: Send an unknown method call
    • 300: Send email
    • 301: Selectively replace multiple instances of a character within a string
    • 302: Self numbers
    • 303: Sequence of primes by trial division
    • 304: Semiprime
    • 305: Semordnilap
    • 306: Sequence of non-squares
    • 307: Set
    • 308: Self-describing numbers
    • 309: Sieve of Eratosthenes
    • 310: SEDOLs
    • 311: Search in paragraphs text
    • 312: Secure temporary file
    • 313: Scope modifiers
    • 314: Scope/Function names and labels
    • 315: Sailors, coconuts and a monkey problem
    • 316: Same fringe
    • 317: Search a list of records
    • 318: Safe addition
    • 319: Safe primes and unsafe primes
    • 320: Ruth-Aaron numbers
    • 321: Runtime evaluation
    • 322: Runtime evaluation/In an environment
    • 323: Search a list
    • 324: RSA code
    • 325: Rosetta Code/Rank languages by number of users
    • 326: Runge-Kutta method
    • 327: S-expressions
    • 328: RPG attributes generator

搜集汇总

数据集介绍

构建方式

该数据集构建于Rosetta Code社区的多语言编程任务资源之上，通过系统性地收集同一编程问题在不同编程语言中的实现代码，形成跨语言代码对。每条数据包含一个代码片段及其对应的标签，标签为具体编程任务的名称，例如“Ascending primes”或“Array concatenation”，从而将语义上等价但语法迥异的代码片段归并为同一类别。数据集以HuggingFace的Datasets格式存储，包含code和label两个字段，其中label为整数索引，映射到预定义的任务名称列表。

使用方法

该数据集适用于监督学习场景下的代码分类与相似性检测任务。用户可通过HuggingFace的datasets库直接加载，使用load_dataset('guychuk/code-duplicates-across-languages')即可获取。数据划分为训练集与测试集，其中code字段作为模型输入，label字段作为分类目标。研究者可基于此构建跨语言代码嵌入模型，或用于训练代码重复检测器，以识别不同语言中实现相同功能的代码片段。建议在使用前对代码进行适当的预处理，如去除注释或统一标识符命名风格，以提升模型泛化能力。

背景与挑战

背景概述

在软件工程与自然语言处理交叉领域，跨语言代码重复检测是一项极具挑战性的任务，它对于代码克隆检测、程序理解、以及跨语言迁移学习等研究具有重要价值。guychuk/code-duplicates-across-languages数据集由研究者构建，旨在系统性地收集和标注不同编程语言间功能等价的代码片段。该数据集以Rosetta Code社区中丰富的多语言编程任务为基础，涵盖了从基础算法到复杂数据结构的数百个编程问题，每个问题均配有多种语言的实现。其核心研究问题在于如何识别并量化不同语言间代码的语义等价性，从而为跨语言代码分析提供可靠的基准。该数据集的发布，不仅推动了代码相似性度量方法的发展，也为跨语言程序翻译和代码搜索等应用提供了关键支撑。

当前挑战

该数据集所面临的挑战主要源于跨语言代码重复检测的固有复杂性。首先，不同编程语言在语法结构、类型系统、以及标准库使用上存在巨大差异，使得基于文本或抽象语法树的传统克隆检测方法难以直接适用，需要设计能够捕获语义等价性的高级特征表示。其次，构建过程中，从Rosetta Code中提取并标注的代码片段虽功能相同，但可能因语言习惯差异导致实现风格迥异，如何确保标注的准确性和一致性是一大难题。此外，数据集中包含的数百个任务标签虽然丰富，但部分任务仅有一种或少数几种语言的实现，导致数据分布不均衡，增加了模型训练的难度。最后，代码片段可能包含非功能性差异（如注释、命名规范），如何有效滤除这些噪声，聚焦于核心逻辑的等价性判断，也是亟待解决的技术瓶颈。

常用场景

经典使用场景

在跨语言代码分析与理解领域，guychuk/code-duplicates-across-languages数据集为研究者提供了弥足珍贵的资源。其核心使用场景在于检测和识别不同编程语言中实现相同功能的代码片段，即跨语言代码克隆检测。通过将来自Rosetta Code社区的数千个编程任务解决方案汇聚一堂，该数据集以任务标签为锚点，将多语言实现代码组织成平行语料，使得研究者能够系统性地探索代码在不同语言范式下的语义等价性与语法变异性。这一场景不仅支撑了跨语言代码检索、代码迁移与自动翻译等前沿课题，还为理解编程语言间的表达差异与共性规律奠定了坚实的数据基础。

解决学术问题

该数据集精准回应了代码克隆检测领域长期存在的跨语言瓶颈。传统克隆检测研究多聚焦于单语言环境，难以捕捉异构语言间功能等价的代码对，而guychuk/code-duplicates-across-languages通过提供标注明确的跨语言代码副本，使学术界得以突破语言壁垒，深入探究语义相似性建模、跨语言抽象语法树对齐以及语言无关的代码表示学习等核心问题。其意义在于推动了从‘语法匹配’向‘语义理解’的范式转变，显著提升了代码检索与复用系统的泛化能力，对软件工程、程序语言理论与自然语言处理交叉领域产生了深远影响。

实际应用

在实际应用中，该数据集赋能了多项具有高价值的工程实践。例如，在大型软件系统的维护与重构中，开发者可利用基于此数据集训练的模型，自动识别不同语言编写的遗留代码模块与新型实现之间的功能重复，从而制定精准的迁移或淘汰策略。此外，在线编程教育平台可借助跨语言代码匹配技术，为学习者提供多语言实现对照，辅助理解算法本质与语言特性差异。在代码搜索引擎与智能编程助手中，该数据集支撑了跨语言代码推荐与问答功能，使用户能够以自然语言或示例代码为查询，获取多种语言的等效实现，极大提升了开发效率。

数据集最近研究

最新研究方向

该数据集聚焦于跨语言代码重复检测这一前沿方向，在软件工程与编程语言研究领域具有重要价值。随着多语言编程生态的蓬勃发展，开发者常在不同语言间复用逻辑片段，而传统代码克隆检测多局限于单一语言。本数据集基于Rosetta Code社区中同一编程任务的多语言实现，构建了包含数百种算法题目的跨语言代码对，为研究语义等价的代码片段在不同语法体系下的表征与匹配提供了宝贵资源。当前热点事件如大规模语言模型在代码生成与理解中的广泛应用，使得跨语言代码重复问题成为评估模型泛化能力的关键基准。该数据集不仅有助于推动代码相似性分析、跨语言迁移学习等前沿研究，还为构建更鲁棒的多语言代码智能工具奠定了数据基础，对提升软件复用效率与代码质量具有深远意义。