gitgud-code
收藏资源简介：
# GitGud Code Dataset ## Dataset Description This dataset was compiled from code repositories hosted on [GitGud.io](https://gitgud.io), a GitLab-based code hosting platform. GitGud.io serves as an alternative git hosting service used by various developer communities and open-source projects. ### Dataset Summary | Statistic | Value | |-----------|-------| | **Total Files** | 16,322,315 | | **Total Repositories** | 7,204 | | **Total Size** | 17.46 GB (compressed Parquet) | | **Programming Languages** | 2,185 | | **File Format** | Parquet with Zstd compression (17 files) | ### Key Features - **Diverse code corpus**: Contains code from over 7,000 repositories across various domains - **Wide language coverage**: Spans 2,185 programming languages and file types detected by file extension mapping - **Rich metadata**: Includes repository name, file path, detected language, license information, and file size - **Quality filtered**: Filtering applied to remove binary files, overly long lines, and license files ### Languages The dataset includes 2,185 programming languages and file types. The top 30 languages by file count: | Rank | Language | File Count | |------|----------|------------| | 1 | tw (Twine) | 3,301,366 | | 2 | XML | 3,281,566 | | 3 | svg | 1,744,500 | | 4 | C# | 1,367,799 | | 5 | JavaScript | 1,252,710 | | 6 | C++ | 731,619 | | 7 | erb | 710,279 | | 8 | JSON | 398,139 | | 9 | Text | 377,948 | | 10 | twee | 300,576 | | 11 | csv | 205,230 | | 12 | HTML | 170,711 | | 13 | Markdown | 160,735 | | 14 | TypeScript | 147,173 | | 15 | Lua | 117,079 | | 16 | PHP | 116,059 | | 17 | none | 111,791 | | 18 | pal | 110,626 | | 19 | CSS | 108,664 | | 20 | Python | 106,261 | | 21 | dm | 98,333 | | 22 | Ruby | 93,685 | | 23 | _comment | 91,730 | | 24 | Java | 81,190 | | 25 | YAML | 63,289 | | 26 | ActionScript | 62,210 | | 27 | Git | 43,748 | | 28 | mdwn | 42,654 | | 29 | mk | 41,789 | | 30 | INI | 39,760 | ### Licenses The dataset includes files from repositories with various licenses: | License | File Count | |---------|------------| | mit | 9,517,343 | | bsd-3-clause | 3,315,732 | | unknown | 2,935,736 | | mpl-2.0 | 338,040 | | gpl-2.0 | 79,415 | | lgpl-2.1 | 38,429 | | gpl-3.0 | 25,964 | | apache-2.0 | 20,562 | | cc-by-4.0 | 18,703 | | agpl-3.0 | 15,367 | | cc-by-nc-4.0 | 6,362 | | wtfpl | 6,163 | | bsd-2-clause | 3,749 | | zlib | 482 | | unlicense | 261 | | cc-by-sa-4.0 | 7 | ## Dataset Structure ### Data Fields | Field | Type | Description | |-------|------|-------------| | `code` | string | Content of the source file (UTF-8 encoded) | | `repo_name` | string | Name of the GitGud repository (format: `username/repo`) | | `path` | string | Path of the file within the repository (relative to repo root) | | `language` | string | Programming language detected by file extension mapping | | `license` | string | License of the repository (SPDX identifier or "unknown") | | `size` | int64 | Size of the source file in bytes | ### Data Format - **Format**: Apache Parquet with Zstd compression (level 19) - **File Structure**: 17 files (`gitgud-00000.parquet` to `gitgud-00016.parquet`) - **Rows per shard**: ~1,000,000 (except last shard: 322,315) ### Data Splits All examples are in the train split. There is no validation or test split. ### Example Data Point ``` { 'code': 'using System;\nusing System.Collections.Generic;\n...', 'repo_name': 'username/game-mod', 'path': 'src/GameMod/Player.cs', 'language': 'C#', 'license': 'mit', 'size': 2048 } ``` ## Dataset Creation ### Pipeline Overview The dataset was created through a multi-stage pipeline: 1. **Repository Discovery**: Scraping public repository URLs from GitGud.io's GitLab API v4 endpoint using multiple sort orderings (`id`, `name`, `path`, `updated_at`, `star_count`, `last_activity_at`, `similarity`) 2. **Branch Enumeration**: Fetching all branches for each repository via the GitLab API 3. **Archive Download**: Downloading `.tar.gz` archives for each repository/branch combination 4. **Content Extraction**: Extracting and filtering source code files from archives 5. **Parquet Generation**: Writing filtered records to Parquet shards with Zstd compression ### Language Detection Programming languages are detected using file extension mapping. The pipeline maps ~80 programming languages by their file extensions, including: - **Major languages**: Python, JavaScript, TypeScript, C, C++, C#, Java, Go, Rust, Ruby, PHP - **Configuration**: JSON, YAML, TOML, XML, INI - **Markup**: HTML, CSS, Markdown, LaTeX - **Game development**: GLSL, HLSL, GDScript - **And many more** Files with unrecognized extensions are labeled with the extension itself (without the dot prefix). Files without extensions are labeled as "none" or by special filename matching (e.g., "Dockerfile", "Makefile"). ### License Detection Licenses are detected by: 1. Scanning for license files (`LICENSE`, `LICENSE.txt`, `LICENSE.md`, `COPYING`, `COPYING.txt`, `COPYING.md`) 2. Matching license text against known patterns (MIT, Apache 2.0, GPL variants, BSD, Creative Commons, MPL, ISC, Unlicense, Artistic, WTFPL, Zlib, etc.) 3. Defaulting to "unknown" if no license can be detected ### File Filtering Filtering is applied to ensure data quality: #### Size Limits | Limit | Value | |-------|-------| | Max repository archive size | 64 MB | | Max line length | 1,000 characters | #### Content Filtering - **Binary Detection**: Files with null bytes in the first 1KB are excluded - **UTF-8 Validation**: Files must be decodable as UTF-8 (with fallback to latin-1, cp1252, iso-8859-1) - **Long Lines**: Files with any line exceeding 1,000 characters are excluded - **License Files**: License files (LICENSE, COPYING, etc.) are excluded from the dataset (but used for license detection) ### Source Data All data originates from public repositories hosted on [GitGud.io](https://gitgud.io). ## Considerations for Using the Data ### Personal and Sensitive Information The dataset may contain: - Email addresses in code comments or configuration files - API keys or credentials that were accidentally committed - Personal information in comments or documentation Users should exercise caution and implement appropriate filtering when using this data. ### Licensing Information This dataset is a collection of source code from repositories with various licenses. Any use of all or part of the code gathered in this dataset must abide by the terms of the original licenses, including attribution clauses when relevant. The license field in each data point indicates the license of the source repository.
# gitgud.io 代码数据集卡片 ## 数据集摘要 本数据集收录自免费代码托管平台gitgud.io上的26,610个仓库与分支的源代码文件，涵盖多种编程语言，包含多样化的开源项目与个人仓库集合。 ## 数据集结构 ### 数据字段 本数据集包含以下字段： - `code`：源代码文件内容（字符串类型） - `repo_name`：gitgud.io 仓库名称（字符串类型） - `path`：文件在仓库内的路径（字符串类型） - `language`：通过文件扩展名推断得到的编程语言（字符串类型） - `license`：仓库对应的开源许可证（如可获取）（字符串类型） - `size`：源代码文件的字节大小（整数类型） ### 数据划分 所有样本均归属训练集，无验证集划分。 ### 数据格式 - **格式**：采用Zstandard压缩的JSON Lines（JSONL）格式（文件后缀为`.jsonl.zst`） - **文件结构**：共194个数据文件，命名格式为`gitgud_0000.jsonl.zst`至`gitgud_0193.jsonl.zst` - **总仓库数**：26,610个仓库与分支 - **筛选规则**：排除了单行长度超过1,000字符的文件 - **去重处理**：本数据集未执行任何去重操作
