matlok/python-text-copilot-training-instruct

--- license: - other pretty_name: >- python copilot instructions on how to code using alpaca and yaml dataset_info: - config_name: view_01_transformers_src splits: - name: view_01_transformers_src - config_name: view_02_pytorch_fsdp splits: - name: view_02_pytorch_fsdp - config_name: view_03_deepspeed_runtime splits: - name: view_03_deepspeed_runtime - config_name: view_schema splits: - name: view_schema configs: - config_name: view_01_transformers_src data_files: - split: view_01_transformers_src path: files/lok-python-copilot-text.instruct-v1_00000053.parquet - config_name: view_02_pytorch_fsdp data_files: - split: view_02_pytorch_fsdp path: files/lok-python-copilot-text.instruct-v1_00000040.parquet - config_name: view_03_deepspeed_runtime data_files: - split: view_03_deepspeed_runtime path: files/lok-python-copilot-text.instruct-v1_00000019.parquet - config_name: view_schema data_files: - split: view_schema path: files/lok-python-copilot-text.instruct-v1_00000002.parquet size_categories: - 1M<n<10M tags: - python-copilot - python-coding - python-architecture - knowledge-graphs - multimodal - text-image-audio - fine-tuning - training - question-answering - image-knowledge-graph - alpaca - mp3 - png - text - instruct - coding - task - prompt - response - yaml # supported task_categories # text-classification, token-classification, table-question-answering, question-answering, zero-shot-classification, translation, summarization, conversational, feature-extraction, text-generation, text2text-generation, fill-mask, sentence-similarity, text-to-speech, text-to-audio, automatic-speech-recognition, audio-to-audio, audio-classification, voice-activity-detection, depth-estimation, image-classification, object-detection, image-segmentation, text-to-image, image-to-text, image-to-image, image-to-video, unconditional-image-generation, video-classification, reinforcement-learning, robotics, tabular-classification, tabular-regression, tabular-to-text, table-to-text, multiple-choice, text-retrieval, time-series-forecasting, text-to-video, visual-question-answering, document-question-answering, zero-shot-image-classification, graph-ml, mask-generation, zero-shot-object-detection, text-to-3d, image-to-3d, other task_categories: - text-generation - question-answering # supported task_ids # acceptability-classification, entity-linking-classification, fact-checking, intent-classification, language-identification, multi-class-classification, multi-label-classification, multi-input-text-classification, natural-language-inference, semantic-similarity-classification, sentiment-classification, topic-classification, semantic-similarity-scoring, sentiment-scoring, sentiment-analysis, hate-speech-detection, text-scoring, named-entity-recognition, part-of-speech, parsing, lemmatization, word-sense-disambiguation, coreference-resolution, extractive-qa, open-domain-qa, closed-domain-qa, news-articles-summarization, news-articles-headline-generation, dialogue-generation, dialogue-modeling, language-modeling, text-simplification, explanation-generation, abstractive-qa, open-domain-abstractive-qa, closed-domain-qa, open-book-qa, closed-book-qa, slot-filling, masked-language-modeling, keyword-spotting, speaker-identification, audio-intent-classification, audio-emotion-recognition, audio-language-identification, multi-label-image-classification, multi-class-image-classification, face-detection, vehicle-detection, instance-segmentation, semantic-segmentation, panoptic-segmentation, image-captioning, image-inpainting, image-colorization, super-resolution, grasping, task-planning, tabular-multi-class-classification, tabular-multi-label-classification, tabular-single-column-regression, rdf-to-text, multiple-choice-qa, multiple-choice-coreference-resolution, document-retrieval, utterance-retrieval, entity-linking-retrieval, fact-checking-retrieval, univariate-time-series-forecasting, multivariate-time-series-forecasting, visual-question-answering, document-question-answering task_ids: - parsing --- ## Python Copilot Instructions on How to Code using Alpaca and Yaml This dataset is a subset of the matlok python copilot datasets. Please refer to the [Multimodal Python Copilot Training Overview](https://huggingface.co/datasets/matlok/multimodal-python-copilot-training-overview) for more details on how to use this dataset. ### Details Each row contains python code, either a class method or a global function, imported modules, base classes (if any), exceptions (ordered based off the code), returns (ordered based off the code), arguments (ordered based off the code), and more. - Rows: 1737704 - Size: 28.6 GB - Data type: text - Format: Introduction on code usage using alpaca and yaml response ### Schema The instruction alpaca text with yaml response is in the **desc** column: ```json { "active": "bool", "args": "string", "args_len": "float64", "audio_file": "string", "audio_path": "string", "class_bases": "string", "class_name": "string", "code": "string", "code_len": "float64", "desc": "string", "desc_docstr": "string", "desc_docstr_len": "float64", "desc_len": "int64", "docstr": "string", "docstr_len": "int64", "file_path": "string", "file_type": "string", "function_names": "string", "gen_bytes": "int64", "gen_data_type": "string", "gen_mode": "string", "gen_size": "int64", "gen_valid": "string", "height": "int64", "image_file": "string", "image_path": "string", "method_names": "string", "name": "string", "num_all_bases": "int64", "num_bases": "int64", "num_classes": "int64", "num_functions": "float64", "num_imports": "int64", "num_methods": "float64", "prompts": "string", "raises": "string", "raises_len": "float64", "recsize": "int64", "repo": "string", "returns": "string", "returns_len": "float64", "size": "int64", "src_object": "string", "sub_file": "string", "total_objects": "int64", "usage": "string", "usages": "string", "width": "int64" } ``` ### How to use the dataset ```python from datasets import load_dataset ds = load_dataset("matlok/python-text-copilot-training-instruct", data_dir="files") ```

原始信息汇总

数据集概述

数据集信息

• 配置名称和分割：
  • view_01_transformers_src
    • 分割名称：view_01_transformers_src
  • view_02_pytorch_fsdp
    • 分割名称：view_02_pytorch_fsdp
  • view_03_deepspeed_runtime
    • 分割名称：view_03_deepspeed_runtime
  • view_schema
    • 分割名称：view_schema

数据文件

• 配置名称和数据文件路径：
  • view_01_transformers_src
    • 分割：view_01_transformers_src
    • 路径：files/lok-python-copilot-text.instruct-v1_00000053.parquet
  • view_02_pytorch_fsdp
    • 分割：view_02_pytorch_fsdp
    • 路径：files/lok-python-copilot-text.instruct-v1_00000040.parquet
  • view_03_deepspeed_runtime
    • 分割：view_03_deepspeed_runtime
    • 路径：files/lok-python-copilot-text.instruct-v1_00000019.parquet
  • view_schema
    • 分割：view_schema
    • 路径：files/lok-python-copilot-text.instruct-v1_00000002.parquet

数据集规模

• 大小类别：
  • 1M<n<10M

标签

• python-copilot
• python-coding
• python-architecture
• knowledge-graphs
• multimodal
• text-image-audio
• fine-tuning
• training
• question-answering
• image-knowledge-graph
• alpaca
• mp3
• png
• text
• instruct
• coding
• task
• prompt
• response
• yaml

支持的任务类别

• text-generation
• question-answering

支持的任务ID

• parsing

数据集详细信息

• 行数：1737704
• 大小：28.6 GB
• 数据类型：text
• 格式：Introduction on code usage using alpaca and yaml response

数据集模式

• 描述列：desc
• 字段类型：
  • active: bool
  • args: string
  • args_len: float64
  • audio_file: string
  • audio_path: string
  • class_bases: string
  • class_name: string
  • code: string
  • code_len: float64
  • desc: string
  • desc_docstr: string
  • desc_docstr_len: float64
  • desc_len: int64
  • docstr: string
  • docstr_len: int64
  • file_path: string
  • file_type: string
  • function_names: string
  • gen_bytes: int64
  • gen_data_type: string
  • gen_mode: string
  • gen_size: int64
  • gen_valid: string
  • height: int64
  • image_file: string
  • image_path: string
  • method_names: string
  • name: string
  • num_all_bases: int64
  • num_bases: int64
  • num_classes: int64
  • num_functions: float64
  • num_imports: int64
  • num_methods: float64
  • prompts: string
  • raises: string
  • raises_len: float64
  • recsize: int64
  • repo: string
  • returns: string
  • returns_len: float64
  • size: int64
  • src_object: string
  • sub_file: string
  • total_objects: int64
  • usage: string
  • usages: string
  • width: int64

如何使用数据集

python from datasets import load_dataset

ds = load_dataset("matlok/python-text-copilot-training-instruct", data_dir="files")