biligcrawl/WikipediaTR

# Title This product is for ... ### Table of Contents - [Prerequisites](#prerequisites) - [Architecture](#architecture) - [Features](#features) - [Demo](#demo) - [Reports](#reports) - [Installation](#installation) - [File Structure](#file-structure) - [Version Control System](#version-control-system) - [Upcoming](#upcoming) - [Documentations](#documentations) - [License](#license) - [Links](#links) - [Team](#team) - [Contact](#contact) - [Resources](#resources) - [Citation](#citation) ###### Github Action Test ### Prerequisites - Python 3.10 - Nvidia T4 GPU or higher - Microsoft SQL - Llama 2 - Whisper ### Architecture  ### Features - [x] ................ - [x] ................ - [x] ................ ### Demo [Video](videoURL) [](videoURL)  ### Reports  ### Installation *__for Linux/Ubuntu__* - Do the following operations in order. #### 1. Install Nvidia Drivers - System Update & Upgrade ```bash sudo apt update ``` ```bash sudo apt upgrade ``` - Install Nvidia Drivers ```bash sudo apt install ubuntu-drivers-common ``` ```bash ubuntu-drivers devices ``` ```bash sudo ubuntu-drivers autoinstall ``` - System Reboot ```bash sudo reboot ``` - Check Nvidia Driver and Note CUDA Version ```bash nvidia-smi ``` - *__Note:__* Yukarıdaki adımda hem Nvidia Driver'ın yüklenip yüklenmediğini kontrol edin. Hem de yüklemeniz gereken CUDA versiyonunu öğreninin. ### 2. Install CUDA 12.2 for Ubuntu 22.04 ```bash wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-ubuntu2204.pin ``` ```bash sudo mv cuda-ubuntu2204.pin /etc/apt/preferences.d/cuda-repository-pin-600 ``` ```bash wget https://developer.download.nvidia.com/compute/cuda/12.2.0/local_installers/cuda-repo-ubuntu2204-12-2-local_12.2.0-535.54.03-1_amd64.deb ``` ```bash sudo dpkg -i cuda-repo-ubuntu2204-12-2-local_12.2.0-535.54.03-1_amd64.deb ``` ```bash sudo cp /var/cuda-repo-ubuntu2204-12-2-local/cuda-*-keyring.gpg /usr/share/keyrings/ ``` ```bash sudo apt-get update ``` ```bash sudo apt-get -y install cuda ``` ```bash sudo find / -name nvcc ``` ```bash echo 'export PATH=/usr/local/cuda-12.3/bin:$PATH' >> ~/.bashrc ``` ```bash source ~/.bashrc ``` ```bash nvcc --version ``` --- #### 3. Python 3.10.0 Installation ```bash sudo apt install build-essential zlib1g-dev libncurses5-dev libgdbm-dev libnss3-dev libssl-dev libreadline-dev libffi-dev libsqlite3-dev wget ``` ```bash sudo apt-get install libbz2-dev ``` ```bash sudo apt-get install libgdbm-compat-dev ``` ```bash sudo apt-get install liblzma-dev ``` ```bash sudo apt-get install tk-dev ``` ```bash sudo apt-get install uuid-dev ``` ```bash sudo apt-get install zlib1g-dev ``` ```bash wget https://www.python.org/ftp/python/3.10.0/Python-3.10.0.tar.xz ``` ```bash tar -xf Python-3.10.0.tar.xz ``` ```bash cd Python-3.10.0 ``` ```bash ./configure --enable-optimizations --prefix=/usr/local ``` ```bash make -j $(nproc) ``` ```bash sudo make altinstall ``` --- ##### 4. Get Repository - Clone Repository ```bash git clone https://github.com/organization_name/repo_name ``` - Navigate to the project directory: ```bash cd Repository_Directory ``` - Create Virtual Environment named `.venv` ```bash python3.10 -m venv .venv ``` - Activate virtual environment ```bash source .venv/bin/activate ``` --- #### 5. Install Pytorch ```bash pip install torch --index-url https://download.pytorch.org/whl/cu122 ``` - *__Note:__* Lütfen farklı CUDA verdiyonu kullanıyorsanız Pytorch versiyonunuzu değiştirin. #### 6. Download Llama and Install - Visit the `https://ai.meta.com/llama/` and get download link for Llama 2 ```bash git clone https://github.com/facebookresearch/llama.git ``` ```bash cd llama ``` ```bash pip install -e . ``` ```bash chmod +x download.sh ``` ```bash ./download.sh ``` - *__Note__*: If you get the error `"The file is already fully retrieved; nothing to do."`, delete the `--continue` commands in the `download.sh` file and try again. Related issue link: `https://github.com/facebookresearch/llama/issues/760` --- #### 7. Install Requirements for Audio Process ```bash sudo apt-get install unixodbc-dev ``` ```bash pip install Cython ``` ```bash pip install -r requirements/requirements_audio.txt ``` --- #### 8. Install Requirements for Architecture ```bash pip install -r requirements/requirements_arctitecture.txt ``` --- #### 9. Install ODBC Driver 18 ```bash curl https://packages.microsoft.com/keys/microsoft.asc | sudo tee /etc/apt/trusted.gpg.d/microsoft.asc ``` ```bash curl https://packages.microsoft.com/config/ubuntu/$(lsb_release -rs)/prod.list | sudo tee /etc/apt/sources.list.d/mssql-release.list ``` ```bash sudo apt-get update ``` ```bash sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18 ``` --- #### 10. Install FFmpeg ```bash sudo apt install ffmpeg ``` --- #### 11. Install Linux Services ```bash sudo cp /home/azureuser/directory/services/audio.service /etc/systemd/system/ ``` ```bash sudo cp /home/azureuser/directory/services/text.service /etc/systemd/system/ ``` ```bash sudo systemctl daemon-reload ``` ```bash sudo systemctl enable audio.service ``` ```bash sudo systemctl enable text.service ``` ```bash sudo systemctl start audio.service ``` ```bash sudo systemctl start text.service ``` ```bash sudo systemctl status audio.service ``` ```bash sudo systemctl status text.service ``` --- ### File Structure ```text . ├── README.md ├── TODO.md ├── app.py ├── examples │ ├── example_1.wav │ ├── example_2.wav ├── flagged ├── gradio_app.py ├── outputs │ ├── example_1.json ├── requirements.txt └── test.py ``` --- ### Version Control System #### Releases - [v0.1](https://github.com/organization_name/repo_name/archive/refs/tags/v0.1.zip) #.zip - [v0.1](https://github.com/organization_name/repo_name/archive/refs/tags/v0.1.tar.gz) #.tar.gz #### Branches - [BranchName](https://github.com/organization_name/repo_name/tree/Stream) #### Related Repos - [RepoName](https://github.com/organization_name/repo_name) --- ### Upcoming - [ ] .... --- ### Documentations #### Turkish Readme - [README_TR](docs/README_TR.md) --- #### TO DO list - [TODO](TODO.md) --- #### Contributing Guidence - [Contribute](CONTRIBUTE.md) ### License - [LICENSE](LICENSE) --- ### Links - [Github](https://github.com/repo) - [Website](https://company_name.com/en) - [Linkedin](https://www.linkedin.com/company/company_name/) --- ### Team - [Bunyamin Ergen](https://www.linkedin.com/in/bunyaminergen) --- ### Contact - [Mail](mailto:[e-mail]) --- ### Resources - [Resources](RESOURCES.md) --- ### Citation - Reference to cite if you use this work in a paper or research project: ``` text @software{software_2024, author = {Ergen}, doi = {00.0000/zenodo.0000}, month = {01}, title = {{software_2024}}, url = {https://github.com/repo_name}, year = {2024} } ``` --- ---

原始信息汇总

数据集概述

数据集内容

• 文件结构： text . ├── README.md ├── TODO.md ├── app.py ├── examples │ ├── example_1.wav │ ├── example_2.wav ├── flagged ├── gradio_app.py ├── outputs │ ├── example_1.json ├── requirements.txt └── test.py

数据集安装指南

• 系统要求：

  • Python 3.10
  • Nvidia T4 GPU或更高
  • Microsoft SQL
  • Llama 2
  • Whisper

• 安装步骤：

  1. Nvidia驱动安装：
     • 更新系统并安装Nvidia驱动。
  2. CUDA 12.2安装：
     • 为Ubuntu 22.04安装CUDA 12.2。
  3. Python 3.10.0安装：
     • 在Ubuntu系统上安装Python 3.10.0。
  4. 获取仓库：
     • 克隆仓库并设置虚拟环境。
  5. 安装Pytorch：
     • 根据CUDA版本安装Pytorch。
  6. 下载并安装Llama：
     • 从指定链接下载Llama 2并安装。
  7. 安装音频处理要求：
     • 安装必要的音频处理库。
  8. 安装架构要求：
     • 安装架构相关的依赖。
  9. 安装ODBC驱动18：
     • 安装Microsoft ODBC驱动18。
  10. 安装FFmpeg：
  • 安装FFmpeg用于音频处理。
  11. 安装Linux服务：
  • 配置并启用音频和文本服务。

版本控制

• 发布版本：

  • v0.1 (zip格式)
  • v0.1 (tar.gz格式)

• 分支：

  • Stream分支

文档与资源

• 土耳其语README：

  • README_TR

• TODO列表：

  • TODO

• 贡献指南：

  • Contribute

许可证

• LICENSE

引用信息

• 引用格式： text @software{software_2024, author = {Ergen}, doi = {00.0000/zenodo.0000}, month = {01}, title = {{software_2024}}, url = {https://github.com/repo_name}, year = {2024} }