Llama3-Tutorial（Llama 3 超等讲堂） 学习条记
课程资料[*]
[*] 五一 Llama 3 超等讲堂 | 第二节 Llama 3 微调个人小助手认知（XTuner版）_哔哩哔哩_bilibili
[*] SmartFlowAI/Llama3-Tutorial: Llama3-Tutorial（XTuner、LMDeploy、OpenCompass） (github.com)
[*] 开发机 (intern-ai.org.cn)
[*] Llama3-Tutorial/docs/hello_world.md at main · SmartFlowAI/Llama3-Tutorial (github.com)
[*] 预先预备
[*] 注册InternStudio、创建开发机并启动，这里要选择 “资源设置”，指的是GPU的资源。
[*] 安装VSCODE
[*] 创建长途链接
[*] 由于代码经常更新，以是须要实时同步最新的代码，假如那段步伐调试出现标题，可以先实验更新长途代码库。
git branch -r #检察分支
git checkout main
git pull origin main
第一节 本地 Web Demo 摆设
#安装环境
conda create -n llama3 python=3.10
conda activate llama3
conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=12.1 -c pytorch -c nvidia
#通过软毗连链接大模子文件。
ln -s /root/share/new_models/meta-llama/Meta-Llama-3-8B-Instruct ~/model/Meta-Llama-3-8B-Instruct
#运行，下面有两种方式， 发起采取quant模式，占用资源较少。在最小的“资源设置”的开发机上也可以跑。
streamlit run ~/Llama3-Tutorial/tools/internstudio_web_demo.py ~/model/Meta-Llama-3-8B-Instruct
streamlit run ~/Llama3-Tutorial/tools/internstudio_quant_web_demo.py ~/model/Meta-Llama-3-8B-Instruct (quant模式)
第二节 Llama 3 微调个人小助手认知（XTuner版）
#通过gdata.py，创建微调的数据集。这里可以任意起个名字，比方：“南方蓝天”
cd ~/Llama3-Tutorial
python tools/gdata.py
conda activate llama3 （激活环境）
cd ~/Llama3-Tutorial
# 开始练习,使用 deepspeed 加快，A100 40G显存 耗时24分钟
xtuner train configs/assistant/llama3_8b_instruct_qlora_assistant.py --work-dir /root/llama3_pth
运行完成后的目次
# Adapter PTH 转 HF 格式
xtuner convert pth_to_hf /root/llama3_pth/llama3_8b_instruct_qlora_assistant.py \
/root/llama3_pth/iter_500.pth \
/root/llama3_hf_adapter
# 模子归并
export MKL_SERVICE_FORCE_INTEL=1
xtuner convert merge /root/model/Meta-Llama-3-8B-Instruct \
/root/llama3_hf_adapter\
/root/llama3_hf_merged Adapter
第三节 Llama 3 图片明确本领微调（XTuner+LLaVA）
启动环境
conda create -n llama3 python=3.10
conda activate llama3
conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=12.1 -c pytorch -c nvidia
#单独安装lfs
pip install lfs
#须要的 openai/clip-vit-large-patch14-336，权重，即 Visual Encoder 权重。
cd ~/model
ln -s /root/share/new_models/openai/clip-vit-large-patch14-336 .
#然后我们预备 Llava 将要用到的 Image Projector 部分权重。
ln -s /root/share/new_models/xtuner/llama3-llava-iter_2181.pth .
#数据预备
cd ~
git clone https://github.com/InternLM/tutorial -b camp2
python ~/tutorial/xtuner/llava/llava_data/repeat.py \
-i ~/tutorial/xtuner/llava/llava_data/unique_data.json \
-o ~/tutorial/xtuner/llava/llava_data/repeated_data.json \
-n 200
微调过程
启动练习：（！！！这一步须要50M的显存，由于开发机的显存不敷，以是无法完成。有些遗憾）
xtuner train ~/Llama3-Tutorial/configs/llama3-llava/llava_llama3_8b_instruct_qlora_clip_vit_large_p14_336_lora_e1_finetune.py \
--work-dir ~/llama3_llava_pth --deepspeed deepspeed_zero2
转换为 HuggingFace 格式
xtuner convert pth_to_hf ~/Llama3-Tutorial/configs/llama3-llava/llava_llama3_8b_instruct_qlora_clip_vit_large_p14_336_lora_e1_finetune.py \
~/model/llama3-llava-iter_2181.pth \
~/llama3_llava_pth/pretrain_iter_2181_hf
第四节 Llama 3 高效摆设实践（LMDeploy版）
环境设置
# 假如你是InternStudio 可以直接使用
# studio-conda -t lmdeploy -o pytorch-2.1.2
# 初始化环境
conda create -n lmdeploy python=3.10
conda activate lmdeploy
conda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=12.1 -c pytorch -c nvidia
安装lmdeploy最新版。
cd ~
pip install -U lmdeploy
LMDeploy Chat CLI 工具
conda activate lmdeploy #切换环境
lmdeploy chat /root/model/Meta-Llama-3-8B-Instruct
3. LMDeploy模子量化(lite)
本部分内容告急先容怎样对模子举行量化。告急包罗 KV8量化和W4A16量化。
3.1 设置最大KV Cache缓存巨细
模子在运行时，占用的显存可大抵分为三部分：模子参数自己占用的显存、KV Cache占用的显存，以及中央运算结果占用的显存。LMDeploy的KV Cache管理器可以通过设置--cache-max-entry-count参数，控制KV缓存占用剩余显存的最大比例。默认的比例为0.8。
lmdeploy chat /root/model/Meta-Llama-3-8B-Instruct/
内存根本都使用了。
lmdeploy chat /root/model/Meta-Llama-3-8B-Instruct/ --cache-max-entry-count 0.5
用量低沉了一些
lmdeploy chat /root/model/Meta-Llama-3-8B-Instruct/ --cache-max-entry-count 0.01
3.2 使用W4A16量化
lmdeploy lite auto_awq \
/root/model/Meta-Llama-3-8B-Instruct \
--calib-dataset 'ptb' \
--calib-samples 128 \
--calib-seqlen 1024 \
--w-bits 4 \
--w-group-size 128 \
--work-dir /root/model/Meta-Llama-3-8B-Instruct_4bit
lmdeploy chat /root/model/Meta-Llama-3-8B-Instruct_4bit --model-format awq
lmdeploy chat /root/model/Meta-Llama-3-8B-Instruct_4bit --model-format awq --cache-max-entry-count 0.01
已经降到了6G的内存。
3.3 在线量化 KV
自 v0.4.0 起，LMDeploy KV 量化方式有原来的离线改为在线。而且，支持两种数值精度 int4、int8。量化方式为 per-head per-token 的非对称量化。它具备以下上风：
量化不须要校准数据集
kv int8 量化精度险些无损，kv int4 量化精度在可担当范围之内
推理高效，在 llama2-7b 上参加 int8/int4 kv 量化，RPS 相较于 fp16 分别提拔近 30% 和 40%
支持 volta 架构（sm70）及以上的全部显卡型号：V100、20系列、T4、30系列、40系列、A10、A100 等等 通过 LMDeploy 应用 kv 量化非常简单，只须要设定 quant_policy 参数。LMDeploy 规定 qant_policy=4表现 kv int4 量化，quant_policy=8 表现 kv int8 量化。
4. LMDeploy服务（serve）
4.1 启动API服务器
lmdeploy serve api_server \
/root/model/Meta-Llama-3-8B-Instruct \
--model-format hf \
--quant-policy 0 \
--server-name 0.0.0.0 \
--server-port 23333 \
--tp 1
设定SSH的转发
ssh -CNg -L 23333:127.0.0.1:23333 [email protected] -p 46647
4.2 下令行客户端毗连API服务器
在“4.1”中，我们在终端里新开了一个API服务器。 本节中，我们要新建一个下令行客户端去毗连API服务器。起首通过VS Code新建一个终端： 激活conda环境
1， 下令行方式
conda activate lmdeploy
lmdeploy serve api_client http://localhost:23333
2 网页方式
conda activate lmdeploy
pip install gradio==3.50.2
lmdeploy serve gradio http://localhost:23333 \
--server-name 0.0.0.0 \
--server-port 6006
5. 推理速率
克隆堆栈
cd ~
git clone https://github.com/InternLM/lmdeploy.git
下载测试数据（642M）
cd /root/lmdeploy
wget https://hf-mirror.com/datasets/anon8231489123/ShareGPT_Vicuna_unfiltered/resolve/main/ShareGPT_V3_unfiltered_cleaned_split.json
实行 benchmark 下令(假如你的显存较小，可以调低--cache-max-entry-count)
python benchmark/profile_throughput.py \
ShareGPT_V3_unfiltered_cleaned_split.json \
/root/model/Meta-Llama-3-8B-Instruct \
--cache-max-entry-count 0.8 \
--concurrency 256 \
--model-format hf \
--quant-policy 0 \
--num-prompts 10000
我在这运行出了一个错误提示， 让后将/root/lmdeploy/benchmark/profile_throughput.py 中的286行的“ArgumentHelper.enable_prefix_caching(pt_group)”表明掉就可以了。
测试的时间开发机还没有跑满
第五节 Llama 3 Agent 体验微调（XTuner版）
2.1 环境设置
环境设置还是llama3
conda activate llama3
安装XTuner（完成之前的课程，已经已经装完了）
cd ~
git clone -b v0.1.18 https://github.com/InternLM/XTuner
cd XTuner
pip install -e .
2.2 模子预备
2.3 数据集预备
由于 HuggingFace 上的 Agent-FLAN 数据集暂时无法被 XTuner 直接加载，因此我们起主要下载到本地，然后转换成 XTuner 直接可用的格式。
cd ~
cp -r /root/share/new_models/internlm/Agent-FLAN .
chmod -R 755 Agent-FLAN
2.4 微调启动
由于练习时间太长，我们也为各人预备好了已经练习好且转换为 HuggingFace 格式的权重，可以直接使用。路径位于
/share/new_models/agent-flan/iter_2316_hf。
假如要使用已经练习好的权重，可以使用如下指令归并权重：
export MKL_SERVICE_FORCE_INTEL=1
xtuner convert merge /root/model/Meta-Llama-3-8B-Instruct \
/share/new_models/agent-flan/iter_2316_hf \
~/llama3_agent_pth/merged
4. Lagent Web Demo（在课本的背面，但是要先使用的内容，发起课本可以提到前面）
pip install lagent
streamlit run ~/Llama3-Tutorial/tools/agent_web_demo.py /root/model/Meta-Llama-3-8B-Instruct
streamlit run ~/Llama3-Tutorial/tools/agent_web_demo.py /root/llama3_agent_pth/merged
启动后对比标题，这次使用的标题是“查找关于InternLM2的论文”
对比之前的结果，这次已经可以主动调用ArxivSearch接口了。
第六节 Llama 3 本领评测（OpenCompass 版）
这次继续使用llama3环境
conda activate llama3
安装 OpenCompass。简介一下，OpenCompass是上海人工智能实验室开源的大模子评测平台，它涵盖了学科、语言、知识、明确、推理等五大评测维度，可以全面评估大模子的本领。上海人工智能实验室 (shlab.org.cn)
cd ~
git clone https://github.com/open-compass/opencompass opencompass
cd opencompass
pip install -e .
数据预备
下载数据集到 data/ 处
wget https://github.com/open-compass/opencompass/releases/download/0.2.2.rc1/OpenCompassData-core-20240207.zip
unzip OpenCompassData-core-20240207.zip
下载速率着实是太慢了，背面没法利用了，以是实战部分就到此竣事， 后续的只是课程的内容。
下令行快速评测
OpenCompass 预界说了许多模子和数据集的设置，你可以通过 工具 列出全部可用的模子和数据集设置。
# 列出全部设置
# python tools/list_configs.py
# 列出全部跟 llama (模子)及 ceval（数据集） 相干的设置
python tools/list_configs.py llama ceval
以 C-Eval_gen 为例：
python run.py --datasets ceval_gen --hf-path /root/model/Meta-Llama-3-8B-Instruct --tokenizer-path /root/model/Meta-Llama-3-8B-Instruct --tokenizer-kwargs padding_side='left' truncation='left' trust_remote_code=True --model-kwargs trust_remote_code=True device_map='auto' --max-seq-len 2048 --max-out-len 16 --batch-size 4 --num-gpus 1 --debug
下令剖析
python run.py \
--datasets ceval_gen \
--hf-path /root/model/Meta-Llama-3-8B-Instruct \ # HuggingFace 模子路径
--tokenizer-path /root/model/Meta-Llama-3-8B-Instruct \ # HuggingFace tokenizer 路径（假如与模子路径类似，可以省略）
--tokenizer-kwargs padding_side='left' truncation='left' trust_remote_code=True \ # 构建 tokenizer 的参数
--model-kwargs device_map='auto' trust_remote_code=True \ # 构建模子的参数
--max-seq-len 2048 \ # 模子可以担当的最大序列长度
--max-out-len 16 \ # 天生的最大 token 数
--batch-size 4 \ # 批量巨细
--num-gpus 1 \ # 运行模子所需的 GPU 数目
--debug
评测完成后，将会看到：
dataset version metric mode opencompass.models.huggingface.HuggingFace_meta-llama_Meta-Llama-3-8B-Instruct
---------------------------------------------- --------- ------------- ------ --------------------------------------------------------------------------------
ceval-computer_network db9ce2 accuracy gen 63.16
ceval-operating_system 1c2571 accuracy gen 63.16
ceval-computer_architecture a74dad accuracy gen 52.38
ceval-college_programming 4ca32a accuracy gen 62.16
ceval-college_physics 963fa8 accuracy gen 42.11
ceval-college_chemistry e78857 accuracy gen 29.17
ceval-advanced_mathematics ce03e2 accuracy gen 42.11
ceval-probability_and_statistics 65e812 accuracy gen 27.78
ceval-discrete_mathematics e894ae accuracy gen 25
ceval-electrical_engineer ae42b9 accuracy gen 32.43
ceval-metrology_engineer ee34ea accuracy gen 62.5
ceval-high_school_mathematics 1dc5bf accuracy gen 5.56
ceval-high_school_physics adf25f accuracy gen 26.32
ceval-high_school_chemistry 2ed27f accuracy gen 63.16
ceval-high_school_biology 8e2b9a accuracy gen 36.84
ceval-middle_school_mathematics bee8d5 accuracy gen 31.58
ceval-middle_school_biology 86817c accuracy gen 71.43
ceval-middle_school_physics 8accf6 accuracy gen 57.89
ceval-middle_school_chemistry 167a15 accuracy gen 80
ceval-veterinary_medicine b4e08d accuracy gen 52.17
ceval-college_economics f3f4e6 accuracy gen 45.45
ceval-business_administration c1614e accuracy gen 30.3
ceval-marxism cf874c accuracy gen 47.37
ceval-mao_zedong_thought 51c7a4 accuracy gen 50
ceval-education_science 591fee accuracy gen 51.72
ceval-teacher_qualification 4e4ced accuracy gen 72.73
ceval-high_school_politics 5c0de2 accuracy gen 68.42
ceval-high_school_geography 865461 accuracy gen 42.11
ceval-middle_school_politics 5be3e7 accuracy gen 57.14
ceval-middle_school_geography 8a63be accuracy gen 50
ceval-modern_chinese_history fc01af accuracy gen 52.17
ceval-ideological_and_moral_cultivation a2aa4a accuracy gen 78.95
ceval-logic f5b022 accuracy gen 40.91
ceval-law a110a1 accuracy gen 33.33
ceval-chinese_language_and_literature 0f8b68 accuracy gen 34.78
ceval-art_studies 2a1300 accuracy gen 54.55
ceval-professional_tour_guide 4e673e accuracy gen 55.17
ceval-legal_professional ce8787 accuracy gen 30.43
ceval-high_school_chinese 315705 accuracy gen 31.58
ceval-high_school_history 7eb30a accuracy gen 65
ceval-middle_school_history 48ab4a accuracy gen 59.09
ceval-civil_servant 87d061 accuracy gen 34.04
ceval-sports_science 70f27b accuracy gen 63.16
ceval-plant_protection 8941f9 accuracy gen 68.18
ceval-basic_medicine c409d6 accuracy gen 57.89
ceval-clinical_medicine 49e82d accuracy gen 54.55
ceval-urban_and_rural_planner 95b885 accuracy gen 52.17
ceval-accountant 002837 accuracy gen 44.9
ceval-fire_engineer bc23f5 accuracy gen 38.71
ceval-environmental_impact_assessment_engineer c64e2d accuracy gen 45.16
ceval-tax_accountant 3a5e3c accuracy gen 34.69
ceval-physician 6e277d accuracy gen 57.14
ceval-stem - naive_average gen 46.34
ceval-social-science - naive_average gen 51.52
ceval-humanities - naive_average gen 48.72
ceval-other - naive_average gen 50.05
ceval-hard - naive_average gen 32.65
ceval - naive_average gen 48.63
config 快速评测，在config 下添加模子设置文件 eval_llama3_8b_demo.py
from mmengine.config import read_base
with read_base():
from .datasets.mmlu.mmlu_gen_4d595a import mmlu_datasets
datasets = [*mmlu_datasets]
from opencompass.models import HuggingFaceCausalLM
models = [
dict(
type=HuggingFaceCausalLM,
abbr='Llama3_8b', # 运行完结果展示的名称
path='/root/model/Meta-Llama-3-8B-Instruct', # 模子路径
tokenizer_path='/root/model/Meta-Llama-3-8B-Instruct', # 分词器路径
model_kwargs=dict(
device_map='auto',
trust_remote_code=True
),
tokenizer_kwargs=dict(
padding_side='left',
truncation_side='left',
trust_remote_code=True,
use_fast=False
),
generation_kwargs={"eos_token_id": },
batch_padding=True,
max_out_len=100,
max_seq_len=2048,
batch_size=16,
run_cfg=dict(num_gpus=1),
)
]
运行python run.py configs/eval_llama3_8b_demo.py
评测完成后，将会看到：
dataset version metric mode Llama3_8b
------------------------------------------------- --------- -------- ------ -----------
lukaemon_mmlu_college_biology caec7d accuracy gen 66.67
lukaemon_mmlu_college_chemistry 520aa6 accuracy gen 37
lukaemon_mmlu_college_computer_science 99c216 accuracy gen 53
lukaemon_mmlu_college_mathematics 678751 accuracy gen 36
lukaemon_mmlu_college_physics 4f382c accuracy gen 48.04
lukaemon_mmlu_electrical_engineering 770ce3 accuracy gen 43.45
lukaemon_mmlu_astronomy d3ee01 accuracy gen 68.42
lukaemon_mmlu_anatomy 72183b accuracy gen 54.07
lukaemon_mmlu_abstract_algebra 2db373 accuracy gen 31
lukaemon_mmlu_machine_learning 0283bb accuracy gen 43.75
lukaemon_mmlu_clinical_knowledge cb3218 accuracy gen 58.87
lukaemon_mmlu_global_facts ab07b6 accuracy gen 39
lukaemon_mmlu_management 80876d accuracy gen 78.64
lukaemon_mmlu_nutrition 4543bd accuracy gen 72.55
lukaemon_mmlu_marketing 7394e3 accuracy gen 90.17
lukaemon_mmlu_professional_accounting 444b7f accuracy gen 49.65
lukaemon_mmlu_high_school_geography 0780e6 accuracy gen 75.25
lukaemon_mmlu_international_law cf3179 accuracy gen 62.81
lukaemon_mmlu_moral_scenarios f6dbe2 accuracy gen 38.66
lukaemon_mmlu_computer_security ce7550 accuracy gen 35
lukaemon_mmlu_high_school_microeconomics 04d21a accuracy gen 62.18
lukaemon_mmlu_professional_law 5f7e6c accuracy gen 47.91
lukaemon_mmlu_medical_genetics 881ef5 accuracy gen 62
lukaemon_mmlu_professional_psychology 221a16 accuracy gen 69.44
lukaemon_mmlu_jurisprudence 001f24 accuracy gen 69.44
lukaemon_mmlu_world_religions 232c09 accuracy gen 74.85
lukaemon_mmlu_philosophy 08042b accuracy gen 71.06
lukaemon_mmlu_virology 12e270 accuracy gen 43.98
lukaemon_mmlu_high_school_chemistry ae8820 accuracy gen 42.86
lukaemon_mmlu_public_relations e7d39b accuracy gen 60
lukaemon_mmlu_high_school_macroeconomics a01685 accuracy gen 57.95
lukaemon_mmlu_human_sexuality 42407c accuracy gen 74.05
lukaemon_mmlu_elementary_mathematics 269926 accuracy gen 28.84
lukaemon_mmlu_high_school_physics 93278f accuracy gen 26.49
lukaemon_mmlu_high_school_computer_science 9965a5 accuracy gen 63
lukaemon_mmlu_high_school_european_history eefc90 accuracy gen 74.55
lukaemon_mmlu_business_ethics 1dec08 accuracy gen 51
lukaemon_mmlu_moral_disputes a2173e accuracy gen 70.81
lukaemon_mmlu_high_school_statistics 8f3f3a accuracy gen 52.78
lukaemon_mmlu_miscellaneous 935647 accuracy gen 54.15
lukaemon_mmlu_formal_logic cfcb0c accuracy gen 42.86
lukaemon_mmlu_high_school_government_and_politics 3c52f9 accuracy gen 86.01
lukaemon_mmlu_prehistory bbb197 accuracy gen 64.2
lukaemon_mmlu_security_studies 9b1743 accuracy gen 75.51
lukaemon_mmlu_high_school_biology 37b125 accuracy gen 74.84
lukaemon_mmlu_logical_fallacies 9cebb0 accuracy gen 68.1
lukaemon_mmlu_high_school_world_history 048e7e accuracy gen 83.12
lukaemon_mmlu_professional_medicine 857144 accuracy gen 72.43
lukaemon_mmlu_high_school_mathematics ed4dc0 accuracy gen 31.48
lukaemon_mmlu_college_medicine 38709e accuracy gen 56.65
lukaemon_mmlu_high_school_us_history 8932df accuracy gen 82.84
lukaemon_mmlu_sociology c266a2 accuracy gen 76.12
lukaemon_mmlu_econometrics d1134d accuracy gen 55.26
lukaemon_mmlu_high_school_psychology 7db114 accuracy gen 65.14
lukaemon_mmlu_human_aging 82a410 accuracy gen 62.33
lukaemon_mmlu_us_foreign_policy 528cfe accuracy gen 70
lukaemon_mmlu_conceptual_physics 63588e accuracy gen 26.38
opencompass 官方已经支持 Llama3
https://github.com/open-compass/opencompass/commit/a256753221ad2a33ec9750b31f6284b581c1e1fd#diff-e446451cf0c8fc747c5c720f65f8fa62d7bd7f5c88668692248517d249c798b5
课后总结
[*] 非常感谢ModelScope魔搭社区提供的这次时机，可以从一个小白开始一步步学习AI的一些根本知识。开始的时间由于多人创作，一些目次调解导致了一些紊乱， 但是很快就举行了修正。现在是一套非常完满，过细，可以手把手教小白入门的课程了。
[*] 第一节、第二节是根本课程，相对轻易，但是有许多环境设置的工作，很轻易堕落，涉及的知识面也比力多，这部分各人要耐烦细致。 学习后对于任何AI模子都可以轻松摆设了。
[*] 第三节有些痛惜，免费提供的InterStudio开发机上的显存不敷，以是无法举行练习。幸亏我工作中对于图片的练习需求很少。
[*] 第四节通过LMDeploy可以很快摆设AI服务， 可以使用两种方式低沉显存的使用量，可以通过下令行或API提供服务。
[*] 第五节事智能体，固然跟随课程可以完成练习，但是对智能体的团体知识的先容在课程中是比力少的，偶然机还要自己学习。
[*] 第六节由于下载速率的标题没有完成，盼望OpenCompass首页_上海人工智能实验室 (shlab.org.cn)可以对国内的用户可以提供更多的带宽。
[*] 这次学习团体上是小白入门教程，仅仅打开了AI的一个窗口，后续须要扩展的知识另有许多，盼望官方可以提供一个发展路径，为后续学习提供引导。
页:
[1]