利用xinference摆设自界说embedding模子（docker）

利用xinference摆设自界说embedding模子（docker）

阐明：

• 初次发表日期：2024-08-27
  
• 官方文档： https://inference.readthedocs.io/zh-cn/latest/index.html
  

利用docker摆设xinference

1. FROM nvcr.io/nvidia/pytorch:23.10-py3
3. # Keeps Python from generating .pyc files in the container
4. ENV PYTHONDONTWRITEBYTECODE=1
6. # Turns off buffering for easier container logging
7. ENV PYTHONUNBUFFERED=1
9. RUN python3 -m pip uninstall -y transformer-engine
10. RUN python3 -m pip install --upgrade pip
12. RUN python3 -m pip install torch==2.3.0 torchvision==0.18.0 torchaudio==2.3.0 --no-cache-dir --index-url https://download.pytorch.org/whl/cu121
14. # If there are network issue, you can download torch whl file and use it
15. # ADD torch-2.3.0+cu121-cp310-cp310-linux_x86_64.whl /root/torch-2.3.0+cu121-cp310-cp310-linux_x86_64.whl
16. # RUN python3 -m pip install /root/torch-2.3.0+cu121-cp310-cp310-linux_x86_64.whl
18. RUN python3 -m pip install packaging setuptools==69.5.1 --no-cache-dir -i https://mirror.baidu.com/pypi/simple
19. RUN python3 -m pip install -U ninja --no-cache-dir -i https://mirror.baidu.com/pypi/simple
20. RUN python3 -m pip install flash-attn==2.5.8 --no-build-isolation --no-cache-dir
21. RUN python3 -m pip install "xinference[all]" --no-cache-dir -i https://repo.huaweicloud.com/repository/pypi/simple
23. EXPOSE 80
25. CMD ["sh", "-c", "tail -f /dev/null"]

复制代码

构建镜像

1. docker build -t myxinference:latest .

复制代码

参照 https://inference.readthedocs.io/zh-cn/latest/getting_started/using_docker_image.html#mount-your-volume-for-loading-and-saving-models 摆设docker服务
别的，如果利用huggingface的话，发起利用 https://hf-mirror.com/ 镜像（记得docker摆设时设置HF_ENDPOINT环境变量）。
以下假设摆设后的服务所在为 http://localhost:9997
摆设自界说 embedding 模子

准备embedding模子自界说JSON文件

创建文件夹custom_models/embedding：

1. mkdir -p custom_models/embedding

复制代码

然后创建以下模子自界说JSON文件：
360Zhinao-search.json:

1. {
2.     "model_name": "360Zhinao-search",
3.     "dimensions": 1024,
4.     "max_tokens": 512,
5.     "language": ["en", "zh"],
6.     "model_id": "qihoo360/360Zhinao-search",
7.     "model_format": "pytorch"
8. }

复制代码

gte-Qwen2-7B-instruct.json：

1. {
2.     "model_name": "gte-Qwen2-7B-instruct",
3.     "dimensions": 4096,
4.     "max_tokens": 32768,
5.     "language": ["en", "zh"],
6.     "model_id": "Alibaba-NLP/gte-Qwen2-7B-instruct",
7.     "model_format": "pytorch"
8. }

复制代码

zpoint_large_embedding_zh.json:

1. {
2.     "model_name": "zpoint_large_embedding_zh",
3.     "dimensions": 1792,
4.     "max_tokens": 512,
5.     "language": ["zh"],
6.     "model_id": "iampanda/zpoint_large_embedding_zh",
7.     "model_format": "pytorch"
8. }

复制代码

注意：对于下载到当地的模子可以设置 model_uri参数，比方 “[file:///path/to/llama-2-7b](file:///path/to/llama-2-7b)”。
注册自界说 embedding 模子

1. xinference register --model-type embedding --file custom_models/embedding/360Zhinao-search.json --persist --endpoint http://localhost:9997
3. xinference register --model-type embedding --file custom_models/embedding/gte-Qwen2-7B-instruct.json --persist --endpoint http://localhost:9997
5. xinference register --model-type embedding --file custom_models/embedding/zpoint_large_embedding_zh.json --persist --endpoint http://localhost:9997

复制代码

启动自界说 embedding 模子

1. xinference launch --model-type embedding --model-name gte-Qwen2-7B-instruct --model-engine transformers  --model-format pytorch --endpoint http://localhost:9997
3. xinference launch --model-type embedding --model-name 360Zhinao-search --model-engine transformers  --model-format pytorch --endpoint http://localhost:9997
5. xinference launch --model-type embedding --model-name zpoint_large_embedding_zh --model-engine transformers  --model-format pytorch --endpoint http://localhost:9997

复制代码

启动bge-m3和bge-reranker-base模子

bge-m3和bge-reranker-base是比力常用的embedding模子和reranking模子。

1. xinference launch --model-name bge-m3 --model-type embedding --endpoint http://localhost:9997
3. xinference launch --model-name bge-reranker-base --model-type rerank --endpoint http://localhost:9997

复制代码

curl调用测试

embedding:

1. curl http://localhost:9997/v1/embeddings \
2.   -H "Content-Type: application/json" \
3.   -d '{
4.     "input": "The food was delicious and the waiter...",
5.     "model": "360Zhinao-search",
6.     "encoding_format": "float"
7.   }'

复制代码

reranking:

1. curl http://localhost:9997/v1/rerank \
2.   -H "Content-Type: application/json" \
3.   -d '{
4.   "model": "bge-reranker-base",
5.   "query": "I love you",
6.   "documents": [
7.     "I hate you",
8.     "I really like you",
9.     "天空是什么颜色的",
10.     "黑芝麻味饼干"
11.   ],
12.   "top_n": 3
13. }'

复制代码