vLLM部署Qwen1.5-32B-Chat
部署环境
操作系统：Ubuntu 20.04.2 LTS
Docker：v24.0.0
Docker Compose：v2.17.3
GPU：NVIDIA A100-SXM4-80GB
Nvidia Driver Version: 535.154.05
CUDA Version: 12.2
部署过程
模型文件下载
下载地址：通义千问1.5-32B-对话 · 模型库 (modelscope.cn)
下载方法：
- SDK下载
#模型下载 from modelscope import snapshot_download model_dir = snapshot_download('qwen/Qwen1.5-32B-Chat', cache_dir='/path/to/your/model/dir/') # cache_dir指定模型保存目录 - Git下载
git clone https://www.modelscope.cn/qwen/Qwen1.5-32B-Chat.git
docker镜像构建
克隆Qwen1.5仓库：
git clone https://github.com/QwenLM/Qwen1.5.git
构建镜像：
![[大模型]<span style='color:red;'>Qwen</span>1.5-7<span style='color:red;'>B</span>-<span style='color:red;'>Chat</span>-GPTQ-Int4 <span style='color:red;'>部署</span>环境](https://img-blog.csdnimg.cn/direct/a00a5637ff5e459a9e274da85c6afac7.png#pic_center)
![[大模型]<span style='color:red;'>Qwen</span>1.5-7<span style='color:red;'>B</span>-<span style='color:red;'>Chat</span> FastApi <span style='color:red;'>部署</span>调用](https://img-blog.csdnimg.cn/direct/fb653ca9da0e410f9956dd21a5aa8e31.png#pic_center)
![[大模型]<span style='color:red;'>Qwen</span>-7<span style='color:red;'>B</span>-<span style='color:red;'>Chat</span> WebDemo](https://img-blog.csdnimg.cn/direct/568fd79bf1fe4e10ba83ff9bcf545de9.png#pic_center)
![[大模型]<span style='color:red;'>Qwen</span>-Audio-<span style='color:red;'>chat</span> WebDemo <span style='color:red;'>部署</span>](https://img-blog.csdnimg.cn/direct/e2935baa06674badacba9928a372ab80.png#pic_center)
