@lobehub/chat/docs/usage/providers/vllm.mdx

Lobe Chat - an open-source, high-performance chatbot framework that supports speech synthesis, multimodal, and extensible Function Call plugin system. Supports one-click free deployment of your private ChatGPT/LLM web application.

---
title: Using vLLM API Key in LobeChat
description: Learn how to configure and use the vLLM language model in LobeChat, obtain an API key, and start a conversation.
tags:
  - LobeChat
  - vLLM
  - API Key
  - Web UI
---

# Using vLLM in LobeChat

<Image alt={'Using vLLM in LobeChat'} cover src={'https://github.com/user-attachments/assets/1d77cca4-7363-4a46-9ad5-10604e111d7c'} />

[vLLM](https://github.com/vllm-project/vllm) is an open-source local large language model (LLM) deployment tool that allows users to efficiently run LLM models on local devices and provides an OpenAI API-compatible service interface.

This document will guide you on how to use vLLM in LobeChat:

<Steps>
  ### Step 1: Preparation

  vLLM has certain requirements for hardware and software environments. Be sure to configure according to the following requirements:

  | Hardware Requirements |                                                                         |
  | --------------------- | ----------------------------------------------------------------------- |
  | GPU                   | - NVIDIA CUDA <br /> - AMD ROCm <br /> - Intel XPU                      |
  | CPU                   | - Intel/AMD x86 <br /> - ARM AArch64 <br /> - Apple silicon             |
  | Other AI Accelerators | - Google TPU <br /> - Intel Gaudi <br /> - AWS Neuron <br /> - OpenVINO |

  | Software Requirements                   |
  | --------------------------------------- |
  | - OS: Linux <br /> - Python: 3.9 – 3.12 |

  ### Step 2: Install vLLM

  If you are using an NVIDIA GPU, you can directly install vLLM using `pip`. However, it is recommended to use `uv` here, which is a very fast Python environment manager, to create and manage the Python environment. Please follow the [documentation](https://docs.astral.sh/uv/#getting-started) to install uv. After installing uv, you can use the following command to create a new Python environment and install vLLM:

  ```shell
  uv venv myenv --python 3.12 --seed
  source myenv/bin/activate
  uv pip install vllm
  ```

  Another method is to use `uv run` with the `--with [dependency]` option, which allows you to run commands such as `vllm serve` without creating an environment:

  ```shell
  uv run --with vllm vllm --help
  ```

  You can also use [conda](https://docs.conda.io/projects/conda/en/latest/user-guide/getting-started.html) to create and manage your Python environment.

  ```shell
  conda create -n myenv python=3.12 -y
  conda activate myenv
  pip install vllm
  ```

  <Callout type={'note'}>
    For non-CUDA platforms, please refer to the [official
    documentation](https://docs.vllm.ai/en/latest/getting_started/installation/index.html#installation-index)
    to learn how to install vLLM.
  </Callout>

  ### Step 3: Start Local Service

  vLLM can be deployed as an OpenAI API protocol-compatible server. By default, it will start the server at `http://localhost:8000`. You can specify the address using the `--host` and `--port` parameters. The server currently runs only one model at a time.

  The following command will start a vLLM server and run the `Qwen2.5-1.5B-Instruct` model:

  ```shell
  vllm serve Qwen/Qwen2.5-1.5B-Instruct
  ```

  You can enable the server to check the API key in the header by passing the parameter `--api-key` or the environment variable `VLLM_API_KEY`. If not set, no API Key is required to access.

  <Callout type={'note'}>
    For more detailed vLLM server configuration, please refer to the [official
    documentation](https://docs.vllm.ai/en/latest/).
  </Callout>

  ### Step 4: Configure vLLM in LobeChat

  - Access the `Application Settings` interface of LobeChat.
  - Find the `vLLM` settings item under `Language Model`.

  <Image alt={'Fill in the vLLM API Key'} inStep src={'https://github.com/user-attachments/assets/669c68bf-3f85-4a6f-bb08-d0d7fb7f7417'} />

  - Open the vLLM service provider and fill in the API service address and API Key.

  <Callout type={'warning'}>
    * If your vLLM is not configured with an API Key, please leave the API Key blank. \* If your vLLM
      is running locally, please make sure to turn on `Client Request Mode`.
  </Callout>

  - Add the model you are running to the model list below.
  - Select a vLLM model to run for your assistant and start the conversation.

    <Image alt={'Select vLLM Model'} inStep src={'https://github.com/user-attachments/assets/fcdfb9c5-819a-488f-b28d-0857fe861219'} />
</Steps>

Now you can use the models provided by vLLM in LobeChat to have conversations.