This repository was archived by the owner on Jun 13, 2025. It is now read-only.
Repository navigation
This repository was archived by the owner on Jun 13, 2025. It is now read-only.
[Bug]: Failure to loading lora adapter crashes the whole vllm instance #7
Description
What
Start vllm:
HF_TOKEN="<FILL_IN_YOUR_HUGGINGFACE_TOKEN>"
LLAMAFACTORY_LORA_DIR="<FILL_IN_LLAMAFACTORY_LORA_OUTPUT>"
docker run --runtime nvidia --gpus all -v ${LLAMAFACTORY_LORA_DIR}:/eda-test/lora-test --env "HUGGING_FACE_HUB_TOKEN=${HF_TOKEN}" --env "VLLM_ALLOW_RUNTIME_LORA_UPDATING=True" -p 8000:8000 --ipc=host powerml/inference-engine-nvidia:v0.6.5-000 --model meta-llama/Meta-Llama-3.1-8B-Instruct --dtype=half --enable_lora --max-model-len 192Then send lora inference request, note /eda-test is incorrect:
curl http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{
"model": "meta-llama/Meta-Llama-3.1-8B-Instruct",
"prompt": "San Francisco is a",
"lora_request": {
"lora_name": "test_adapter",
"lora_path": "/eda-test"
}
}'The stack trace from vllm:
INFO: 172.17.0.1:54434 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error
ERROR 01-30 09:48:33 engine.py:135] RuntimeError('Error in model execution (input dumped to /tmp/err_execute_model_input_20250130-094833.pkl): Loading lora /eda-test failed')
ERROR 01-30 09:48:33 engine.py:135] Traceback (most recent call last):
ERROR 01-30 09:48:33 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/lora/worker_manager.py", line 95, in _load_adapter
ERROR 01-30 09:48:33 engine.py:135] lora = self._lora_model_cls.from_local_checkpoint(
ERROR 01-30 09:48:33 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-30 09:48:33 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/lora/models.py", line 215, in from_local_checkpoint
ERROR 01-30 09:48:33 engine.py:135] with open(lora_config_path) as f:
ERROR 01-30 09:48:33 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-30 09:48:33 engine.py:135] FileNotFoundError: [Errno 2] No such file or directory: '/eda-test/adapter_config.json'
ERROR 01-30 09:48:33 engine.py:135]
ERROR 01-30 09:48:33 engine.py:135] The above exception was the direct cause of the following exception:
ERROR 01-30 09:48:33 engine.py:135]
ERROR 01-30 09:48:33 engine.py:135] Traceback (most recent call last):
ERROR 01-30 09:48:33 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner_base.py", line 116, in _wrapper
ERROR 01-30 09:48:33 engine.py:135] return func(*args, **kwargs)
ERROR 01-30 09:48:33 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^
ERROR 01-30 09:48:33 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1624, in execute_model
ERROR 01-30 09:48:33 engine.py:135] self.set_active_loras(model_input.lora_requests,
ERROR 01-30 09:48:33 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1342, in set_active_loras
ERROR 01-30 09:48:33 engine.py:135] self.lora_manager.set_active_adapters(lora_requests, lora_mapping)
ERROR 01-30 09:48:33 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/lora/worker_manager.py", line 137, in set_active_adapters
ERROR 01-30 09:48:33 engine.py:135] set_active_adapters_worker(requests, mapping, self._apply_adapters,
ERROR 01-30 09:48:33 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/adapter_commons/utils.py", line 52, in set_active_adapters_worker
ERROR 01-30 09:48:33 engine.py:135] apply_adapters_func(requests)
ERROR 01-30 09:48:33 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/lora/worker_manager.py", line 197, in _apply_adapters
ERROR 01-30 09:48:33 engine.py:135] self.add_adapter(lora)
ERROR 01-30 09:48:33 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/lora/worker_manager.py", line 206, in add_adapter
ERROR 01-30 09:48:33 engine.py:135] lora = self._load_adapter(lora_request)
ERROR 01-30 09:48:33 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-30 09:48:33 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/lora/worker_manager.py", line 108, in _load_adapter
ERROR 01-30 09:48:33 engine.py:135] raise RuntimeError(f"Loading lora {lora_path} failed") from e
ERROR 01-30 09:48:33 engine.py:135] RuntimeError: Loading lora /eda-test failed
ERROR 01-30 09:48:33 engine.py:135]
ERROR 01-30 09:48:33 engine.py:135] The above exception was the direct cause of the following exception:
ERROR 01-30 09:48:33 engine.py:135]
ERROR 01-30 09:48:33 engine.py:135] Traceback (most recent call last):
ERROR 01-30 09:48:33 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 133, in start
ERROR 01-30 09:48:33 engine.py:135] self.run_engine_loop()
ERROR 01-30 09:48:33 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 196, in run_engine_loop
ERROR 01-30 09:48:33 engine.py:135] request_outputs = self.engine_step()
ERROR 01-30 09:48:33 engine.py:135] ^^^^^^^^^^^^^^^^^^
ERROR 01-30 09:48:33 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 214, in engine_step
ERROR 01-30 09:48:33 engine.py:135] raise e
ERROR 01-30 09:48:33 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/multiprocessing/engine.py", line 205, in engine_step
ERROR 01-30 09:48:33 engine.py:135] return self.engine.step()
ERROR 01-30 09:48:33 engine.py:135] ^^^^^^^^^^^^^^^^^^
ERROR 01-30 09:48:33 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 1405, in step
ERROR 01-30 09:48:33 engine.py:135] outputs = self.model_executor.execute_model(
ERROR 01-30 09:48:33 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-30 09:48:33 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/executor/gpu_executor.py", line 88, in execute_model
ERROR 01-30 09:48:33 engine.py:135] output = self.driver_worker.execute_model(execute_model_req)
ERROR 01-30 09:48:33 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-30 09:48:33 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 343, in execute_model
ERROR 01-30 09:48:33 engine.py:135] output = self.model_runner.execute_model(
ERROR 01-30 09:48:33 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 01-30 09:48:33 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 01-30 09:48:33 engine.py:135] return func(*args, **kwargs)
ERROR 01-30 09:48:33 engine.py:135] ^^^^^^^^^^^^^^^^^^^^^
ERROR 01-30 09:48:33 engine.py:135] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner_base.py", line 152, in _wrapper
ERROR 01-30 09:48:33 engine.py:135] raise type(err)(
ERROR 01-30 09:48:33 engine.py:135] RuntimeError: Error in model execution (input dumped to /tmp/err_execute_model_input_20250130-094833.pkl): Loading lora /eda-test failed
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [1]
Activity