LCORE-1496: Inference providers e2e tests failing (Watson, Vertex, Azure, RHAIIS, RHELAI)#1358
LCORE-1496: Inference providers e2e tests failing (Watson, Vertex, Azure, RHAIIS, RHELAI)#1358are-ces wants to merge 16 commits intolightspeed-core:mainfrom
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
WalkthroughConfiguration files for multiple LLM providers (Azure, VertexAI, WatsonX) and container runtime are updated to add Model Context Protocol (MCP) support, restructure storage backends, update embedding models from nomic to sentence-transformers, and standardize provider endpoint naming conventions (api_base/url → base_url). Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes 🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
tests/e2e/configs/run-watsonx.yaml (1)
157-162:⚠️ Potential issue | 🔴 CriticalDuplicate
vector_storeskey will break vector store registration.Same issue as in
examples/watsonx-run.yaml- thevector_stores: []at line 162 overwrites the FAISS configuration above it.🐛 Proposed fix - remove the duplicate key
vector_stores: - embedding_dimension: 768 embedding_model: sentence-transformers/all-mpnet-base-v2 provider_id: faiss vector_store_id: ${env.FAISS_VECTOR_STORE_ID} - vector_stores: [] datasets: []🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@tests/e2e/configs/run-watsonx.yaml` around lines 157 - 162, The YAML contains a duplicate top-level key "vector_stores" which causes the configured FAISS entry (the block with embedding_dimension, embedding_model, provider_id, and vector_store_id) to be overwritten by the later empty `vector_stores: []`; remove the trailing duplicate `vector_stores: []` so the FAISS configuration block remains the sole "vector_stores" entry (same fix as in examples/watsonx-run.yaml).
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@examples/watsonx-run.yaml`:
- Around line 157-162: The YAML contains a duplicate top-level key
"vector_stores" which causes the configured FAISS block (embedding_dimension,
embedding_model, provider_id, vector_store_id with ${env.FAISS_VECTOR_STORE_ID})
to be overwritten by the later "vector_stores: []"; remove the redundant
"vector_stores: []" entry so the FAISS vector store configuration remains (keep
the block containing embedding_dimension, embedding_model, provider_id,
vector_store_id).
---
Outside diff comments:
In `@tests/e2e/configs/run-watsonx.yaml`:
- Around line 157-162: The YAML contains a duplicate top-level key
"vector_stores" which causes the configured FAISS entry (the block with
embedding_dimension, embedding_model, provider_id, and vector_store_id) to be
overwritten by the later empty `vector_stores: []`; remove the trailing
duplicate `vector_stores: []` so the FAISS configuration block remains the sole
"vector_stores" entry (same fix as in examples/watsonx-run.yaml).
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 761aed89-4a09-46d4-b1c2-c76b08c57a02
📒 Files selected for processing (10)
.github/workflows/e2e_tests_providers.yaml.github/workflows/e2e_tests_rhaiis.yaml.github/workflows/e2e_tests_rhelai.yamlexamples/azure-run.yamlexamples/vertexai-run.yamlexamples/watsonx-run.yamltest.containerfiletests/e2e/configs/run-azure.yamltests/e2e/configs/run-vertexai.yamltests/e2e/configs/run-watsonx.yaml
746337e to
f788a9c
Compare
f788a9c to
621b5c8
Compare
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Bump max_attempts in wait_for_container_health from 3 to 6 (30s instead of 15s) to prevent ConnectionResetError after config switch restarts in library mode. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
6277ed5 to
0f3f5c8
Compare
WatsonX models are registered without the provider prefix in llama-stack. Add workaround to return provider_resource_id when the selected model is a watsonx model. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add a 40-minute wait for watsonx server mode to avoid concurrent requests hitting the Lite plan rate limit. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The library-mode default inference uses openai/gpt-4o-mini but the model was not registered in Llama Stack, causing requests to be misrouted to the vLLM provider which rejects them as a non-chat model. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This reverts commit 3ae9527.
In library mode, Llama Stack auto-discovers 80 OpenAI models and the test framework picks babbage-002 (a completions-only model) as the default, causing all query tests to fail with "not a chat model". Set E2E_DEFAULT_MODEL_OVERRIDE and E2E_DEFAULT_PROVIDER_OVERRIDE to bypass auto-detection and use the correct RHAIIS vLLM model. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Description
Several issues are addressed in this PR:
srcto the PYTHONPATHType of change
Tools used to create PR
Identify any AI code assistants used in this PR (for transparency and review context)
Related Tickets & Documents
Checklist before requesting a review
Testing
e2e tests passing for the providers.
Summary by CodeRabbit
New Features
Configuration & Storage