A High-Performance LLM Inference Engine with vLLM-Style Continuous Batching
-
Updated
Jan 2, 2026 - C++
#
continuous-batching
Here are 6 public repositories matching this topic...
gLLM: Global Balanced Pipeline Parallelism System for Distributed LLM Serving with Token Throttling
pipeline-parallelism tensor-parallelism llm-serving llm-inference pagedattention continuous-batching qwen3 token-throttling chunked-prefill
-
Updated
Jan 12, 2026 - Python
OpenAI-compatible server with continuous batching for MLX on Apple Silicon
-
Updated
Dec 4, 2025 - Python
Fork of OpenAI and Anthropic compatible server for Apple Silicon. Native MLX backend, 500+ tok/s. Run LLMs and vision-language models with continuous batching, MCP tool calling, and multimodal support.
inference-server mlx multimodal apple-silicon llm vllm local-ai continuous-batching tool-calling openai-compatible
-
Updated
Mar 9, 2026 - Python
PagedAttention + Continuous Batching Inference Engine Prototype (Rust): Paged KV Cache & Dynamic Scheduling | PagedAttention + Continuous Batching 推理引擎原型（Rust），KV Cache 分页管理与动态调度
-
Updated
Mar 9, 2026 - Rust
-
Updated
Jun 19, 2024 - Jupyter Notebook
Improve this page
Add a description, image, and links to the continuous-batching topic page so that developers can more easily learn about it.
Add this topic to your repo
To associate your repository with the continuous-batching topic, visit your repo's landing page and select "manage topics."