News

5/16/2026 Preprint release of Nitsum: Serving Tiered LLM Requests with Adaptive Tensor Parallelism (blog post)

1/26/2025 🎉 OSWorld-Human was accepted to MLSys 2026!

1/26/2025 🎉 Beat the long tail: Distribution-Aware Speculative Decoding for RL Training was accepted to MLSys 2026!

9/23/2025 🎉 FarSight was accepted to the Workshop on Machine Learning for Systems at NeurIPS 2025!

9/23/2025 🎉 Demystifying Delays in Reasoning was accepted to the Workshop on Efficient Reasoning at NeurIPS 2025!

Efficient Augmented LLM Serving With InferCept

February 10, 2024 - 6 mins read

Today’s large language models (LLMs) are being paired with various tools and environments to satisfy increasingly complex user queries. Augmenting models with these capabilities means LLM inference can be intercepted by external actions. We designed InferCept [ICML ‘24], the first serving framework designed for augmented LLMs. InferCept minimizes resource waste and sustains a 1.6x-2x higher serving load, completing twice as many requests compared to state-of-the-art serving systems. Try InferCept here.

Read More…