Parallel CPU-GPU Execution for LLM Inference on Constrained GPUs
arxiv.org · 6,622 words · saved by 1 readers
N/A
APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs Jiakun Fan1 , Yanglin Zhang2 , Xiangchen Li1 , and Dimitrios S. Nikolopoulos1 1 Department of Computer Science, Virginia Tech…
saved by
related reading
- Look Ma, No Bubbles! Designing a Low-Latency Megakernel for Llama-1B · Hazy Researchhazyresearch.stanford.edu
- LLM Inference Performance Engineering: Best Practices | Databricks Blogdatabricks.com
- ⭐️ Fast LLM Inference From Scratchandrewkchan.dev
- Transformers Inference Optimization Toolset | AstraBlogastralord.github.io
- Context Parallelism for Scalable Million-Token Inferencearxiv.org
- Paged Attention from First Principles: A View Inside vLLM – Hamza's Bloghamzaelshafie.bearblog.dev
- [2510.13786] The Art of Scaling Reinforcement Learning Compute for LLMsarxiv.org
- HeteroLLM: Accelerating Large Language Model Inference on Mobile SoCs with Heterogeneous AI Acceleratorsarxiv.org
- A guide to LLM inference and performancebaseten.co
- Decoding Speculative Decoding from First Principlesjwlabs.vercel.app
- How is LLaMa.cpp possible?finbarr.ca
- Fast Inference from Transformers via Speculative Decodingarxiv.org