Support with vLLM

[Aniketto16]

Hello!
Thank you for your great work, its amazing how much hard work you put for this algorithm. I just had one question is it possible to integrate this with vLLM serving ?

This will really boost the inference time in limited resources setting once you cross the 8192 token mark, is there a way ? Thank you in advance for your help!!

[Mooler0410]

We are not very familiar with vLLM and its internal mechanism. We will check its compatibility with SelfExtend. Thanks for your suggestion!

[K-Mistele]

+1, would love to see this in vLLM!

[linchen111]

+1, would love to see this in vLLM too!

[WeixuanXiong]

+1, would love to see this in vLLM, since lots of online services are based on vllm! It will be so ideal if we can easily use self extend trick on our online service!

[yxchng]

can this be used in vllm now?