Paul Balanca

A Practical Investigation of Training-free Relaxed Speculative Decoding

Guoxuan Xia, Luka Ribar, Paul Balanca

Speculative decoding accelerates sampling from an autoregressive LLM by using a faster auxiliary model to draft tokens which are then verified in parallel by the LLM. Standard speculative decoding is lossless: its rejection and resampling steps exactly preserve the LLM's sampling…Read more