How does 速 come to mean invite?
多功能成语词典 explains:

解释 速:邀请｡指未经邀请而突然自己来的客人｡

and also gives the note:

提示 “速&quot;不要解释成“速度&quot;或“迅速&quot;｡

A Student's Dictionary of Classical and Medieval Chinese also notes that 速 has the meanings

2 lead to, induce to; beckon toward; invite.

汉语大词典 also notes this meaning, with many examples

5 召, 请。
《诗·小雅·伐木》
既有肥羜, 以速諸父。
郑玄 笺
速, 召也。
《文选·张衡〈思玄赋〉》
速燭龍令執炬兮, 過 鍾山 而中休。
旧注
速, 徵也。
清 顾禄 《清嘉录·冬至大如年》
是夜, 人家更速燕飲, 謂之節酒。
杨树达 《〈凌虚岛〉小说序》
近者得君札, 速余序君書。

It’s not that 速 came to mean invite - but rather that it meant this as early as I Ching, which came out roughly 1100 BC - over three thousand years ago:

《周易·需》:“有不速之客三人来｡&quot;