Can HKDF be instantiated with HMAC-SWIFFT?
The reasoning is wrong: the whole point of the paper is to establish a definition for secure general-purpose KDFs. 

In definition 7 of the paper, it is defined that a KDF is secure, if it can withstand distinguishing game from an adversary with capability to query the KDF oracle - that's what something linear like SWIFFT cannot provide. 

As for the quoted text in OP, it just means the author is proving his ideas using as few and as straightforwardly useful assumptions as possible, it did not imply what I thought it implied in OP.