Help me describe/identify this challenge-response protocol/algorithm?
The intend appears to be that vector is secret and the main key; and version (that, we are told, is secret) is an extension of that key (or variant selector).

1) I could say that ResetCode is a MAC of quote with key vector and version.

2) Never met this particular one. Anyone with common sense should laugh at it as snake oil if it pretends to be military-grade cryptography.

3a) When seen as a MAC, an obvious algorithmic flaw is that adding 1 (mod 256) to version and subtracting 11111 from any quote which has each of its digits non-zero leaves the ResetCode unchanged; this can be classified as a related-key weakness. That observation can make a usable paper tabulation of the function over twice less bulky (only entries with quote including a 0 need to be tabulated). Adding 10 (mod 256) to version also has special properties.

3a') Update: there's much worse! For constant version and vector, changing a particular digit in quote into another digit causes a difference (mod 10) in any particular digit of ResetCode that is independent of the 4 other digits of quote. With this observation, from the ResetCode for a small set of quote values, we can compute the ResetCode for any quote value, without knowing vector or version. The set of 46 quote values having at most one non-zero digit is convenient for hand calculation, but even random quote from a significantly smaller set often allows trivial linear algebra to pull that trick. In fact that MAC is so linear and with poor mixing that it could be broken from examples alone, without knowledge of the algorithm or reverse engineering the code or extracting vector!

3b) In the usage context, after vector has become public (e.g. if it is common to all alarms of the type), most often a single intercept of (ResetCode, quote) is enough to find version, leading to total break. Even a theoretically perfect cryptographic MAC is expected to have this property, given the ridiculously small size of what remains of the key (version), and the larger size of ResetCode. Any theoretical weaknesses of the algorithm seen as a MAC is moot compared to this system weakness of a small version when an adversary can obtain all key material beside version, or has access to a black box computing ResetCode for any chosen input (quote, version).