Is multi-byte mapping to one byte a viable encryption technique? The idea is to randomly map byte values eg

\begin{equation}
  Sub(x)=\left\{
  \begin{array}{@{}ll@{}}
    (a, b), &amp; \text{if}\ r=1 \\
    (c, d), &amp; \text{r=0}
  \end{array}\right.
\end{equation} 

Where $r$ is the random value that determines the output, $x$ is the input byte, and $a, b$ or $c, d$ is the output bytes.

A concrete example might be:

\begin{equation}
  Sub(97)=\left\{
  \begin{array}{@{}ll@{}}
    (12, 57), &amp; \text{if}\ r=1 \\
    (34, 54), &amp; \text{r=0}
  \end{array}\right.
\end{equation} 

Each byte has a unique byte reference.

To encrypt choose random map and set

encrypt[97] = 12,57
encrypt[97] = 34,54


To decrypt find matching values

decrypt[12,57] = 97
decrypt[34,54] = 97


Is this kind of multi-byte mapping to one byte a viable encryption technique?
You seem to be describing a homophonic substitution cipher, except on bytes instead of letters.  Such a cipher is slightly better (i.e. harder to break) than a simple substitution cipher, but not by much.

To analyze the security of your cipher in more detail, one would need to know how the encryption map is chosen (e.g. is it derived from a key somehow), how many possible encryptions each plaintext byte has and how the encryption of each plaintext byte is chosen.  Still, even without knowing the details, most ciphers of this type tend to be vulnerable to some pretty simple attacks if the attacker can collect enough ciphertext (and possibly corresponding plaintext) to analyze.

For example, if the encryption of each plaintext byte is chosen randomly from among the possible alternatives, this implies that each of the encryptions of a given plaintext byte occurs with the same average frequency.  If the attacker can collect enough ciphertext, they can determine which pairs of bytes occur with approximately the same frequency in the ciphertext, and guess that those are probably alternative encryptions of the same plaintext byte.  (How much is enough depends strongly on what the plaintext is, but a megabyte or two should be plenty for most kinds of data.)  Having done that, they can then attack the cipher using basic frequency analysis, just like one would attack a simple non-homophonic substitution cipher.

Also, if the attacker has access to some ciphertext and to the corresponding plaintext, they can simply compile a partial decryption table by comparing them.  With this method, even a few dozen kilobytes or less may be enough to get a reasonably good coverage of the possible encryptions of most common plaintext bytes.  Armed with such a partial encryption table, the attacker may then be able to partially decrypt other ciphertexts and to use those partial decryptions to guess the plaintext bytes corresponding to other ciphertext byte pairs and thereby extend their table further.

Basically, a cipher like this could be quite a challenging puzzle for an amateur cryptanalyst to break.  You shouldn't rely on it for any real security, though, especially not if you want to encrypt more than very small amounts of data.