Transforming a byte with a subset of a small, fixed set of values and xor into any other value
You have recalled the correct category, "linear-algebra".

Let us see how we can interpret xor in the terms of linear-algebra. Here is the computation rules for xor.
$$0\oplus0=0$$
$$0\oplus1=1$$
$$1\oplus0=1$$
$$1\oplus1=0$$
If we consider 0 and 1 as the zero element and the unit element of the binary field $\Bbb F_2$, the simplest finite field, then we can see $\oplus$ is none other than the addition in $\Bbb F_2$.
$$0+0=0$$
$$0+1=1$$
$$1+0=1$$
$$1+1=0$$

That means, if we xor a string of 3 bits to another string of 3 bits, we are actually adding two vectors $\vec a$ and $\vec b$ in the three-dimensional linear space $\Bbb F_2^3,$
$$\vec a+\vec b=\vec c$$
For example, $(0,1,0) + (1,1,0)=(1,0,0)$.

If we fix the second vector, the addition by the second vector is a linear transformation from $\Bbb F_2^3$ to $\Bbb F_2^3$. All these transformations form a 3-dimensional linear space over $\Bbb F_2$ with basis 
$$\{\text{addition by (1,0,0)}, \text{ addition by } (0,1,0), \text{ addition by (0,0,1)}\}.$$ 
Given a set of vector in $\Bbb F_2^3$, all combinations of the additions by those vectors can transform any vector to another vector if and only those vectors are linearly independent. 

For example, "three bits 0b101, 0b110 and 0b011" "do not have this property" because, $(1,0,1)+(1,1,0)+(0,1,1)=(0,0,0)$ in $\Bbb F_2^3$, i.e, those three vectors are not linearly independent.



Here is a counting exercise.

(Exercise) Show the number of sets of such strings of 3 bits is $(2^3-1)*(2^3-2)*(2^3-2^2) = 168.$ In general the number of set of such string of n bits is $(2^n-1)(2^n-2)\cdots(2^n-2^{n-1}).$