How to define enumeration of the set of finite state machines?
One simple encoding is to represent each of the $N$ states by a partial function $f_n:\{0,1\} \rightarrow \{1, \dots, N\}$. Input $x$ in state $n$ will cause a transisition to state $f_n(x)$ if $f_n(x)$ exists and an error condition if not. You can assume that the initial state is always state $1$.
For $N$ states there are $(N+1)^2$ such partial functions (because either $0$ or $1$ or both may fail to have an image), so there are $(N+1)^{2N}$ encodings. However, some of the encodings will represent FSMs that are isomorphic (because they represent the &quot;same&quot; FSM with a re-labelling of states).
For example, for $2$ states there are $9$ partial functions from $\{0,1\}$ to $\{1,2\}$ and each state can be represented by one of these partial functions, so there are $9^2=81$ representations of FSMs. The $2$ state FSM which stays in the same state on input $0$ and changes to the opposite state on input $1$ is represented by the two functions
$f_1(0)=1 \quad f_1(1)=2\\
f_2(0)=2 \quad f_2(1)=1$