How to effectively represent and generate 2D cellular automaton rules that are invariant under rotation and reflection of the input matrix?
I won't go into the details of your specific case but try to answer the general problem.

In the unrestricted case there is a mapping from each of the $n$ (=512) input states to one of 2 output states and you want to restrict the function as follows:


Create a partitioning of the initial $n$ input states into subsets.
Your adapted function maps each of these subsets to an output


Proceed as follows:


Enumerate all input states from 1 to $n$ (this is then the index of that state)
Write a function $f(i)$ that generates all rotations/symmetric versions of an input state $i$ and take the lowest index of that set
Partition the set $1..n$ according to $f$, i.e. if $f(x) = f(y)$, they are in the same class. The result is a pairwise disjoint set of sets, where each set consists of 'similar' input states that should therefore also have the same output. 
Let $P$ be the number of these sets.


Answering your questions should then be easy:


You could assign each of the $P$ sets an arbitrary output to get a unique function. $\Rightarrow 2^P$ functions
A bitstring of length $P$
Generate random integers (with more than $P$ bits each) and take the last $P$ bits. 

a. Maybe consider the distribution of values generated by your RNG, i.e.: are the last $P$ bits uniformly (or whatever you desire) distributed?