Balance groups based on average, element constraints and size
This is a combinatorial optimization problem.  It is NP-hard (it is easy to see it is as hard as the partition problem).  Consequently, you should not expect any general problem that is efficient, scales to large problem sizes, and always returns the optimal answer.

Instead, you probably will want to use some kind of heuristics.  One approach worth considering is to use integer linear programming or some variant of it.  If your objective function is minimize the spread between largest group average and the smallest group average, then the problem can be encoded as an instance of integer linear programming (ILP) and then you could use an off-the-shelf ILP solver.  If your objective function is to minimize the standard deviation of the group averages, then the problem can be encoded as an instance of integer quadratic programming (MIQP) and then you could use a MIQP solver.  You might want to run them for a limited amount of time and use the best solution found within that time period, rather than run as long as needed to find the exact optimal solution.

The encoding of the problem as integer programming is tedious but fairly straightforward.  You introduce zero-or-one integer variables $x_{i,j}$, with the intended meaning that $x_{i,j}=1$ means that the $i$th item is assigned to the $j$th group.  (When you have multiple elements of the same color, treat them all as a single item with the same value of $i$; they just add more than one to the number of elements in that group.)  Then the number of elements in the group is a linear function of the $x$'s, and the group average of each group is a linear function of the $x$'s, so you can write down constraints for these to be a valid solution as linear inequalities.