Can I run a GA to optimize wavelet transform?
You should be able to use GA or any other derivative-free optimization methods to solve your problem [1]. The correct parameter values can be determined using your application. If I'm trying to optimize for compression then I will look into the entropy of the resultant coefficients. Deciding on the actual values for your parameters will depend on how you plan to use them. The correct parameter values are simply the ones that work better.