Synthesize Digital Filter with Arbitrary Frequency-Response
For FIR filter design, common algorithms include Remez exchange or Parks-McClellan :
What is the difference between Remez exchange and Parks-McClellan filter design?

For IIR filters, one possibility is the Differential Evolution algorithm, as described in some IEEE journals.  If you have a realistic desired phase response and can guess the filter order, then Berchin's FDLS method may be suitable.