How can I make a phase shift in an audio signal without changing the frequency response too much?
You want an audio delay line. These basically sample the incoming signal, remember it for a little while, and then send it back out again. As long as you sample at 2X the highest frequency you care about, then there should not be any changes to the spectrum across the delay (but you might need to low pass filter the input to avoid aliasing). The delay lines can be digital or analog, hardware and/or software. 

The Teensy ecosystem has an amazing support for audio processing that would make building something like this as easy and connecting some processing blocks on the screen.

If you want to go old school, you could use this amazing chip that stores the samples as analog voltages. It can add a programmable delay between 5ms and 51ms. 

If you want something more interesting, you might consider this approach that over-samples a single bit for its simplicity and elegance. I remember back in the day when single bit digital amplifiers first came out using this same trick.

If you just want to buy something, here is a delay board with audio connectors and potentiometers ready to go.