Optimization Teensy 3.6 Arduino IDE fast I/O Multiplexing Led Matirx
I have also written a driver for LED matrices based on shift registers. To quickly shift out the bits to the shift registers for each scan update, I used SPI. On the Teensy 3.6, this works quite well. 

But what I suspect your issue is (since you didn't post your whole code) is that you are calculating each of the bit updates in between each shifting out. If you are doing this, I suggest that you don't. Instead, buffer the bit calculations, and use a timer interrupt to shift out each row update. This approach takes advantage of the fact that you are likely to recalculate the bit layout for the matrix overall at a much slower frequency than the need to send our row updates to effect the multiplexing of the LED rows. I explain this design in much more detail in this Instructable.

Using the interrupt allows you to cleanly do the heavy lifting of recalculating the bit layout into a buffer while the row updates will consistently go out, creating a very stable image. It's the closest thing you'd to true threading on Arduino-like platforms. Also, this approach will allow a much smaller base interval for row updates. In my code, I use a 5 µs update interval, that gets modulated in order to be able to create a 12-bit color palette. I probably could get a richer color depth on the Teensy 3.6 given it's clock speed, but I wanted my code to be universal, so I am able to get the 12-bit color on 8-bit AVR micro controllers too (e.g., Arduino Mega). 

You can study (or use) my code at this GitHub repository, and read more about my LED matrices (hardware and software) at my website.