AVR C 74hc595 rgb matrix multiplexing and brightness
Your program appears to have a problem with ghosting- I don't see you shutting down the anode drive before changing the cathode drive. 

As far as brightness goes- it looks to me from a cursory look at the code that you have LEDs powered all the time so the duty cycle is about 50% for 'on' LEDs. 
Do verify this somehow, preferably with an oscilloscope. 

In that case, you would have to increase the current (or use much better LEDs). The 74HC595 is not great at sourcing current and you are asking it to source current for (right now) 7 LEDs so the loading is like an LED with 31 ohms in series. If you only turn on a single LED you will probably see much more brightness. 

You can add anode drivers (for example, use the 74HC595s to switch P-channel MOSFETs). You will then be limited by the abs max 70mA GND current of the 595. If you go 60mA that's 1.8mA per LED in a 2 of 4 matrix. Maybe enough with good LEDs, otherwise you have to add cathode drivers too. 

As you've discovered, the brightness does not change significantly with timing until you fall below the flicker fusion frequency, it's basically the average LED current. No free lunches.