Improving brightness of multiplexed 7-segment led display
You definitely have to use series resistors!  
Don't use the 74HC244 others suggested. Read the datasheet, they can only source and sink 6mA, not 20mA. 


You have a common anode 4\$\times\$4 matrix, so for a 20mA display the anode driver has to supply 80mA if all LEDs are on. The most simple solution is to use 4 PNP transistors which you drive via a base resistor from your microcontroller. A low level will activate a column. The BC327 has an \$H_{FE}\$ of 100 minimum, so a 2mA base current will be enough to switch it on. Then your base resistor is 2.2k\$\Omega\$.  

Most microcontrollers' I/O can sink 20mA so for port B you only need 4 series resistors. If your LEDs drop 2V you need 150\$\Omega\$. If the 20mA should be too much for the controller you can use NPN transistors as drivers. A 4.7k\$\Omega\$ base resistor will do.  

This is a cheaper solution than using drivers like the ULN2803.