Can oracle use more memory than memory_max_target?
The output of free includes caches and buffers, and a lot of people fall into the trap of thinking that their memory is used while it is not. Visit this site for explanation: http://www.linuxatemyram.com/

On the other hand though, yes, Oracle can definitely use more memory than configured with the memory_max_target parameter. Parameters, such as memory_max_target, memory_max_size limit the size of the SGA only. 
Prior to 12c, PGA could not be limited, even though there is the hidden parameter _pga_max_size. That parameter limits the PGA size per session, but it does not account for all kinds of allocations, thus even 1 "rouge" session can eat up all the memory of the database server. 

Starting with 12c, there is the parameter pga_aggregate_limit, which is a real hard limit for the PGA.

References:


Using Automatic Memory Management (Oracle Docs 11g)
6.3 Using Automatic Memory Management (Oracle Docs 12c)
6.4.4 Using Automatic PGA Memory Management (Oracle Docs 12c)