Oracle 12c Database swapping after set up huge pages
The steps I did to setup this hugepage configuration on ReDHat 7.3 64GBRAM and the mistake. (Link from oracle base posted above provides a better Manual for reuse)
Check total RAM of OS:

grep MemTotal /proc/meminfo
MemTotal:       65809456 kB


Check Oracle Database – AMM needs to be off:
MEMORY_TARGET und MEMORY_MAX_TARGETneed to be set to zero.

SQL&gt; select value from v$parameter where name = 'memory_target';
VALUE
---------------------------
0


Calculate SGA size, for example 32GB:

select value/1024 from v$parameter where name = 'sga_target';

VALUE/1024
----------
  32768000


Calculate Huge Pages,
for a  x86_64 Red Hat Enterprise Linux Server we can use a default 2MB hugepagesize: 

SGA / Hugepagesize=Hugepage +1
  32768000/2048=16001


This need to be set in /etc/sysctl.conf as root:

vm.nr_hugepages =16001


Also we need to set the /etc/security/limits.conf. This value hard and soft memlock should be greater than the the SGA and smaller than the total RAM. 

Hugepages * Hugepagesize = minimum Memlock or something like 90% of total RAM.


Setting memlock to a lower value then SGA leads to unpredictable behavior of the Oracle Database. As commented above Oracle Database will allocate memory without using the huge pages. In my case this leaded to fatal errors when using impdp. Datapump did not provide errors which made sense.  

grep oracle /etc/security/limits.conf
oracle  soft    nproc   2047
oracle  hard    nproc   16384
oracle  soft    nofile  1024
oracle  hard    nofile  65536
oracle  soft    stack   10240
#oracle  soft    memlock 19857408
#oracle  hard    memlock 19857408
oracle  soft    memlock 57671680
oracle  hard    memlock 57671680


For RedHat we also need to turn off Transparent Hugepages

cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
[always] madvise never

echo never &gt; /sys/kernel/mm/redhat_transparent_hugepage/enabled
echo never &gt; /sys/kernel/mm/redhat_transparent_hugepage/defrag

cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
always madvise [never]


Reboot System and start Oracle Database check:

grep Huge /proc/meminfo
AnonHugePages:     40960 kB
HugePages_Total:   16001
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB