title: Optimizing mysql my.cnf - memory usage is dangerously high
body: <p>I have tunet up mysql by following some tips on the internet. </p>
  
  <p>but the websites are loading slow and time time-to-first-byte is very high. therefore i started investigation and as far as i see,  it is casued by mysql and high memory usage</p>
  
  <p>every time i make changes to the my.cnf according to the suggestions i get from 
  <strong>./mysqltuner.pl</strong> AND <strong>./tuning-primer.sh</strong>
  it gives another suggestions.  Actually some values should have balance to each other.   I hope someone have an idea how to get high performance usage of mysql and in the same time, take care of the server health</p>
  
  <p><strong>MariaDB server version: 10.0.20-MariaDB-log MariaDB Server</strong>
  Server information:
  Intel Core i7-3770
  2x HDD 3,0 TB SATA
  4x RAM 8192 MB DDR3
  Cloudlinux + Cpanel installed
  Apache/2.4.16 + eAccelerator + mod_pagespeed</p>
  
  <pre><code>SLOW QUERIES
  Error: slow_query_log_file=/var/log/mysql/log-slow-queries.log
  Current long_query_time = 10.000000 sec.
  You have 1944 out of 25401054 that take longer than 10.000000 sec. to complete
  **Your long_query_time seems to be fine**
  
  BINARY UPDATE LOG
  The binary update log is NOT enabled.
  **You will not be able to do point in time recovery**
  See http://dev.mysql.com/doc/refman/10.0/en/point-in-time-recovery.html
  
  WORKER THREADS
  Current thread_cache_size = 8
  Current threads_cached = 6
  Current threads_per_sec = 0
  Historic threads_per_sec = 0
  **Your thread_cache_size is fine**
  
  MAX CONNECTIONS
  Current max_connections = 151
  Current threads_connected = 6
  Historic max_used_connections = 43
  The number of used connections is 28% of the configured maximum.
  **Your max_connections variable seems to be fine.**
  
  No InnoDB Support Enabled!
  
  MEMORY USAGE
  Max Memory Ever Allocated : 20.96 G
  Configured Max Per-thread Buffers : 24.81 G
  Configured Max Global Buffers : 13.89 G
  Configured Max Memory Limit : 38.70 G
  Physical Memory : 31.12 G
  
  **Max memory limit exceeds 90% of physical memory**
  
  KEY BUFFER
  Current MyISAM index space = 29 M
  Current key_buffer_size = 384 M
  Key cache miss rate is 1 : 870
  Key buffer free ratio = 78 %
  **Your key_buffer_size seems to be fine**
  
  QUERY CACHE
  Query cache is enabled
  Current query_cache_size = 512 M
  Current query_cache_used = 222 M
  Current query_cache_limit = 1.00 G
  Current Query cache Memory fill ratio = 43.41 %
  Current query_cache_min_res_unit = 4 K
  **MySQL won't cache query results that are larger than query_cache_limit in size**
  
  SORT OPERATIONS
  Current sort_buffer_size = 16 M
  Current read_rnd_buffer_size = 8 M
  **Sort buffer seems to be fine**
  
  JOINS
  ./tuning-primer.sh: line 402: export: `2097152': not a valid identifier
  Current join_buffer_size = 128.00 M
  You have had 10199 queries where a join could not use an index properly
  join_buffer_size &gt;= 4 M
  **This is not advised
  You should enable "log-queries-not-using-indexes"
  Then look for non indexed joins in the slow query log.**
  
  OPEN FILES LIMIT
  Current open_files_limit = 16162 files
  The open_files_limit should typically be set to at least 2x-3x
  that of table_cache if you have heavy MyISAM usage.
  **Your open_files_limit value seems to be fine**
  
  TABLE CACHE
  Current table_open_cache = 8000 tables
  Current table_definition_cache = 8000 tables
  You have a total of 7347 tables
  You have 8000 open tables.
  **Current table_cache hit rate is 21%
  , while 100% of your table cache is in use
  You should probably increase your table_cache**
  
  TEMP TABLES
  Current max_heap_table_size = 16 M
  Current tmp_table_size = 16 M
  Of 592173 temp tables, 36% were created on disk
  **Perhaps you should increase your tmp_table_size and/or max_heap_table_size
  to reduce the number of disk-based temporary tables
  Note! BLOB and TEXT columns are not allow in memory tables.
  If you are using these columns raising these values might not impact your
  ratio of on disk temp tables.**
  
  TABLE SCANS
  Current read_buffer_size = 16 M
  Current table scan ratio = 74 : 1
  **read_buffer_size is over 8 MB there is probably no need for such a large read_buffer**
  
  TABLE LOCKING
  Current Lock Wait ratio = 1 : 4366
  **You may benefit from selective use of InnoDB.
  If you have long running SELECT's against MyISAM tables and perform
  frequent updates consider setting 'low_priority_updates=1'
  If you have a high concurrency of inserts on Dynamic row-length tables
  consider setting 'concurrent_insert=ALWAYS'.**
  
  root@my [~]# ./mysqltuner.pl
  
   &gt;&gt;  MySQLTuner 1.4.0 - Major Hayden &lt;major@mhtx.net&gt;
   &gt;&gt;  Bug reports, feature requests, and downloads at http://mysqltuner.com/
   &gt;&gt;  Run with '--help' for additional options and output filtering
  [!!] Currently running unsupported MySQL version 10.0.20-MariaDB-log
  [OK] Operating on 64-bit architecture
  
  -------- Storage Engine Statistics -------------------------------------------
  [--] Status: +ARCHIVE +Aria +BLACKHOLE +CSV +FEDERATED +InnoDB +MRG_MyISAM
  [--] Data in MyISAM tables: 82M (Tables: 925)
  [--] Data in InnoDB tables: 7G (Tables: 6334)
  [--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 52)
  [--] Data in MEMORY tables: 0B (Tables: 2)
  [!!] Total fragmented tables: 159
  
  -------- Security Recommendations  -------------------------------------------
  [OK] All database users have passwords assigned
  
  -------- Performance Metrics -------------------------------------------------
  [--] Up for: 14h 42m 18s (25M q [480.374 qps], 81K conn, TX: 71B, RX: 6B)
  [--] Reads / Writes: 98% / 2%
  [--] Total buffers: 13.9G global + 168.3M per thread (151 max threads)
  **[!!] Maximum possible memory usage: 38.7G (124% of installed RAM)**
  [OK] Slow queries: 0% (1K/25M)
  [OK] Highest usage of available connections: 28% (43/151)
  [OK] Key buffer size / total MyISAM indexes: 384.0M/29.8M
  [OK] Key buffer hit rate: 99.9% (12M cached / 14K reads)
  [OK] Query cache efficiency: 44.9% (20M cached / 44M selects)
  **[!!] Query cache prunes per day: 2013573**
  [OK] Sorts requiring temporary tables: 0% (1K temp sorts / 1M sorts)
  **[!!] Joins performed without indexes: 10207
  [!!] Temporary tables created on disk: 58% (345K on disk / 592K total)**
  [OK] Thread cache hit rate: 98% (1K created / 81K connections)
  [OK] Table cache hit rate: 21% (8K open / 38K opened)
  [OK] Open file limit used: 12% (1K/16K)
  [OK] Table locks acquired immediately: 99% (10M immediate / 10M locks)
  [OK] InnoDB buffer pool / data size: 10.0G/7.7G
  [OK] InnoDB log waits: 0
  -------- Recommendations -----------------------------------------------------
  General recommendations:
      **Run OPTIMIZE TABLE to defragment tables for better performance
      Reduce your overall MySQL memory footprint for system stability
      Increasing the query_cache size over 128M may reduce performance
      Adjust your join queries to always utilize indexes
      When making adjustments, make tmp_table_size/max_heap_table_size equal
      Reduce your SELECT DISTINCT queries without LIMIT clauses**
  Variables to adjust:
    *** MySQL's maximum memory usage is dangerously high ***
    *** Add RAM before increasing MySQL buffer variables ***
      query_cache_size (&gt; 512M) [see warning above]
      join_buffer_size (&gt; 128.0M, or always use indexes with joins)
      tmp_table_size (&gt; 16M)
      max_heap_table_size (&gt; 16M)
  </code></pre>
  
  <p>And here is the my.cnf settings which i have set</p>
  
  <pre><code>[mysqld]
  #http://blog.secaserver.com/2011/08/mysql-recommended-my-cnf-settings-innodb-engine/
  # GENERAL #
  default-storage-engine=InnoDB
  tmpdir=/tmp_mysql
  group_concat_max_len=10000000
  local-infile=1
  
  # LOGGING #
  slow_query_log = 1
  slow_query_log_file=/var/log/mysql/log-slow-queries.log
  long_query_time = 10
  log-error = /var/log/error.log
  log-queries-not-using-indexes
  
  # CACHES AND LIMITS AND SAFETY #
  max_allowed_packet = 512M #16
  query_cache_size = 512M
  query_cache_limit = 1024M
  thread_cache_size = 8
  table_definition_cache = 8000
  table_open_cache = 8000
  sort_buffer_size = 16M
  read_buffer_size = 16M   #2
  read_rnd_buffer_size = 8M
  join_buffer_size = 128M
  thread_concurrency = 0 # Try number of CPU's*2 for thread_concurrency
  key_buffer = 256M
  
  
  # INNODB #
  innodb_file_per_table=1
  innodb_file_format = Barracuda
  innodb_sort_buffer_size = 128M
  innodb_data_home_dir = /var/lib/mysql
  innodb_log_group_home_dir = /var/lib/mysql
  innodb_thread_concurrency=0
  innodb_flush_method=O_DIRECT 
  innodb_lock_wait_timeout = 120
  innodb_buffer_pool_size=10G
  innodb_log_file_size = 1536M   # Set .. innodb_log_file_size to 25 % of innodb_buffer_pool_size -You can set .. innodb_buffer_pool_size up to 50 - 80 %
  innodb_log_buffer_size = 3072M
  innodb_additional_mem_pool_size = 20M
  #innodb_read_io_threads=16
  #innodb_write_io_threads=16
  #innodb_io_capacity=500
  #innodb_flush_log_at_trx_commit=1
  #sync_binlog=1
  #innodb_data_file_path = ibdata1:2000M;ibdata2:10M:autoextend
  
  
  # MyISAM #
  key_buffer_size = 384M
  myisam_sort_buffer_size = 64M
  
  
  [mysqldump]
  quick
  max_allowed_packet = 16M
  
  [mysql]
  no-auto-rehash
  # Remove the next comment character if you are not familiar with SQL
  #safe-updates
  
  [myisamchk]
  key_buffer_size = 256M
  sort_buffer_size = 256M
  read_buffer = 2M
  write_buffer = 2M
  
  [mysqlhotcopy]
  interactive-timeout
  </code></pre>