Optimizing MariaDB for 5-10K small databases Running a SaaS based on MariaDB LTS (1:10.1.44-0ubuntu0.18.04.1) with nearly 5000 databases (1 per tenant). Average per tenant: 8MB of data (on disk) with ~50 tables. CPU/Mem load is negligible. It's been running fine for a couple years, with some 5-20 tenants added per day.

However. When the nr of dbs grows to about 5150, the MariaDB server process crashes. So far I haven't been able to distill a useful log message, unfortunately. Furthermore, it takes about 10 minutes for MariaDB to start up, which is quite long when I want to experiment with new settings. 

What settings would you recommend to reduce the risk of crashing, and speed up starting time?

Some more characteristics:


SHOW GLOBAL STATUS
SHOW VARIABLES
Server has 2CPU and 8GB ram, of which 25% is available during peak times.
Load per CPU core never exceeds 0.20 during peak times 
Connections are short lived, I expect that there are never more than 20 concurrent db connections. 
All tables are InnoDB
Schema may differ per database (but are probably identical for 98% of tenants)
I use stock configuration, except for:


innodb_file_per_table
open_files_limit        = 65536
log_warnings            = 3
innodb_buffer_pool_size = 1G


FWIW, I'm willing to upgrade to a newer version of MariaDB, but given the risk of new uncertainties, I'd rather make an informed decision than take a leap of faith. 

Thanks for any advice!
Your output for GLOBAL_STATUS is showing:

opened_tables 2098690


The definition for this status is defined as:


  Number of tables the server has opened...


Reference: Opened_tables (MariaDB documentation) 

Your configuration for innodb_file_per_table is set to:

innodb_file_per_table   ON


And MariaDB defines this setting to be:


  If set to ON, then new InnoDB tables are created with their own InnoDB file-per-table tablespaces. If set to OFF, then new tables are created in the InnoDB system tablespace instead. Page compression is only available with file-per-table tablespaces. Note that this value is also used when a table is re-created with an ALTER TABLE which requires a table copy.


Reference: innodb_file_per_table (MariaDB documentation)

So basically, you have 2'098'690 open files on your Ubuntu Server, but your settings for innodb_open_files is set to 2000:

innodb_open_files   2000


Now there could be an issue that Ubunutu is hitting a limit or your MariaDB is in some race condition opening and closing the database/table files. 

The 2 million + open files is a counter and there is a recommendation that if your counter keeps on increasing that: 


  If the number of opened tables is increasing rapidly, you should look at increasing the table_open_cache value. Try to find a value that sees a slow, or possibly even no, increase in the number of opened tables.


Reference: Optimizing table_open_cache (MariaDB documentation)

Additionally you might want to have a look at your Ubunutu instance and see what limits are set for system-wide resources:

 bash#&gt; ulimit -a


This will return a list of currently set limits. You might have to increase these limits to achieve a stable MariaDB environment. The one setting being:

-n   The maximum number of open file descriptors. 


Reference: ulimit (SS64.com)

You can set this setting according to this Q&amp;A: 

How to increase max open files limit on Ubuntu 18.04? (Ask Ubuntu)