title: How to discover why MariaDB crashes?
body: <p>I use Nextcloud (18 currently). Everything was working fine until this day when I got this message when accessing my cloud via the webbrowser.</p>
  
  <blockquote>
    <p>Internal Server Error  The server encountered an internal error  and was unable to complete your request. Please contact the server  administrator if this error reappears multiple times, please include the  technical details below in your report. More details can be found in  the server log.</p>
  </blockquote>
  
  <p>I am using Docker, or more specific Docker Compose for almost 2 years now. fpm alpine, split into 5 parts: mariadb, redis, app, web, and cron.</p>
  
  <pre><code>version: '3'
  
  services:
    nextcloud_db:
      image: mariadb
      command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
      restart: always
      volumes:
        - /etc/localtime:/etc/localtime:ro
        - nextcloud_db:/var/lib/mysql
      environment:
        - MYSQL_ROOT_PASSWORD=
      env_file:
        - db.env
  
    nextcloud_redis:
      image: redis:alpine
      restart: always
  
    nextcloud_app:
      build: ./app
      restart: always
      volumes:
        - /etc/localtime:/etc/localtime:ro
        - nextcloud:/var/www/html
      environment:
        - MYSQL_HOST=nextcloud_db
        - REDIS_HOST=nextcloud_redis
      env_file:
        - db.env
      depends_on:
        - nextcloud_db
        - nextcloud_redis
  
    nextcloud_web:
      build: ./web
      restart: always
      volumes:
        - /etc/localtime:/etc/localtime:ro
        - nextcloud:/var/www/html:ro
      environment:
        - VIRTUAL_HOST=
        - LETSENCRYPT_HOST=
      depends_on:
        - nextcloud_app
  
    nextcloud_cron:
      image: nextcloud:fpm-alpine
      restart: always
      volumes:
        - /etc/localtime:/etc/localtime:ro
        - nextcloud:/var/www/html
      entrypoint: /cron.sh
      depends_on:
        - nextcloud_db
        - nextcloud_redis
  
  volumes:
    nextcloud_db:
    nextcloud:
  </code></pre>
  
  <p>The mariadb one has an error currently.</p>
  
  <pre><code>nextcloud_db_1     | 2020-05-15 01:28:44+02:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.4.13+maria~bionic started.
  nextcloud_db_1     | 2020-05-15 01:28:49+02:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
  nextcloud_db_1     | 2020-05-15 01:28:49+02:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 1:10.4.13+maria~bionic started.
  nextcloud_db_1     | 2020-05-15  1:28:50 0 [Note] mysqld (mysqld 10.4.13-MariaDB-1:10.4.13+maria~bionic) starting as process 1 ...
  nextcloud_db_1     | 2020-05-15  1:28:50 0 [Warning] You need to use --log-bin to make --binlog-format work.
  nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] InnoDB: Using Linux native AIO
  nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
  nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] InnoDB: Uses event mutexes
  nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
  nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] InnoDB: Number of pools: 1
  nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] InnoDB: Using SSE2 crc32 instructions
  nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] mysqld: O_TMPFILE is not supported on /tmp (disabling future attempts)
  nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
  nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] InnoDB: Completed initialization of buffer pool
  nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
  nextcloud_db_1     | 2020-05-15  1:28:51 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=8312011756
  nextcloud_db_1     | 2020-05-15  1:28:52 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
  nextcloud_db_1     | 2020-05-15  1:28:52 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
  nextcloud_db_1     | 2020-05-15  1:28:52 0 [Note] InnoDB: Creating shared tablespace for temporary tables
  nextcloud_db_1     | 2020-05-15  1:28:52 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
  nextcloud_db_1     | 2020-05-15  1:28:52 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
  nextcloud_db_1     | 2020-05-15  1:28:52 0 [Note] InnoDB: Waiting for purge to start
  nextcloud_db_1     | 2020-05-15 01:28:52 0x7f010affd700  InnoDB: Assertion failure in file /home/buildbot/buildbot/build/mariadb-10.4.13/storage/innobase/include/fut0lst.ic line 67
  nextcloud_db_1     | InnoDB: Failing assertion: addr.page == FIL_NULL || addr.boffset &gt;= FIL_PAGE_DATA
  nextcloud_db_1     | InnoDB: We intentionally generate a memory trap.
  nextcloud_db_1     | InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
  nextcloud_db_1     | InnoDB: If you get repeated assertion failures or crashes, even
  nextcloud_db_1     | InnoDB: immediately after the mysqld startup, there may be
  nextcloud_db_1     | InnoDB: corruption in the InnoDB tablespace. Please refer to
  nextcloud_db_1     | InnoDB: https://mariadb.com/kb/en/library/innodb-recovery-modes/
  nextcloud_db_1     | InnoDB: about forcing recovery.
  nextcloud_db_1     | 200515  1:28:52 [ERROR] mysqld got signal 6 ;
  nextcloud_db_1     | This could be because you hit a bug. It is also possible that this binary
  nextcloud_db_1     | or one of the libraries it was linked against is corrupt, improperly built,
  nextcloud_db_1     | or misconfigured. This error can also be caused by malfunctioning hardware.
  nextcloud_db_1     |
  nextcloud_db_1     | To report this bug, see https://mariadb.com/kb/en/reporting-bugs
  nextcloud_db_1     |
  nextcloud_db_1     | We will try our best to scrape up some info that will hopefully help
  nextcloud_db_1     | diagnose the problem, but since we have already crashed,
  nextcloud_db_1     | something is definitely wrong and this may fail.
  nextcloud_db_1     |
  nextcloud_db_1     | Server version: 10.4.13-MariaDB-1:10.4.13+maria~bionic
  nextcloud_db_1     | key_buffer_size=134217728
  nextcloud_db_1     | read_buffer_size=2097152
  nextcloud_db_1     | max_used_connections=0
  nextcloud_db_1     | max_threads=102
  nextcloud_db_1     | thread_count=4
  nextcloud_db_1     | It is possible that mysqld could use up to
  nextcloud_db_1     | key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 760255 K  bytes of memory
  nextcloud_db_1     | Hope that's ok; if not, decrease some variables in the equation.
  nextcloud_db_1     |
  nextcloud_db_1     | Thread pointer: 0x7f00fc000c08
  nextcloud_db_1     | Attempting backtrace. You can use the following information to find out
  nextcloud_db_1     | where mysqld died. If you see no messages after this, something went
  nextcloud_db_1     | terribly wrong...
  nextcloud_db_1     | stack_bottom = 0x7f010affcc08 thread_stack 0x49000
  nextcloud_db_1     | 2020-05-15  1:28:52 0 [Note] InnoDB: 10.4.13 started; log sequence number 8312011765; transaction id 20582626
  nextcloud_db_1     | 2020-05-15  1:28:52 0 [Note] InnoDB: !!! innodb_force_recovery is set to 1 !!!
  nextcloud_db_1     | 2020-05-15  1:28:52 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
  nextcloud_db_1     | 2020-05-15  1:28:52 0 [Note] Plugin 'FEEDBACK' is disabled.
  nextcloud_db_1     | mysqld(my_print_stacktrace+0x2e)[0x561046981e8e]
  nextcloud_db_1     | mysqld(handle_fatal_signal+0x515)[0x5610463fd915]
  nextcloud_db_1     | /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7f0148c84890]
  nextcloud_db_1     | /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7f01475b8e97]
  nextcloud_db_1     | /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7f01475ba801]
  nextcloud_db_1     | mysqld(+0x57c53c)[0x56104610a53c]
  nextcloud_db_1     | mysqld(+0xb35506)[0x5610466c3506]
  nextcloud_db_1     | mysqld(+0xb35d5d)[0x5610466c3d5d]
  nextcloud_db_1     | mysqld(+0xb39a55)[0x5610466c7a55]
  nextcloud_db_1     | mysqld(+0xb1c16e)[0x5610466aa16e]
  nextcloud_db_1     | /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7f0148c796db]
  nextcloud_db_1     | /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7f014769b88f]
  nextcloud_db_1     |
  nextcloud_db_1     | Trying to get some variables.
  nextcloud_db_1     | Some pointers may be invalid and cause the dump to abort.
  nextcloud_db_1     | Query (0x0):
  nextcloud_db_1     | Connection ID (thread ID): 2
  nextcloud_db_1     | Status: NOT_KILLED
  nextcloud_db_1     |
  nextcloud_db_1     | Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
  nextcloud_db_1     |
  nextcloud_db_1     | The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
  nextcloud_db_1     | information that should help you find out what is causing the crash.
  nextcloud_db_1     | Writing a core file...
  nextcloud_db_1     | Working directory at /var/lib/mysql
  nextcloud_db_1     | Resource Limits:
  nextcloud_db_1     | Limit                     Soft Limit           Hard Limit           Units
  nextcloud_db_1     | Max cpu time              unlimited            unlimited            seconds
  nextcloud_db_1     | Max file size             unlimited            unlimited            bytes
  nextcloud_db_1     | Max data size             unlimited            unlimited            bytes
  nextcloud_db_1     | Max stack size            8388608              unlimited            bytes
  nextcloud_db_1     | Max core file size        unlimited            unlimited            bytes
  nextcloud_db_1     | Max resident set          unlimited            unlimited            bytes
  nextcloud_db_1     | Max processes             unlimited            unlimited            processes
  nextcloud_db_1     | Max open files            1048576              1048576              files
  nextcloud_db_1     | Max locked memory         16777216             16777216             bytes
  nextcloud_db_1     | Max address space         unlimited            unlimited            bytes
  nextcloud_db_1     | Max file locks            unlimited            unlimited            locks
  nextcloud_db_1     | Max pending signals       1545098              1545098              signals
  nextcloud_db_1     | Max msgqueue size         819200               819200               bytes
  nextcloud_db_1     | Max nice priority         0                    0
  nextcloud_db_1     | Max realtime priority     0                    0
  nextcloud_db_1     | Max realtime timeout      unlimited            unlimited            us
  nextcloud_db_1     | Core pattern: co...
  nextcloud_db_1     | Fatal signal 11 while backtracing
  This could be because you hit a bug. It is also possible that this binary
  or one of the libraries it was linked against is corrupt, improperly built,
  or misconfigured. This error can also be caused by malfunctioning hardware.
  
  To report this bug, see https://mariadb.com/kb/en/reporting-bugs
  
  We will try our best to scrape up some info that will hopefully help
  diagnose the problem, but since we have already crashed,
  something is definitely wrong and this may fail.
  
  Server version: 10.4.13-MariaDB-1:10.4.13+maria~bionic
  key_buffer_size=134217728
  read_buffer_size=2097152
  max_used_connections=0
  max_threads=102
  thread_count=4
  It is possible that mysqld could use up to
  key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 760255 K  bytes of memory
  Hope that's ok; if not, decrease some variables in the equation.
  
  Thread pointer: 0x7fa650000c08
  Attempting backtrace. You can use the following information to find out
  where mysqld died. If you see no messages after this, something went
  terribly wrong...
  stack_bottom = 0x7fa674ff0c08 thread_stack 0x49000
  2020-05-15  0:01:32 0 [Note] InnoDB: 10.4.13 started; log sequence number 8312011765; transaction id 20582626
  2020-05-15  0:01:32 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
  mysqld(my_print_stacktrace+0x2e)[0x55e945e98e8e]
  mysqld(handle_fatal_signal+0x515)[0x55e945914915]
  2020-05-15  0:01:32 0 [Note] Plugin 'FEEDBACK' is disabled.
  /lib/x86_64-linux-gnu/libpthread.so.0(+0x12890)[0x7fa6aeddc890]
  /lib/x86_64-linux-gnu/libc.so.6(gsignal+0xc7)[0x7fa6ad710e97]
  /lib/x86_64-linux-gnu/libc.so.6(abort+0x141)[0x7fa6ad712801]
  mysqld(+0x57c53c)[0x55e94562153c]
  mysqld(+0xb35506)[0x55e945bda506]
  mysqld(+0xb35d5d)[0x55e945bdad5d]
  mysqld(+0xb39a55)[0x55e945bdea55]
  mysqld(+0xb1c16e)[0x55e945bc116e]
  /lib/x86_64-linux-gnu/libpthread.so.0(+0x76db)[0x7fa6aedd16db]
  /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f)[0x7fa6ad7f388f]
  
  Trying to get some variables.
  Some pointers may be invalid and cause the dump to abort.
  Query (0x0):
  Connection ID (thread ID): 4
  Status: NOT_KILLED
  
  Optimizer switch: index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,derived_merge=on,derived_with_keys=on,firstmatch=on,loosescan=on,materialization=on,in_to_exists=on,semijoin=on,partial_match_rowid_merge=on,partial_match_table_scan=on,subquery_cache=on,mrr=off,mrr_cost_based=off,mrr_sort_keys=off,outer_join_with_cache=on,semijoin_with_cache=on,join_cache_incremental=on,join_cache_hashed=on,join_cache_bka=on,optimize_join_buffer_size=on,table_elimination=on,extended_keys=on,exists_to_in=on,orderby_uses_equalities=on,condition_pushdown_for_derived=on,split_materialized=on,condition_pushdown_for_subquery=on,rowid_filter=on,condition_pushdown_from_having=on
  
  The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
  information that should help you find out what is causing the crash.
  Writing a core file...
  Working directory at /var/lib/mysql
  Resource Limits:
  Limit                     Soft Limit           Hard Limit           Units
  Max cpu time              unlimited            unlimited            seconds
  Max file size             unlimited            unlimited            bytes
  Max data size             unlimited            unlimited            bytes
  Max stack size            8388608              unlimited            bytes
  Max core file size        unlimited            unlimited            bytes
  Max resident set          unlimited            unlimited            bytes
  Max processes             unlimited            unlimited            processes
  Max open files            1048576              1048576              files
  Max locked memory         16777216             16777216             bytes
  Max address space         unlimited            unlimited            bytes
  Max file locks            unlimited            unlimited            locks
  Max pending signals       1545098              1545098              signals
  Max msgqueue size         819200               819200               bytes
  Max nice priority         0                    0
  Max realtime priority     0                    0
  Max realtime timeout      unlimited            unlimited            us
  Core pattern: co...
  Fatal signal 11 while backtracing
  </code></pre>
  
  <p>This is in the Docker logs. this sounds way too complicated for me and I have no clue.</p>
  
  <p>I have not made any changes to the setup. As I said I am using docker for almost 2 years, with some changes made to the docker-compose.yml file over the time. But this issue just appeared and I haven't changed anything for about 3 or 4 months.</p>
  
  <p>Is it possible, that it has something to do with a big database? I am using Nextcloud News and do not purge any items because I use them for machine learning data processing. So the database might be a little big.</p>
  
  <p>After amysqldump the oc_news_items table is about 170MB in size and the last automated dump (this morning) doesn't seem corrupted or something like this, but it has suspiciously 99991 rows. So maybe there might be a problem with over 99999 rows of nextcloud news items in the db? Or maybe only coincidence.</p>