<p dir="auto">Configurations in <code class="notranslate">server.yaml</code>:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="mode:
  type: Memory

rules:
  - !AUTHORITY
    users:
      - root@%:root
      - sharding@:sharding
    provider:
      type: ALL_PRIVILEGES_PERMITTED
  - !TRANSACTION
    defaultType: XA
    providerType: Atomikos
  - !SQL_PARSER
    sqlCommentParseEnabled: true
    sqlStatementCache:
      initialCapacity: 2000
      maximumSize: 65535
      concurrencyLevel: 4
    parseTreeCache:
      initialCapacity: 128
      maximumSize: 1024
      concurrencyLevel: 4

props:
  max-connections-size-per-query: 1
  kernel-executor-size: 16  # Infinite by default.
  proxy-frontend-flush-threshold: 128  # The default value is 128.
  proxy-opentracing-enabled: false
  proxy-hint-enabled: false
  sql-show: false
  check-table-metadata-enabled: false
  show-process-list-enabled: false
    # Proxy backend query fetch size. A larger value may increase the memory usage of ShardingSphere Proxy.
    # The default value is -1, which means set the minimum value for different JDBC drivers.
  proxy-backend-query-fetch-size: -1
  check-duplicate-table-enabled: false
  proxy-frontend-executor-size: 0 # Proxy frontend executor size. The default value is 0, which means let Netty decide.
    # Available options of proxy backend executor suitable: OLAP(default), OLTP. The OLTP option may reduce time cost of writing packets to client, but it may increase the latency of SQL execution
    # and block other clients if client connections are more than `proxy-frontend-executor-size`, especially executing slow SQL.
  proxy-backend-executor-suitable: OLAP
  proxy-frontend-max-connections: 0 # Less than or equal to 0 means no limitation.
  sql-federation-enabled: false
    # Available proxy backend driver type: JDBC (default), ExperimentalVertx
  proxy-backend-driver-type: JDBC"><pre class="notranslate"><code class="notranslate">mode:
  type: Memory

rules:
  - !AUTHORITY
    users:
      - root@%:root
      - sharding@:sharding
    provider:
      type: ALL_PRIVILEGES_PERMITTED
  - !TRANSACTION
    defaultType: XA
    providerType: Atomikos
  - !SQL_PARSER
    sqlCommentParseEnabled: true
    sqlStatementCache:
      initialCapacity: 2000
      maximumSize: 65535
      concurrencyLevel: 4
    parseTreeCache:
      initialCapacity: 128
      maximumSize: 1024
      concurrencyLevel: 4

props:
  max-connections-size-per-query: 1
  kernel-executor-size: 16  # Infinite by default.
  proxy-frontend-flush-threshold: 128  # The default value is 128.
  proxy-opentracing-enabled: false
  proxy-hint-enabled: false
  sql-show: false
  check-table-metadata-enabled: false
  show-process-list-enabled: false
    # Proxy backend query fetch size. A larger value may increase the memory usage of ShardingSphere Proxy.
    # The default value is -1, which means set the minimum value for different JDBC drivers.
  proxy-backend-query-fetch-size: -1
  check-duplicate-table-enabled: false
  proxy-frontend-executor-size: 0 # Proxy frontend executor size. The default value is 0, which means let Netty decide.
    # Available options of proxy backend executor suitable: OLAP(default), OLTP. The OLTP option may reduce time cost of writing packets to client, but it may increase the latency of SQL execution
    # and block other clients if client connections are more than `proxy-frontend-executor-size`, especially executing slow SQL.
  proxy-backend-executor-suitable: OLAP
  proxy-frontend-max-connections: 0 # Less than or equal to 0 means no limitation.
  sql-federation-enabled: false
    # Available proxy backend driver type: JDBC (default), ExperimentalVertx
  proxy-backend-driver-type: JDBC
</code></pre></div>
<p dir="auto">Exception log:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="Exception in thread &quot;main&quot; org.apache.shardingsphere.spi.exception.ServiceProviderNotFoundException: No implementation class load from SPI `org.apache.shardingsphere.infra.yaml.config.swapper.mode.PersistRepositoryConfigurationYamlSwapper` with type `Memory`.
	at org.apache.shardingsphere.spi.typed.TypedSPIRegistry.getRegisteredService(TypedSPIRegistry.java:76)
	at org.apache.shardingsphere.infra.yaml.config.swapper.mode.PersistRepositoryConfigurationYamlSwapperEngine.swapToObject(PersistRepositoryConfigurationYamlSwapperEngine.java:57)
	at org.apache.shardingsphere.infra.yaml.config.swapper.mode.ModeConfigurationYamlSwapper.swapToObject(ModeConfigurationYamlSwapper.java:44)
	at org.apache.shardingsphere.proxy.initializer.BootstrapInitializer.init(BootstrapInitializer.java:70)
	at org.apache.shardingsphere.proxy.Bootstrap.main(Bootstrap.java:47)"><pre class="notranslate"><code class="notranslate">Exception in thread "main" org.apache.shardingsphere.spi.exception.ServiceProviderNotFoundException: No implementation class load from SPI `org.apache.shardingsphere.infra.yaml.config.swapper.mode.PersistRepositoryConfigurationYamlSwapper` with type `Memory`.
	at org.apache.shardingsphere.spi.typed.TypedSPIRegistry.getRegisteredService(TypedSPIRegistry.java:76)
	at org.apache.shardingsphere.infra.yaml.config.swapper.mode.PersistRepositoryConfigurationYamlSwapperEngine.swapToObject(PersistRepositoryConfigurationYamlSwapperEngine.java:57)
	at org.apache.shardingsphere.infra.yaml.config.swapper.mode.ModeConfigurationYamlSwapper.swapToObject(ModeConfigurationYamlSwapper.java:44)
	at org.apache.shardingsphere.proxy.initializer.BootstrapInitializer.init(BootstrapInitializer.java:70)
	at org.apache.shardingsphere.proxy.Bootstrap.main(Bootstrap.java:47)
</code></pre></div>
<p dir="auto">MetadataCenter Design5.x is a part of Orchestration 5.x Design(<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="572604912" data-permission-text="Title is private" data-url="https://github.com/apache/shardingsphere/issues/4515" data-hovercard-type="issue" data-hovercard-url="/apache/shardingsphere/issues/4515/hovercard" href="https://github.com/apache/shardingsphere/issues/4515">#4515</a>).</p>
<h2 dir="auto">Metadata Center Design</h2>
<p dir="auto">The purpose of this document is to illustrate the metadata center design of the Apache ShardingSphere governance module.</p>
<p dir="auto">[TOC]</p>
<h3 dir="auto">1. Definition</h3>
<p dir="auto">The word 'metadata' in this article is the metadata of the data source used by Sharding-JDBC / Sharding-Proxy. These metadata are the core data objects to ensure the correct operations of each component of ShardingSphere. Currently, they are scattered in each part of the system. They should be reorganized and managed in a unified manner using a new component such like 'metadata center', and coordinate changes when metadatas changes.</p>
<h3 dir="auto">2. Types</h3>
<p dir="auto">The current metadata object models are mainly defined in:</p>
<blockquote>
<p dir="auto">org.apache.shardingsphere.sql.parser.binder</p>
<p dir="auto">├─column<br>
│      ColumnMetaData.java<br>
│      ColumnMetaDataLoader.java<br>
├─index<br>
│      IndexMetaData.java<br>
│      IndexMetaDataLoader.java<br>
├─schema<br>
│      SchemaMetaData.java<br>
│      SchemaMetaDataLoader.java<br>
└─table<br>
TableMetaData.java<br>
TableMetaDataLoader.java</p>
<p dir="auto">Hierarchical relationship is <code class="notranslate">schema &gt; table &gt; column + index</code></p>
<p dir="auto">At the same time, the scaling module also has a subset of models and loaders that need to be merged (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="584974626" data-permission-text="Title is private" data-url="https://github.com/apache/shardingsphere/issues/4866" data-hovercard-type="issue" data-hovercard-url="/apache/shardingsphere/issues/4866/hovercard" href="https://github.com/apache/shardingsphere/issues/4866">#4866</a>).</p>
</blockquote>
<p dir="auto">One issue to further discuss is:</p>
<blockquote>
<p dir="auto">Question 1: Should sharding rules and so on be put in the metadata center uniformly?</p>
<p dir="auto">It is recommended to only process datasource metadatas right now, and rule datas should still be placed in config center. See if we want to adjust them later.</p>
</blockquote>
<h3 dir="auto">3. Current Metadata Loading</h3>
<p dir="auto">The unified entry for loading metadatas in：</p>
<blockquote>
<p dir="auto">org.apache.shardingsphere.sql.parser.binder.metadata.schema.SchemaMetaDataLoader</p>
</blockquote>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/807508/77133013-76df2780-6a9c-11ea-9a98-54d69bd43599.png"><img src="https://user-images.githubusercontent.com/807508/77133013-76df2780-6a9c-11ea-9a98-54d69bd43599.png" alt="image-20200319190724018" style="max-width: 100%;"></a></p>
<p dir="auto">Metadatas are loaded in three places:</p>
<p dir="auto">1、Sharding-JDBC</p>
<p dir="auto">2、Sharding-Proxy</p>
<blockquote>
<p dir="auto">Bootstrap.startWithRegistryCenter-&gt;LogicSchemas.init/initSchemas-&gt;LogicSchemas.initSchemas(for)</p>
<p dir="auto">-&gt;LogicSchemaFactory.newInstance-&gt;XXSchema-&gt;XXSchema.createMetaData/loadSchemaMetaData</p>
<p dir="auto">-&gt;SchemaMetaDataLoader.load(dataSource, maxConnectionsSizePerQuery)</p>
</blockquote>
<p dir="auto">Each XXSchema class implements guava's @ Subscribe's renew method, which can execute the corresponding rules to refresh them(rules, not metadatas) when an event is received.</p>
<p dir="auto">ShardingSchema and MasterSlaveSchema can additionally support the disableEvent to disable the data source. (This section have a lot codes those can be optimized.)</p>
<p dir="auto">sharding table metadatas loading:</p>
<blockquote>
<p dir="auto">org.apache.shardingsphere.core.metadata.ShardingMetaDataLoader</p>
</blockquote>
<p dir="auto">Load the logic part and the default part separately, and then call SchemaMetaDataLoader to load according to the hierarchy.</p>
<p dir="auto">3、Sharding-scaling</p>
<p dir="auto">There are an independent TableMetaDataLoader/ColumnMetaDataLoader here, used to load its own TableMetaData/ColumnMetaData.</p>
<h3 dir="auto">4. Metadata Usage</h3>
<p dir="auto"><strong>Route Module</strong></p>
<ul dir="auto">
<li>If it is SelectStatementContext, where condition exists, it will participate in obtaining ShadingConditions</li>
<li>If it is DDLStatement, DCLStatement, it will get the table data from it.</li>
</ul>
<p dir="auto"><strong>Rewrite Module</strong></p>
<ul dir="auto">
<li>Determine if the column exists in metadata.
<ul dir="auto">
<li>EncryptPredicateParameterRewriter</li>
<li>EncryptPredicateColumnTokenGenerator</li>
<li>EncryptPredicateRightValueTokenGenerator</li>
</ul>
</li>
</ul>
<p dir="auto"><strong>Execute Module</strong></p>
<ul dir="auto">
<li>If it is the following types, it will refresh metaData.</li>
</ul>
<div class="highlight highlight-source-java notranslate position-relative overflow-auto" dir="auto" data-snippet-clipboard-copy-content="if (sqlStatementContext instanceof CreateTableStatementContext) {
           refreshTableMetaData(runtimeContext, ((CreateTableStatementContext) sqlStatementContext).getSqlStatement());
       } else if (sqlStatementContext instanceof AlterTableStatementContext) {
           refreshTableMetaData(runtimeContext, ((AlterTableStatementContext) sqlStatementContext).getSqlStatement());
       } else if (sqlStatementContext instanceof DropTableStatementContext) {
           refreshTableMetaData(runtimeContext, ((DropTableStatementContext) sqlStatementContext).getSqlStatement());
       } else if (sqlStatementContext instanceof CreateIndexStatementContext) {
           refreshTableMetaData(runtimeContext, ((CreateIndexStatementContext) sqlStatementContext).getSqlStatement());
       } else if (sqlStatementContext instanceof DropIndexStatementContext) {
           refreshTableMetaData(runtimeContext, ((DropIndexStatementContext) sqlStatementContext).getSqlStatement());
       }"><pre class="notranslate"><span class="pl-k">if</span> (<span class="pl-s1">sqlStatementContext</span> <span class="pl-k">instanceof</span> <span class="pl-smi">CreateTableStatementContext</span>) {
           <span class="pl-en">refreshTableMetaData</span>(<span class="pl-s1">runtimeContext</span>, ((<span class="pl-smi">CreateTableStatementContext</span>) <span class="pl-s1">sqlStatementContext</span>).<span class="pl-en">getSqlStatement</span>());
       } <span class="pl-k">else</span> <span class="pl-k">if</span> (<span class="pl-s1">sqlStatementContext</span> <span class="pl-k">instanceof</span> <span class="pl-smi">AlterTableStatementContext</span>) {
           <span class="pl-en">refreshTableMetaData</span>(<span class="pl-s1">runtimeContext</span>, ((<span class="pl-smi">AlterTableStatementContext</span>) <span class="pl-s1">sqlStatementContext</span>).<span class="pl-en">getSqlStatement</span>());
       } <span class="pl-k">else</span> <span class="pl-k">if</span> (<span class="pl-s1">sqlStatementContext</span> <span class="pl-k">instanceof</span> <span class="pl-smi">DropTableStatementContext</span>) {
           <span class="pl-en">refreshTableMetaData</span>(<span class="pl-s1">runtimeContext</span>, ((<span class="pl-smi">DropTableStatementContext</span>) <span class="pl-s1">sqlStatementContext</span>).<span class="pl-en">getSqlStatement</span>());
       } <span class="pl-k">else</span> <span class="pl-k">if</span> (<span class="pl-s1">sqlStatementContext</span> <span class="pl-k">instanceof</span> <span class="pl-smi">CreateIndexStatementContext</span>) {
           <span class="pl-en">refreshTableMetaData</span>(<span class="pl-s1">runtimeContext</span>, ((<span class="pl-smi">CreateIndexStatementContext</span>) <span class="pl-s1">sqlStatementContext</span>).<span class="pl-en">getSqlStatement</span>());
       } <span class="pl-k">else</span> <span class="pl-k">if</span> (<span class="pl-s1">sqlStatementContext</span> <span class="pl-k">instanceof</span> <span class="pl-smi">DropIndexStatementContext</span>) {
           <span class="pl-en">refreshTableMetaData</span>(<span class="pl-s1">runtimeContext</span>, ((<span class="pl-smi">DropIndexStatementContext</span>) <span class="pl-s1">sqlStatementContext</span>).<span class="pl-en">getSqlStatement</span>());
       }</pre></div>
<p dir="auto"><strong>Merge Module</strong></p>
<ul dir="auto">
<li>Not used, use SQL directly to return MetaData from ResultSet</li>
</ul>
<h3 dir="auto">5. Metadata Changes</h3>
<p dir="auto">Currently, metadatas are loaded and managed by each started sharding-JDBC or proxy node.</p>
<p dir="auto">If a DDL is executed through one node, the following refresh method is directly called to refresh the metadata of the current node.</p>
<p dir="auto">JDBC:</p>
<blockquote>
<p dir="auto">org.apache.shardingsphere.shardingjdbc.executor.AbstractStatementExecutor.refreshMetaDataIfNeeded</p>
</blockquote>
<p dir="auto">Proxy:</p>
<blockquote>
<p dir="auto">org.apache.shardingsphere.shardingproxy.backend.schema.impl.ShardingSchema.refreshTableMetaData</p>
</blockquote>
<p dir="auto">A tip: The codes in these two places are also heavily duplicated.</p>
<h3 dir="auto">6. Metadata Center Design</h3>
<p dir="auto">From the above sections, we know that there are some points that need improvement:</p>
<p dir="auto">Improvement 1:</p>
<p dir="auto">If there are multiple nodes started at the same time, a large number of metadatas will be repeatedly loaded from the DB, and maybe lead to a performance issue.</p>
<p dir="auto">Improvement 2:</p>
<p dir="auto">Once a node executed a DDL, other nodes don't know that the metadatas had changed, and lead to a consistency issue.</p>
<p dir="auto">We hope to solve these two problems via metadata center design.</p>
<h4 dir="auto">6.1 Definition</h4>
<p dir="auto">The metadata center is a mechanism for unified loading of metadatas, change notifications, and data synchronization through the unified management of all metadatas.</p>
<h4 dir="auto">6.2 Feature</h4>
<p dir="auto">It is planned to sort out the existing metadata loading and all usage scenarios, uniformly manage the metadata to be persisted to the CenterRepository since the metadatas are loaded for the first time, and subsequent nodes will start to obtain metadatas from the metadata center (Improvement 1). When a node performs a DDL operation, the node's metadatas is refreshed and synchronized to CenterRepository, and then all other nodes are notified to synchronize new datas from the metadata center (Improvement 2).</p>
<h4 dir="auto">6.3 API</h4>
<p dir="auto">On the basis of clearing up, the existing metadata loading logic will be restructured, and some loader code will be migrated to the new metadata center module:</p>
<blockquote>
<p dir="auto">sharding-orchestration-center-metadata</p>
</blockquote>
<p dir="auto">And the following new API:：</p>
<ol dir="auto">
<li>MetadataCenter</li>
<li>MetadataLoader</li>
<li>MetadataNode</li>
<li>MetadataListener</li>
<li>MetadataChangeEvent</li>
</ol>
<h4 dir="auto">6.4 Data Structure</h4>
<p dir="auto">The current metadata memory structure is as follows:</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/807508/77133014-78105480-6a9c-11ea-972d-e4dad9f7bf08.png"><img src="https://user-images.githubusercontent.com/807508/77133014-78105480-6a9c-11ea-972d-e4dad9f7bf08.png" alt="image-20200319194148562" style="max-width: 100%;"></a></p>
<p dir="auto">There are two style available metadata structure configurations.</p>
<p dir="auto">CenterRepository metadata structure configuration style one:</p>
<blockquote>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="├─orchestration-namespace
├─orchestration-name
│  ├─metadata
│  │  ├─schema01
│  │  │  ├─table1
│  │  │  │  ├─columns
│  │  │  │  └─indexs
│  │  │  ├─table2
│  │  │  │  ├─columns
│  │  │  │  └─indexs
│  │  ├─schema02
│  │  │  ├─table3
│  │  │  │  ├─columns
│  │  │  │  └─indexs
│  │  │  ├─table4
│  │  │  │  ├─columns
│  │  │  │  └─indexs"><pre class="notranslate"><code class="notranslate">├─orchestration-namespace
├─orchestration-name
│  ├─metadata
│  │  ├─schema01
│  │  │  ├─table1
│  │  │  │  ├─columns
│  │  │  │  └─indexs
│  │  │  ├─table2
│  │  │  │  ├─columns
│  │  │  │  └─indexs
│  │  ├─schema02
│  │  │  ├─table3
│  │  │  │  ├─columns
│  │  │  │  └─indexs
│  │  │  ├─table4
│  │  │  │  ├─columns
│  │  │  │  └─indexs
</code></pre></div>
</blockquote>
<p dir="auto">CenterRepository metadata structure configuration style two:</p>
<blockquote>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="├─orchestration-namespace
├─orchestration-name
│  ├─metadata
│  │  ├─schemas
│  │  │  ├─ [json/yaml text contents]"><pre class="notranslate"><code class="notranslate">├─orchestration-namespace
├─orchestration-name
│  ├─metadata
│  │  ├─schemas
│  │  │  ├─ [json/yaml text contents]
</code></pre></div>
</blockquote>
<p dir="auto">Style one: It is intuitive and fine-grained.</p>
<p dir="auto">Style two: It is simpler and easier to manage.</p>
<p dir="auto">(It is recommended to use style 2.)</p>
<p dir="auto">One question:</p>
<blockquote>
<p dir="auto">Question 2: Will the logic table be displayed here?</p>
<p dir="auto">I think the answer is no.</p>
<p dir="auto">Only the actual real tables metadatas here, logic table is not a part of metadata.</p>
</blockquote>
<h4 dir="auto">6.5 Loading</h4>
<p dir="auto">Current loading process:</p>
<blockquote>
<p dir="auto">Load sharding tables first, then load default tables, and check whether the metadatas of all tables is consistent according to the check.metadata.enable parameter.</p>
</blockquote>
<p dir="auto">New loading process:</p>
<blockquote>
<p dir="auto">After loading metadata, write to CenterRepository, then trigger global notification.</p>
</blockquote>
<blockquote>
<p dir="auto">Question 3: Are the metadatas of this step written to CenterRepository after each group or datasource is loaded, or after all metadatas loaded and written them once (related to triggering one or more events).</p>
<p dir="auto">Consider to implement a full synchronizing at first, and then see if it can be persisted and notified in batches, based on loading speed.</p>
</blockquote>
<h4 dir="auto">6.6 Synchronization</h4>
<p dir="auto">After DDL is executed in one node, the metadatas of this node is refreshed through the Event mechanism(instead of local method calling),  synchronized to CenterRepository, and then a global change notification is triggered to other nodes.</p>
<blockquote>
<p dir="auto">Question 4: Do we need to hold the loading process of other nodes to prevent concurrency.</p>
<p dir="auto">Do not consider concurrency at first, implement the load and notify function, and finally solve this problem, maybe involving distributed locks.</p>
</blockquote>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/807508/77132931-2ff13200-6a9c-11ea-82a4-dbb10195b1c8.PNG"><img src="https://user-images.githubusercontent.com/807508/77132931-2ff13200-6a9c-11ea-82a4-dbb10195b1c8.PNG" alt="幻灯片4" style="max-width: 100%;"></a></p>
<h4 dir="auto">6.7 Notification</h4>
<p dir="auto">Through the Event mechanism, other nodes are notified and then update metadatas from CenterRepository.</p>
<p dir="auto"><a target="_blank" rel="noopener noreferrer nofollow" href="https://user-images.githubusercontent.com/807508/77132933-31225f00-6a9c-11ea-9e97-12593880da3e.PNG"><img src="https://user-images.githubusercontent.com/807508/77132933-31225f00-6a9c-11ea-9e97-12593880da3e.PNG" alt="幻灯片5" style="max-width: 100%;"></a></p>
<h3 dir="auto">7. Task list</h3>
<ul dir="auto">
<li>7.1 metadata carding</li>
<li>7.2 metadata refactoring (code abstraction and cleanup)</li>
<li>7.3 Add center-metadata related project structure</li>
<li>7.4 Implementing metadata persistence</li>
<li>7.5 Migrating some loader codes</li>
<li>7.6 Implementing the event notification mechanism</li>
<li>7.7 Implementing Global Synchronization</li>
<li>7.8 Optimize loading and usage</li>
<li>7.9 Improve Unittests</li>
<li>7.10 Implementing examples</li>
<li>7.11 Implementing documents</li>
</ul>