abstract: Smart contract transactions demonstrate issues of performance and correctness that application programmers must work around. Although the blockchain consensus mechanism approaches ACID compliance, use cases that rely on frequent state changes are impractical due to the block publishing interval of $O(10^1)$ seconds. The effective isolation level is Read-Committed, only revealing state transitions at the end of the block interval. Values read may be stale and not match program order, causing many transactions to fail when a block is committed. This paper perceives the blockchain as a transactional data structure, using this analogy in the development of a new algorithm, *Hash-Mark-Set* (HMS), that improves transaction throughput by providing a Read-Uncommitted view of state variables. HMS creates a directed acyclic graph (DAG) from the pending transaction pool. The transaction order derived from the DAG is used to provide a Read-Uncommitted view of the data for new transactions, which enter the DAG as they are received. An implementation of HMS is provided, interoperable with Ethereum and ready for use in smart contracts. Over a wide range of transaction mixes, HMS is demonstrated to improve throughput. A side product of the implementation is a new technique, *Runtime Argument Augmentation* (RAA), that allows smart contracts to communicate with external data services before submitting a transaction. RAA has use cases beyond HMS and can serve as a lightweight replacement for blockchain oracles.
author:
title: Read-Uncommitted Transactions for
  Smart Contract Performance [^1]

# Introduction

Blockchains rely on a consensus mechanism to agree upon the sequencing of client transactions in a block, committing transactions as a group to the distributed ledger. *Smart contracts* are the interface to process client requests and send transactions to the blockchain peer network. A block of transactions must be validated to ensure that the sequence is consistent. All peers on the network perform the validation step by re-executing the transactions within the block and checking that the initial and final states match, introducing latency.

Latency resulting from the publishing and validation of a block decreases the success rate for the transactions in the block due to the possibility of stale reads of state variables, also known as *storage variables*. Changes to storage variables are only visible after they are committed to a published block. This isolation level of intra-block transactions is called Read-Committed. Transactional reads of storage variables can become outdated while waiting on the validation step since other published blocks may update the storage variables, leading to transaction failure. Additionally, since read operations can only access the published storage variable value, intra-block changes can also cause a transaction to fail due to a stale read.

A smart contract transaction is a concurrent method, often with semantic dependencies. The way block publishing commits multiple smart contract transactions simultaneously is analogous to the way a transactional data structure  commits multiple concurrent methods in what appears to be a single atomic step. Using this analogy, the blockchain is a blind transactional data structure that selects and sequences concurrent method calls without regard for their semantics, causing many to fail due to the restrictive Read-Committed isolation level. An ideal algorithm for blockchain transactions would consider transaction semantics and include all related transactions as a series in a block commit.

In this paper, we present *Hash-Mark-Set* (HMS), an algorithm that increases the throughput of smart contract transactions by providing a Read-Uncommitted view of the storage variables. HMS organizes the pool of pending transactions (TxPool) on specific storage variables in a directed acyclic graph (DAG) that establishes an ordering among the transactions and enables an uncommitted view of the storage variables to be retrieved. HMS reduces transactional failures because the Read-Uncommitted view increases the likelihood that a transaction has consistent inputs. Latency is also reduced because concurrent actors will no longer need to wait until a block is committed to see a change in storage variables that is likely to be committed in the next block or two. We integrate HMS into smart contracts through *Runtime Argument Augmentation* (RAA), our proposed technique that allows smart contracts to communicate with external data services prior to sending a transaction.

An interoperable implementation of the solution is provided and demonstrated on the Ethereum blockchain. State throughput, defined in Section  as throughput of *successful* blockchain transactions, increases by a factor of five across a range of transaction mixes. By adding the cooperation of blockchain miners, state throughput increases by an order of magnitude.

We make the following contributions:

1.  Present *Hash-Mark-Set* (HMS), an algorithm organizing a pool of pending transactions that share state variables.

2.  Introduce *Runtime Argument Augmentation* (RAA), a novel technique for smart contracts to communicate with external data services.

3.  Demonstrate improvements to latency and state throughput when HMS provides a Read-Uncommitted view to smart contract clients.

4.  Provide an interoperable implementation for Ethereum:
    https://github.com/area67/sereth

# Background

A few issues specific to the blockchain are discussed in this section.

## Blockchain Transactions

A *blockchain* is a distributed ledger maintained by one or more peers following a communication protocol and agreeing on a consensus mechanism. The ledger is written in chunks called *blocks* that are linked in a practically unforgeable cryptographic *chain*, replicated among many peers to avoid dependence upon a single entity. State variables that are recorded on the blockchain are called *storage* variables in Ethereum. Bitcoin was the first blockchain , providing transactions on a store and exchange of value, *i.e.* a currency. A transaction is a concurrent method call that if successful, changes the state of the ledger. A block may contain from zero up to a finite number of transactions, typically $O(10^1)$ to $O(10^3)$. Well known database transaction models such as ACID and BASE are applicable to the blockchain, motivating our use of isolation levels Read-Uncommitted and Read-Committed.

## Concurrent Smart Contracts

Going beyond exchange of value, later blockchains added the ability to program arbitrary instruction sequences in a transaction. Their programming languages are Turing complete and their programs are called *smart contracts* . Concurrency is framed in the words of Sergey and Hobor, "Accounts using smart contracts in a blockchain are like threads using concurrent objects in shared memory" . Herlihy endorsed this line of reasoning in a keynote address , exhorting concurrency researchers to "civilize" the blockchain.

Invoking a smart contract function that *may* change ledger state creates a transaction and sends it to the network. It should be noted that some smart contract functions, designated *pure* or *view*, cannot change ledger state and they do not create transactions. The unprocessed transaction pool of pending transactions is referred to as the *TxPool*. The network of peers is a concurrent system and it follows that its incoming transactions, found in the TxPool, is a concurrent history. The *real time ordering* of a concurrent history is a total ordering over the transactions in a concurrent history such that transaction $T_1$ is ordered before transaction $T_2$ if $T_1$ is received by the TxPool before $T_2$.

## Miner Privilege

On Ethereum, Bitcoin and many other blockchains, the inclusion and sequencing of transactions in a block does not follow real time order, rather transactions are arranged in a total order that is arbitrary and subject to the same economic incentives that drive blockchain progress . This is called the *block order*. Special peers, called *miners* have the privilege of deciding what goes into a block and in what order. Each transaction is isolated and a miner generally has no way of knowing if one may depend upon another, so the rules for selecting transactions are flexible. Miners generally favor transactions with higher fees, but they may favor some peers, including themselves. They may use altruistic criteria such as including only small transactions or those from peers with low bandwidth.

The discretion given to miners in the protocol works as if the scheduler of a CPU could favor particular threads. Ethereum miners read the TxPool grouped by peer addresses (aka threads) with transactions ordered by a counter called a *nonce*. Miners may favor an address and include its transactions before another peer without regard for the real time order in which they were received. Miners may refuse to include any transactions sent from particular addresses. But a miner may not commit a transaction from a given address to a block out of nonce order. This means that blockchain transactions from the same address are executed in the order they are sent, while the order of transactions from different addresses is not defined. Since a blockchain transaction is a concurrent method, we can describe this behavior as being equivalent to *sequential consistency*, a correctness property such that history of methods is equivalent to a legal sequential history, and all methods take effect in program order .

The TxPool is shared by peers on the network, including miners. Intuitively, if communication were instantaneous, all peers would see the same TxPool, and the order in which the transactions were received would match their real time ordering, i.e. the order in which they were sent. Miner privilege would still allow the transactions to be placed in a block in an order different from the real time order. The outcome of our Read-Uncommmitted view of state is subject to network synchronization and miner privilege. Information about the TxPool is not available to the smart contract as it submits transactions.

## Block Publishing and Validation

Blocks of selected transactions are committed all at once in a super transaction called *block publishing*. Transactions are interpreted sequentially within a block according to the block order, using the previous ledger state (block) as the initial context. Changes to storage variables are not visible until they are committed to a block and the block is published. The changes to storage variables during the interval of block publication are called *intra-block* changes. Transactions within the block are affected by the intra-block changes, but post-publication transactions read the block final value of a storage variable from the previous block, none of the current changes. Once published there is no opportunity to re-order the concurrent methods. These values were read from the previous block, published *block interval* seconds ago. The block interval defines the latency.

Block publishing is effectively a read lock until the next block is committed. Dirty reads are not allowed. In database terms the isolation level of intra-block transactions is Read-Committed. To accept a published block every peer must perform *block validation*, the task of checking that the block is consistent with the state of the network. Transactions committed to a block must be consistent in that they must include the effects of all previous transactions. The process of peers redundantly validating transactions in a block is called transaction *replay*. Block publishing and validation takes a significant amount of time $O(10^1)$ to $O(10^2)$ seconds, creating latency.

Since only the final state of the block is published, intermediate states become invisible without a detailed replay of the transactions in the block, something that a typical smart contract cannot do. The loss of intermediate states during a block update is a consequence of the Read-Committed view of state variables. This low isolation level avoids blocking but may allow a great number of transactions to be rejected later as inconsistent. Transactions that seem valid when submitted are rejected because the values on which they are based were stale. The number of transactions that are rejected impacts state throughput. Where state changes are frequent and there are many transactions in the pool to be interleaved in a block, a large percentage of transactions fail. To say a transaction failed means that it would have violated the consistency of the sequential history of the block in which it is embedded. To keep the sequential history of the block consistent, the transaction is included in the block, but has no effect on the system state. In database terms the transaction was *rolled back*. A principle cause of failure is the high latency imposed on reading changes to persistent storage variables.

## Blockchain Oracles

A characteristic of the blockchain is that security concerns related to the adversarial distributed environment impose restrictions on information transfer. Unassisted, smart contracts operate in a bubble, allowed to view only public blockchain state variables via getter functions and not allowed to call any outside sources of information. The discussion in Section  about peers replaying blocks can explain this. Since all peers must replay and validate the block, they all must see the same state changes. If a contract is using an outside source of information, no matter how reliable, it may change with time or due to corrections or it may become unavailable. This would cause some peers to see a different state than others, and the block could not be validated. The problem can be solved with a smart contract that mediates a secure and verifiable connection to external data feeds . Such a service is also called a *blockchain oracle* , .

## Challenging Use Cases

Blockchain performance, measured in terms of transaction throughput and latency, is a limiting factor for many use cases . Latency and throughput are considered together in this paper because the Read-Committed latency of state variable limits the throughput of successful transactions. This ubiquitous blockchain latency has been dubbed, 'the long system freeze'' . Our example use case is a decentralized market to buy and sell assets, a core use case driving blockchain research and investment. This example also represents the general case of concurrent actors reading a time sensitive shared state variable.

Say that trading opens at a certain price, visible to all buyers. Orders are received on the network to be processed. To simplify, orders must be at the exact price, *i.e.* there are no limit or market orders. The price changes frequently and unpredictably due to market dynamics. If 100 orders are received at the published price near the start of a block interval and the price changes after the first order, then only one will be accepted. Blockchain correctness (safety, consistency) is preserved by the expedient of invalidating 99 of the 100 transactions in this example, clearly an inefficient mechanism.

Due to miner privilege, the first order submitted in time may not be the first included in the block. Progress of the system cannot be fair in any case because there is not enough information in the TxPool on which to base a real time order of the requests from different peers. Even with such information, miners are not bound to prevent starvation, quite the contrary they may cause it. Information is also hidden from the buyers querying the smart contract for the price. Block replay is not available within the smart contract. Unless it is separately analyzed, 98 of the 99 price changes are invisible to participants and valuable market information about intermediate price changes is lost. The arbitrary transaction priority combined with read latency also creates a vulnerability known as *blockchain frontrunning* .

# Methodology

This paper presents Hash-Mark-Set (HMS), an algorithm that overcomes the limitations of the Read-Committed isolation level by providing a Read-Uncommitted view of storage variables. The Read-Uncommitted view alleviates the problems in the example of Section . Clients can observe partial changes within the block prior to publishing, reducing the chance that a transaction will fail due to a stale read. The *Mark* in HMS also establishes a partial intra-block order that a cooperating miner can enforce. Such cooperation is reasonable given financial incentives that might be offered by decentralized asset exchanges.

HMS provides a Read-Uncommitted view by maintaining the transactions in a directed acyclic graph (DAG) that represents an ordering among the transactions in the unprocessed transaction pool, *TxPool*, and applying a topological sort to the longest branch to retrieve the value of an unpublished storage variable. To enable the Read-Uncommitted view to be accessible through smart contracts, we propose Runtime Argument Augmentation (RAA), our proposed technique that modifies the Ethereum Virtual Machine (EVM) interpreter to apply the HMS algorithm and access the value of an unpublished storage variable. The RAA technique is made available to users through our proposed smart contract *Sereth*.

To evaluate the performance benefits of our proposed methodology, we present a new metric, *state throughput*, which measures the throughput of successful transactions. State throughput disregards failed transactions in the throughput measurement, which provides a better representation of the rate at which state changes are made in comparison to raw throughput. In the following subsections, we define state throughput, provide the Sereth smart contract application programming interface, and explain HMS and RAA, the two innovations of this paper.

## State Throughput

Blockchains are different from databases in the following way: failed transactions are included in the persistent shared ledger. Because a block may include a large percentage of failed transactions, raw throughput of transactions per second is not an adequate measure of performance. In the example described in Section , raw throughput was 100 per interval, but 99 of 100 transactions fail. In a database these rolled back transactions would not count in throughput, but in a blockchain they are included in the block. A new metric, *state throughput*, $T_{state}$, is defined here as the product of the raw throughput and the ratio of transactions included in a block that successfully make state changes. State throughput divided by raw throughput yields the transaction efficiency $\eta$. $$\frac{T_{state}}{T_{raw}} = \eta$$

Transactions in the TxPool form a concurrent history, with a non-deterministic outcome. We observed that transaction failure can be reduced by obtaining a view of state that is more likely to be consistent at the moment the transaction is committed to a block. To maximize $\eta$, transactions are organized to provide a predictive view of state, ordering transactions such that the order closely matches the real time order in which the transactions were received.

## Sereth Smart Contract

Our implementation of HMS for Ethereum is called Sereth, a variation of Geth, the name of the standard client. Sereth is implemented as an interoperable Ethereum client that can be substituted for one or more peers in any standard Ethereum network, public or private. The Sereth smart contract shown in Listing  manages the price and accepts the $set$ and $buy$ transactions from addresses on the blockchain. The $mark$ and $get$ functions are read only. They do not create transactions but are used to return the intra-block state that will be used in $set$ and $buy$. This intra-block state view uses RAA to get the results of the HMS algorithm. The values are written into the function arguments using RAA and then returned to the calling address.

```

pragma solidity ^0.4.24;

contract Sereth {
...
// Mark, Set and Get are methods on state variables
// managed by the Hash-Mark-Set algorithm.

    function mark(bytes32[3] raa)
            private pure returns(bytes32) {
        return raa[1];
    }

    function set(bytes32[3] fpv) public {
        // If mark is valid, set new mark and value.
        if (keccak256(fpv[1]) == keccak256(p[1])) {
            nSet++;
            p[0] = bytes32(msg.sender);
            p[1] = keccak256(fpv[1], fpv[2]);
            p[2] = fpv[2];
        }
    }

    function get(bytes32[3] raa)
            public pure returns(bytes32) {
        return raa[2];
    }

// Function buy() demonstrates a dynamic pricing use case
// for the Hash-Mark-Set transactional data structure.

    function buy(bytes32[3] offer) public {
        // If mark and price match then buy() succeeds.
        if ((keccak256(offer[1]) == keccak256(p[1])) &&
            (keccak256(offer[2]) == keccak256(p[2]))) {
            nBuy++;
            p[0] = bytes32(msg.sender);
        }
    }
}
```

## Hash-Mark-Set

Hash-Mark-Set takes advantage of an underutilized communication channel among the peers on a blockchain, the transaction pool (TxPool). We created a smart contract, `Sereth.sol`, to manage the state variables. In Sereth, function arguments are formatted so they contain three key elements within the transaction, $address$, $mark$, and $value$. The $address$ field contains the address of the sender of the transaction. The $mark$ field contains a Keccak256 hash which solidifies a transactions place in a series of Sereth transactions. The $value$ field indicates how the sender would like to modify the state variable. Together, these elements are referred to as a transaction's $AMV$. To create a transaction using the Sereth contract, one must pass in three parameters: $flag$, $previous\_mark$, and $value$. These parameters are referred to as the $FPV$. The $FPV$ is easily visible as a string of bytes within the transactions $input$ field.

We define a transaction's $mark$ such that given $Txn_1$ which follows $Txn_0$, $Txn_1.mark = Keccak256(Txn_0.mark, Txn_1.val)$. This creates a sequentially consistent ordering between any number of transactions in what we call a $series$. To create a $series$, the $FPV$ of each transaction in the TxPool is extracted from their respective $Data$ fields. By matching the $previous\_mark$ of a transaction with the $mark$ of a different transaction, we can determine a strict order of all Sereth transactions in the current TxPool. This provides the smart contract with a *Read-Uncommitted* view of the intra-block state. In addition, because every state change is linked by a unique hash that includes the value, multiple state changes sequenced in the atomic block update are preserved.

Algorithm  shows the HMS algorithm as implemented on the Ethereum blockchain. Users interact with the algorithm through an Ethereum contract. We refer to line $x$ of algorithm $A$ as $A$:$x$.

A call to HashMarkSet() is made from the EVM interpreter when the transaction being processed has a function signature that matches that of a Sereth transaction. The $RAA$ variable on line : represents the storage variable value obtained using the RAA technique. We first extract the $RAA$ from the given $input$ field of the transaction we are processing. This process is simple, as each element is stored in a contiguous 32 bytes within $input$. By writing the result of HashMarkSet() to $RAA$, the result will be made visible within the contract's execution.

Algorithm details how the current transaction pool is filtered and then returned to the main function for handling.

For each transaction in the pool, we check that the function signature is equal to one of the write functions from our HMS contract. Additionally, we check the first 32 bytes of the FPV for a flag indicating one of several possible states for the transaction. Due to this filtering only a small percentage of the TxPool requires processing, so the overhead of HMS is relatively small.

First, the transaction may be one of the first HMS transactions that appeared during the current block. In this case, we consider the transaction a $head~candidate$, meaning that it or another transaction with the same flag will serve as the head of the serialized list of transactions for the current block. This allows us to easily continue the list from the previous block without being able to view the state variable. The second possible state indicates that the transaction is not a head candidate, and at the time of the transaction's submission, it was found to be the successor to the current tail of the series. If a transaction contains neither of these flags, it is considered rejected and is not included in the list of relevant transactions. If a transaction is accepted, The $FPV$ is then extracted from the $input$ field. The $FPV$ contains $previous\_mark$ and $value$, which are the two values needed to calculate the $mark$ of a transaction and determine its place in the series. A node is created from the transaction for later inclusion in a linked data structure.

Once $txnList$ has been populated by transactions from the TxPool on line :, we check on line : if the list is empty. If so, the submitted transaction is the first Sereth transaction sent in the current block, and the way to know if it matches the previous mark is to check the state variable within the contract. In this case, a flag is written to the $data$ field, which will be visible to the contract. The contract value will be written in the last 32 bytes of the transaction FPV by the sender.

If the list contains one or more transactions, then we know that there already exists at least one series for the current block. Algorithm contains the functions which return the most valid series from a list of Sereth transactions.

Line : refers to Series(), which iterates through each transaction in the list of Sereth transactions and forms graph relations between all transactions with corresponding mark/value hashes. Due to the uncertain nature of concurrency, it is possible for a transaction to have multiple potential successors, but only one predecessor.

At line : we locate from multiple potential head nodes the one that produces the deepest graph. From that graph, the deepest branch is our series. This logic mirrors that of the blockchain, in which branches are resolved by taking the longest branch.

## Runtime Argument Augmentation

Blockchain oracles provide a secure and verifiable medium for smart contracts to access external data feeds, but still suffer from stale reads due to latency. In our implementation of HMS it became clear that a traditional oracle would not satisfy the requirement for intra-block data. To overcome the limitations of oracles, we propose *Runtime Argument Augmentation* (RAA), a technique that provides data to a smart contract by using the argument list as a channel to pass information. RAA is a modification to the Ethereum Virtual Machine (EVM) interpreter, written in Golang. Figure 1 is an activity diagram showing the modified processing. In activity $E2$ the EVM interpreter checks to see if a function is requesting external data items using RAA. If so, the interpreter calls the RAA provider in activities $R1$ to $R3$, implemented as a Golang service compiled into the EVM. Data is obtained from the RAA provider and written into the function arguments. The data types of the items being requested must match the data types of the arguments. In $E3$ the function returns the result of evaluation using the modified arguments to the smart contract for use in activity $S3$. RAA is flexible: any computation can be accomplished by the RAA provider, and the information can flow in both directions. RAA is fast because it is written as an extension of the EVM. A smart contract using RAA is indistinguishable to unmodified clients running Geth, who merely see that arguments are passed in and a result returned.

There are some limitations. RAA cannot be used to modify the arguments of a smart contract function that may send a transaction. This is because transactions along with their inputs are cryptographically signed by the sending address, stored in parameters `msg.hash` and `msg.sender`. Without this protection a malicious Geth client could modify the inputs of a transaction, for example doubling the price offered for an item or changing the delivery address. In testing the limits of RAA we found that the modified transactions would still be mined, but would not be accepted by peers who must validate the newly created block. In order to use RAA information in a transaction, a smart contract or other blockchain actor calls the RAA function first, then uses the information provided to improve the subsequent transaction. This is the process used to obtain the experimental results that follow.

# Correctness

Concurrent systems are expected to satisfy correctness (safety) and progress (liveness) properties. Correctness is determined according to a defined correctness condition presented in literature . HMS is designed for the sequential consistency correctness condition because miners are required to preserve the nonce order when committing a transaction from a given thread to a block. Since the nonce is a counter that reflects the sequential ordering of transactions issued by the same thread and a blockchain transaction is analogous to a concurrent method, the blockchain is inherently sequentially consistent. In the following lemma we show that HMS generates a series that provides a sequentially consistent ordering of the transactions in the longest branch. The benefit of generating a series of transactions in the longest branch is that it offers the greatest potential for optimum state throughput.

**Lemma 1**. *The series generated from HMS preserves a sequentially consistent ordering of transactions invoked in the longest branch of the directed acyclic graph.*

*Proof.* For each transaction $T$ in the transaction pool, if the signature is a set operation, and $T$ is either a possible head candidate or is a successor to the current tail in the series, then $T$'s mark is updated by hashing the predecessor transaction's mark and value, and the list of transactions to be considered for the series is amended to include transaction $T$. If the length of the list of transactions is larger than one, then HMS generates a series of transactions by calling the Series function with the transaction list as input. It now must be shown that the generated series is both sequentially consistent and the longest branch. The Series function creates an adjacency list of all transactions in the transaction list such that a transaction $T_2$ that is a member of $T_1$'s list indicates that $T_2$ is a successor to $T_1$. The Series function then iterates through the potential head candidates and applies the DeepestBranch algorithm. Each recursive call to DeepestBranch will iterate through the list of successor transactions in the adjacency list and apply DeepestBranch to each successor transaction until a transaction with no successors is reached. At each recursive call to DeepestBranch, transaction $txn$ passed as an input parameter is amended to the path. Since the exploration of the adjacency list guarantees that all successor transactions are visited after a predecessor transaction, any path generated from DeepestBranch will be sequentially consistent because the program order established in the adjacency list is preserved. Since the depth at each recursive call of DeepestBranch is incremented by one, and a path that exceeds the maximum depth is recorded upon termination of the recursive calls, the final recorded path by DeepestBranch will be the longest branch within the adjacency list. ◻

Progress of the underlying blockchain (the computer) is assumed. We focus here on the progress of smart contract methods using a view of state variables managed by HMS. *Lock freedom* is defined as ensuring that some concurrent actor makes progress, and this is true for the blockchain as a whole but not for an individual smart contract. Miners may assign a low priority to a particular contract so it makes no progress. At peak times, many more transactions are sent to the network than can be included in a block. Transactions sent may be lost due to network failures, memory limitations or peers not replaying them. Miners may refuse to include transactions for arbitrary reasons. As a result, the progress guarantee provided by Ethereum is smart contract termination . Since the TxPool is a finite list of transactions, Algorithm  trivially terminates. Algorithm  and Algorithm  terminate given that the recursive function DeepestBranch terminates. We now show in the following lemma that DeepestBranch terminates.

**Lemma 2**. *DeepestBranch presented in Algorithm  is guaranteed to terminate.*

*Proof.* The $txnList$ in the Series function is a finite list of transactions because it is a subset of the TxPool generated by the Process function. Therefore, each list within the adjacency list of transactions constructed by the nested for-loop on line  of Algorithm  will also contain a finite number of transactions. Since the $txn.mark$ computed by Process establishes an ordering among the transactions in $txnList$, the adjacency list of transactions will not contain any cycles due to the if-statement on line  of Algorithm . DeepestBranch will be invoked by the Series function no more than the number of transactions contained in $txnList$. For each invocation of DeepestBranch, a recursive call to DeepestBranch is made for each transaction in $head$'s list of successor transactions. Since DeepestBranch is only invoked on the successors of $head$, and each list in the adjacency list of transactions is finite, it is guaranteed that every invocation of DeepestBranch will eventually reach a transaction with no successors. Upon reaching a transaction with no successors, DeepestBranch terminates on line  of Algorithm . ◻

# Results

This section shows experimental results of tests of the HMS algorithm on a private Ethereum blockchain. The chain used for testing is a fork of an open source multi-peer private network configuration . Experiments were hosted on Ubuntu 16.04 EC2 servers in the AWS cloud. The private network was configured to be a model of the Ethereum mainnet or the Ropsten testnet. Proof of work was used as the consensus mechanism. The block difficulty, transaction fees, processing power of the peers and peering topology were adjusted to produce block size and interval in the range of production Ethereum blockchains.

Interoperability was tested by running experiments with a mix of peers running standard Geth and modified Sereth clients. The first experiments were qualitative to demonstrate practical use of the two innovations of this paper: HMS and RAA. Smart contract functions that created transactions were followed through the process of invocation, interpretation, transactions sent to the TxPool, replay, mining and validation. The Sereth client operated interchangeably with Geth clients on the same network. This is not surprising because Ethereum already supports a variety of clients with subtle differences, all following the same protocol. Deployment of Sereth in the wild would not require a fork or any special permission from the network. The Solidity smart contract equipped with RAA also functioned even when deployed to a Geth client, although of course the substitution of arguments did not take place and they were returned unchanged.

Next we demonstrated that a sequential history was properly handled by sending a series of test transactions from the address of a single peer so that there is only one possible history, where real time order equals nonce order equals block order. As expected, the transaction failure rate was zero and the transaction efficiency $\eta$ was 1.0.

The quantitative experiments using concurrent peers demonstrated the effectiveness of HMS and the importance of transaction efficiency. Experiments considered the history of program execution on a single shared variable P where P is an object containing the AMV tuple described in the HMS algorithm. The dynamic pricing exchange from Section  is used to motivate the experiments, with the value of P representing the price. Two transaction types are used in the experiments: $buy$ (buys one item at the current price) and $set$ (changes the price). A ratio of buys (Read-Uncommitted) to sets (Writes) was used as a non-dimensional parameter that would scale up to larger servers as the absolute number of transactions increased. The number of set transactions was varied from 100 to 5, yielding a buy to set ratio of from 1:1 to 20:1.

Figure 2 depicts a plot of state throughput measured at different buy set ratios. Each data point represents the result of 100 buy transactions, so state throughput is equivalent to $\eta$ expressed as a percentage. Transactions were submitted at an interval of one second, resembling a moderate throughput smart contract use case. This interval was sufficient to demonstrate the problem of stale reads and can easily be reproduced with ordinary servers using the provided source code. The sets are evenly spaced over the processing of the buys. The lines are smoothed averages of the points shown, with the shaded areas representing the 90 percent confidence interval for the lines.

## Standard Geth client

The baseline scenario sends transactions to an unmodified peer running the standard Geth client. The transaction efficiency at different buy to set ratios is labeled as "geth_unmodified" in Figure 2. In this scenario, buy transactions that read the price P from block $n-1$ and are included in block $n$ before the price is modified will be successful, while all other buys will fail. When there are many price sets, as in the experiments with 1:1 and 2:1 ratios, only a few buys are successful. In some runs no buy transactions succeeded at all. The efficiency increases somewhat as the ratio of buys to sets goes above 10:1 because there are more buys reading correct values before an intra-block set occurs. However it remains poor for two reasons.

First, with a low ratio of buys it is unlikely for a buy to land in the very beginning of the block before any sets take place. Thus many will fail. Second, even as the ratio increases, because of the large transaction pool there are often no buys going into block $n+1$ that have a valid view of block $n$. Instead, block $n$ is assembled from buys that were submitted a few blocks ago, so they may have a view of block $n-2$, $n-3$ or older blocks. These buys fail because the blockchain state has passed them by before they were included. This phenomenon is frequently observed in public blockchains .

Although not shown in the plot, it was also observed that with few state changes (high ratio of buys) transaction efficiency becomes more sensitive to the transaction interval, as miners may sequence a large number of buys together.

Sets are not plotted. All of the sets succeed because they are sent from the owner of the contract and they do not depend on the previous price. If sets came from different addresses some might have failed, but it is reasonable that the owner of the contract is the only one allowed to set the price.

## Hash-Mark-Set without miner assistance

The second experimental scenario, labeled as "sereth_client" in Figure 2, used the modified Sereth client on the network, implementing the HMS algorithm. The set transactions were ordered with HMS while buy transactions were sent exactly as in the baseline scenario. Interleaved with the sets, any buy at the right mark and price succeeded. The benefit of HMS in this scenario is that the buy transactions have a Read-Uncommitted view of the likely state of the storage variable P when they will be evaluated. This allows many more transactions to succeed. A sequentially consistent ordering of the set operations was established and their dependent buys have a view of the state provided by HMS. Figure  shows an improvement in throughput by approximately a factor of five over the entire range of read /write ratios. These results were achieved without miner assistance, so they could be accomplished simply by running the modified client on the public Ethereum blockchain, as long as access to the smart contract was via these clients.

This experiment also demonstrates how HMS alleviates the intra-block lost update problem. The FPV arguments in each buy include the previous mark, a hash that relates it to an interval between two sets. If a sequence occurs such as: set(5), buy(5), set(7), set(5), buy(5), a particular buy(5) can prove that it was sent during the first or the second interval the price was set to 5. Linking each buy transactions to a particular set price prevents the frontrunning attack mentioned in Section .

## Hash-Mark-Set with semantic mining

In the third experimental scenario, the inputs of the second scenario were repeated with the miner using the HMS algorithm to determine the block order of transactions. In this scenario HMS information about the TxPool is available to both smart contract users and miners. Since the miner now has awareness of the semantics of the transactions, we call this *semantic mining*. Previously miners would not reorder transactions to increase transaction efficiency, but the semantic miners have this capability. The line labeled "semantic_mining" in Figure  shows the results. About 80 percent of transactions succeed due to semantic mining providing interleaving in conformance to the Read-Uncommitted view used by the smart contract clients when they sent the transactions. Relative improvement in throughput was greatest with a high frequency of price changes, i.e. where there are 1 or 2 buys per set. At these ratios the advantage of having the miner interleave transactions increases transaction efficiency from a few percent to almost 90 percent, resulting in a factor of six over the unassisted case. Overall, 10-20 percent of transactions were lost due to the fact that the TxPool no longer contains marked transactions immediately after the block is published. Transaction efficiency could approach 100 percent if HMS were extended to include the final values from replaying each block. Other factors that would impact efficiency is if only a fraction of the miners were assisting, or if communication of the TxPool were impeded among the Sereth enabled peers. Performance would be degraded in these cases but there would still be benefits proportional to the participation.

# Related Work

There are five main approaches to improve blockchain throughput and latency: Reparameterization, sidechains, sharding, leader election and invalid state tagging.

Reparameterization involves tuning the block size and interval to network bandwidth and peer computing power . HMS does not use reparameterization, but could influence tuning trade-offs by decreasing the significance of block interval.

Sidechains increase throughput by creating transaction channel networks such as Lightning . As the name implies they exploit parallelism by running multiple chains, merging them to the main chain as needed to ensure correctness. Sidechains have been implemented at scale on existing blockchains. Recently generalized formally as *state channels* , they can provide throughput gains of several orders of magnitude. However the authors note state channels do not solve the latency, or as they call it, "time granularity" problem. The Read-Uncommitted view provided by HMS does solve this for specific state variables.

Sharding increases throughput by isolating segments of the blockchain peer network. Sharding requires changes to consensus protocol but has been accepted by Ethereum with significant progress and a target implementation date in 2020. Like state channels, sharding is inherently parallel and offers performance gains of several orders of magnitude. Sharding is a global solution but would need customization to address state throughput of individual smart contracts as does HMS.

Bitcoin-NG "Next Generation" uses leader election with continuous serialization to modify the consensus protocol in proof of work blockchains such as Bitcoin. Performance gains scaling to the limits of network latency and individual peer processing power are reported. Recent work notes the history of improvement and elaboration on the original proposal. Our solution shares with Bitcoin-NG the concept of continuous serialization to reduce the "long freeze" of latency, but HMS does not require protocol changes to interoperate with current blockchains.

The scope of our review was public blockchains, however a Byzantine Fault Tolerant (BFT) proposal for Hyperledger is relevant because it focused on the bottleneck of transaction signing and ordering in block creation . BFT uses a leader to coordinate block creation achieving transaction rates of over 2000 per second on private blockchains. Unlike Ethereum, Hyperledger tags transactions known to be based on invalid states before they are ordered in a block, so time is not wasted replaying the failed transactions. However the authors do not consider transaction efficiency and the BFT ordering service does not use semantics to reduce failures as HMS does.

Software Transactional Memory (STM) algorithms have also been applied to blockchain throughput. The original ideas in using STM to enable concurrent processing of smart contract methods were continued by . These researchers note the EVM is not parallel and the difficulty of determining transaction dependencies in a block, so in both papers smart contracts were translated into C++ which is supported by the STM library. Simulated miners then interleave and order smart contract methods in STM to create a concurrent execution. Speedups of up to 2x were achieved. Concurrency based throughput gains in which "any sequential execution will do" are different from HMS, which sequences smart contract methods for transaction efficiency and increased state throughput.

A parallel to HMS is found in an earlier STM paper whose language about "publishing" is prescient as it was written before the blockchain was invented. A correctness condition called Selective Strict Serialization (SSS) is introduced, in which some transactions are strictly serialized and others are not, but are marked to the serialized history. In Section  above we applied sequential consistency as the correctness condition for our HMS algorithm. In our experiments HMS establishes a fixed ordering for the state changes (sets) while allowing the dependent transactions (buys) to be arbitrarily interleaved. Multiple buys can occur in a price interval and are not dependent upon each other. Within the interval any order of buys is valid so they do not require an established ordering constraint. Further work might show that SSS is a correctness condition suitable for HMS.

There is relevant work on improving throughput and latency of concurrent systems by reducing *abort rate*, defined as how many times a transaction is retried before success . This is different from our *state throughout*, which measures efficiency of blockchain commits that are not repeated. High abort rates due to delayed write visibility, where transaction writes may only be read after commit, is addressed by Faleiro et al.  in the proposal of piece-wise visibility (PWV), a deterministic concurrency control protocol designed to enable early write visibility. PWV divides a transaction into a set of sub-transactions which are scheduled to be executed in a serializable order. Each sub-transaction write is made visible as soon as it commits, enabling the original transaction writes to be visible prior to commit time. A DAG is used to order database sub-transactions based on data dependencies. HMS uses a DAG to order blockchain transactions in a sequentially consistent fashion, and the final series of transactions is derived from the deepest branch.

The fundamental difference between PWV and HMS is that PWV enables writes to be visible inside the commit protocol while HMS enables Read-Uncommitted isolation for smart contracts through our proposed RAA technique, described in Section . The PWV commit protocol only provides write visibility after a transaction is submitted to the database system, which limits the potential performance gains in comparison to HMS that provides write visibility to smart contract clients such that the requested data from RAA can be utilized prior to transaction submission.

# Conclusions

State throughput, the throughput of successful transactions, is proposed as the appropriate metric for smart contract performance. An algorithm, Hash-Mark-Set, and a novel architectural technique, Runtime Argument Augmentation, are presented and demonstrated together on the Ethereum blockchain to improve state throughput.

HMS provides smart contracts a Read-Uncommitted view of state. At the same time, HMS provides information about transaction dependencies to the miners so they can adjust the block order, called semantic mining. Miners cooperating with smart contracts using the HMS algorithm to order dependent transactions were able to create blocks in which most transactions were successful. This is demonstrated to improve transaction efficiency from less than 5 percent to over 80 percent in cases where state changes are frequent, more than an order of magnitude improvement in state throughput. Even without semantic mining, the Read-Uncommitted view is helpful, increasing state throughput by a factor of five across the full range of tested read to write ratios from 1:1 to 20:1. Latency (as a function of correct reads) was also reduced in both scenarios, client modifications only and semantic mining. In addition to the performance gains, HMS solves the blockchain lost update and frontrunning attack problems because transactions using Read-Uncommitted values keep a unique hash validated record of the particular interval during which the value was read.

RAA is presented as a new technique to provide smart contracts rapid communication with external data services. In our experiments smart contracts used RAA to access Read-Uncommitted views of data necessary for transaction success and thus increase transaction efficiency. RAA works at the architectural level of the EVM, using the interpreter to achieve high performance. Peers running the RAA modified client were demonstrated to work interoperably with standard peers.

# Acknowledgment

Thanks to Raul Jordan and Andreas Olofsson for assistance in understanding the intricacies of Geth client software.

[^1]: This work was funded by the National Science Foundation (NSF) under Grant Numbers 1717515 and 1740095.


**Where and How Do They Happen ?**

**Elemér E Rosinger**

Department of Mathematics
and Applied Mathematics
University of Pretoria
Pretoria
0002 South Africa
email@example.com

**Abstract**
This is a two part paper. The first part, written somewhat earlier, presented standard processes which cannot so easily be accommodated within what are presently considered as physical type realms. The second part further elaborates on that fact. In particular, it is argued that quantum superposition and entanglement may better be understood in extensions of what we usually consider as physical type realms, realms which, as it happens, have so far never been defined precisely enough.

**Part I**

**Abstract**
It has for ages been a rather constant feature of thinking in science to take it for granted that the respective thinking happens in realms which are totally outside and independent of all the other phenomena that constitute the objects of such thinking. The imposition of this divide on two levels may conflict with basic assumptions of Newtonian and Einsteinian mechanics, as well as with those in Quantum Mechanics. It also raises the question whether the realms in which thinking happens have no any other connection with the realms science deals with, except to host and allow scientific thinking.

**0. The Yet Undefined Physical Realms ...**
In the sequel, based on rather obvious and simple, even if so far seldom considered facts within, or related to Physics, we shall argue that what are usually assumed to be the Physical Realms may have to be extended. Such possible additional realms, however, are not along those infinitely many of Everett's "many-worlds" view of Quantum Mechanics. Instead, they are suggesting a finite number of further physical type realms, thus they can be seen as a development of the classical Cartesian realm of "res extensa".
As for what Physical Realms may actually mean, or rather, Physics itself, here is a recent and quite appropriate view on that never yet clarified issue, \[8, pp. 153,154\] :

> "Physics is the study of those phenomena that are successfully treatable with well-specified and testable models.
> For example, Physics treats atoms and simple molecules. Chemistry, on the other hand, deals with all molecules, most of whose electron distributions cannot be well specified. A physicist might study a well specified biological system, but the functioning of a complex organism lies in the domain of biologists.
> Anything not successfully treatable with a well-specified and testable model is rather quickly defined out of Physics."

It is quite clear in this spirit that, even if no one seems to care much about a more precise definition of Physics, and thus, of Physical Realms, phenomena such as human thinking, let alone, human consciousness or awareness, are not expected to concern Physics any time soon. Consequently, what for Descartes constituted "res cogitans", that is, the realms of thinking, are supposed to remain in the splendour of their undisturbed solitude, as far as Physics is concerned. And then, anything that may be seen as remotely acceptable from a physical point of view, may be but a refinement, or rather, a structural enrichment of the Cartesian "res extensa", that is, of the realms which, at least intuitively, are supposed to have to do with Physics.
And yet, as seen in the sequel, the story is not quite that simple, not even from a strictly physical point of view ...

**1. Conflict with Newtonian Mechanics**
Instant action at arbitrary distance, such as in the case of gravitation, is one of the basic assumptions of Newtonian mechanics. This certainly does not appear to conflict with the fact that we can think instantly and simultaneously about phenomena which are no matter how far apart from one another in space or in time. However, absolute space is also a basic assumption of Newtonian Mechanics. And it is supposed to contain absolutely everything that may exist in Creation, be it in the past, present or future. Consequently, it is supposed to contain, among others, the physical body of the thinking scientist as well.
Yet it is not equally clear whether it also contains scientific thinking itself which, traditionally, is assumed to be totally outside and independent of all phenomena under its consideration, therefore in particular, of the Newtonian absolute space, and also, of absolute time.
And then the question arises :

> Where and how does such a scientific thinking take place or happen ?

**2. A difference with Mathematics**
Mathematical thinking, especially in its modern and abstract variants, does not appear to need the assumption of any absolute space, or for that matter, absolute time. Such thinking may appear to unfold during appropriate local time intervals. However, when seen all in itself, and unrelated to the physical body of the respective mathematician, it is quite likely that such thinking has no location in any space, be it relative or absolute.

**3. Conflict with Einsteinian Mechanics**
In Einsteinian Mechanics a basic assumption is that there cannot be any propagation of action faster than light.
Yet just like in the case we happen to think in terms of Newtonian Mechanics, our thinking in terms of Einsteinian Mechanics can again instantly and simultaneously be about phenomena no matter how far apart from one another in space or time.
Consequently, the question arises :

> Given the mentioned relativistic limitation, how and where does such a thinking happen ?

**4. Conflict with Quantum Mechanics**
Let us consider the classical EPR, or Einstein-Podolsky-Rosen entanglement phenomenon, and for simplicity, do so in the terms of quantum computation. For that purpose it suffices to consider double qubits, that is, elements of $\mathbb{C}^2 \bigotimes \mathbb{C}^2$, such as for instance the EPR pair
(4.1)    $\begin{array}{l}
               |~ \omega_{00} > ~=~ |~ 0, 0 > ~+~ |~ 1, 1 > ~=~ \\ \\
                ~~~=~ |~ 0 > \bigotimes |~ 0 > ~+~ |~ 1 > \bigotimes |~ 1 >
                         \, \in  \mathbb{C}^2 \bigotimes \mathbb{C}^2
            \end{array}$
which is well known to be *entangled*, in other words, $|~ \omega_{00} >$ is *not* of the form
$~~~~~~ ( \alpha |~ 0 > ~+~ \beta |~ 1 > ) \bigotimes ( \gamma |~ 0 > ~+~ \delta |~ 1 > )
                                           \, \in  \mathbb{C}^2 \bigotimes \mathbb{C}^2$
for any $\alpha, \beta, \gamma,\delta \in \mathbb{C}^2$.
Here we can turn to the usual and rather picturesque description used in quantum computation, where two fictitious personages, Alice and Bob, are supposed to exchange information, be it of classical or quantum type.
Alice and Bob can each take their respective qubit from the entangled, or EPR pair of qubits $|~ \omega_{00} >$, and then go away with it no matter how far apart from one another. And the two qubits thus separated in space will remain entangled, unless of course one or both of them get involved in further classical or quantum interactions. For clarity, however, we should note that the single qubits which, respectively, Alice and Bob take away with them from the EPR pair $|~ \omega_{00} >$ are neither one of the terms $|~ 0, 0 >$ or $|~ 1, 1 >$ in (4.1), since both these are themselves already pairs of qubits, thus they cannot be taken away as mere single qubits, either by Alice, or by Bob. Consequently, the single qubits which Alice and Bob take away with them cannot be described in any other form, except that which is implicit in (4.1).
Now, after that short detour into the language of quantum computation, we can note that, according to Quantum Mechanics, the entanglement in the EPR double qubit $|~ \omega_{00} >$ implies that the states of the two qubits which compose it are correlated, no matter how far from one another Alice and Bob would be with them. Consequently, knowing the state of one of these two qubits can give information about the state of the other qubit. On the other hand, in view of General, or even Special Relativity, such a knowledge, say by Alice, cannot be communicated to Bob faster than the velocity of light.
And yet, anybody who is familiar enough with Quantum Mechanics, can instantly know and understand all of the above, no matter how far away from one another Alice and Bob may be with their respective single but entangled qubits.
So that, again, the question arises :

> How and where does such a thinking happen ?

**5. Two, Among Other Possible Alternatives**
Let us first assume that scientific thinking does indeed happen in realms outside and independent of all the realms in which the variety of phenomena studied by scientific thinking takes place. Then the very existence of scientific thinking proves the existence of realms transcendental to those which at present are customarily the object of that scientific thinking.
In this case, one may ask whether the realms in which scientific thinking happens have, indeed, no any other connection whatsoever with the realms which are the object of study of science, except to host and allow such scientific thinking.
A cautious answer is of course not one of categorical negation. Furthermore, any answer, including a categorically negative one, may need some supporting evidence, and possibly of experimental or empirical kind as well.
If alternatively, we assume that, after all, there is only one overall realm in which everything happens, then quite likely we may have to extend rather significantly, if not in fact dramatically, the list of entities, phenomena, or processes which are, or can be relevant in Physics, Chemistry, Biology, and so on. Certainly, in such a case it can no longer be taken for granted - and done so without any supporting evidence - that the whole range of entities and their interactions which form the object of science are isolated in some subdomain of that unique overall realm. And very much isolated they appear to be, since usual scientific thinking itself is assumed to be outside and independent of them, plus we deal with all those entities and their interactions as if they were perfectly self-contained.

**6. Conclusions**
It may be useful to ask the following four questions :

> 1\. Do we believe that whatever in Creation which may be relevant to science is already accessible to our awareness ?
> 2. And if not - which is most likely the case - then do we believe that it may become accessible during the lifetime of our own generation ?
> 3. And if not - which again is most likely the case - then do we believe that we should nevertheless try some sort of two way interactions with all that which may never ever become accessible to the awareness of our generation, yet may nevertheless be relevant to science even in our own days ?
> 4. And if yes - which most likely is the minimally wise approach - then how do we intend to get into a two way interaction with all those realms about which our only awareness can be that they shall never ever be within our awareness, or perhaps, not even of human awareness as such, no matter how long our species may live ?

**Part II**

**Abstract**
It is further argued that quantum superposition and entanglement may better be understood in extensions of what we usually consider to be physical type realms, realms which in fact have never been defined precisely enough.

**1. Superposition : a Typically Quantum
Fundamental Phenomenon**
In the non-relativistic Quantum Mechanics of a finite system $S$ described by states in a Hilbert space $H$, if for instance $\psi_1,~ \psi_2 \in H$ are two possible orthogonal states of the system $S$, then further states of $S$ are given by the arbitrary linear combinations
(1.1)    $\psi = c_1 \psi_1 + c_2 \psi_2,~~~ c_1, c_2 \in \mathbb{C}$
where the usual normalizing conditions are assumed
(1.2)    $|| \psi_1 || = || \psi_2 || = 1,~~~ | c_1 |^2 + | c_2 |^2 = 1$
hence resulting as well in
(1.3)    $|| \psi || = 1$
So far, in no other theory of Physics is such a property present. As for the importance of that property in Quantum Mechanics it suffices to recall two facts : it leads to yet unsolved foundational controversies, as in the celebrated argument in Schrödinger's Cat, and it is considered to be one of the basic resources of quantum computers, a resource which allows them unprecedented computational power, a power not possible to attain with usual electronic computers.

**2. Realms Physical, and Other Ones Less So ?**
It is nowadays a fundamental assumption that the Physical Realms do surely contain all there is, or at least, all there is of interest to Physics. And as with many a fundamental assumption, this one is so deeply ingrained that hardly anyone finds any reason at all to make it explicit to any extent.
One of the amusing aspects of such an approach is the convenient *circularity* of the argument, a circularity which, however, does not seem to concern in the least its proponents ...
Another amusing aspect is the recently emerging credo, according to which "information is physical" ...
This credo does, of course, reflect an awareness that what earlier were perceived, mostly tacitly, as the possible boundaries of the Physical Realms should now be extended in order not to leave out such an entity of fast growing importance like information.
And needless to say, such a move to encompass information within the Physical Realms is rather easy to accomplish, since the latter remains as undefined as it has always been ...
Indeed, there is here a significant mismatch between the rather clear definition of information in present day science and technology, and on the other hand, the actual, and quite convenient vagueness of what the Physical Realms are supposed to be about. Not to mention that, in spite of the insistent propagation of that newly emerged credo, the concept of information is in fact treated as a second class one at best in most of the present day fundamental theories of Physics, including in Quantum Mechanics.
On the other hand, and despite of the above, it is quite clear that the so called Physical Realms, even in their ever vague and latest extended sense, do *not* contain all that is of interest. And on top of it, they happen to *fail* to do so precisely on their own terms.
Several such instances were discussed in Part I, and here we recall one of them :

> Anybody, and even more so a physicist, can at the same time think about two arbitrarily far away places in the universe, for instance, two galaxies.
> On the other hand, according to Relativity Theory, no physical interaction can take place with arbitrary velocity.
> Thus such a thinking, so easily and so commonly available to quite everybody, *cannot* be of a physical nature.

And then, the question arises :

> Where and how does such a thinking happen, if not within the Physical Realms, and definitely not there, in view of Physics itself ?

And while such a question remains unanswered, and in fact, not even considered by present day Physics, perhaps one may as well compound the issue with the following.
It was Descartes in the early 1600s, who suggested the existence of two distinct realms. His "res extensa" was more or less what has been meant by the Physical Realms. On the other hand, his "res cogitans" was a realm beyond and outside of "res extensa", and it encompassed thinking.
As a consequence, Descartes has for long been ridiculed as being a dualist ...
Such a judgment misses, however, the fundamental fact that, as so many major European scientists of his time, Descartes himself was a deeply religious person in the Christian tradition. Consequently, he could not possibly be less removed from dualism than anybody else, since he saw God as underlying all Creation, and thus in particular, both "res extensa" and "res cogitans".
Now of course, Descartes himself did not advocate the study of "res cogitans" by the means of Physics, whatever the latter may mean under reasonable conditions.
And Classical Physics, that is, prior to the 20th century, did not in any way seem to require a more direct involvement of "res cogitans" than it would usually happen in the customary thinking process of normal humans, among them, physicists.
Relativity Theory, in spite of the above question, has not changed that classical situation, and it did not appear to need to do so. What it does instead, and even if not yet seriously considered, is to point quite sharply to the existence of at least two very different realms. And for the lack of better terms, as well as a homage to Descartes, we can still call those two realms as "res extensa" and "res cogitans", respectively.

**3. Does Superposition Need a Third Realm ?**

This may not be such an easy to answer question as one would like it. Indeed, Schrödinger's Cat already shows that it is not trivial. Therefore, let us consider it with some care.
What is obvious from (1.1) - (1.3) is that superposition takes place in the Hilbert space $H$, that is, within the mathematical model of the quantum system $S$. And as mathematical models go, they may hopefully reflect their respective system which, of course, is supposed to be situated in "res extensa", but on the other hand, as mere models, are *not* supposed to be identical with such a system.
This failure to distinguish between a physical system and its model is precisely one of the reasons one ends up with the controversy about Schrödinger's Cat.
And then, the question arises :

> Are superpositions (1.1) - (1.3) bona fide physical phenomena, or on the contrary, they are merely convenient features of the respective mathematical model ?

Well, as far as one can understand, this question does not have a clear enough answer in present day Quantum Mechanics.
However, as it may happen with not a few physicists, in case one tends to consider superpositions as genuine physical phenomena, then the foundational controversy around Schrödinger's Cat may simply be set aside by considering a *third* realm which we may call "res super-extensa", and in which such superpositions take place. This realm contains the usual "res extensa", in the sense that $\psi$ in (1.1), as a superposition of $\psi_1$ and $\psi_2$, belongs to it, without however belonging to "res extensa", while $\psi_1$ and $\psi_2$ belong to the latter. Clearly, just as with "res extensa", there is no need for any overlapping between "res super-extensa" and "res cogitans".
Here, however, one should note that the mathematical model (1.1) - (1.3), assumed to be in "res cogitans", need not always distinguish between "res extensa" and "res super-extensa". Indeed, $\psi$ in (1.1), as an element of the Hilbert space $H$, can in itself belong to "res extensa", as long as it is not seen as being constituted as a superposition.
The point to note with the above is that it is precisely the preference to see superpositions as physically real, that is, as having genuine physical existence, and not merely being representations in a mathematical model, which, when considered together with conundrums such as Schrödinger's Cat, can suggest the consideration of a third realm, such as that of "res super-extensa".

**4. And How About Entanglements ?**
As seen in Part I, entanglement also raises a question as to where and how it happens, given what appears to be its instantaneous nonlocal manifestation.
And yet, it may appear that entanglement, even more than superposition, is seen by physicists as a genuine physical phenomenon, and not merely as some occurrence in the mathematical model.
In this regard, no less than superpositions, entanglements are typical quantum phenomena, as well as unprecedented resources in quantum computation. As for their foundational importance, it suffices to recall the celebrated EPR paper, with all the related subsequent developments.
Thus a fundamental and still controversial issue which entanglements bring up is that of *nonlocality*. This fact, as is well known, was brought forward most starkly with the celebrated Bell Inequalities.
Here however, once one may consider the possibility of a third realm, like for instance, the above "res super-extensa", which is in fact but a *larger* instance of the customary "res extensa", the very issue of nonlocality may benefit from a new view and understanding.
Indeed, it may simply happen that in "res super-extensa" the dichotomy "local - nonlocal" is meaningless.
And here we should recall that such a possibility is not at all strange, since in a bounded system modelled mathematically by a compact space, the very concept of "nonlocal" loses much of its usual difficulties, if not in fact, its meaning. And in this regard we can recall that, so far, the very question whether the whole of the universe itself is in fact bounded is still open.
But then, and as if to complicate the issues, entanglements need *not* necessarily happen in the same extension of "res extensa" in which superposition may happen. Consequently, we may yet have to consider another, namely, third physical type realm as well.
A further possible consequence of considering physical extensions of the usual "res extensa" is that the foundational controversy related to the so called "hidden variables" in Quantum Mechanics may give way in favour of whole physical realms which, so far, were themselves hidden. In other words, it may well happen that what has been missing were not some hidden variables within this or that quantum entity, but rather whole physical type *realms* within which the very quantum processes as a whole may actually take place.
And with the acceptance in String Theory of the fact that the so called Physical Realms may have highly counterintuitive large finite dimensions, some of them so contracted as to make the dichotomy "local - nonlocal" quite meaningless, there is no longer any particular reason to be so parsimonious when considering the possible realms, beyond the usual "res extensa", that may be relevant to Physics.

**5. Conclusions**
Several extensions of what usually is meant by the otherwise undefined concept of Physical Realms were argued, based on rather obvious, simple, as well as fundamental physical considerations. In section 2, such an extension is motivated by the limitation of velocity of physical interactions, as follows from Relativity. In section 3, it was argued that, precisely to the extent that quantum superposition is not a mere feature of a mathematical model, but a genuine physical phenomenon, an extension of the customary concept of Physical Realms may be needed. In section 4, it was argued that quantum entanglement may need yet another such extension.
And as suggested, such possible extensions of the concept of Physical Realms need not necessarily be given by one and the same additional realm.
In case such a multiplicity of realms, beyond the two classical Cartesian ones, may raise certain concerns, one can always remember that, as thinking humans, thus in particular, physicists, our basic realm is in fact the "res cogitans". No wonder that Descartes insisted on what he considered as the fundamental ontological fact for us humans, namely, "cogito, ergo sum" ...
And therefore, without much further intellectual effort, we may at a certain stage subsume all other possible realms to that one. In other words, we may as well consider that everything is but a model, including what for so long we considered as having "objective" existence, whatever "objective" may happen to mean, namely, the Physical Realms.
The only major difference such a subsummation may imply is that we should redefine accordingly what we mean by "experimental evidence", and in particular, by "falsifiability".