title: TZ4Fabric: Executing Smart Contracts with ARM TrustZone
authors: Christina MĂźller Marcus Brandenburger Christian Cachin Pascal Felber Christian GĂśttel Valerio Schiavoni
text: 
  
  Disk Illumination and Jet Variability of the Herbig Ae Star HD 163296 Using Multi-Epoch HST/STIS Optical, Near-IR, and Radio Imagery and Spectroscopy
      Misato Fukagawa
      December 30, 2023
  =====================================================================================================================================================
  
  
  Blockchain technology promises to revolutionize manufacturing industries.  
  For example, several supply chain use cases may benefit from transparent asset tracking and automated processes using smart contracts.  
  Several real-world deployments exist where the transparency aspect of a blockchain is both an advantage and a disadvantage at the same time.
  The exposure of assets and business interaction represent critical risks.
  However, there are typically no confidentiality guarantees to protect the smart contract logic as well as the processed data.
  Trusted execution environments (TEE) are an emerging technology available in both edge or mobile-grade processors (e.g., ARM TrustZone) and server-grade processors (e.g., Intel SGX).
  TEEs shield both code and data from malicious attackers.
  This practical experience report presents , an extension of Hyperledger Fabric to leverage ARM TrustZone for the secure execution of smart contracts.
  Our design minimizes the trusted computing base executed by avoiding the execution of a whole Hyperledger Fabric node inside the TEE, which continues to run in untrusted environment.
  Instead, we restrict it to the execution of only the smart contract.
  The prototype exploits the open-source OP-TEE framework, as it supports deployments on cheap low-end devices (e.g., Raspberry Pis). 
  Our experimental results highlight the performance trade-off due to the additional security guarantees provided by ARM TrustZone.
  will be released as open source.
  
  
  
  § INTRODUCTION
  
  
  
  Industry 4.0 <cit.> is among the primary pushing factors for the adoption of blockchain technologies.
  Beyond the well-known case of cryptocurrencies <cit.>, several applications are being developed and deployed across diverse domains, including asset trading <cit.>, insurance claim processing <cit.>, cross-border payments <cit.>, innovative money lending <cit.>, transparent asset tracking along a given supply chain (e.g., farm tracking <cit.> or trusted food chain <cit.>) and so on.
  Typically, all of these scenarios involve a very large and heterogeneous set of mutually untrusted entities which still must collaborate to validate and execute a common (application-specific) transaction.
  
  When deployed over a blockchain infrastructure, the interaction between these entities can be controlled by smart contracts <cit.>, i.e., programs that execute on top of a distributed ledger in a fully decentralized manner <cit.>. 
  The correct execution of the smart contract is enforced by consensus protocols.
  Smart contracts can be deployed and executed over “Internet of things” (IoT) devices in a verifiable and efficient manner <cit.>.
  Nevertheless, the confidentiality of the contract itself is at stakes, especially in environments where the hardware and software integrity of the device can be compromised by powerful attackers, including compromised operating systems or malicious human operators.
  
  
  
  Hardware-based trusted execution environments (TEEs) offer an exciting new opportunity to overcome such limitations.
  In a nutshell, TEEs isolate data and code by shielding it from malicious users, compromised system libraries or thwarted operating systems.
  Notable examples include Intel software guard extensions (SGX) for server-grade processors <cit.> and ARM TrustZone <cit.> (from here on referred to as ARM TZ) for IoT- and edge-based devices.
  
  Hyperledger Fabric private chaincode (FPC) <cit.> is an open source project which enables nodes on a channel (a subset of nodes in the blockchain network) the execution of smart contracts inside TEEs, specifically inside Intel SGX enclaves.
  Since IoT devices are mostly small, (potentially) battery-powered and embedding low-power processors <cit.>, FPC cannot easily be deployed in this context, which greatly hinders its applicability of our target deployment scenarios.
  
  <Ref> depicts our envisioned deployment scenario. 
  A heterogeneous and possibly distributed set of nodes joins the blockchain.
  A smart contract must be executed across a channel and only few nodes are shielded against attackers.
  For example, an attacker could compromise a channel, gain access to its nodes and gather sensitive information.
  In order to contain the attack and prevent it from spreading to other channels, smart contracts of some channels could be executed inside a TEE.
  Nodes protected by TEEs are shielded against malicious attacks, including a compromised operating system or an attacker with physical access to the device. 
  Especially nodes that are deployed as IoT devices in “the wild” are prone to such kinds of attacks.
  In the case of TEE-enabled nodes, the processor package is the security perimeter.
  
  In this paper, we present the design, implementation and evaluation of , a prototype for Hyperledger Fabric <cit.> chaincode execution that integrates with ARM TZ.
  We discuss in particular the challenges that arise from executing smart contracts in an embedded TEE much less powerful than Intel SGX, as well as the limitations that result from 's design and the underlying platform.
  As we detail further (<Ref>), architecture is inspired by FPC, but it isolates the TEE component so that only the smart contract execution can be offloaded to it.
  This design allows not only to leverage ARM TZ, but also to possibly extend to future (yet unreleased) TEEs, by leveraging our modular architecture.
  We believe this modularity represents a major differentiation, but can also be seen as a supplement for FPC's design.
  
  This paper is organized as follows.
  In <Ref>, we present background on blockchain and Hyperledger Fabric, as well as ARM TZ and the OP-TEE runtime.
  <Ref> describes 's threat model.
  We then describe the architecture of our system in <Ref> and elaborate on its most important implementation details in <Ref>.
  <Ref> presents the evaluation of our prototype, reporting on performance as well as energy-related results.
  We finally discuss related work in <Ref> and conclude with some open challenges in <Ref>.
  
  
  § BACKGROUND
  
  
  
  Blockchain and Smart-contracts.
  A blockchain is a type of distributed ledger <cit.>.
  It records data (i.e., transactions) in a decentralized way.
  The data is appended in blocks and connected (“chained”) via hashes.
  Each transaction is signed by the party it was invoked.
  Before appending transactions, nodes of the blockchain network must validate and agree on a unique order of these transactions.
  The latter can be achieved via some consensus mechanism <cit.>.
  Thanks to its characteristics, a blockchain guarantees availability, transparency, immutability and integrity of the stored data.
  Data privacy and scalability are rather limited in blockchains.
  We distinguish between the following two types of blockchains: permissionless (also called public) and permissioned <cit.>.
  In a public blockchain (e.g., Bitcoin <cit.> and Ethereum <cit.>), anyone can participate in the network (read data from the blockchain, invoke transactions, validate transactions, etc.), whereas in a permissioned blockchain (e.g., Ripple <cit.> and Hyperledger Fabric <cit.>), the access of the network is restricted and entities are known.
  The concept of smart contracts was described first by Nick Szabo <cit.>: A smart contract is a computerized transaction protocol that executes the terms of a contract.
  With the emergence of blockchain technology, this idea was put into practice.
  A smart contract is an agreement translated into program code and stored in a blockchain network <cit.>.
  It is automatically executed when the defined conditions are met.
  As they are integrated into a blockchain, smart contracts inherit from its features of availability, transparency, immutability and integrity <cit.>.
  Furthermore, they are efficient, reduce cost and save time bypassing any third party.
  However, the execution of smart contracts present several challenges, e.g., it is not trivial to guarantee data privacy, avoid bugs in the contract code or protect from attacks, as it happened to Ethereum <cit.>.
  Hyperledger Fabric (HF) <cit.> is a permissioned blockchain supporting smart contracts.
  It belongs to the Hyperledger <cit.> open source project that includes different frameworks and tools related to blockchain technologies.
   In HF, a smart contract is called chaincode.
  Currently, developers can choose among three general-purpose programming languages (Go, Java and Node.js) to implement chaincodes.
  To facilitate deployments, HF uses Docker containers.
  There exist three types of nodes in a HF network: clients, peers and orderers.
  <Ref> presents the high-level architecture and workflow of HF.
  
  Before a chaincode function can get called (invoked), it must be installed (stored on the file system) and instantiated at the peer.
  A client (application) sends a request (transaction proposal) to a peer in order to invoke a chaincode function (<Ref>-202 and <Ref>-203). 
  In a first phase, i.e., execution or endorsement, the peer executes the called chaincode function (<Ref>-203192 and <Ref>-203193)
  and sends a transaction response 
  back to the client (<Ref>-204). 
  The transaction response is signed by the peer and contains the execution response message, as well as the readset and writeset.
  The readset represents all values of the keys a peer has queried from the ledger via GetState during the execution.
  The writeset contains all key-value pair updates a peer has generated via PutState.
  When the client has collected enough transaction responses as defined by the so called endorsement policy it sends them to an orderer (<Ref>-205). 
  The orderer puts the transaction responses into blocks and disseminates them to all peers (<Ref>-205192). 
  This is the ordering phase.
  In the third phase, the validation phase, each peer checks if the endorsement policy is satisfied and if there is no read-write conflict between the different transactions.
  Finally, they put the transaction on the ledger (<Ref>-205193). 
  The ledger has two components: a blockchain and a world state.
  The world state is a pluggable database, to store and efficiently retrieve later on the current values of the keys in the blockchain.
  
  ARM TrustZone provides hardware components for enabling TEEs on ARM processors <cit.>.
  OP-TEE <cit.> is a popular open source runtime with native support for ARM TZ.
  OP-TEE follows the TEE architecture and API standardized by GlobalPlatform (GP) <cit.>.
  ARM TZ enables a single TEE—called secure world—per system <cit.>.
  The other part of the system is called normal world.
  The processor can be in one of two security states: secure (for the secure world) and non-secure (for the normal world).
  Switching happens via a secure monitor call (SMC).
  System resources are strictly isolated: the normal world cannot access the resources (e.g., memory, peripherals, etc.) reserved for the secure world.
  During the bootstrap of the secure world, a chain of trust is established and there is an integrity check of the secure world software images—a process called secure boot.
  Additional details regarding OP-TEE framework are given while discussing implementation in <Ref>.
  
  
  § THREAT MODEL
  
  
  For our thread model we consider a powerful attacker with administrative rights as well as physical access to all nodes supporting ARM TZ.
  There, we rely on the protection mechanisms offered by the TEE to shield the privacy of the smart-contract code, as well as its privacy upon execution.
  We further assume that the operating system and the user space in the normal world cannot be trusted.
  However, we do assume that the TEE, which includes bootloader, firmware, OP-TEE, and the secure monitor are trusted.
  The HF blockchain network can run different consensus algorithms, among which PBFT <cit.>, and can therefore tolerate Byzantine failures.
  
  Since ARM TZ does not natively support remote attestation mechanisms, an attacker might try to compromise the contract before it is executed.
  The chaincode is stored in the normal world as a trusted application (TA) and signed with the build key.
  In addition, it is possible to encrypt the TA to add further security measures.
  Without the build key an attacker cannot easily tamper with the chaincode, as its signature is verified in the secure world before execution.
  Furthermore, this attack can be mitigated by integrating schemes such as Fides <cit.>.
  Notice that ARM TZ does not provide integrity protection at runtime, hence cannot easily prevent unnoticed malicious corruptions of the chaincode.
  
  While recent side-channel attacks <cit.> have been unveiled against ARM TZ, and mitigations continue to be released <cit.>, we consider them out of the scope of this work, as also mentioned in <cit.>.
  Mitigations will eventually be released by means of architectural microcode updates (as happened for Intel SGX to mitigate Spectre/Meltdown attacks <cit.>).
  
  
  § ARCHITECTURE
  
  
  ARM TZ-enabled devices are typically low-power embedded devices of which many are battery powered.
  Hence, these devices have to get by with limited resources in terms of persistent and volatile memory.
  Given these limitations, it is oftentimes challenging to port applications or systems that are being used in a desktop or sever environment – in particular if these tend to have many dependencies.
  In our case, depends on packages such as Go programming language (golang) environment, HF, OP-TEE, as well as gRPC <cit.>.
  Not only do these packages (and their dependencies) require several hundreds of megabyte of persistent memory, but also do they need a few hundred megabytes of volatile memory at runtime.
  While there are IoT devices that can satisfy these requirements (e.g., Raspberry Pi models), many cannot.
  For this reason we have settled for an approach where we decouple large system components (i.e., HF and Docker) from lightweight security-relevant components (i.e., chaincode) by means of a proxy.
  
  The design of is inspired by the architecture of FPC <cit.>.
  In our design, sensitive information can be contained in the chaincode operation and its response, which have to be shielded from malicious attacks.
  We therefore offload the chaincode to a TEE-enabled embedded device.
  To facilitate our design, we divide into three major components: (1) a wrapper that resides as a chaincode at a peer, (2) a proxy and (3) the chaincode itself.
  The wrapper communicates with the proxy via gRPC.
  The proxy and the chaincode run on an ARM-based environment with TrustZone.
  In particular, the proxy resides in the normal world, whereas the chaincode is located in the secure world.
  Our design leverages the OP-TEE framework to drive the interaction between the proxy and the chaincode.
  The wrapper is installed and instantiated as chaincode and is used as an interface towards the peer and the ledger.
  It forwards incoming invocations from the clients to the chaincode in the secure world, handles the communication towards the ledger and sends transaction responses back to the peer.
  The proxy acts as an intermediary and forwards the calls between the wrapper and the chaincode.
  It is responsible for context switching from the normal world to the secure world.
  A chaincode implements the blockchain application logic and is invoked by the clients via the wrapper.
  During execution, a chaincode has access to the ledger via getState and putState commands through the proxy.
  
  Our design allows us to instantiate and run multiple chaincodes in a single secure world, which provides isolation from the normal word, i.e., the operating system does not have direct access to the chaincode resources and can only interact with a chaincode through the proxy.
  However, as all chaincodes reside in the same secure world, there are no isolation guarantees among the chaincodes, something that is instead provided by FPC as a result of the ability to run multiple SGX enclaves on the same processor.
  
  Our prototype focuses on the inclusion of ARM TZ in HF.
  This means that we have not implemented a mechanism in order to replicate chaincodes in our network.
  The principal reason for this choice lies in the execution of chaincodes inside ARM TZ.
  Chaincodes are executed in the prototype as TAs (see <Ref>) inside OP-TEE.
  This implies that the chaincode is compiled using the same build system as the OP-TEE running on the machine and that the TA is signed with the original build key and optionally encrypted.
  Therefore, every chaincode would have to be compiled to a TA and installed on the target machine, before it could be invoked.
  Alternatively, implementing a virtual machine as TA inside ARM TZ that is capable to execute smart contracts (e.g., Solidity <cit.>) would overcome the previously explained restriction.
  Chaincodes could then be replicated on the target machine and directly be invoked without having to go through the process of generating a TA.
  In the prototype we have generated a TA for the chaincode and we have deployed it on all ARM TZ-enabled proxies.
  As mentioned before, there is also the possibility to use smart contracts which are written in domain-specific languages (DSL) (e.g., Solidity) and that are executed by a virtual machine (e.g., Ethereum Virtual Machine).
  Since our goal is to shield the chaincode with a TEE, this would also require that the virtual machines executing the chaincode is shielded as well.
  Such an approach would inevitably increase the trusted computing base (TCB) and take up some of the already limited resources on embedded devices.
  On top of that our design would strongly depend on the design of the DSL and the potential flaws that come with it.
  
  Another aspect to consider is that the ledger might hold sensitive information, which could be shielded by the approach used in FPC.
  However, for our prototype we did not shield the ledger using Intel SGX.
  
  
  § IMPLEMENTATION DETAILS
  
  
  
  Our implementation leverages the OP-TEE framework, depicted in <Ref>.
  OP-TEE contains the following components: OP-TEE Client, OP-TEE Linux driver and OP-TEE OS.
  
  The OS of the normal world is also referred to as rich execution environment (REE).
  The OP-TEE Linux driver provides the driver for the normal world.
  An application running inside the normal world is referred to as host application.
  The TEE client API and the TEE internal API enable the communication between a host application and an application of the secure world, i.e., the TA.
  Both APIs are defined by GP <cit.>: the TEE client API is implemented by the OP-TEE client component, the TEE internal API is implemented by the OP-TEE OS. 
  
  
  
  Interactions between the different components of the system happens as follows (see also <Ref>).
  Before initiating any communication, the host application establishes a connection towards the secure world (TEEC_InitializeContext) and opens a session towards the TA (TEEC_OpenSession) with the unique identifier of the TA () as parameter.
  Then, the host application can call functions of the TA with the TEE client APIs TEEC_InvokeCommand, as this allows to pass data between the host application and the TA via shared memory reference or by value.
  
  Once the host application has finished communication with the TA, it needs to close the session (TEEC_CloseSession) and finalize the context (TEEC_FinalizeContext) to release any allocated resources. 
  
  In our implementation, all components are deployed within containers.
  Since there is no native Docker support in the normal world (untrusted part) of OP-TEE, the peer is decoupled from the ARM TZ node.
  Communication between a chaincode_wrapper at the peer (specific wrapper around the actual chaincode) and a chaincode inside the secure world is enabled via gRPC remote procedure calls and through the API provided by OP-TEE between normal and secure world.
  
  Finally, it is worth mentioning that the availability of remote attestation is one of the most important difference between Intel SGX and ARM TZ with OP-TEE, where the former natively supports it <cit.>.
  While this represents one of the main limitation of compared to its SGX-based FPC counterpart, we highlight in <Ref> possible workarounds.
  The code of is released as open source and available from <https://github.com/piachristel/open-source-fabric-optee-chaincode>.
  
  
  § EVALUATION
  
  
  
  
  This section presents the experimental evaluation of the prototype.
  Our main goal is to show the performance trade-offs of the system.
  In particular, we investigate the throughput and latency impact of our implementation.
  Further, to clarify the performance penalties of the system, we conduct a latency breakdown to identify the bottleneck in the system, as well as energy measurements.
  
  
  
  
  
   §.§ Experimental settings
  
  
  
  
  Each experiment is executed with OP-TEE (v3.8) using QEMU for ARM TZ emulation and with OP-TEE running on the Raspberry Pi.
  We use a set of 27 server machines with two Intel Xeon CPU L5420 at 2.5 and 8 of RAM.
  The orderer, 8 wrapper and 8 proxy (QEMU instance) each run on their own server.
  The Raspberry Pi model 3B+ embeds ARM TZ (ARMv8-A). 
  We employ the Linux CPUFreq powersave governor on all Raspberry Pi.
  @christian: add which CPU governor is used
  Hence, we emulate the same processor using QEMU.
  Specifically, QEMU emulates a Cortex-A53 CPU with symmetric multiprocessing (SMP) set to 4 cores, and the number of trusted threads in OP-TEE is set to 4 in order to have an emulated environment to closely mimic the one found in the physical Raspberry Pi 3B+.
  Clients invoke transactions from 8 dedicated machines.
  
  
  We use two auxiliary machines in order to record the power consumption of the different types of nodes that form our HF network.
  We rely on a LINDY iPower Control 2x6M power distribution unit (PDU) to monitor energy consumptions.
  Due to the limited number of available ports, we restrict the measurements the following nodes: the orderer, 3 wrapper, and 3 proxy (QEMU instances).
  The 8 Raspberry Pi 3B+ are all equipped with a Raspberry Pi
  PoE-HAT[<https://www.raspberrypi.org/products/poe-hat/>, last access: May 29 2020] and make use of 802.3.af Power-over-Ethernet <cit.>.
  They are connected to a Ubiquiti Networks UniFi USW-48P-750 switch, which also allows to directly query their power consumption.
  The nodes are connected via 1 Gbit/s switched network and run Ubuntu 18.04 LTS, except for the client machines which run Ubuntu 20.04 LTS.
  The clocks of the auxiliary and the client machines are synchronized using NTP, which is necessary in order to relate the power measurements to the benchmark measurements in our setup.
  We implemented and deploy in our benchmarks a simple chaincode toy-example, e.g., tracking the coffee consumption in an office.
  Clients submit transactions to track their coffee consumption and query the current coffee statistics.
  The chaincode and the proxy are written in ISO C and C++, whereas the clients and the wrapper are written in golang.
  Our coffee tracking chaincode is simple enough to conduct a read/write benchmark on the HF blockchain network.
  should we say why this example is representative? baseline numbers?
  Each client machine runs a peer. 
  We exploit its capability to connect to a remote peer in the network.
  Clients repeatedly invoke transactions on the their local peer which get forwarded to the HF network peer.
  The communication between the wrapper and the proxy uses gRPC (v.1.28.1).
  
  The execution time of a client measurement begins before the client invokes the first transaction and terminates after the client has received the corresponding response from the last transaction.
  Furthermore, for each client we record the number of invoked transactions.
  We support experimental reproducibility, and detailed instructions to setup and reproduce these experiments are given at git repository (<Ref>).
  
  
  
  
   §.§ TCB Size
  
  
  The TCB consists of all components residing in the secure world.
  With our design we limit this to the chaincode and other components (i.e., wrapper, proxy).
  The peers are considered to be untrusted and thus run outside the secure world.
  In total, our prototype contains ∼22000 lines of untrusted C++ code (most of it generated by gRPC) and only ∼400 lines of trusted C code.
  Additionally, the OP-TEE OS itself consists of ∼233600 lines of C and assembler code, as well as the Trusted Firmware-A which sums to ∼31400 lines of C and assembler code.
  For OP-TEE OS and Trusted Firmware-A we considered only the platform-specific source code.
  
  In contrast, the peer consists of ∼80000 lines of Go code (v1.4.1).
  This shows that by only executing the chaincode itself inside the TEE and not the entire peer, we can drastically minimize the TCB and thereby reduce the attack surface.
  
  
  
  
   §.§ Throughput and Latency Impact
  
  
  
  We begin with measuring the throughput and latency of , by deploying and running the coffee tracking chaincode.
  We perform multiple rounds with up to 8 clients.
  In every round the clients repeatedly invoke transactions over at least 30 seconds.
  We expect the throughput to increase with increasing number of clients, while latency to be stable until the number of clients matches the CPU cores in our machine.
  
  A preliminary evaluation using a single peer and a single Raspberry Pi 3B has otherwise shown that the throughput only slightly increases, by approximately 1.2×, from 1 to 2 clients and then already stagnates.
  
  To investigate this unexpected result and identify the root cause of the bottlenecks, we conducted the experiment with different scenarios.
  Our baseline consists of all components (i.e., chaincode, clients, etc.) running in the normal (untrusted) world on the same machine.
  This scenario does not benefit from the security properties introduced by ARM TZ, and it does not use the gRPC messaging library.
  In a second scenario we place the chaincode in the secure world, but run the components on the same machine.
  Finally, we locate the clients and the wrapper on a dedicated machine, and proxy and chaincode runs with QEMU and on a Raspberry Pi.
  
  <Ref> summaries the throughput and latency for the baseline scenario and the final scenario.
     
  In the baseline scenario, we look into the impact of gRPC.
  We tried different configurations and the best results are achieved with NUM_CQS set to half the number of cores, and MIN_POLLERS and MAX_POLLERS with the default values.
  Using these settings, throughput increases by 1.9× for 2 clients, 2.5× for 4 clients and 2.8× for 8 clients.
  As these factors are far above those observed in our preliminary experiment, we can therefore rule out gRPC as the bottleneck.
     
  In the second scenario, we notice that the throughput only increases by 1.3× for 2 clients, 1.4× for 4 clients and 1.5× for 8 clients.
  These factors are slightly above our observation during our preliminary experiment, but still well below the baseline case.
  We therefore conclude that the bottleneck mainly originates from the secure world calls. 
  
  
  
  In the final scenario we observe that the throughput does not improve with an increasing number of clients in case of the Raspberry Pi.
  In contrast with QEMU, there is an increase of  1.2× from 1 to 2 clients.
  For more clients, throughput stagnates.
  This is due to the bottleneck we have already observed in the second scenario.
  In contrast, the throughput of the baseline increases up to 8 clients.
  With the proxy and the wrapper running on the Raspberry Pi, we can increase throughput by 3.1× for 8 clients compared to 1 client.
  For QEMU, it is increased by 2.6× from 1 to 8 clients.
  
  Considering the experiment on the Raspberry Pi for 1 client, the throughput of the baseline is about 27× higher compared to original .
  In case of 8 clients, we even have an increase of about 130×.
  For QEMU, the difference is less extreme: the baseline throughput is between 5× to 10× higher compared to the original .
  The observations show that the execution of the chaincode inside the secure world comes with non-negligible cost in terms of throughput.
  
  
  
   §.§ Latency Breakdown
  
  
  Next, we provide a detailed breakdown of the latency overheads incurred by .
  We divide the execution latency in several phases as shown in <Ref>:
  
  A initializes the wrapper;
  B initializes the proxy and includes TEEC_OpenSession to allocate memory;
  D prepares GetState commands in the chaincode;
  E forwards the command via GetStateRequest to the wrapper;
  G processes GetStateRequest;
  H post-processes GetStateResponse in the proxy;
  J executes the response message in the chaincode;
  K concludes the chaincode execution via a InvocationResponse;
  L releases the resources, for closing the session (TEEC_CloseSession) and for finalizing the context (TEEC_FinalizeContext) at the proxy;
  M finalizes execution at the wrapper.
  The phases C, I, N, and O are measured for calculating the round-trip times.
  
  
  We perform this experiment with the same settings as in the original experiment. 
  
  For both settings with QEMU and the Raspberry Pi, phase B contributes up 65-75% of the total time.
  We observe that the vast majority of phase B is needed for TEEC_OpenSession.
  We leverage this information to identify throughput bottlenecks observed throughout our evaluation.
  
  The measured latency for the phases D, E, G, H, J, K and M are smaller than 1 ms, and overall contribute to less than 1% of the total latency.
  
  
  
  
  
   §.§ Throughput-Latency Benchmark
  
  In this benchmark we evaluate the coffee tracking chaincode on the setup described in <Ref>.
  We perform multiple rounds with up to 128 clients.
  In every round the clients repeatedly invoke read or write transactions over a duration of 5 minutes.
  Clients are distributed evenly among the 8 available peers.
  We use the same naming convention for the scenarios introduced in <Ref> with the exception that all components (excluding clients) run on their own server.
  
  The throughput-latency plot is shown in <Ref>.
  We observe that for both read and write transactions, the baselines are very similar to each other.
  In the baseline scenario, the chaincode is executed on the wrapper in the normal world and not shielded by ARM TZ.
  There is also no communication going on from the wrapper to the proxy since this scenario does not make use of gRPC.
  Therefore, the setup for these two baselines is identical, in line with our measurements.
  
  Notice that our benchmark is not saturating the system with the baseline settings.  
  We consistently observe across transactions that the ARM TZ-enabled
  environments start saturating already with 8 clients around 65tx, which corresponds to a single client per wrapper-proxy pair.
  This observation was made earlier during the preliminary evaluation in <Ref>.
  Client transactions on the chaincode are essentially invoked sequentially.
  One reason for the low throughput is the substantial overhead due to shielding TAs with ARM TZ.
  There is also potential to improve the load times for TAs with OP-TEE by caching their contexts and session to improve reusability.
  We intend to explore these optimizations in future work.
  
  Using a shielded QEMU instance (emulating ARM TZ),  the latency increases by a factor of 2.5 for reads and by a factor of 2.6 for writes respectively, before reaching the saturation point.
  On the Raspberry Pi the latency increases by a factor of 2.0 for reads and by a factor of 2.1 for writes before reaching the saturation point.
  While the throughput and latency of read transactions is better by a factor of 1.04 than write transactions with QEMU, the throughput and latency of read transaction improves by a factor of 1.02 than write transactions on the Raspberry Pi.
  
  
  
  
  
   §.§ Energy consumption
  
  Finally, we conclude our experimental evaluation with a study of the power footprint of .
  We achieve this by recording the power consumption of the nodes running the HF network with the coffee tracking chaincode with .
  We expect the execution to be faster and require less energy per transaction when running OP-TEE on the Raspberry Pi as compared to emulating ARM TZ with QEMU.
  
  The energy consumption of the orderer, wrapper, and proxy are depicted in <Ref>, and we distinguish between read (top) and write (bottom) transactions.
  We observe that the energy consumption across the nodes during the entire benchmark is rather stable, only slightly increasing when the number of clients grow.
  There are two exceptions occurring under different situations for 16 clients: (1) the orderer and wrapper energy for the baselines rise (<Ref>-[a,b,d,e]) as well as (2) the energy of the proxy (<Ref>-[c,f]).
  
  The first exception is an indication of saturation of the HF network.
  This is further supported by the decline of average transactions per client in the network and the fact that the system starts approaching the saturation point as shown in <Ref>.
  Both the orderer and wrapper nodes begin to struggle for resources under the load of the increasing number of clients.
  
  The second exception is a result of the ARM TZ world-switching overhead when a chaincode is invoked.
  As the load increases the performance governor has to increase the frequency and voltage of the CPU over longer intervals.
  Although barely visible, the Raspberry Pi is experiencing the same issue.
  This is better highlighted in the energy per transaction (see <Ref>), where the energy increases proportionally to the QEMU instance.
  This steadily raises the energy consumption of the proxy machines by a factor of 1.25 with QEMU and by a factor of 1.05 on a Raspberry Pi beyond the saturation point.
  In terms of energy there is no significant difference between read and write transactions on both platforms.
  
  
  § RELATED WORK
  
  
  
  
  
    
  *Smart contract execution with a TEE
  Apart from FPC, there are some other works about confidential smart contract execution with a TEE.
  Confidential consortium framework (CCF) <cit.>, Ekiden <cit.>, ShadowEth <cit.> and private data objects (PDOs) <cit.>, just to mention a few recent ones.
  In contrast to , they all use Intel SGX as underlying TEE technology for a (prototype) implementation.
  The authors of Ekiden state that their technology may use any TEE which is similar to Intel SGX and supports attestation.
  Furthermore, SGX can provide means to mitigate rollback attacks <cit.>, for which the necessary hardware support might not be provided in ARM TZ.
  
  
  
    
  *Confidentiality in context of blockchain and IoT
  In , we are concerned with confidentiality of smart contracts (their logic) and data in context of IoT networks.
  Some other research efforts are concerned with the confidentiality of data produced and processed by IoT devices and stored on the blockchain.
  
  
  
    
  *Trust for data generated by IoT devices
  AnyLedger <cit.> is a platform for connecting physical devices to the blockchain.
  The key feature is an ARM TZ-based wallet for IoT devices.
  Key generation, private key storage and signing process of (smart contract) transactions are all placed inside the secure world of ARM TZ.
  Hence, AnyLedger wallet guarantees that IoT data hash / address (linking to the interplanetary file system IPFS) placed on the blockchain is integrity protected and authenticated.
  Furthermore, data stored on the IPFS is encrypted.
  AnyLedger is pluggable to any blockchain technology (for example Ethereum or Bitcoin).
  <cit.> is another system which equips IoT devices with a TEE to guarantee integrity and confidentiality of the IoT data.
  
  
  
    
  *Confidential computation
  BeeKeeper 2.0 <cit.> is a blockchain network for IoT systems that consists of IoT devices, servers and validator nodes.
  It enables IoT devices to share data with each other.
  Furthermore, devices can use servers for performing homomorphic computations on encrypted data.
  Computation results are verified by the validator nodes and recorded on the blockchain after successful verification.
  Since homomorphic encryption is used, the confidentiality of data sent to and processed by servers is guaranteed.
  BeeKeeper 2.0 can be added on top of any blockchain technology (Hyperledger Fabric, Ethereum etc.).
  In the paper, authors use Hyperledger Fabric for deployment.
  
  
  
    
  *ARM TZ and blockchain technology
  Some related systems use ARM TZ not for smart contracts directly but in context of the blockchain technology (which is the underlying technology of smart contracts).
  Secure blockchain lightweight wallets (SBLWT) <cit.> use ARM TZ to guarantee confidentiality and integrity for information generated and stored in the Bitcoin wallet (wallet's private key, wallet addresses, block headers used for Simplified Payment Verification).
  Synchronization of block headers and the verification process of transactions is executed in the secure world to avoid any manipulation by an attacker.
  SBLWT is safer than often used software wallets but still more portable than hardware wallets.
  An implementation of SBLWT using ARM TZ with OP-TEE has been deployed on Raspberry Pi 3 Model B.
  The Bitcoin wallet <cit.> and sensitive Bitcoin wallet information are backed by ARM TZ technology.
  
  
  § CONCLUSION AND OPEN CHALLENGES
  
  
  
  This work reports on our practical experience with the combination of Hyperledger Fabric and ARM TZ, a particularly useful setting in the context of IoT.
  We presented the design and implementation of a prototype, , for Hyperledger Fabric chaincode execution with ARM TZ and OP-TEE, demonstrating a fully-working mechanism to execute smart contracts inside the secure world. 
  Our design minimizes the TCB executed by avoiding execution of a whole Hyperledger Fabric node inside the TEE, which is assumed to be running in an untrusted environment.
  Instead, we restrict it to the execution of only the smart contract.
  The prototype exploits the open source OP-TEE framework, as it supports deployments on cheap low-end devices (e.g., Raspberry Pis).
  However, we also report on several challenges faced while building our prototype, in particular the missing support for remote attestation in ARM TZ and OP-TEE, and missing hardware support for critical security features. 
  Also specifications, such as GP's TEE APIs, can limit the potential of TEEs.
  For example, according to the specification a TEE is only capable of doing integer-based computation, therefore limiting the scope of smart contracts that could be executed inside a TEE.
  For this reason it is important to reevaluate and update such specifications regularly.
  Furthermore, support for a validation mechanism in the blockchain which ensures that the smart contract has been executed correctly and has not been tampered with can be of use.
  
  Our experimental results highlight the performance and energy trade-off due to additional security guarantees provided by ARM TZ.
  We leave the final design and replication of chaincodes in our prototype as future work.
  The is released as open source.
   
  
  
  § ACKNOWLEDGMENTS
  
  The research leading to these results has received funding from the European Union's Horizon 2020 research and innovation programme under the LEGaTO Project (https://legato-project.eu/legato-project.eu), grant agreement No 780681.
  
  IEEEtranN
  
  
  get arXiv to do 4 passes: Label(s) may have changed. Rerun
  