Stress-SGX: Load and Stress your Enclaves for Fun and Profit
Sébastien Vaucher Valerio Schiavoni Pascal Felber






University of Neuchâtel, Neuchâtel, Switzerland

{mailto:sebastien.vaucher@unine.chsebastien.vaucher,mailto:valerio.schiavoni@unine.chvalerio.schiavoni,mailto:pascal.felber@unine.chpascal.felber}@unine.ch

: Load and Stress your
 Enclaves for Fun and Profit
    Sébastien Vaucher Valerio Schiavoni Pascal Felber
    December 30, 2023
=====================================================




[remember picture, overlay]
[anchor=north,xshift=60pt,rotate=90,black!70] at (current page.west) 420pt
	This is a post-peer-review, pre-copyedit version of an article published in Networked Systems. The final authenticated version is available online at https://doi.org/10.1007/978-3-030-05529-5_24.
;

The latest generation of Intel processors supports SGX, a set of instructions that implements TEE right inside the CPU, by means of so-called enclaves.
This paper presents , an easy-to-use stress-test tool to evaluate the performance of SGX-enabled nodes.
We build on top of the popular tool, while only keeping the workload injectors (stressors) that are meaningful in the SGX context.
We report on several insights and lessons learned about porting legacy code to run inside an SGX enclave, as well as the limitations introduced by this process.
Finally, we use to conduct a study comparing the performance of different SGX-enabled machines.








§ INTRODUCTION



The latest generation of Intel processors (starting from the Skylake microarchitecture) features a new set of instructions: SGX.
This instruction set allows programs to execute securely inside hardware enclaves, hence creating TEE.
Specifically, SGX enclaves protect the code from external threats, including privileged system software.
Given the novelty of the technology (the first compatible CPU was released in August 2015) and the lack of in-depth literature, it is still challenging to validate the performance of code running inside enclaves.
It becomes even harder to study the problem under varying conditions, such as different hardware revisions or workloads. 
Moreover, as we show in this paper, microcode updates issued by CPU vendors can introduce performance degradations that are difficult to promptly detect.

The main contribution of this paper is , a stress tool capable of artificially putting SGX enclaves under high load.
It supports workloads of different nature, as explained in <ref>.
As far as we know, is the first tool that can be leveraged to induce hybrid workloads (with and without SGX).
We believe that many researchers can benefit from using our tool, simplifying design plans for their evaluation settings.
is GPL-licensed free software[<https://github.com/sebva/stress-sgx>].



  
Motivating scenarios.
We introduce three use-cases for and how researchers could leverage this tool.
First, there exist several real-world cluster traces readily available for research, such as the ones released by Google Borg <cit.> or Microsoft Azure <cit.>.
However, these traces do not have the same properties as typical SGX workloads.
Due to anonymization, hardware characteristics of the original machines remain undisclosed.
Finally, the nature of jobs deployed on these clusters is also confidential.
allows to directly map behaviors described in such traces onto similar, SGX-specific ones.

A second motivating scenario stems from the availability of a range of SGX-enabled CPUs on the market, each with their own characteristics and associated performance.
Apart from the expected differences due to hardware revision, cache size, available instruction sets or frequency, one still needs to hand-craft micro-benchmarks in order to evaluate SGX-specific performance.
facilitates this process by providing the same interface as  <cit.> to inject load on a CPU under controlled conditions. 

A third motivating scenario directly derives from the necessity to measure the electric energy consumption of code executing inside enclaves, by means of software <cit.> or hardware power meters.
offers a common code-base for both SGX and native contexts, making it easier to isolate the energy requirements of SGX-specific workloads.



  
Roadmap.

The remainder of the paper is organized as follows: <ref> describes some implementation details and lessons learned.
<ref> presents our preliminary evaluation. 
Finally, <ref> discusses future work and concludes.



§ IMPLEMENTATION



is implemented as a fork of (version 0.09.10).
It directly reuses the same compilation and runtime foundations.
From a user perspective, SGX-enabled stressors (specific pieces of executable code that exercise a given CPU functionality or low-level operation) are selected by specifying command-line options starting with .
We concentrate on porting CPU stressing methods to run in an enclave in a way that makes both native and SGX versions comparable from a performance standpoint.
All features offered by the CPU stressors shipped with are supported by , with the exception of partial load.
Specifying a fixed load percentage is not possible (it is locked to 100), because this feature is based on precise timing, which is currently not available inside the enclave.
Our prototype supports 54 enclave-enabled stress methods out of the 68 currently shipped with .
The exhaustive list is presented in <ref>.
We note that support for each stressor depends on the availability of its required functionalities within the enclave, as well as their support by the SDK.








 


  
Porting CPU stressors to run inside an enclave.

The Intel SGX SDK is designed in a way that allows existing code to be ported to run in an enclave with reasonable engineering efforts <cit.>.
We take advantage of this fact to port the CPU stressors of in our own fork.
We detail here the particular complications that we encountered throughout this porting effort, and the specific adjustments applied to solve each of them.

After creating an enclave using the template given by the SDK, we copy the relevant source code for the CPU stressors in the aforementioned template.
defines several macros in its global header file.
It is not possible to include the code verbatim as it depends on numerous system-specific features.
Our solution is to define the needed symbols on a case-by-case basis.

The next obstacle is the need to split the code in trusted and untrusted parts.
We decompose the code in a way that limits the number of enclave transitions (entering and exiting the enclave) required to run a stressor.
Transitions are costly <cit.>, so it is crucial that none happen while stressing is in progress, to ensure a consistent behavior.

The user can gracefully abort the execution of using standard Linux signals such as .
includes a mechanism to catch the majority of signals and react accordingly for its built-in stressors.
We leverage the fact that SGX enclaves can access the untrusted memory of their enclosing process to pass a pointer to the  variable to the enclave.
This variable is later used to indicate when to stop the execution of stressors.
Code running inside the enclave periodically polls the flag, and stops the execution if asked by the user.
The same flag is also used to make a stressor run for a given duration.
Timekeeping is done outside the enclave, with the indication to stop the execution notified by changing the value of the variable.



  
Ensuring byte-per-byte equivalence of native and SGX code.

During the initial testing phase of , we observed vast differences in performance for the same stressor executed in native and enclave modes.
As a matter of fact, while the source code was identical in both instances, the resulting compiled binaries slightly differ.
We believe that these slight differences are inevitable, as different linking rules are needed by each execution mode.

Conveniently, an enclave compiled using the official SGX SDK will produce a statically-compiled shared library.
We leverage this aspect to guarantee that both native and SGX versions of a stressor execute a perfectly identical binary by dynamically linking this shared object.
Choosing this optional approach limits to stressors that are available in enclave mode.





§ EVALUATION


pre-sents

This section presents our preliminary evaluation of .
<ref> lists the SGX machines used for our experiments.
We chose these machines due to their different form factors, hardware features, and widespread market availability. 
We expect them to represent a meaningful sample of SGX machines that are in use nowadays.   
These machines are configured to run Ubuntu Linux 17.10, along with v2.0 of the Intel SGX SDK, PSW and driver.



  
Cost of enclave transitions.
When programming for SGX, it is important to keep in mind that the cost required to enter and exit an enclave is significant <cit.>.
<ref> presents the time needed to perform 100 million enclave transitions on a single core at a time (left) and on all available cores (right).
As expected, the server-class machine is consistently faster than the other ones, while the Intel Compute Stick performs the worst. 
We also observe marginal differences in single-core performance, in which case the processor can run at its maximal turbo frequency.
Given the notable price difference between the various machines, the Intel NUC offers the best price-performance ratio.



  
Cost of latest microcode update. The second suite of microbenchmarks highlights a rather surprising effect of a recent microcode update, recently issued by Intel (on 2018-01-08) to mitigate the Spectre attack <cit.>.
To observe the performance impact of this microcode update, we execute all 54 supported stressors before and after the microcode update.
Using the previous microcode, all stressors display the same performance in SGX and native modes.
Under the updated microcode, on the other hand, we observe a significant difference in SGX versus native performance.
<ref> presents the results for the 27 tests for which SGX performance is affected.
We measure slowdowns up to 3.8× ( running on a single core).
Given the undisclosed nature of microcode updates, it is difficult to identify the root cause of this performance degradation.


 



§ CONCLUSION AND FUTURE WORK



The expanding availability of SGX-enabled machines calls for new tools to evaluate the performance of secure clusters.
The current lack of ready-to-deploy SGX applications usually forces researchers to implement single-use workloads.
This paper presented , an easy-to-use tool capable of stressing SGX enclaves and report on diverse metrics.
We plan to extend our prototype along the following directions: first, we intend to evaluate the performance of reading and writing encrypted memory using the memory stressors of , ported in (not presented in this paper due to lack of space).
Second, we will integrate these stressors within a large-scale monitoring and performance framework for containerized microservices, to easily monitor performance regressions.
 



  
Acknowledgement. The research leading to these results has received funding from the European Union's Horizon 2020 research and innovation programme under the LEGaTO Project (https://legato-project.eu/legato-project.eu), grant agreement No 780681.

splncs04


