APACHE: A Processing-Near-Memory Architecture for Multi-Scheme Fully Homomorphic Encryption
Lin Ding Song Bian Penggao He Yan Xu Gang Qu Jiliang Zhang
=1

















  








 

 
 
 























stheorem
sproof




=8.5in
=11in












arabic


 



1]Lin Ding
2]Song Bian
1]Penggao He
1]Yan Xu
3]Gang Qu
1]Jiliang ZhangCorresponding author: Jiliang Zhang
[1]Hunan University
[2]Beihang University
[3]University of Maryland, College Park


































: A Processing-Near-Memory Architecture for 
Multi-Scheme Fully Homomorphic Encryption
    [
    April 29, 2024
======================================================================================


plain






Fully Homomorphic Encryption (FHE) allows one to outsource 
computation over encrypted data to untrusted servers without worrying
about data breaching. 
Since FHE is known to be extremely computationally-intensive, 
application-specific accelerators emerged as a powerful solution
to narrow the performance gap. Nonetheless, due to the 
increasing
complexities in FHE schemes per se and multi-scheme
FHE algorithm designs 
in end-to-end privacy-preserving tasks, existing FHE 
accelerators often face the challenges of low hardware 
utilization rates and insufficient memory bandwidth. 
In this work, we present , a layered near-memory 
computing hierarchy tailored for multi-scheme FHE acceleration.
By closely inspecting the data flow across different FHE
schemes, we propose a layered near-memory
computing architecture with fine-grained functional 
unit design to
significantly enhance the utilization rates
of both computational resources and memory bandwidth.
In addition, we propose a multi-scheme operator compiler
to efficiently schedule high-level FHE computations
across lower-level functional units.
In the experiment, we evaluate on various FHE 
applications, such as Lola MNIST, HELR, fully-packed bootstrapping, and fully homomorphic processors.
The results illustrate that outperforms 
the state-of-the-art ASIC FHE accelerators by 
2.4× to 19.8× over a variety of
operator and application benchmarks. 




§ INTRODUCTION



With the increasing popularity of cloud-based
storage and computation outsourcing,
fully homomorphic encryption (FHE) <cit.> 
rises as one of the most promising solutions to secure 
data confidentiality while maintaining its usability. 
Equipped with recent advances in fundamental cryptographic 
designs, modern FHE allows an extremely broad spectrum of 
complicated algorithms to be directly applied to ciphertexts, 
without the service provider ever knowing the decryption key.

To further enhance its usability, increasing attention focuses on
designing protocols that adopt multi-scheme FHE constructions. 
In such case, multiple FHE schemes, e.g., TFHE <cit.> and
CKKS <cit.>, are used to (jointly or separately) establish 
privacy-preserving computation protocols,
such as federated learning 
<cit.>,
fully homomorphic processors
<cit.>, private 
medicine <cit.>, and 
private data inquery
<cit.>.
Here, schemes such as CKKS and BFV <cit.> are often used to evaluate polynomial
functions (e.g., additions, multiplications), whereas 
TFHE-like schemes <cit.> are employed to evaluate non-polynomial
functions over ciphertexts.

While protocols based on multi-scheme FHE can be much 
more expressive and usable,  we face
new computational and communicational obstacles 
against the associated architectural designs.
Here, we first categorize basic FHE algorithms into data-heavy 
operators and computation-heavy operators,


as illustrated in <ref>.
Here, we see that multi-scheme-FHE-based protocols 
need to include a much wider range of fundamental FHE operators
to support complex computations across different
privacy-preserving tasks. 
However, as also sketched in <ref>, 
due to varying ciphertexts parameters and 
evaluation key sizes (e.g., bootstrapping keys, 
key-switching keys, rotation keys), FHE operators of different
schemes exhibit distinct data-heavy and computation-heavy 
characteristics, necessitating new explorations in
the architectural design space of FHE accelerators.

In addition, we point out that separating accelerator designs
for BFV/CKKS-like and TFHE-like schemes can suffer from low
hardware utilization rates.
As <ref> illustrates, in
privacy-preserving database applications <cit.>, 
the runtime of TFHE-like scheme dominate the overall protocol latency.
On the other hand, protocols like privacy-preserving neural 
network inference
can be implemented solely based on BFV/CKKS-like schemes, without 
the need of instantiating any TFHE operator.
Consequently, the effective acceleration of multi-scheme
FHE protocols demands further studies on a unified
architecture for the acceleration of both 
computation-heavy and data-heavy operators across 
different FHE schemes.













































































Based on the above analyses, we derive three
essential design principles to enable the multi-scheme
applicability of FHE accelerators: 1) 
layered memory access hierarchy for
the minimization of I/O load, 2) reconfigureble 
functional unit (FU) design for the support
of flexible operator granuality, 
and 3) effective operator scheduling for 
the maximization of data parallelism.
Using such critiques, we evaluated
the qualitative characteristics of existing
FHE accelerators in <ref>.
and draw two main insights.  First, prior 
accelerators <cit.> mostly depend
on a two-level memory hierarchy, namely, the 
off-chip and of-chip memory 
levels with low and high communication
bandwidths, respectively.






For instance, existing BFV/CKKS accelerators, such 
as <cit.>,

utilize High-Bandwidth 
Memory (HBM) to buffer key and ciphertexts data, where
HBM can provide a maximum of roughly 2 TB/s of I/O 
throughput. However, as <ref> highlights, a 
throughput of 8 TB/s or above is required to fully exhaust the computation
resources of a fully pipelined circuit bootstrapping unit for TFHE.
It is obvious that, while simply mounting additional memory bandwidth 
can solve the I/O bottleneck, due to the constraint on 
the area and the computational resources, the expensive 
bandwidth added are mostly 
wasted when executing other FHE operators.
Second, the interconnect topology of existing architectures are 
generally fixed for a given FHE scheme without operator-level 
task scheduling, resulting in less optimal resource utilization 
rates. For example, in <cit.>, a 
BConv unit is explicitly instantiated to accelerate the base conversion 
algorithm in the BFV/CKKS bootstrapping procedure. Yet, since 
TFHE-like FHE schemes generally do not use the residual number system (RNS),
the BConv unit is completely unused when accelerating TFHE, leaving
as much as 30% of chip area unoccupied during task execution.
Hence, a natural question to ask is that, how can we co-design 
the memory and computation architectures to support multi-scheme FHE
operators, where both the memory bandwidth and the computing resources can be fully 
exploited?
















 
 
 




 
Our Contribution: In this work, 
we propose , a processing-near-memory 
architecture designed for general-purpose FHE 
acceleration. First, we leverage a three-level
memory hierarchy to fully exploit the 
internal bandwidth of the DIMM 
units <cit.>.


By lifting the large-volume communication from 
external I/O loads, we are able to avoid mounting an 
excessive amount of expensive HBM units. 
Moreover, we propose a new circuit topology to 
interconnect functional units, such that 
a high hardware utilization rate can be retained
across the diverse set of FHE operators.




The key contributions of our work can be summarized as follows.


    
  * Multi-Level PNM Architecture: 
    Our key observation is that, as illustrated in <ref>,
    data-heavy operators generally have low computational circuit depth.
    Based on such insight, we allocate different functional units into
    a total of three memory levels: 
    the external I/O level, the near-memory level, and 
    the in-memory level. By placing computation-light but 
    memory-heavy units closer (or even into) the memory die,
    
    
    
    we are able to reduce the data external I/O bandwidth 
    by up to 3.15 × 10^5 times. 
    
    
    
    
    
    
    
  * Configurable Internconnect Topology: 
    We propose a configurable interconnect topology with
    fine-grained functional unit designs 
    to simultaneously ensure multi-scheme operator support and 
    high hardware utilization rates. we show that the utilization rates of the (I)NTT FUs can be kept at least 90% across multi-scheme FHE tasks, while that of existing accelerators only range from 50%
    to 85%.
    
    
    
    
    
    
  * Operator- and Task-level Scheduling: 
    To fully exploit the proposed hardware
    topology, we design a scheduler that extracts control 
    and data flow of multi-scheme FHE operators to 
    determine the proper resource allocations and the 
    associated datapath configurations.
    Moreover, we also exploit task-level operator parallelism to 
    further take advantage of the massive parallel processing
    capability of multi-channel dual in-line memory modules (DIMMs).
    
    
    
    
  * Implementation and Evaluation: 
    We thoroughly evaluate the performance of on a
    set of multi-scheme FHE operator and application benchmarks. 
    The results show that achieves 2.4× to 19.8× 
    speedup, compared to the state-of-the-art FHE accelerators.


The remaining part of this paper is organized as follows. 
In <ref>, we first provide
basics on multi-scheme FHE ciphertext types and operator.
Then, we overview our architecture in 
<ref>. Next, we show how to design
and configure the underlying pipelined circuits and memory 
layouts in <ref>. 
Based on the lower-level functional units, we detail the 
scheduling strategy in <ref>.
Subsequently, we provide the detailed experiment setup and results in 
<ref>, and present the related works in <ref>. Finally, we conclude our work in <ref>.




§ PRELIMINARIES






Similar to existing
works <cit.>, we divide existing FHE schemes into
two primary categories: BFV/CKKS-like and TFHE-like. In what
follows, we provide a brief summary on the fundamental 
ciphertext types and key homomorphic operators for 
multi-scheme FHE. 



 §.§ Notations

For basic notations, we use 𝔹, ℝ, and ℤ to 
denote the set of {0, 1}, real numbers, and integers, respectively. Let q/Q/Q' be different sizes of ciphertext modulus, p is for the plaintext modulus, and ℤ_q/ℤ_Q/ℤ_Q' refer to the set of integers modulo q/Q/Q'. 

We use R and R_Q to represent ℤ[X]/(X^N + 1) and ℤ[X]/(X^N + 1) mod Q, respectively, for some ciphertext modulus Q and polynomial degree N. Throughout this paper, we use bold lowercase letters (e.g., 𝐚) for vectors, tilde lowercase letters (e.g., a) for polynomials, and bold uppercase letters (e.g., 𝐀) for matrices. 



 §.§ Multi-Scheme FHE Ciphertext









Most existing lattice-based FHE construct ciphertexts
based on either (or both) the learning with error (LWE) and 
ring learning with error (RLWE) hardness problems <cit.>. 
Based on their algebraic properties, we summarize three fundamental
types of ciphertext formats that cover most FHE applications, namely, 
the LWE ciphertext, the RLWE ciphertext, and the RGSW ciphertext.













    
       
    

- LWE Ciphertext: Here, we use a secret key 𝐬∈𝔹^n to encrypt a single integer message m ∈ℤ_p, and get a LWE ciphertext as follows,
    
    𝖫𝖶𝖤^n,q_𝐬 (m) = (b, 𝐚) = (<-𝐚, 𝐬> + Δ· m + e, 𝐚),

where 𝐚 is chosen uniformly at random, e is a noise sampled from discretized Gaussian, and Δ is a scale factor to protect the least significant bits of the message m from the noises. 

- RLWE Ciphertext: A RLWE ciphertext can be defined by the following equation,
    
    𝖱𝖫𝖶𝖤^N,Q_s (m) = (b, a) = (<-a, s> + Δ·m + e, a).

Here, a vector of messages m∈ R_p is encrypted under the secret key s∈𝔹[X]/(X^N + 1). Similar to the LWE ciphertext, a∈ R_Q is also uniformly and randomly sampled from discretized Gaussian. Δ is a scaling factor. 

- RGSW Ciphertext: A RGSW ciphertext is a matrix consisting of 2d RLWE ciphertexts and is noted as 𝖱𝖦𝖲𝖶^N',Q'_s (m).




















 §.§ Computational Optimizations for FHE
 
To boost the computation of FHE schemes, the current FHE implementations have adopted two main technique: number theoretic transform (NTT) and residue number system (RNS).

NTT and Polynomial Multiplication: In 
FHE, the most expensive routine is the polynomial multiplication.
In most cases, we adopt NTT to reduce the complexity of
polynomial multiplication to 𝒪(N log N), by 
computing a·b = ((a) ∘(b)), where ∘ is coefficient-wise multiplication.


RNS and Base Conversion: In CKKS/BFV-like schemes, 
RNS is used to decompose polynomials with the large
moduli Q which can be hundreds to thousands of bits 
to a set of sub-polynomials that are of smaller moduli
{q_i}. 
CKKS/BFV-like FHE schemes often need to raise or reduce
the moduli, known as the 𝖬𝗈𝖽𝗎𝗉 and 
𝖬𝗈𝖽𝖽𝗈𝗐𝗇 operators. To carry out
𝖬𝗈𝖽𝗎𝗉 and 𝖬𝗈𝖽𝖽𝗈𝗐𝗇, a large 
number of RNS base conversion (𝖡𝖢𝗈𝗇𝗏) operations 
have to be invoked. <ref> shows the process of generating a 
new subpolynomial [a]_p_j from a 
of moduli Q.
    
    𝖡𝖢𝗈𝗇𝗏([a]_Q,p_j): [a]_p_j= (∑_i=0^L-1[[a]_q_i·q̂_i^-1]_q_i·q̂_i) mod p_j,

where L is the number of modulus q_i of Q.
Based on 𝖡𝖢𝗈𝗇𝗏, 𝖬𝗈𝖽𝗎𝗉 and 𝖬𝗈𝖽𝖽𝗈𝗐𝗇 can be defined by <ref> and <ref>, respectively.
    
    𝖬𝗈𝖽𝗎𝗉([a]_Q,P · Q): [a]_p_j = 𝖡𝖢𝗈𝗇𝗏([a]_Q,p_j), 
       𝖬𝗈𝖽𝖽𝗈𝗐𝗇([a]_P · Q,Q): 
       [a]_q_j = ([a]_q_j - 𝖡𝖢𝗈𝗇𝗏([a]_P,q_j))· P^-1 mod q_j ,

where j ∈ [0,M) and M is the number of sub-modulus of P.









 §.§ Multi-Scheme FHE Operators
 

In this subsection, we provide a detailed explanation of the BFV/CKKS-like and TFHE-like homomorphic operators. Meanwhile, we decompose such homomorphic operators into lower-level operators, including (inverse) number theoretic transform ((𝖨)𝖭𝖳𝖳), modular addition (𝖬𝖠𝖽𝖽), modular multiplication (𝖬𝖬𝗎𝗅𝗍), and coefficient automorphism. Considering the computation overhead and data size, we further classify such homomorphic operators into data-heavy and computation-heavy operators, as <ref> illustrates.






























  §.§.§ Homomorphic BFV/CKKS-like Operators



The homomorphic arithmetic evaluation is conducted on the 𝖱𝖫𝖶𝖤 (m) ciphertext over ℝ_Q. The basic homomorphic arithmetic operators are homomorphic addition (𝖧𝖠𝖽𝖽) and homomorphic multiplication (𝖧𝖬𝗎𝗅𝗍).

- Homomorphic Addition (𝖧𝖠𝖽𝖽): 𝖧𝖠𝖽𝖽 of two ciphertexts only requires 𝖬𝖠𝖽𝖽 between the ciphertext polynomials: 𝖱𝖫𝖶𝖤 (m_0 + m_1) = (b_0 + b_1, a_0 + a_1) (mod   Q).




- Key switching (): We use  to transform a 
cipher polynomial a encrypted by s to that of 
s'. 





 includes 𝖬𝗈𝖽𝗎𝗉 and 
𝖬𝗈𝖽𝖽𝗈𝗐𝗇, requiring a large number of , , NTT, and 
INTT operations.

- Homomorphic Multiplication (𝖧𝖬𝗎𝗅𝗍): 
𝖧𝖬𝗎𝗅𝗍 supports the plaintext-ciphertext multiplication 
(𝖯𝖬𝗎𝗅𝗍) and ciphertext-ciphertext multiplication 
(𝖢𝖬𝗎𝗅𝗍). For  calculations, we have 𝖱𝖫𝖶𝖤 (m_0 
) ·m_1 = (b_0·m_1, 
 a_0·m_1) (mod   Q). 
For 𝖢𝖬𝗎𝗅𝗍, we compute 𝖱𝖫𝖶𝖤 (m_0·m_1) = 𝖪𝖾𝗒𝖲𝗐𝗂𝗍𝗁(a_0·a_1, evk) + (b_0·b_1, a_0·b_1 + a_1·b_0) (mod   Q). 

- Homomorphic Rotate (𝖧𝖱𝗈𝗍): 
𝖧𝖱𝗈𝗍 performs a circular left shift by r slots on a ciphertext 𝖱𝖫𝖶𝖤 = (b, a), by
executing 𝖧𝖱𝗈𝗍(𝖱𝖫𝖶𝖤, r, evk^(r)) = 
(ψ_r(a), evk^(r)_rot) + 
(ψ_r(b),0), 
where evk^(r) is a  key for 𝖧𝖱𝗈𝗍 and ψ_r(·) represents the automorphism, i,e, 
the i^th coefficient of a polynomial moves to the slot of i · (5^r) mod N. Low-level operators of 𝖧𝖱𝗈𝗍 includes ψ_r(·) besides these of .


 































	
	
	
 
 
 
 
 
	
 
 
 
 
 
 
 
 
 
	
	
	

- BFV/CKKS Bootstrapping: The 
above 𝖧𝖬𝗎𝗅𝗍 operations 
only support the limited number of homomorphic multiplications. 
For the deeper multiplicative depth, the 𝖻𝗈𝗈𝗍𝗌𝗍𝗋𝖺𝗉𝗉𝗂𝗇𝗀 
is proposed to recover the level budgets of homomorphic 
multiplications. 𝖻𝗈𝗈𝗍𝗌𝗍𝗋𝖺𝗉𝗉𝗂𝗇𝗀 is the most 
complex and expensive
operation in both BFV and CKKS. Due to the space limit, 
we refer readers to <cit.> for further details on
the exact arithmetic characteristics of BFV and CKKS
bootstrapping. 













  §.§.§ Homomorphic TFHE-like Operators



In TFHE, the most fundamental homomorphic operators are the controlled multiplexer (𝖢𝖬𝖴𝖷) and key switching. Moreover, based on both operators, gate bootstrapping (GB) and circuit bootstrapping (CB) have been built to enable various homomorphic functionalities in constructing diverse applications.








- Controlled Multiplexer (𝖢𝖬𝖴𝖷): This operator is the homomorphic counterpart of a multiplexer. 
It outputs a new 𝖱𝖫𝖶𝖤 ciphertext which shares the same plaintext as either input 𝖱𝖫𝖶𝖤 ciphertexts for 𝐜𝐭_0 or 𝐜𝐭_1, depending on the given 𝖱𝖦𝖲𝖶 ciphertext of 𝐂 encrypting a selection bit. The 𝖢𝖬𝖴𝖷 function can be represented as 𝖢𝖬𝖴𝖷 (𝐜𝐭_0, 𝐜𝐭_1, 𝐂) = 𝐂⊡ (𝐜𝐭_0 - 𝐜𝐭_1) + 𝐜𝐭_1, where ⊡ presents an external product. 





- Key Switching (KS): To switch keys and ciphertext formats, TFHE has defined two key-switching flavours, namely the public functional key switching (𝖯𝗎𝖻𝖪𝖲) and private functional key switching (𝖯𝗋𝗂𝗏𝖪𝖲). And the main computation producers can be characterized by the two following equations,

    𝖯𝗎𝖻𝖪𝖲 (f,𝐊𝐒,𝐜) = (0,f(b^1,⋯,b^p)) - ∑_i=1^n∑_j=1^tâ_i,j·𝐊𝐒_i,j,
       𝖯𝗋𝗂𝗏𝖪𝖲 (𝐊𝐒^(f),𝐜) = - ∑_z=1^p∑_i = 1^n+1∑_j=1^tĉ^(z)_i,j·𝐊𝐒^(f)_z,i,j,

where 𝐜 is the set of p 𝖫𝖶𝖤 ciphertexts, and f is
some Lipschitz continuous function. â_i,j is the j^th bit 
of a_i, in which a_i = f(a_i^(1),⋯,a_i^(p)), while 
ĉ^(z)_i,j is the j^th bit of the i^th coefficient in the 
z^th 𝖫𝖶𝖤 ciphertext. 
𝐊𝐒 and 𝐊𝐒^(f) are the key-switching keys, each of which consists of either 𝖫𝖶𝖤 ciphertexts or 𝖱𝖫𝖶𝖤 ciphertexts. 


- Bootstrapping of TFHE: The operations inherently introduce a 
certain degree of noise into the resulting ciphertext, thereby increasing 
the risk of decryption failure. To guarantee the correct decryption, 
bootstrapping is designed to remove the noise at the end of each operation. 


We can combine 
𝖻𝗈𝗈𝗍𝗌𝗍𝗋𝖺𝗉𝗉𝗂𝗇𝗀 and  𝖯𝗎𝖻𝖪𝖲 to construct various 
homomorphic logic gates (𝖧𝗈𝗆𝖦𝖺𝗍𝖾), such as 𝖧𝗈𝗆𝖠𝖭𝖣, 
𝖧𝗈𝗆𝖮𝖱, and 𝖧𝗈𝗆𝖷𝖮𝖱. Similarly, by 
jointly using 𝖻𝗈𝗈𝗍𝗌𝗍𝗋𝖺𝗉𝗉𝗂𝗇𝗀 and 𝖯𝗋𝗂𝗏𝖪𝖲, a circuit 
bootstrapping (CB) functionality can be achieved, which provides a higher 
precision and enhanced flexibility for constructing complex and advanced 
applications, as exemplified by such <cit.>. 






§ MOTIVATION AND ARCHITECTURAL OVERVIEW









 §.§ Challenges and Motivations



Before delving into the architectural details, we first provide
a thorough analyses on the computation and memory characteristics
for existing multi-scheme FHE operators, and discuss the
primary challenges as well as key insights for .

As summarized in <ref>, we focus on the 
most frequently used operators for both BFV/CKKS-like and TFHE-like
schemes, and classify the operators into two types, namely,
computation-heavy and data-heavy. Here, we point out two main 
challenges for accelerating multi-scheme FHE. First, the amount 
of cache data vary by as much as 48× between the operators. 
As a result, existing accelerators with the typical on-chip and
off-chip memory architecture has to either cache all the data on chip 
(which can be impractical for large evaluation keys with up to 1.8 GB
of data) or incur excessive off-chip I/O bandwidth that are left
unused for computation-heavy operators. Second, while most 
high-level operators can be broken down to the a set of basic 
functional units, (i.e., NTT, modular addition and multiplication, 
and automorphism), the exact operator composition and control 
path do vary from one to the other.
However, since existing FHE accelerators generally target on
only one FHE scheme, their circuit topologies are fixed at
design time (e.g., for <cit.>,
FUs are in the pre-determined order of
NTT, modular multiplication, modular addition, and automorphism).
Consequently, some of the FUs are under-utilized, or completely
unused, when executing certain operators
(e.g., 𝖯𝗋𝗂𝗏𝖪𝖲 and 𝖯𝗎𝖻𝖪𝖲 does not
use the NTT units).

Motivated by the above challenges,
our main goal in this work is thus to decouple and
decompose data-heavy and 
computation-heavy FHE operators based using a multi-level 
memory hierarchy with
configurable FU interconnects, so that we can achieve
high hardware utilization rates while significantly
reducing I/O bandwidth.





 §.§ Architectural Overview


As illustrated in <ref>(a),  is comprised 
of two main modules: the scheduling module and the DIMM module.
When a particular FHE program is received by a host CPU, the control and data flow graphs
of the FHE operators are first extracted by the scheduler module. The 
scheduler then breaks down the data dependency between the operators
and construct the task queue, where operator-level parallelism is 
extracted to exploit the computational capabilities of the multi-channel
DRAM module. Next, the operators are fed into the DRAM module, where
the majority of the data buffering and task execution happen. 
Lastly, the execution results for each of the operators
in the task are naturally stored inside the DRAM module for
the execution of the subsequent operators.
Hence, the overall goal of is to significantly reduce
the off-chip I/O loads for multi-scheme FHE acceleration
by properly allocating functional units into the DRAM module.

To carry out highly complex cryptographic operations inside the 
DRAM module, computing and storage elements are 
carefully organized to form three fundamental 
memory levels:  the external I/O level, the near-memory computing level, 
and the in-memory computing level. In what follows, we briefly summarize 
the memory and computing topology in each of the layers.

192 I/O Level: First of all, on the I/O level,
we expose the set of operators that constitute the 
application programming interface (API) of the overall
accelerator, including all the FHE operators 
described in <ref>.
When executing an FHE operator, 
we assume that all ciphertexts data 
and evaluation keys are pre-loaded into the DIMMs, which
we believe is a reasonable assumption since 
the same thing holds true for the regular computation 
model. 

193 Near-Memory Computing Level: 
The NMC level is where most of the low-level FUs locate.
We have three main components here: i) the standard
memory chips, ii) the NMC module, and iii) the modified
in-memory computing module. First,
for the native memory ranks,
we employ the standard ×8 DRAM chips, where
the internal architecture of these DRAMs remain unchanged. 
The only difference here is that the data bus 
of each rank is connected separately to one of the data
buffers inside the NMC module. 
 
By parallelizing the data bus of multiple DRAM ranks, 
provides large internal bandwidth to the NMC module, 
such that an excessive amount of I/O communication can be avoided. 
Inside the NMC  module, we incorporate configurable 
FUs implementing the respective underlying operators, and the FUs
are structured with a flexible 
interconnect topology to accommodate various data flows of FHE 
operators efficiently. 

194 In-Memory Computing Level: 
As sketched in <ref>(b) and <ref>(c), 
we place application-specific
circuits on the in-memory computing level to mainly reduce the bandwidth 
bottleneck introduced by the TFHE key switching algorithms, namely,
the  and  operators. As illustrated in 
<ref>,
the sizes of keys needed to be loaded on-chip is extremely
large for  and , while the computation circuit
is extremely shallow (only a couple of adders). Hence, 
we insert the accumulation adders at the bank level of the
×8 DRAM chip, where the evaluation keys are pre-loaded. 


 








§ DESIGN EXPLORATION OF FUNCTIONAL UNITS



To design a near-computing module for multi-scheme FHE, we see two significant design challenges. First, 
as illustrated in <ref>(a) and (b),
TFHE-like and BFV/CKKS-like schemes exhibit distinct data and control flows. Consequently, 
regardless of which scheme we use as the reference for the NMC 
design, there will always be inefficiencies when the other scheme is used.
Second, within the BFV/CKKS-like schemes, many tasks only adopt the
 and  operators without invoking NTT. 
As a result, if a the NTT, , and  circuits pipelined together,
the NTT modules become idle when only
 and  are called. 
To tackle the above challenges, we propose
a flexible NMC module design with configurabilities on both the interconnect
level and the internal FU level.






 §.§ The NMC Module and its Interconnection













Here, we first describe the instantiated FUs in the NMC module, and then discuss
how the FUs need to be interconnected to form the overall NMC architecture.
As outlined in <ref>, the NMC module
is composed of three subcomponents, namely, the interconnect controller,
the NMC core, and the data buffer. During the execution of an FHE
operator, the associated data are first loaded into the data buffer.
Then, the data path in the NMC core is properly configured by the
controller to execute the exact computations. Within the NMC core,
we have six FUs: the register file, the decomposition
unit , the (I)NTT unit , the modular multiplication
unit , the modular addition , and the automorphism
unit . The  module can implement the NTT and INTT operators by feeding different twiddle factors, respectively.

While the designs of the FUs themselves are close
to those of the existing works <cit.>,
the interconnects need to be completely re-designed due to the
complex arithmetic characteristics of the multi-scheme FHE operators.
First, we observe that the -- 
routine is one of the most frequently used control flows for both 
BFV/CKKS and TFHE, 
as depicted in <ref> (a) and (b). 
Furthermore, it is observed that automorphism and decomposition operators 
are comparatively less utilized in both schemes, thus requiring lower 
bandwidth. Based on this observation, we propose a configurable interconnect 
topology for our NMC modules, as depicted in 
<ref>. This topology incorporates 
two separate pipelines: the
-- routine and the - routine. The 
former routine, coupled with a central register file of 8MB, is capable of
executing operators such as 𝖢𝖬𝖴𝖷 of 
𝖧𝗈𝗆𝖦𝖺𝗍𝖾 for TFHE-like schemes and 𝖢𝖬𝗎𝗅𝗍 of BFV/CKKS-like schemes. 
Conversely, the latter routine can efficiently compute
𝖧𝖠𝖽𝖽 and 𝖯𝖬𝗎𝗅𝗍 without interrupting the  pipeline. 
Notably, the central register file ensures adequate internal bandwidth for the 
-- routine, while the register file of 1MB
supplies operands to the - routine. 
Concretely, we can formulate the utilization rate of the
 FU for a single fixed pipeline routine and our
configurable interconnect topology by <ref> and 
<ref>, respectively.

    𝐔𝐭𝐥_NTT = 𝐓_ALL-𝐓_nonNTT/𝐓_ALL, 
       𝐔𝐭𝐥'_NTT = R1.𝐓_ALL - R1.𝐓_nonNTT/R1.𝐓_ALL⋃R2.𝐓_ALL,

where 𝐓_ALL stands for the overall latency of a routine, 
and R1.𝐓_ALL⋃R2.𝐓_ALL is the 
union of timing segments of the two pipeline routines R1 and R2. 
The union runtime is no bigger than 𝐓_ALL. Meanwhile, it 
holds R1.𝐓_nonNTT≥ 0.5 ×𝐓_nonNTT (resp. R1.𝐓_nonNTT =
𝐓_nonNTT) for / (resp. 𝖢𝖬𝖴𝖷). 
Thus, we can observe a significant improvement in the utilization rate of 
the  FU for both CKKS/BFV-like and TFHE-like schemes for the 
proposed interconnect topology. 















 §.§ FU Configuration in the NMC Module

As illustrated in Table II, a general-purpose FHE accelerator must accommodate 
operands of 32 and 64 bits. However, existing FHE accelerators predominantly 
confine modular multiplication, addition, and (I)NTT circuits to fixed bitwidths. 
For example, BTS <cit.>, ARK <cit.>, and Strix <cit.> uses a fixed bit width of 64
for the (I)NTT/FFT designs. Such limitations inevitably result in notable 
resource under-utilization  when handling 32-bit operands. 
To accommodate the pivot bit-widths, we have designed 
and integrated the FUs with configurable bitwidth within our . 

(1) Configurable 𝖬𝖬𝗎𝗅𝗍 and  circuits (<ref>): 
𝖬𝖬𝗎𝗅𝗍 <cit.> and 𝖬𝖠𝖽𝖽 
<cit.> play the critical roles in the construction of 
various low-level operators and homomorphic operators, such as , 
𝖧𝖬𝗎𝗅𝗍, 𝖢𝖬𝖴𝖷, and high-level operators
such as bootstrapping. Therefore, all modular multiplication and
addition units in the proposed architecture have configurable
operand bit width that can be flexibly switched between the
64-bit operand mode and 32-bit operand mode, where one execution
of the multiplier (or adder) produces one 64-bit output
or two 32-bit outputs.






The structure of the proposed configurable  is depicted in 
<ref>. 
First, we point out that a 64-bit adder can be easily divided into
two 32-bit adders by cutting the pipeline at the carry bit. 
Second, our key observation is that the Karatsuba multiplier processes a k-bit 
multiplication using three k/2-bit multipliers. Consequently, by 
modifying the structure of a 64-bit Karatsuba multiplier, it becomes feasible 
to compute either a 64-bit multiplication or at least two parallel 32-bit 
multiplications. More concretely, 
we can use three k-bit multipliers and three k-bit subtractors 
to construct a k-bit   <cit.>. Hence, we arrange 
our configurable 64-bit adders and 
64-bit multiplier to construct the 64-bit configurable  as 
shown in <ref>. Lastly, we can simply
use the 64-bit configurable adder and subtractor to implement a configurable
 FU.










(2) Configurable NTT FU: The computation of (I)NTT 
<cit.> is akin to the Discrete Fourier 
Transform (DFT) <cit.>, but operates over a integer polynomial 
ring. In existing FPGA and ASIC accelerators of FHE, the operand bitwidths of the 
customized (I)NTT circuits are specific to 28 bits, 32 bits, 64 bits, or others. 
Nevertheless, due to the substantial hardware overhead associated with implementing (I)NTT, 
it is impractical to directly incorporate multiple (I)NTT FUs for all possible operand 
bitwidths. Hence, the most viable solution is to devise the configurable (I)NTT module. In 
this work, we utilize our configurable  and  circuits to construct the (I)NTT
FUs. By such an approach, we can ensure that one configurable (I)NTT circuit can be 
flexibly configured into either one high bit-width (up to 64 bits) (I)NTT FU or two 
parallel low bit-width (each supporting up to 32 bits) (I)NTT FUs.


(3) Automorphism (<ref>): In addition to supporting multiple operand bitwidths, the automorphism unit needs to address a new design challenge posed by the distinct coefficient rotation rules in the CKKS and TFHE encryption schemes.
Specifically, in CKKS, the coefficient shift is determined by τ = i · k    N.
<cit.> has illustrated that implementing automorphism for BGV and CKKS can be achieved effectively by utilizing four permute/transpose operations within SRAM buffers.
In contrast, in TFHE, we have a fixed automorphism of τ = i + k 
   2N for computing the operation of 
X^-𝐚_i·𝐀𝐂𝐂.
It is observed that automorphism for TFHE can be implemented using only a few shift registers and a single SRAM buffer.
Thus, we can see that there is an inevitable disparity between the automorphism operators employed in CKKS and TFHE. 







§ DATA AND TASK SCHEDULING



As mentioned, for the general acceleration of multi-scheme FHE, the design 
of an efficient and effective scheduling strategy is essential to 
facilitate data reuse and minimize unnecessary data movement <cit.>. 
Moreover, in order to exploit the massive parallelism of
multiple DIMM cards, the mapping and aggregation intermediate results
has to be carefully treated to avoid excessive inter-DIMM communication.



To address the operator and data scheduling issue, we define
scheme-specific scheduling strategies at both the FHE application
level (i.e., task-level) and multi-scheme operator level.


Based on the scheduled data and control sequences, we generate 
the micro-instructions to configure the respective FUs and
the overall execution process.








 §.§ Task-Level Scheduling



For task-level scheduling, consider two basic cases of homomorphic 
evaluation tasks: those without a specific ciphertext sequence and 
those with. In both cases, the plaintext content remains undisclosed 
to the service provider. It is important to emphasize that 
more complex tasks can be constructed by the combinations of the
above two cases.

When there is no explicit data dependency, the ciphertext can be processed concurrently in multiple DIMMs. <ref> (a) shows a demo involving two DIMMs in the absence of specific ciphertext dependency. In this case, the same homomorphic operators (i.e., tree-shape 𝖢𝖬𝖴𝖷 operators in the example of <ref> (a)) would be executed within each DIMM, independently. Then, only the local result is transmitted to another DIMM to obtain the aggregation outcome. 
In another case, the homomorphic task exhibits a specific order, as illustrated in <ref> (b). If the amount of ciphertext operands does not exceed the storage capacity of a single DIMM, we would process the task in a single DIMM. Conversely, if the storage capacity is exceeded, the local result of DIMM 0 is transmitted to DIMM 1 as the input to launch the subsequent evaluation.




By implementing the above scheduling strategy, primary data movements 
occur within each DIMM. Only small local results are communicated across 
DIMMs via the host bus. Although our scheme significantly reduces the 
IO load compared to AISC, FPGA, and GPU, the data transmission still 
hinders the underlying circuits to maintain the full pipeline work for a 
single task. This is because branch prediction 
<cit.> and out-of-order execution mechanisms 
<cit.> of the host CPU render that the local results are 
delivered at a random time point and in order. An important insight is that 
it is unnecessary to complete one end-to-end task before commencing the 
next one. Instead, servers commonly receive massive tasks, and thus we can 
perform other tasks while awaiting the delivery of local results, as 
<ref> (c) and (d) shown. Namely, using the data scheduling 
strategy and the task breakdown, we can ensure that our DIMMs provide 
the capability of parallelism and keep computing fully pipelined.



 §.§ Operator-level Scheduling

Once the task sequence is determined, the next step is to organize the 
homomorphic operators in a systematic manner to achieve high
FU utilization rates and data reuse.





As mentioned in <ref>, at this stage, if the 
homomorphic operators that share the same evaluation
key (e.g., evk of CKKS 
or 𝖯𝗋𝗂𝗏𝖪𝖲 key of TFHE), such operators are clustered
to be executed together. Based on the operator clustering results, we 
determine the operator batching size and the exact order of operator
execution. Finally, the scheduler generates the sequences of 
the micro-instructions 
along with the associated data to the DIMM module. In the follows, 
we detail the operator-level scheduling for
BFV/CKKS-like and TFHE-like schemes in a fine-grained manner.

Scheduling CKKS/BFV-like Operators: For BFV/CKKS-like schemes, 
we observe that directly executing operators according to the task-level
schedule can lead to low resource utilization rates. For example, as 
<ref>(b) shows, 𝖢𝖬𝗎𝗅𝗍 and  
can be decomposed into a sequence of 𝖬𝗈𝖽𝗎𝗉 
(174-175-176-177)
and 𝖬𝗈𝖽𝖽𝗈𝗐𝗇 
(178-179-180). 
If we map the 174 and 175 operators
to the -- routine for the 
𝖬𝗈𝖽𝗎𝗉 operation, the  and 
 FUs experience pipeline bubbles after the  execution
in 176. Similarly,  pipeline bubbles occur for  and 
/ during the execution of  and 
𝖬𝗈𝖽𝖽𝗈𝗐𝗇. To mitigate this issue, 
the scheduler rearranges and 
divides the underlying operators into three groups: (174-175
), (176-177), and 
(178-179). This approach 
can significantly reduce the occurrence of pipeline bubbles, 
generating only one instance of 
 bubble when executing
176-177.
Moreover, to increase the reuse of operand data 
(such as the 𝖬𝗈𝖽𝗎𝗉 base for 174-175 
and the evaluation key for 177), we batch
operators on the group level, instead of the operator level.





Scheduling for TFHE-like Operators: 
In the TFHE scheme, the bootstrapping key needs to be reused for 
gate and circuit bootstrapping. To reduce the memory bandwidth demand, 
we adopt the design in <cit.>, where a batch of ciphertexts 
are processed by the i^th 𝖢𝖬𝖴𝖷 in parallel before 
iterating to the i+1^th 𝖢𝖬𝖴𝖷, to ensure the fully reuse of 
BK_i. For a single 𝖢𝖬𝖴𝖷, the data path
is configured as illustrated in <ref>
to keep the utilization rates of the  FU as high as possible. 
Concretely, a 𝖢𝖬𝖴𝖷 needs to decompose the input polynomial 
into d sub-polynomials (d = 2 in <ref>). 
The decomposed polynomials fed into the  FU followed by
a multiplication of BK_i in the subsequent  FU,
where BK_i is the i-th decomposed TFHE bootstrapping key.
Since one decomposed polynomial needs is multiplied by two 
different shares of BK_i, the two separate -
pipeline routines shown in <ref> are simultaneously
activated,
where the  completes the computation of a 𝖢𝖬𝖴𝖷
by accumulating the d partial products.
When the last accumulation of the batch of i^th 𝖢𝖬𝖴𝖷 is 
completed, we then call the  FU again to perform the INTT and
generate the final result.
Finally, we note that our scheduling scheme is compatible with a wide range
values of d with a high resource utilization.
Whereas, in many existing TFHE accelerators, such as MATCHA <cit.>, 
FPT <cit.>, and Strix <cit.>, the NTT and INTT FUs 
are configured specifically to d : 1, which leads to a 
under-utilization when d varies.











 §.§ Data Packaging and Parallelism Extraction


As discussed in <ref>, all FHE 
schemes operate over three different types of ciphertexts: 
, , and 𝖱𝖦𝖲𝖶. When <ref> holds, we pack t  ciphertexts into an  for an efficient transfer across DIMMs, 




    𝐓_Pack(t·𝖫𝖶𝖤) + . 𝐓_Transfer≤ t ·.𝐓_Transfer.

Here, we take 
the  ciphertext as an example to illustrate how we manage data across 
DIMMs. As <ref> shows, there are three data packing 
methods of  ciphertext: vertical packing, horizontal packing, and mixed 
packing. 




Vertical Packing: Vertical packing is commonly used in 
applications that analyze and compare input samples within a
given dimension (i.e., number of samples). For this class of 
applications, we ensure that the 
ciphertexts of the same dimension are stored within the same
DIMM as much as possible.
In this way, the computation of different dimensions of input
samples can be parallelized in 
multiple DIMMs. Subsequently, the computed results of all dimensions are 
passed onto a DIMM for the final data aggregation.

Horizontal Packing: This method packs all the features of the same 
sample in an  ciphertext. However, in many applications, the number of plaintext 
samples rarely reaches the lattice dimension, i.e., 2^12 to 2^16. 
Hence, an  ciphertext can be made to contain
multiple plaintext samples. The data delivery overhead of this 
type of packing depends 
on requirements of the application. For example, when computing the 
iteration of the K-means clustering <cit.>, we need to transfer 
the  ciphertexts of K centers and K distance sums, where 
each sum is the distance between a sample and a center. However, 
if the Euclidean distance is required 
for all sample pairs, the communication overhead approaches
𝒪(# sample^2). 

Mixed Packing: Mixed packing divides a multi-dimensional data matrix 
into multiple sub-matrices and encrypts one or more sub-matrices into an 
 ciphertext. Similar to vertical packing, our system stores the 
 ciphertexts of the same features in a DIMM to achieve high parallelism 
efficiency. 








§ IMPLEMENTATION AND EVALUATION





 §.§ Implementation and Setup

In this section, we present the configurations used in the DIMM. The 
simulation framework is designed with the following components: 
1) Simulation of FHE functionality: A behavioral level simulator is 
developed based on the TFHE and CKKS schemes. This simulator not only performs 
FHE operations but also generates the physical memory access trace required 
for further analysis. 2) Trace-based Memory Simulator: To simulate 
the cache and DRAM behaviors in each DIMM, we employ NVsim <cit.>, 
CACTI <cit.>, and trace-based simulators Ramulator <cit.>, all 
of which are the widely-used tools for memory analysis. In this work, we set 
the cache and memory fabricated under the 22 nm technology node. 
3) NMC Module Analysis: For the NMC module, we estimate its latency, 
area, and power characteristics by employing Synopsys Design Compiler with 22 
nm technology library at a frequency of 1 GHz. By combining the above four 
components along with a workload distribution by our compiler, we are able to 
estimate the end-to-end performance of . The DIMM configuration and 
simulation of hardware overhead are illustrated in <ref>.











 §.§ Baselines and Benchmarks




For comparison purposes, we evaluate with the state-of-the-art FPGA 
<cit.>, as well as six ASIC FHE accelerators 
<cit.>. Meanwhile, we have done a 
evaluation for Circuit bootstrapping on the Strix <cit.> and 
Morhpling <cit.> with each max bandwidth available for the 
𝖯𝗋𝗂𝗏𝖪𝖲 key. To facilitate comprehensive evaluation and comparison, we incorporate various CKKS and TFHE operators, along with five distinct FHE applications. All assessments adhere to a 128-bit security assumption if no additional declaration is provided. Specifically, in the evaluations of homomorphic operators, TFHE parameters conform to <cit.>, while those of CKKS remain consistent with <cit.>.





 
 



1) Mini-benchmarks: To assess the performance of on 
individual FHE operators across different schemes, we 
adopt 𝖢𝖬𝗎𝗅𝗍, 
𝖧𝖠𝖽𝖽, 𝖯𝖬𝗎𝗅𝗍, , and , 
𝖧𝗈𝗆𝖦𝖺𝗍𝖾, and circuit bootstrapping as the
set of mini-benchmarks.

2) CKKS benchmarks: In this experiment, we employ three 
CKKS-based schemes for evaluation: Logistic Regression (LR), Lola 
MNIST <cit.>, and fully-packed bootstrapping. First, for LR, 
we use HELR 
<cit.> where a 196-element weight vectors is
trained by 32 iterations. The reported metric is the average 
execution time per iteration. Second, we execute the Lola MINST <cit.>, 
a homomorphic Neural Network based on CKKS, with the same
parameters as in <cit.>. Lastly, we benchmark
the fully-packed CKKS bootstrapping <cit.>.

3) TFHE and mixed benchmarks: For TFHE benchmarks,
we use the homomorphic circuits in <cit.> to benchmark
the performance of end-to-end TFHE applications. In particular,
<cit.> incorporates 
a five-stage pipeline processor with a various of
logic gates such as 𝖧𝗈𝗆𝖭𝖠𝖭𝖣 and 𝖧𝗈𝗆𝖷𝖮𝖱, along with 512 bytes of 
ROM and RAM that are built out of 𝖢𝖬𝖴𝖷's.

HE^3DB <cit.> is the state-of-the-art homomorphic database based on both CKKS and TFHE schemes, supporting the logic and arithmetical data querying and analysis. 
We performed an evaluation of our on “TPC-H Query 6” (i.e., a classical database workload benchmark) of HE^3DB. 














 §.§ Performance and Comparison
 






































 








 
 
 


 




As shown in <ref> and <ref>, we present the performance of architecture for in terms of FHE operators and applications. 
Results are showcased for parallel acceleration using 2/4/8 DIMMs and 
are compared against multiple FHE accelerators, 
including 


Poseidon <cit.>, F1 <cit.>, CraterLake <cit.>, BTS <cit.>, ARK <cit.>, SHARP <cit.>, Strix <cit.>, Morphling <cit.>, and MATCHA <cit.>. 

Performance of FHE Operators: First of all, <ref> shows the remarkable fact that only our can support CKKS and TFHE, while all the comparison work can only perform one of both. 




Moreover, our architecture achieves significant advantages over comparable solutions, when executing TFHE operators. Even with only two parallel DIMMs, we achieve remarkable performance, surpassing the recent work Strix (resp. Morphling) by 6.69× (resp. 3.4×) and 6.75× (resp. 3.38×) for 80-bit and 110-bit security, respectively. When implementing the circuit bootstrapping of 128-bit security, our attains the 19.08× and 6.7× speedup than Strix and Morphling, respectively. 

The significant edge emanates from various design considerations. Primarily, 
our operator-level scheduling ensures the high utilization of (I)NTT FUs, 
which is significantly higher than existing works, where the NTT/FFT units 
often remain idle. Additionally, while the parts of accelerators employ 
fixed 64-bit FFT/NTT (e.g., 64-bit FFT adopted in Strix), our APACHE allows 
each 64-bit NTT FU to be configured into dual 32-bit NTT FUs, thereby 
enhancing resource utilization for 32-bit HomGate. Moreover, we decouple 
𝖯𝗋𝗂𝗏𝖪𝖲 and 𝖯𝗎𝖻𝖪𝖲 from computation clusters and process 
both directly within specific ×8 ranks, reducing bandwidth load by 
3.15 × 10^5 times and 3.05 × 10^4 times, respectively. 
In constrast, Strix (resp. Morphling) takes around 24 ms (resp. 7.7 ms) 
for loading 𝖯𝗋𝗂𝗏𝖪𝖲 key of 1.8 GB when batching 64  
ciphertexts for circuit bootstrapping.
  




Full-system Performance: As <ref> shows, our with 8 DIMMs achieves a speedup of 2.4× and 2.5× for Lola MNIST with encrypted and unencrypted weights, respectively. Moreover, our DIMM × 8 is 8.04× and 15.63× faster than the baseline of BTS for the CKKS schemes of fully-packed bootstrapping and HELR, respectively. 
APACHE has achieved such improvement mainly due to the flexible interconnect topology and fine-grained operator scheduling. Concretely, such real applications of CKKS commonly include a large number of 𝖯𝖬𝗎𝗅𝗍 and 𝖧𝖠𝖽𝖽 independent of NTT. Our allows both operators to be (partially or totally) executed in the pipeline routine 2. Thus, our APACHE significantly improves the utilization of NTT FUs and the overall performance. When carries out the TFHE-based VSP, we observe 18.68× and 6.8× speedup against Strix and Morphling, respectively. The VSP relies on expensive circuit bootstrapping to generate GSW-format addresses. Due to the optimization mismatch in circuit bootstrapping over Strix and Morhpling, our exhibits a more competitive performance in VSP. 
Furthermore, <ref> demonstrates that our is the only work that supports HE^3DB <cit.> based on both TFHE and CKKS schemes, suppressing CPU by 2,304×.

 




 §.§ Architectural Analysis






Utilization Rates of FUs: As presented in <ref>, 
our demonstrates high utilization for hardware resource. 
The utilization of the  FU always keeps above 
90%, while that of existing accelerators only range from 50% to 
85%. The high utilization rate indicates that the proposed interconnect 
topology and operator scheduling strategy can effectively reduce the running
time of NTT-independent operations. Moreover, the high utilization rates of
 also means that our 
the NMC module is supplied with sufficient bandwidth, so 
that the proposed framework can almost fully exploit the
hardware resources of the  FU. 
In addition, we observe that the utilization of the in-memory
computing KS module is around 50%. 
The effective utilization of the KS module indicates that our 
multi-level storage architecture effectively reduces the bandwidth 
caused by loading the key switching key.

Remark on Data Communication across DIMMs: In addition to the circuit utilization and local bandwidth load, another impact determinant of APACHE performance lies in the latency associated with data scheduling across DIMMs. It is pointed out that our APAHCE searches the aggregation point to minimize the data volume that needs to be addressed. Consequently, the time required for data propagation across DIMMs is substantially smaller than that needed by local computations over each DIMM. For example, a APACHE DIMM need 0.38 ms to read a  ciphertext from the VSP RAM, in which 512  ciphertexts are stored. By contrast, forwarding the readout  cipher to another DIMM via the host only requires 0.31 us with given 30GB/s of I/O bandwidth. That is, our APACHE can cover the communication latency within computation duration. 




§ RELATED WORKS



Near/in-memory Computing for FHE Acceleration: Recent studies have proposed to use near/in-memory computing (N/IMC) to optimize FHE by reducing data transfers <cit.>. For instance, ReRAM-based logic gates were used in <cit.> to create extensive pipeline architectures for polynomial operations and the full FHEW scheme <cit.>. 
Alternatively,<cit.> and <cit.> deployed logic circuits adjacent to SRAM sense amplifiers for the schooltext polynomial and NTT computations, respectively. However, these N/IMC approaches have no discussed dividing and conquering FHE operators as data-heavy and computation-heavy traits, leading to lower performance versus ASICs or restricted compatibility with multi-scheme FHE tasks.

GPU-based FHE Acceleration: Research involving GPUs has explored accelerating FHE <cit.>, with notable efforts like using NVIDIA Tesla V100 in <cit.> achieving a 257× speedup over CPUs. Other initiatives include integrating cryptographic functions into GPU operations <cit.> and developing open-source libraries <cit.> for agile FHE development. However, as the general-purpose platform, GPUs lack the specialized hardware of FHE operators, thereby leading to inferior performance compared to FPGA and ASIC.


ASIC-based FHE Acceleration: F1 <cit.> is the first programmable FHE ASIC designed for CKKS/BFV-like schemes, surpassing CPU by over 17,000×. 
Building upon F1, CraterLake <cit.> enables unbounded multiplicative depth by architecture optimization for bootstrapping. 
Further advancements include BTS <cit.> and ARK <cit.>, which enhance performance through optimal parameter selection and key reuse, respectively.  
More recently, SHARP <cit.> built a 36-bit FHE accelerator, attaining high performance with a smaller area and power consumption. Meanwhile, with much effort paid into TFHE acceleration, we also see that the performance of HomGate has improved from 10,000 OP./S <cit.> to 74,000 OP./S <cit.> to the current 147,615 OP./S <cit.>. However, all the above ASIC chips are efficient just on either CKKS/BFV-like schemes or TFHE-like ones.

 
FPGA-based FHE Acceleration: Previous studies have explored FPGA-based acceleration of FHE for specific HE operations <cit.> or leveled HE schemes <cit.>. Recently, literature has introduced FPGA-based approaches for FHE <cit.>. For instance, FAB <cit.> leverages a well-designed datapath to optimize key-switching and effectively utilize its 43 MB on-chip storage. Poseidon <cit.> focuses on fine-grained decomposition of FHE operators,  achieving efficient resource utilization. Moreover, FPT <cit.> stands out as the fastest FPGA accelerator for 𝖧𝗈𝗆𝖦𝖺𝗍𝖾, attaining a 946× speedup over CPU. However, FPGAs generally under-perform when compared to the latest ASIC accelerators of FHE, due to limited on-chip resources.








§ CONCLUSION


In this section, we present a near-memory computing architecture for the acceleration of both BFV/CKKS-like
and TFHE-like FHE schemes. By classifying the key operators of both two FHE lanes into data-heavy and 
computation-heavy operators, we point out that, rather than simply piling up computing resources,
allocating the FUs on the correct memory level is the key in effectively 
accelerating multi-scheme FHE.
Based on such insight, we propose that allocates fully pipelined circuits on both near-memory
and in-memory levels with configurable FU topology to improve the overall hardware utilization rates. 
Through rigorous experiments, we show that can reduce the external I/O bandwidth
by as much as 3.15 × 10^5. 





Finally, we evaluate with multiple FHE applications and prove 
that it behaves better than state-of-the-art ASIC accelerators. 













10

url@samestyle








lattigo
“Lattigo v5,” Online: <https://github.com/tuneinsight/lattigo>, Nov. 2023, ePFL-LDS, Tune Insight SA.

FAB
R. Agrawal, L. de Castro, G. Yang, C. Juvekar, R. T. Yazicigil, A. P. Chandrakasan, V. Vaikuntanathan, and A. Joshi, “FAB: an fpga-based accelerator for bootstrappable fully homomorphic encryption,” in IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2023, pp. 882–895.

BalkanCryptSec2015
S. Akleylek, Ö. Dagdelen, and Z. Y. Tok, “On the efficiency of polynomial multiplication for lattice-based cryptography on gpus using CUDA,” in Cryptography and Information Security in the Balkans - Second International Conference, vol. 9540, 2015, pp. 155–168.

CHES2018
A. A. Badawi, B. Veeravalli, C. F. Mun, and K. M. M. Aung, “High-performance FV somewhat homomorphic encryption on gpus: An implementation using CUDA,” IACR Transactions on Cryptographic Hardware and Embedded Systems, no. 2, pp. 70–95, 2018.

CAV2022
R. Banno, K. Matsuoka, N. Matsumoto, S. Bian, M. Waga, and K. Suenaga, “Oblivious online monitoring for safety LTL specification via fully homomorphic encryption,” in 34th International Conference on Computer Aided Verification (CAV), vol. 13371, 2022, pp. 447–468.

FPT
M. V. Beirendonck, J. D'Anvers, and I. Verbauwhede, “FPT: a fixed-point accelerator for torus fully homomorphic encryption,” IACR Cryptol. ePrint Arch., p. 1635, 2022.

HE3DA
S. Bian, Z. Zhang, H. Pan, R. Mao, Z. Zhao, YierJin, and ZhenyuGuan., “He^3db: An efficient and elastic encrypted database via arithmeticand-logic fully homomorphic encryption,” in Proceedings of the 2023 ACMSIGSAC Conference on Computer and Communications Security (CCS), 2023.

Lola
A. Brutzkus, O. Elisha, and R. Gilad-Bachrach, “Low latency privacy preserving inference,” in Proceedings of the 36th International Conference on Machine Learning (ICML), 2019.

TC2016
X. Cao, C. Moore, M. O'Neill, E. O'Sullivan, and N. Hanley, “Optimised multiplication architectures for accelerating fully homomorphic encryption,” IEEE Transactions on Computers, vol. 65, 2016.

Chatterjee2019
A. Chatterjee and K. M. M. Aung, FURISC: FHE Encrypted URISC Design, 2019, pp. 87–115.

cryptoeprint:2015/699
A. Chatterjee and I. Sengupta, “Furisc: Fhe encrypted urisc design,” Cryptology ePrint Archive, Paper 2015/699, 2015. [Online]. Available: <https://eprint.iacr.org/2015/699>


ASPDAC2022
C. Chen, C. Shen, and J. Zhang, “Lightweight and secure branch predictors against spectre attacks,” in 27th Asia and South Pacific Design Automation Conference, (ASPDAC), 2022, pp. 25–30.

TFHEBoot2018
J. H. Cheon, K. Han, and M. Hhan, “Faster homomorphic discrete fourier transforms and improved FHE bootstrapping,” IACR Cryptol. ePrint Arch., 2018.

CKKS
J. H. Cheon, A. Kim, M. Kim, and Y. S. Song, “Homomorphic encryption for arithmetic of approximate numbers,” in 23rd International Conference on the Theory and Applications of Cryptology and Information Security, vol. 10624, 2017, pp. 409–437.

DBLP:journals/tifs/CheonKK16
J. H. Cheon, M. Kim, and M. Kim, “Optimized search-and-compute circuits and their application to query evaluation on encrypted data,” IEEE Transactions on Information Forensics and Security, vol. 11, no. 1, pp. 188–199, 2016.

tfhe
I. Chillotti, N. Gama, M. Georgieva, and M. Izabachène, “Faster fully homomorphic encryption: Bootstrapping in less than 0.1 seconds,” in 22nd International Conference on the Theory and Application of Cryptology and Information Security, vol. 10031, 2016, pp. 3–33.

cuHE
W. Dai and B. Sunar, “cuhe: A homomorphic encryption accelerator library,” in Cryptography and Information Security in the Balkans - Second International Conference, vol. 9540, 2015, pp. 169–186.

CHET_PLDI2019
R. Dathathri, O. Saarikivi, H. Chen, K. Laine, K. E. Lauter, S. Maleki, M. Musuvathi, and T. Mytkowicz, “CHET: an optimizing compiler for fully-homomorphic neural-network inferencing,” in Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation, (PLDI), 2019, pp. 142–156.

NVSim
X. Dong, C. Xu, Y. Xie, and N. P. Jouppi, “Nvsim: A circuit-level performance, energy, and area model for emerging nonvolatile memory,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 31, no. 7, pp. 994–1007, 2012.

FHEW
L. Ducas and D. Micciancio, “FHEW: bootstrapping homomorphic encryption in less than a second,” in Advances in Cryptology - EUROCRYPT 2015 - 34th Annual International Conference on the Theory and Applications of Cryptographic Techniques, 2015.

BFV
J. Fan and F. Vercauteren, “Somewhat practical fully homomorphic encryption,” IACR Cryptol. ePrint Arch., p. 144, 2012. [Online]. Available: <http://eprint.iacr.org/2012/144>


GPGPU_HPCA2023
S. Fan, Z. Wang, W. Xu, R. Hou, D. Meng, and M. Zhang, “Tensorfhe: Achieving practical computation on encrypted data using gpgpu,” in IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2023, pp. 922–934.

TensorFHE_HPCA2023
S. Fan, Z. Wang, W. Xu, R. Hou, D. Meng, and M. Zhang, “Tensorfhe: Achieving practical computation on encrypted data using gpgpu,” in IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2023, pp. 922–934.

2021Truly
D. Froelicher, J. R. Troncoso-Pastoriza, J. L. Raisaro, M. A. Cuendet, and J. P. Hubaux, “Truly privacy-preserving federated analytics for precision medicine with multiparty homomorphic encryption,” Nature Communications, vol. 12, no. 1, 2021.

MemFHE
S. Gupta, R. Cammarota, and T. Rosing, “Memfhe: End-to-end computing with fully homomorphic encryption in memory,” 2022.

Invited_DAC2021
S. Gupta and T. S. Rosing, “Invited: Accelerating fully homomorphic encryption with processing in memory,” in ACM/IEEE Design Automation Conference (DAC), 2021, pp. 1335–1338.

LR_AAAI2019
K. Han, S. Hong, J. H. Cheon, and D. Park, “Logistic regression on homomorphic encrypted data at scale,” in The 33rd AAAI Conference on Artificial Intelligence, (AAAI), 2019, pp. 9466–9471.

CoxHE_DATE2022
M. Han, Y. Zhu, Q. Lou, Z. Zhou, S. Guo, and L. Ju, “Coxhe: A software-hardware co-design framework for fpga acceleration of homomorphic computation,” in Proceedings of the 2022 Conference & Exhibition on Design, Automation & Test in Europe, (DATE), 2022.

HElib
HElib, “Cuda-accelerated fully homomorphic encryption library,” 2019. [Online]. Available: <https://github.com/homenc/HElib/ tree/master/examples/BGV_country_db_lookup archived at https://perma.cc/ U2MW- QLRJ.>


Intel_optimization
Intel. (2018) Intel® 64 and IA-32 Architectures Optimization Reference Manual. [Online]. Available: <https://www.intel.com/content/dam/doc/manual/64-ia-32-architectures-optimization-manual.pdf>


MAdder_TVLSI2019
F. Jafarzadehpour, A. S. Molahosseini, A. A. E. Zarandi, and L. Sousa, “Efficient modular adder designs based on thermometer and one-hot coding,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 27, no. 9, pp. 2142–2155, 2019.

MATCHA
L. Jiang, Q. Lou, and N. Joshi, “MATCHA: a fast and energy-efficient accelerator for fully homomorphic encryption over the torus,” in ACM/IEEE Design Automation Conference (DAC), 2022, pp. 235–240.

CHES2021
W. Jung, S. Kim, J. H. Ahn, J. H. Cheon, and Y. Lee, “Over 100x faster bootstrapping in fully homomorphic encryption through memory-centric optimization with gpus,” IACR Transactions on Cryptographic Hardware and Embedded Systems, no. 4, p. 114–148, 2021.

2019Homomorphic
A. Khedr and G. Gulak, “Homomorphic processing unit (hpu) for accelerating secure computations under homomorphic encryption,” 2019.

Near-Memory_TCASI2022
H. Kim, J. Mu, C. Yu, T. T.-H. Kim, and B. Kim, “A 1-16b reconfigurable 80kb 7t sram-based digital near-memory computing macro for processing neural networks,” IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 70, no. 4, pp. 1580–1590, 2023.

SHARP_ISCA2023
J. Kim, S. Kim, J. Choi, J. Park, D. Kim, and J. H. Ahn, “Sharp: A short-word hierarchical accelerator for robust and practical fully homomorphic encryption,” in Proceedings of the 50th Annual International Symposium on Computer Architecture (ISCA), 2023.

ARK
J. Kim, G. Lee, S. Kim, G. Sohn, M. Rhu, J. Kim, and J. H. Ahn, “ARK: fully homomorphic encryption accelerator with runtime data generation and inter-operation key reuse,” in 55th IEEE/ACM International Symposium on Microarchitecture (Micro), 2022, pp. 1237–1254.

BTS
S. Kim, J. Kim, M. J. Kim, W. Jung, J. Kim, M. Rhu, and J. H. Ahn, “BTS: an accelerator for bootstrappable fully homomorphic encryption,” in The 49th Annual International Symposium on Computer Architecture (ISCA), 2022, pp. 711–725.

Ramulator
Y. Kim, W. Yang, and O. Mutlu, “Ramulator: A fast and extensible dram simulator,” IEEE Computer Architecture Letters, vol. 15, no. 1, pp. 45–49, 2016.

KU2022103583
H. Ku, W. Susilo, Y. Zhang, W. Liu, and M. Zhang, “Privacy-preserving federated learning in medical diagnosis with homomorphic re-encryption,” Computer Standards & Interfaces, vol. 80, 2022.

ASIACCS2023
C. Lee, S. Min, J. Seo, and Y. Song, “Faster TFHE bootstrapping with block binary keys,” in Proceedings of the 2023 ACM Asia Conference on Computer and Communications Security (ASIACCS), 2023, pp. 2–13.

LeeLLK0NC22
E. Lee, J. Lee, J. Lee, Y. Kim, Y. Kim, J. No, and W. Choi, “Low-complexity deep convolutional neural networks on fully homomorphic encryption using multiplexed parallel convolutions,” in International Conference on Machine Learning (ICML), 2022, pp. 12 403–12 422.

ACCESS9734024
J.-W. Lee, H. Kang, Y. Lee, W. Choi, J. Eom, M. Deryabin, E. Lee, J. Lee, D. Yoo, Y.-S. Kim, and J.-S. No, “Privacy-preserving machine learning with fully homomorphic encryption for deep neural network,” IEEE Access, vol. 10, pp. 30 039–30 054, 2022.

INSPIRE
J. Lin, L. Liang, Z. Qu, I. Ahmad, L. Liu, F. Tu, T. Gupta, Y. Ding, and Y. Xie, “Inspire: In-storage private information retrieval via protocol and architecture co-design,” in Proceedings of the 49th Annual International Symposium on Computer Architecture (ISCA), 2022, p. 102–115.

b10
M. Lipp, M. Schwarz, D. Gruss, T. Prescher, W. Haas, A. Fogh, J. Horn, S. Mangard, P. Kocher, D. Genkin, Y. Yarom, and M. Hamburg, “Meltdown: Reading Kernel Memory from User Space,” in USENIX Security Symposium, 2018, pp. 973–990.

LWE2010
V. Lyubashevsky, C. Peikert, and O. Regev, “On ideal lattices and learning with errors over rings,” in 29th Annual International Conference on the Theory and Applications of Cryptographic Techniques, 2010.

FHESurvey
C. Marcolla, V. Sucasas, M. Manzano, R. Bassoli, F. H. P. Fitzek, and N. Aaraj, “Survey on fully homomorphic encryption, theory, and applications,” Proceedings of the IEEE, vol. 110, pp. 1572–1609, 2022.

VSP
K. Matsuoka, R. Banno, N. Matsumoto, T. Sato, and S. Bian, “Virtual secure platform: A five-stage pipeline processor over TFHE,” in USENIX Security Symposium, 2021, pp. 4007–4024.

MAdder_ISCAS2018
A. S. Molahosseini, A. Asadpoor, A. A. E. Zarandi, and L. Sousa, “Towards efficient modular adders based on reversible circuits,” in IEEE International Symposium on Circuits and Systems (ISCAS), 2018.

CACTI
N. Muralimanohar, R. Balasubramonian, and N. Jouppi, “Optimizing nuca organizations and wiring alternatives for large caches with cacti 6.0,” in IEEE/ACM International Symposium on Microarchitecture (MICRO), 2007, pp. 3–14.

CPU-FPGA_ICCAD2022
K. Nam, H. Oh, H. Moon, and Y. Paek, “Accelerating n-bit operations over tfhe on commodity cpu-fpga,” in Proceedings of the 41st IEEE/ACM International Conference on Computer-Aided Design (ICCAD), 2022.

nuHE
nucyper, “Nufhe, a gpu-powered torus fhe implementation,” <https://github.com/nucypher/nufhe>, 2019.

NTT_TCASI2022
R. Paludo and L. Sousa, “Ntt architecture for a linux-ready risc-v fully-homomorphic encryption accelerator,” IEEE Transactions on Circuits and Systems I: Regular Papers, vol. 69, no. 7, pp. 2669–2682, 2022.

Morphling
Prasetiyo, A. Putra, and J.-Y. Kim, “Morphling: A throughput-maximized tfhe-based accelerator using transform-domain reuse,” in IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2024.

Strix
A. Putra, Prasetiyo, Y. Chen, J. Kim, and J.-Y. Kim, “Strix: An end-to-end streaming architecture with two-level ciphertext batching for fully homomorphic encryption with programmable bootstrapping,” in Proceedings of the 56th Annual IEEE/ACM International Symposium on Microarchitecture (MICRO), 2023.

Cheetah_HPCA2021
B. Reagen, W. Choi, Y. Ko, V. T. Lee, H. S. Lee, G. Wei, and D. Brooks, “Cheetah: Optimizing and accelerating homomorphic encryption for private inference,” in IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2021, pp. 26–39.

TVLSI2020
D. Reis, J. Takeshita, T. Jung, M. T. Niemier, and X. S. Hu, “Computing-in-memory for performance and energy-efficient homomorphic encryption,” IEEE Trans. Very Large Scale Integr. Syst., vol. 28, no. 11, pp. 2300–2313, 2020.

HEAX
M. S. Riazi, K. Laine, B. Pelton, and W. Dai, “HEAX: an architecture for computing on encrypted data,” in Architectural Support for Programming Languages and Operating Systems, Lausanne, (ASPLOS), 2020, pp. 1295–1309.

HPCA2000
S. Rixner, W. J. Dally, B. Khailany, P. R. Mattson, U. J. Kapasi, and J. D. Owens, “Register organization for media processing,” in Proceedings of the Sixth International Symposium on High-Performance Computer Architecture (HPCA), 2000, pp. 375–386.

k-means
L. Rovida, “Fast but approximate homomorphic k-means based on masking technique,” Int. J. Inf. Sec., 2023.

F1
N. Samardzic, A. Feldmann, A. Krastev, S. Devadas, R. G. Dreslinski, C. Peikert, and D. Sánchez, “F1: A fast and programmable accelerator for fully homomorphic encryption,” in 54th Annual IEEE/ACM International Symposium on Microarchitecture (Micro), 2021, pp. 238–252.

CraterLake
N. Samardzic, A. Feldmann, A. Krastev, N. Manohar, N. Genise, S. Devadas, K. Eldefrawy, C. Peikert, and D. Sánchez, “Craterlake: a hardware accelerator for efficient unbounded computation on encrypted data,” in The 49th Annual International Symposium on Computer Architecture (ISCA), 2022, pp. 173–187.

ASPDAC2021
C. Shen, C. Chen, and J. Zhang, “Micro-architectural cache side-channel attacks and countermeasures,” in 26th Asia and South Pacific Design Automation Conference, (ASPDAC), 2021, pp. 441–448.

MMult_ISQED2023
D. Soni, M. Nabeel, H. Gamil, O. Mazonka, B. Reagen, R. Karri, and M. Maniatakos, “Design space exploration of modular multipliers for asic fhe accelerators,” in 24th International Symposium on Quality Electronic Design (ISQED), 2023, pp. 1–8.

TC2023
J. Takeshita, D. Reis, T. Gong, M. Niemier, X. S. Hu, and T. Jung, “Accelerating finite-field and torus fhe via compute-enabled (s)ram,” IEEE Transactions on Computers, pp. 1–14, 2023.

SP2021
S. Tan, B. Knott, Y. Tian, and D. J. Wu, “Cryptgpu: Fast privacy-preserving machine learning on the GPU,” in IEEE Symposium on Security and Privacy (SP), 2021.

MMult_TC2023
W. Tan, A. Wang, X. Zhang, Y. Lao, and K. K. Parhi, “High-speed VLSI architectures for modular polynomial multiplication via fast filtering and applications to lattice-based cryptography,” IEEE Transactions on Computers, vol. 72, no. 9, pp. 2454–2466, 2023.

vernamlab
vernamlab, “Cuda-accelerated fully homomorphic encryption library,” <https://github.com/vernamlab/cuFHE>.

FFT_TSP2020
J. Wang, C. Xiong, K. Zhang, and J. Wei, “Fixed-point analysis and parameter optimization of the radix-2^ pipelined FFT processor,” IEEE Transactions on Signal Processing, vol. 63, no. 18, pp. 4879–4893, 2015.

ISCAS2014
W. Wang, Z. Chen, and X. Huang, “Accelerating leveled fully homomorphic encryption using GPU,” in IEEE International Symposium on Circuits and Systems (ISCAS), 2014, pp. 2800–2803.

TVLSI2014
W. Wang, X. Huang, N. Emmart, and C. Weems, “Vlsi design of a large-number multiplier for fully homomorphic encryption,” IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 22, no. 9, pp. 1879–1887, 2014.

Triangle_TC2022
X. Wang, J. Yang, Y. Zhao, X. Jia, R. Yin, X. Chen, G. Qu, and W. Zhao, “Triangle counting accelerations: From algorithm to in-memory computing architecture,” IEEE Transactions on Computers, vol. 71, no. 10, pp. 2462–2472, 2022.

HE-Booster
Z. Wang, P. Li, R. Hou, Z. Li, J. Cao, X. Wang, and D. Meng, “He-booster: An efficient polynomial arithmetic acceleration on gpus for fully homomorphic encryption,” IEEE Transactions on Parallel and Distributed Systems, vol. 34, no. 4, pp. 1067–1081, 2023.

FFT_ISCAS2022
S.-Y. Wu, K.-Y. Chen, and M.-D. Shieh, “Efficient vlsi architecture of bluestein’s fft for fully homomorphic encryption,” in IEEE International Symposium on Circuits and Systems (ISCAS), 2022, pp. 2242–2245.

MPU_ACM2023
X. Xie, P. Gu, Y. Ding, D. Niu, H. Zheng, and Y. Xie, “Mpu: Memory-centric simt processor via in-dram near-bank computing,” ACM Transactions on Architecture and Code Optimization, vol. 20, 2023.

Poseidon-NDP
Y. Yang, H. Lu, and X. Li, “Poseidon-ndp: Practical fully homomorphic encryption accelerator based on near data processing architecture,” IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, vol. 42, no. 12, pp. 4749–4762, 2023.

Poseidon
Y. Yang, H. Zhang, S. Fan, H. Lu, M. Zhang, and X. Li, “Poseidon: Practical homomorphic encryption accelerator,” in IEEE International Symposium on High-Performance Computer Architecture (HPCA), 2023.

K-means_2023
P. Zhang, T. Huang, X. Sun, W. Zhao, H. Liu, S. Lai, and J. K. Liu, “Privacy-preserving and outsourced multi-party k-means clustering based on multi-key fully homomorphic encryption,” IEEE Transactions on Dependable and Secure Computing, vol. 20, no. 3, 2023.

PipeZK
Y. Zhang, S. Wang, X. Zhang, J. Dong, X. Mao, F. Long, C. Wang, D. Zhou, M. Gao, and G. Sun, “Pipezk: Accelerating zero-knowledge proof with a pipelined architecture,” in 48th ACM/IEEE Annual International Symposium on Computer Architecture (ISCA), 2021, pp. 416–428.









