DCT-CryptoNets: Scaling Private Inference in the Frequency Domain
homomorphic encryption;deep neural networks;privacy-preserving machine learning;frequency-domain learning
Yes
No ethics review needed.
See the weakness.
$\bullet$ Performing TFHE-based inference in the frequency domain allows the usage of lower-resolution inputs without compromising accuracy. This approach significantly decreases both FLOPs and nonlinear operations (ReLU), while also requiring fewer bootstrapping operations. This results in substantial speedup benefits. More importantly, this makes it feasible to perform inference on larger input images, enhancing the practical applicability (such as semantic segmentation) of homomorphic encryption in neural network inference.


$\bullet$ The authors report ciphertext accuracy, specifically on the ImageNet-1K scale, distinguishing this work from most private inference papers, which typically report plaintext accuracy under the assumption that there is no accuracy loss when operations are conducted in field arithmetic.

$\bullet$ The experimental results are extensive and include a detailed sensitivity analysis of the cryptographic hyper-parameters.
This paper introduces DCT-CryptoNets, which performs neural network (CNN) inference entirely using TFHE-based homomorphic encryption in the frequency domain. Unlike hybrid protocols that limit homomorphic encryption to just linear operations, and require MPC for nonlinear operations, this approach leverages TFHE to perform both linear and nonlinear operations under homomorphic encryption. This enables private inference outsourcing to the server without requiring client interaction for intermediate computations. The authors have shown significant speedup over prior TFHE-based implementation (SHE, NeurIPS'19) on CIFAR-10 and ImageNet datasets.
$\bullet$ The lack of sufficient algorithmic contributions and research insights makes it less suitable for the ML conference. Operating in the frequency domain for private inference benefits is not a novel concept (see [1,2]). Also, the usage of quantization-aware training for lower-frequency components is simply an engineering tweak. 

Thus, a more fitting venue for this work might be a cryptography-focused conference. 

$\bullet$ Moreover, the practicality of HE-only private inference remains questionable, especially when compared to hybrid protocol-based approaches. For example, Cheetah [3] achieves ImageNet-1K inference on ResNet-50 in 80.3 seconds in a LAN setting and 134.7 seconds in a WAN setting. Thus, the primary motivation for pursuing HE-only inference appears to be the benefit of non-interactive private inference.




1. Song et al., ENSEI: Efficient secure inference via frequency-domain homomorphic convolution for privacy-preserving visual recognition, CVPR 2020.

2. Li et al. Falcon: A fourier transform based approach for fast and secure convolutional neural network predictions, CVPR 2020.

3. Huang et al., Cheetah: Lean and fast secure Two-Party deep neural network inference, USENIX Security 2022.
Yes
No ethics review needed.
Are there any special scaling or normalization techniques that need to be considered when considering the YCrCb color space components of an image?

Can GPUs be used to any advantage in this approach?

Are there any privacy preservation guarantees or "leakage" guarantees that could be developed around this approach.  (I realize this may be challenging mathematically.)
Novel DCT based insight for unstructured data yielding excellent computational advantages for high resolution images.

Thorough benchmarking and attention to reproducible science.

Excellent comparative analysis of this CKKS based technique relative to competing TFHE approaches.

Excellent and comprehensive list of references that chronicle the current state of the art and prior advances.
This paper presents a very promising strategy for achieving a practical and computationally efficient FHE-based (CKKS) private inference of deep learning models applied to high resolution images.     In a novel approach, the technique presented relies as a starting point on a Discrete Cosine Transform representation of the image.  It also employs a quantization aware training (QAT) framework.  Thorough comparative benchmarking of accuracy and latency for RGB vs DCT techniques a various image resolutions and at various levels of retained low-frequency components (of the DCT) are presented.  

The paper establishes both a detailed methodology and new "high-water mark" performance capability for ML inference generation for high resolution images.  Though performance is less pronounced on smaller images (32x32) it is the capability on larger images that is most important for future practical applications.
I find no obvious weaknesses.

I would emphasize to the reader perhaps new to the field ( in Section 3.2 on page 6)  that model training is done in the plaintext domain. (even though this is most evident in Figure 3 presented on Page 14.
Yes
No ethics review needed.
see weakness
1. **Frequency-Domain Optimization**: DCT-CryptoNets leverage the Discrete Cosine Transform (DCT) to focus on low-frequency components of images, which enhances the model's ability to capture perceptually salient information while reducing computational complexity and improving accuracy compared to traditional RGB-based networks.

2. **Reduced Latency and Improved Scalability**: The proposed method achieves significant latency reductions (up to 5.3×) during inference, especially on large datasets like ImageNet, while demonstrating superior scalability as image resolution increases. This makes DCT-CryptoNets more efficient for real-world applications of privacy-preserving deep learning.

3. **Enhanced Reliability through Reduced Error Accumulation**: By minimizing the need for homomorphic bootstrap operations, DCT-CryptoNets reduce the accumulation of approximation errors, leading to more reliable predictions and improved encrypted accuracy. This addresses challenges faced by earlier fully homomorphic encryption (FHE) schemes.
The paper presents DCT-CryptoNets, a novel framework that enhances privacy-preserving deep learning by utilizing the Discrete Cosine Transform (DCT) to operate in the frequency domain, significantly reducing latency and computational costs associated with fully homomorphic encrypted neural networks (FHENNs). By focusing on low-frequency information, DCT-CryptoNets improve accuracy and reliability in encrypted predictions while demonstrating superior scalability with increasing image resolution. The work addresses key challenges in existing FHE-based neural networks and emphasizes the importance of optimizing cryptographic and quantization parameters for practical applications in secure image processing.
I have a question about the threat model setting here, why the model is trained locally?

If the client could train the model by themselves, why does they do inference locally.

Or if they want to deploy that encrypted model (key from model trainer) on the cloud for other clients as service. How does the key management should be solved?
DCT-CryptoNets substantially reduces latency of fully homomorphic encryption neural networks by learning low-frequency components and operating on these frequency components at inference time.
@inproceedings{
anonymous2024dctcryptonets,
title={{DCT}-CryptoNets: Scaling Private Inference in the Frequency Domain},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=lPJUQsSIxm},
note={under review}
}
The convergence of fully homomorphic encryption (FHE) and machine learning offers unprecedented opportunities for private inference of sensitive data. FHE enables computation directly on encrypted data, safeguarding the entire machine learning pipeline, including data and model confidentiality. However, existing FHE-based implementations for deep neural networks face significant challenges in computational cost, latency, and scalability, limiting their practical deployment. This paper introduces DCT-CryptoNets, a novel approach that operates directly in the frequency-domain to reduce the burden of computationally expensive non-linear activations and homomorphic bootstrap operations during private inference. It does so by utilizing the discrete cosine transform (DCT), commonly employed in JPEG encoding, which has inherent compatibility with remote computing services where images are generally stored and transmitted in this encoded format. DCT-CryptoNets demonstrates a substantial latency reductions of up to 5.3$\times$ compared to prior work on benchmark image classification tasks. Notably, it demonstrates inference on the ImageNet dataset within 2.5 hours (down from 12.5 hours on equivalent 96-thread compute resources). Furthermore, by *learning* perceptually salient low-frequency information DCT-CryptoNets improves the reliability of encrypted predictions compared to RGB-based networks by reducing error accumulating homomorphic bootstrap operations. DCT-CryptoNets also demonstrates superior scalability to RGB-based networks by further reducing computational cost as image size increases. This study demonstrates a promising avenue for achieving efficient and practical private inference of deep learning models on high resolution images seen in real-world applications.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
homomorphic encryption deep neural networks privacy-preserving machine learning frequency-domain learning
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/7ba464ead9aa02c35af07f506146a84942be65b1.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
DCT-CryptoNets: Scaling Private Inference in the Frequency Domain
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission