abstract: Large-scale deep neural networks (DNNs) are both compute and memory intensive. As the size of DNNs continues to grow, it is critical to improve the energy efficiency and performance while maintaining accuracy. For DNNs, the model size is an important factor affecting performance, scalability and energy efficiency. Weight pruning achieves good compression ratios but suffers from three drawbacks: *1)* the irregular network structure after pruning, which affects performance and throughput; *2)* the increased training complexity; and *3)* the lack of rigorous guarantee of compression ratio and inference accuracy.
  .
  To overcome these limitations, this paper proposes CirCNN, a principled approach to represent weights and process neural networks using *block-circulant* matrices. CirCNN utilizes the *Fast Fourier Transform (FFT)*-based fast multiplication, *simultaneously* reducing the computational complexity (both in inference and training) from O($n^2$) to O($n\log n$) and the storage complexity from O($n^2$) to O($n$), with negligible accuracy loss. Compared to other approaches, CirCNN is distinct due to its mathematical rigor: the DNNs based on CirCNN can converge to the same "effectiveness" as DNNs without compression. We propose the CirCNN architecture, a universal DNN inference engine that can be implemented in various hardware/software platforms with configurable network architecture (e.g., layer type, size, scales, etc.). In CirCNN architecture: *1)* Due to the recursive property, *FFT can be used as the key computing kernel*, which ensures universal and small-footprint implementations. *2)* The *compressed but regular* network structure avoids the pitfalls of the network pruning and facilitates high performance and throughput with highly pipelined and parallel design. To demonstrate the performance and energy efficiency, we test CirCNN in FPGA, ASIC and embedded processors. Our results show that CirCNN architecture achieves very high energy efficiency and performance with a small hardware footprint. Based on the FPGA implementation and ASIC synthesis results, CirCNN achieves 6 - 102X energy efficiency improvements compared with the best state-of-the-art results.
author: Caiwen Ding$^{+,1}$, Siyu Liao$^{+,2}$, Yanzhi Wang$^{+,1}$, Zhe Li$^{1}$, Ning Liu$^{1}$, Youwei Zhuo$^3$, Chao Wang$^3$, Xuehai Qian$^3$, Yu Bai$^4$, Geng Yuan$^{1}$, Xiaolong Ma$^{1}$, Yipeng Zhang$^{1}$, Jian Tang$^{1}$, Qinru Qiu$^{1}$, Xue Lin$^5$, Bo Yuan$^{2}$
title: CirCNN: Accelerating and Compressing Deep Neural Networks Using Block-Circulant Weight Matrices

# Introduction

From the end of the first decade of the 21st century, neural networks have been experiencing a phenomenal resurgence thanks to the big data and the significant advances in processing speeds. Large-scale deep neural networks (DNNs) have been able to deliver impressive results in many challenging problems. For instance, DNNs have led to breakthroughs in object recognition accuracy on the ImageNet dataset , even achieving human-level performance for face recognition . Such promising results triggered the revolution of several traditional and emerging real-world applications, such as self-driving systems , automatic machine translations , drug discovery and toxicology . As a result, both academia and industry show the rising interests with significant resources devoted to investigation, improvement, and promotion of deep learning methods and systems.

One of the key enablers of the unprecedented success of deep learning is the availability of very large models. Modern DNNs typically consist of multiple cascaded layers, and at least millions to hundreds of millions of parameters (i.e., weights) for the entire model . The larger-scale neural networks tend to enable the extraction of more complex high-level features, and therefore, lead to a significant improvement of the overall accuracy . On the other side, the layered deep structure and large model sizes also demand increasing computational capability and memory requirements. In order to achieve higher scalability, performance, and energy efficiency for deep learning systems, two orthogonal research and development trends have both attracted enormous interests.

The first trend is the *hardware acceleration of DNNs*, which has been extensively investigated in both industry and academia. As a representative technique, **FPGA-based** accelerators offer good programmability, high degree of parallelism and short development cycle. FPGA has been used to accelerate the original DNNs , binary neural networks , and more recently, DNNs with model compression . Alternatively, **ASIC-based** implementations have been recently explored to overcome the limitations of general-purpose computing approaches. A number of major high-tech companies have announced their ASIC chip designs of the DNN inference framework, such as Intel, Google, etc. . In academia, three representative works at the architectural level are Eyeriss , EIE , and the DianNao family , which focus specifically on the convolutional layers, the fully-connected layers, and the memory design/organization, respectively. There are a number of recent tapeouts of hardware deep learning systems .

These prior works mainly focus on the inference phase of DNNs, and usually suffer from the frequent accesses to off-chip DRAM systems (e.g., when large-scale DNNs are used for ImageNet dataset). This is because the limited on-chip SRAM memory can hardly accommodate large model sizes. Unfortunately, off-chip DRAM accesses consume significant energy. The recent studies show that the per-bit access energy of off-chip DRAM memory is 200$\times$ compared with on-chip SRAM. Therefore, it can easily dominate the whole system power consumption.

The energy efficiency challenge of large models motivates the second trend: *model compression*. Several algorithm-level techniques have been proposed to compress models and accelerate DNNs, including weight quantization , connection pruning , and low rank approximation . These approaches can offer a reasonable parameter reduction (e.g., by $9\times$ to $13\times$ in ) with minor accuracy degradation. However, they suffer from the three drawbacks: *1)* the sparsity regularization and pruning typically result in an irregular network structure, thereby undermining the compression ratio and limiting performance and throughput ; *2)* the training complexity is increased due to the additional pruning process or low rank approximation step , etc.; *3)* the compression ratios depending on network are heuristic and cannot be precisely controlled.

We believe that an *ideal* model compression technique should: *i)* maintain *regular* network structure; *ii)* reduce the complexity for *both inference and training*, and, most importantly, *iii)* retain a *rigorous* mathematical fundation on compression ratio and accuracy.

As an effort to achieve the three goals, we propose CirCNN, a principled approach to represent weights and process neural networks using *block-circulant* matrices . The concept of the block-circulant matrix compared to the ordinary unstructured matrix is shown in Fig. . In a square circulant matrix, each row (or column) vector is the circulant reformat of the other row (column) vectors. A non-squared matrix could be represented by a set of square circulant submatrices (blocks). Therefore, by representing a matrix with a vector, the first benefit of CirCNN is *storage size reduction*. In Fig. , the unstructured $6 \times 3$ weight matrix (on the left) holds 18 parameters. Suppose we can represent the weights using two $3 \times 3$ circulant matrices (on the right), we just need to store *6 parameters*, easily leading to 3x model size reduction. Intuitively, the reduction ratio is determined by the block size of the circulant submatrices: larger block size leads to high compression ratio. In general, the **storage complexity is reduced from O($n^2$) to O($n$)**.

The second benefit of CirCNN is *computational complexity reduction*. We explain the insights using a fully-connected layer of DNN, which can be represented as $\bf{y}=\psi(\bf{W}\bf{x}+\mathbf{\theta})$, where vectors $\bf{x}$ and $\bf{y}$ represent the outputs of all neurons in the previous layer and the current layer, respectively; $\bf{W}$ is the $m$-by-$n$ weight matrix; and $\psi(\cdot)$ is activation function. When $\bf{W}$ is a block-circulant matrix, the *Fast Fourier Transform (FFT)*-based fast multiplication method can be utilized, and the **computational complexity is reduced from O($n^2$) to O($n\log n$)**.

It is important to understand that CirCNN incurs *no conversion* between the unstructured weight matrices and block-circulant matrices. Instead, we *assume* that the layers can be represented by block-circulant matrices and the training generates a vector for each circulant submatrix. The fundamental difference is that: the current approaches apply various compression techniques (e.g., pruning) on the unstructured weight matrices and then retrain the network; while CirCNN directly trains the network assuming block-circulant structure. This leads to two advantages. First, the prior work can only reduce the model size by a heuristic factor, depending on the network, while CirCNN provides the *adjustable but fixed* reduction ratio. Second, with the same FFT-based fast multiplication, the computational complexity of training is also reduced from O($n^2$) to O($n\log n$). Unfortunately, the prior work does not reduce (or even increase) training complexity.

Due to the storage and computational complexity reduction, CirCNN is clearly attractive. The only question is: can a network really be represented by block-circulant matrices with no (or negligible) accuracy loss? This question is natural, because with the much less weights in the vectors, the network may not be able to approximate the function of the network with unstructured weight matrices. Fortunately, the answer to the question is *YES*. CirCNN is *mathematically rigorous*: we have developed a theoretical foundation and formal proof showing that the DNNs represented by block-circulant matrices can converge to the same "effectiveness" as DNNs without compression, fundamentally distinguishing our method from prior arts. The outline of the proof is discussed in Section  and the details are provided in technical reports .

Based on block-circulant matrix-based algorithms, we propose CirCNN architecture, — a universal DNN inference engine that can be implemented in various hardware/software platforms with configurable network architecture (e.g., layer type, size, scales, etc.). Applying CirCNN to neural network accelerators enables notable architectural innovations. *1)* Due to its recursive property and its intrinsic role in CirCNN, FFT is implemented as the *basic computing block*. It ensures universal and small-footprint implementations. *2)* *Pipelining and parallelism* optimizations. Taking advantage of the compressed but regular network structures, we aggressively apply inter-level and intra-level pipelining in the basic computing block. Moreover, we can conduct joint-optimizations considering parallelization degree, performance and power consumption. *3)* *Platform-specific optimizations* focusing on weight storage and memory management.

To demonstrate the performance and energy efficiency, we test CirCNN architecture in *three platforms*: FPGA, ASIC and embedded processors. Our results show that CirCNN architecture achieves very high energy efficiency and performance with a small hardware footprint. Based on the FPGA implementation and ASIC synthesis results, CirCNN achieves 6 - 102X energy efficiency improvements compared with the best state-of-the-art results.

# Background and Motivation

## Deep Neural Networks

Deep learning systems can be constructed using different types of architectures, including deep convolutional neural networks (DCNNs), deep belief networks (DBNs), and recurrent neural networks (RNNs). Despite the differences in network structures and target applications, they share the *same construction principle*: multiple functional layers are cascaded together to extract features at multiple levels of abstraction . Fig. illustrates the multi-layer structure of an example DCNN, which consists of a stack of *fully-connected layers*, *convolutional layers*, and *pooling layers*. These three types of layers are fundamental in deep learning systems.

***The fully-connected (FC) layer*** is the most storage-intensive layer in DNN architectures since its neurons are fully connected with neurons in the previous layer. The computation procedure of a FC layer consists of matrix-vector arithmetics (multiplications and additions) and transformation by the activation function, as described as follows: $$\bf{y}=\psi(\bf{W}\bf{x}+\mathbf{\theta})$$ where $\textbf{W}\in \mathbb{R}^{m\times n}$ is the weight matrix of the synapses between this FC layer (with $m$ neurons) and its previous layer (with $n$ neurons); $\mathbf{\theta}\in \mathbb{R}^{m}$ is the bias vector; and $\psi(\cdot)$ is the activation function. The Rectified Linear Unit (ReLU) $\psi(x)=\max(0,x)$ is the most widely utilized in DNNs.

***The convolutional (CONV) layer***, as the name implies, performs a two-dimensional convolution to extract features from its inputs that will be fed into subsequent layers for extracting higher-level features. A CONV layer is associated with a set of learnable filters (or kernels) , which are activated when specific types of features are found at some spatial positions in inputs. A filter-sized moving window is applied to the inputs to obtain a set of feature maps, calculating the convolution of the filter and inputs in the moving window. Each *convolutional neuron*, representing one pixel in a feature map, takes a set of inputs and the corresponding filter weights to calculate the inner-product. Given input feature map $\textbf{X}$ and the $r\times r$-sized filter (i.e., the *convolutional kernel*) $\textbf{F}$, the output feature map $\textbf{Y}$ is calculated as $$y_{a,b}=\sum_{i=1}^r\sum_{j=1}^r x_{a+i-1,b+j-1}\times f_{i,j},$$ where $y_{a,b}$, $x_{a+i-1,b+j-1}$, and $f_{i,j}$ are elements in $\textbf{Y}$, $\textbf{X}$, and $\textbf{F}$, respectively. Multiple convolutional kernels can be adopted to extract different features in the same input feature map. Multiple input feature maps can be convolved with the same filter and results are summed up to derive a single feature map.

***The pooling (POOL) layer*** performs a subsampling operation on the extracted features to reduce the data dimensions and mitigate overfitting issues. Here, the subsampling operation on the inputs of pooling layer can be realized by various non-linear operations, such as max, average or L2-norm calculation. Among them, the max pooling is the dominant type of pooling strategy in state-of-the-art DCNNs due to the higher overall accuracy and convergence speed .

Among these three types of layers, the majority of computation occurs in CONV and FC layers, while the POOL layer has a relatively lower computational complexity of O($n$). The storage requirement of DNNs is due to the weight matrices $\textbf{W}$'s in the FC layers and the convolutional kernels $\textbf{F}$'s in CONV layers. As a result, the FC and CONV layers become the major research focuses on energy-efficient implementation and weight reduction of DNNs.

## DNN Weight Storage Reduction and Acceleration

Mathematical investigations have demonstrated significant sparsity and margin for weight reduction in DNNs, a number of prior works leverage this property to reduce weight storage. The techniques can be classified into two categories. *1)* *Systematic* methods  such as Singular Value Decomposition (SVD). Despite being systematic, these methods typically exhibit a relatively high degradation in the overall accuracy (by 5%-10% at 10$\times$ compression). *2)* *Heuristic pruning* methods use heuristic weight together with weight quantization. These method could achieve a better parameter reductions, i.e., 9$\times$-13$\times$ , and a very small accuracy degradation. However, the network structure and weight storage after pruning become *highly irregular* (c.f. Fig. ) and therefore indexing is always needed, which undermines the compression ratio and more importantly, the performance improvement.

Besides the pros and cons of the two approaches, the prior works share the following common limitations: *1)* mainly focusing on weight reduction rather than computational complexity reduction; *2)* only reducing the model size by a heuristic factor instead of reducing the Big-O complexity; and *3)* performing weight pruning or applying matrix transformations based on a trained DNN model, thereby adding complexity to the training process. The third item is crucial because it may *limit the scalability of future larger-scale deep learning systems*.

## FFT-Based Methods

LeCun *et al.* has proposed using FFTs to accelerate the computations in the CONV layers, which applies only to a single filter in the CONV layer . It uses FFT to calculate the traditional inner products of filters and input feature maps, and can achieve speedup for large filter sizes (which is less common in state-of-the-art DCNNs ). The underlying neural network structure and parameters remain *unchanged*. The speedup is due to filter reuse and it cannot achieve either asymptotic speedup in big-O notation or weight compressions (in fact additional storage space is needed).

The work most closely related to CirCNN is . It proposed to use circulant matrix in the inference and training algorithms. However, it has a number of limitations. First, it only applied to FC layers, but not CONV layer. It limits the potential gain in weight reduction and performance. Second, it uses a *single* circulant matrix to represent the weights in the whole FC layer. Since the number of input and output neurons are usually not the same, this method leads to the storage waste due to the padded zeros (to make the circulant matrix squared).

## Novelty of CirCNN

Compared with LeCun *et al.* , CirCNN is fundamentally different as it achieves asymptotic speedup in big-O notation and weight compression simultaneously. Compared with , CirCNN generalizes in three significant and novel aspects.

**Supporting both FC and CONV layers**. Unlike FC layers, the matrices in CONV layers are small filters (e.g., $3 \times 3$). Instead of representing each filter as a circulant matrix, CirCNN exploits the *inter-filter sparsity* among different filters. In another word, CirCNN represents a matrix of filters, where input and output channels are the two dimensions, by a vector of filters. The support for CONV layers allow CirCNN to be applied in the whole network.

**Block-circulant matrices**. To mitigate the inefficiency due to the single large circulant matrix used in , CirCNN uses block-circulant matrices for weight representation. The benefits are two-fold. First, it avoids the wasted storage/computation due to zero padding when the numbers of inputs and outputs are not equal. Second, it allows us to derive a *fine-grained tradeoff between accuracy and compression/acceleration*. Specifically, to achieve better compression ratio, larger block size should be used, however, it may lead to more accuracy degradation. The smaller block sizes provide better accuracy, but less compression. There is no compression if the block size is 1.

**Mathematical rigorousness**. Importantly, we perform theoretical analysis to prove that the "effectiveness" of block-circulant matrix-based DNNs will (asymptotically) approach that of original networks without compression. The theoretical proof also distinguishes the proposed method with prior work. The outline of the proof is discussed in Section  and the details are provided in reports .

Fig.  illustrates the difference between the baseline  and CirCNN. The baseline method (a) formulates a large, square circulant matrix by zero padding for FC layer weight representation when the numbers of inputs and outputs are not equal. In contrast, CirCNN(b) uses the block-circulant matrix to avoid storage waste and achieve a fine-grained tradeoff of accuracy and compression/acceleration.

Overall, with the novel techniques of CirCNN, at algorithm level, it is possible to achieve the *simultaneous* and significant reduction of *both computational and storage* complexity, for *both inference and training*.

# CirCNN: Algorithms and Foundation

## FC Layer Algorithm

The key idea of block-circulant matrix-based FC layers is to partition the original arbitrary-size weight matrix $\textbf{W}\in \mathbb{R}^{m\times n}$ into 2D blocks of square sub-matrices, and each sub-matrix is a circulant matrix. The insights are shown in Fig.  . Let $k$ denote the *block size* (size of each sub-matrix) and assume there are $p \times q$ blocks after partitioning $\mathbf{W}$, where $p = m\div k$ and $q=n \div k$. Then $\mathbf{W} = [\mathbf{W}_{ij}]$, $i \in \{1 \dots p\}$, $j \in \{1 \dots q\}$. Correspondingly, the input $\mathbf{x}$ is also partitioned as $\mathbf{x} = [\mathbf{x}^T_1, \mathbf{x}^T_2, \dots, \mathbf{x}^T_q]^T$. Then, the forward propagation process in the inference phase is given by (with bias and ReLU omitted): $$\mathbf{a}
=
\mathbf{Wx}
=
\begin{bmatrix}
         \sum_{j=1}^q \mathbf{W}_{1j} \mathbf{x}_j   \\
         \sum_{j=1}^q \mathbf{W}_{2j} \mathbf{x}_j   \\
         \dots \\
         \sum_{j=1}^q \mathbf{W}_{pj} \mathbf{x}_j
\end{bmatrix}
=
\begin{bmatrix}
         \mathbf{a}_1   \\
         \mathbf{a}_2   \\
         \dots \\
         \mathbf{a}_p
\end{bmatrix},$$ where $\mathbf{a}_i \in \mathbb{R}^{k}$ is a column vector. Assume each circulant matrix $\mathbf{W}_{ij}$ is defined by a vector $\mathbf{w}_{ij}$, i.e., $\mathbf{w}_{ij}$ is the first row vector of $\mathbf{W}_{ij}$. Then according to the *circulant convolution theorem* , the calculation of $\mathbf{W}_{ij} \mathbf{x}_j$ can be performed as $\text{IFFT}\big(\text{FFT}(\mathbf{w}_{ij})\circ\text{FFT}(\mathbf{x}_j)\big)$, where $\circ$ denotes element-wise multiplications. The operation procedure is shown on the right of Fig.  . For the inference phase, the computational complexity of this FC layer will be $O(pqk\log k)$, which is equivalent to $O(n\log n)$ for small $p$, $q$ values. Similarly, the storage complexity will be $O(pqk)$ because we only need to store $\mathbf{w}_{ij}$ or $\text{FFT}(\mathbf{w}_{ij})$ for each sub-matrix, which is equivalent to $O(n)$ for small $p$, $q$ values. Therefore, the simultaneous acceleration and model compression compared with the original DNN can be achieved for the inference process. Algorithm 1 illustrates the calculation of $\bf{Wx}$ in the inference process in the FC layer of CirCNN.

Next, we consider the backward propagation process in the training phase. Let $a_{il}$ be the $l$-th output element in $\mathbf{a}_i$, and $L$ denote the loss function. Then by using the chain rule we can derive the backward propagation process as follows: $$\frac{\partial L}{\partial \mathbf{w}_{ij}}
= \sum_{l=1}^k \frac{\partial L}{\partial a_{il}} \frac{\partial a_{il}}{\partial \mathbf{w}_{ij}}
= \frac{\partial L}{\partial \mathbf{a}_i} \frac{\partial \mathbf{a}_i}{\partial \mathbf{w}_{ij}},$$ $$\frac{\partial L}{\partial \mathbf{x}_{j}}
= \sum_{i=1}^p\sum_{l=1}^k \frac{\partial L}{\partial a_{il}} \frac{\partial a_{il}}{\partial \mathbf{x}_{j}}
= \sum_{i=1}^p \frac{\partial L}{\partial \mathbf{a}_i} \frac{\partial \mathbf{a}_i}{\partial \mathbf{x}_j}.$$ We have proved that $\frac{\partial \mathbf{a}_i}{\partial \mathbf{w}_{ij}}$ and $\frac{\partial\mathbf{a}_i}{\partial \mathbf{x}_j}$ are block-circulant matrices. Therefore, $\frac{\partial L}{\partial \mathbf{w}_{ij}}$ and $\frac{\partial L}{\partial \mathbf{a}_i} \frac{\partial \mathbf{a}_i}{\partial \mathbf{x}_j}$ can be calculated as the "FFT$\rightarrow$element-wise multiplication$\rightarrow$IFFT" procedure and is equivalent to $O(n\log n)$ computational complexity per layer. Algorithm 2 illustrates backward propagation process in the FC layer of CirCNN.

In CirCNN, the inference and training constitute an integrated framework where the reduction of computational complexity can be gained for both. We directly train the vectors $\mathbf{w}_{ij}$'s, corresponding to the circulant sub-matrices $\mathbf{W}_{ij}$'s, in each layer using Algorithm 2. Clearly, the network after such training procedure naturally follows the block-circulant matrix structure. It is a key advantage of CirCNN compared with prior works which require additional steps on a trained neural network.

## CONV Layer Algorithm

In practical DNN models, the CONV layers are often associated with multiple input and multiple output feature maps. As a result, the computation in the CONV layer can be expressed in the format of tensor computations as below: $$\mathcal{Y}(x,y,p)=\sum_{i=1}^r\sum_{j=1}^r\sum_{c=1}^C\mathcal{F}(i,j,c,p)\mathcal{X}(x+i-1,y+j-1,c),$$ where $\mathcal{X}\in\mathbb{R}^{W\times H\times C}$, $\mathcal{Y}\in\mathbb{R}^{(W-r+1)\times(H-r+1)\times P}$, $\mathcal{F}\in\mathbb{R}^{r\times r\times C\times P}$ represent the input, output, and weight "tensors" of the CONV layer, respectively. Here, $W$ and $H$ are the spatial dimensions of the input maps, $C$ is the number of input maps, $r$ is the size of the convolutional kernel, and $P$ is the number of output maps.

We generalize the concept of "block-circulant structure" to the rank-4 tensor ($\mathcal{F}$) in the CONV layer, i.e., *all the slices of the form $\mathcal{F}(\cdot,\cdot,i,j)$ are circulant matrices.* Next, we reformulate the inference and training algorithms of the CONV layer to matrix operations. We use the inference process as an example, and the training process can be formulated in a similar way.

Software tools such as Caffe provide an efficient methodology of transforming tensor-based operations in the CONV layer to matrix-based operations , in order to enhance the implementation efficiency (GPUs are optimized for matrix operations.) Fig.  illustrates the application of the method to reformulate Eqn. (6) to the matrix multiplication $\mathbf{Y}=\mathbf{XF}$, where $\mathbf{X}\in\mathbb{R}^{(W-r+1)(H-r+1)\times Cr^2}$, $\mathbf{Y}\in\mathbb{R}^{(W-r+1)(H-r+1)\times P}$, and $\mathbf{F}\in\mathbb{R}^{Cr^2\times P}$.

Recall that the slice of $\mathcal{F}(\cdot,\cdot,i,j)$ is a circulant matrix. Then according to the reshaping principle between $\mathcal{F}$ and $\mathbf{F}$, we have: $$f_{a+C(i-1)+Cr(j-1),b}=f_{C(i-1)+Cr(j-1),b-a}, \forall a, b$$ which means $\mathbf{F}$ is actually a block-circulant matrix. Hence the fast multiplication approach for block circulant matrix, as the "FFT$\rightarrow$ component-wise multiplication $\rightarrow$IFFT" procedure, can now be applied to accelerate $\mathbf{Y}=\mathbf{XF}$, thereby resulting in the acceleration of (6). With the use of the proposed approach, the computational complexity for (6) is reduced from O($WHr^2CP$) to O($WHQ\log Q$), where $Q=\max(r^2C,P)$.

## Outline of Theoretical Proof

With the substantial reduction of weight storage and computational complexities, we attempt to prove that the proposed block-circulant matrix-based framework will consistently yield the similar overall accuracy compared with DNNs without compression. Only testing on existing benchmarks is insufficient given the rapid emergence of new application domains, DNN models, and data-sets. The theoretical proof will make the proposed method theoretically rigorous and distinct from prior work.

In the theory of neural networks, the "*effectiveness*" is defined using the *universal approximation property*, which states that a neural network should be able to approximate any continuous or measurable function with arbitrary accuracy provided that an enough large number of parameters are available. This property provides the theoretical guarantee of using neural networks to solve machine learning problems, since machine learning tasks can be formulated as finding a proper approximation of an unknown, high-dimensional function. Therefore, the goal is to prove *the universal approximation property of block circulant matrix-based neural networks*, and more generally, for arbitrary structured matrices satisfying the low displacement rank $\gamma$. The detailed proofs for the block circulant matrix-based networks and general structured matrix-based ones are provided in the technical reports .

The proof of the universal approximation property for block circulant matrix-based neural networks is briefly outlined as follows: Our objective is to prove that any continuous or measurable function can be approximated with arbitrary accuracy using a block-circulant matrix-based network. Equivalently, we aim to prove that the *function space* achieved by block-circulant matrix-based neural networks is ***dense*** in the space of continuous or measurable functions with the same inputs. An important property of the activation function, i.e., the *component-wise discriminatory* property, is proved. Based on this property, the above objective is proved using proof by contradiction and Hahn-Banach Theorem .

We have further derived *an approximation error bound of* O($1/n$) when the number of neurons in the layer $n$ is limited, with details shown in . It implies that the approximation error will reduce with an increasing $n$, i.e., an increasing number of neurons/inputs in the network. As a result, we can guarantee the universal "effectiveness" of the proposed framework on different DNN types and sizes, application domains, and hardware/software platforms.

## Compression Ratio and Test Accuracy

In this section, we apply CirCNN to different DNN models *in software* and investigate the weight compression ratio and accuracy. Fig. (a) and (b) show the weight storage (model size) reduction in FC layer and test accuracy on various image recognition datasets and DCNN models: MNIST (LeNet-5), CIFAR-10, SVHN, STL-10, and ImageNet (using AlexNet structure) ). Here, 16-bit weight quantization is adopted for model size reduction. The baselines are the original DCNN models with unstructured weight matrices using 32-bit floating point representations. We see that block-circulant weight matrices enable 400$\times$-4000+$\times$ reduction in weight storage (model size) in corresponding FC layers. This parameter reduction in FC layers is also observed in . The entire DCNN model size (excluding softmax layer) is reduced by 30-50$\times$ when only applying block-circulant matrices to the FC layer (and quantization to the overall network). Regarding accuracy, the loss is negligible and sometimes the compressed models even outperform the baseline models.

Fig. (c) illustrates the further application of block-circulant weight matrices to the CONV layers on MNIST (LeNet-5), SVHN, CIFAR-10, and ImageNet (AlexNet structure) datasets, when *the accuracy degradation is constrained to be 1-2%* by optimizing the block size. Again 16-bit weight quantization is adopted, and softmax layer is excluded. The 16-bit quantization also contributes to 2$\times$ reduction in model size. In comparison, the reductions of the number of parameters in are 12$\times$ for LeNet-5 (on MNIST dataset) and 9$\times$ for AlexNet. Moreover, another crucial property of CirCNN is that *the parameter storage after compression is regular*, whereas result in irregular weight storage patterns. The irregularity requires additional index per weight and significantly impacts the available parallelism degree. From the results, we clearly see the significant benefit and potential of CirCNN: it could produce highly compressed models with regular structure. CirCNN yields more reductions in parameters compared with the state-of-the-art results for LeNet-5 and AlexNet. In fact, the actual gain could even be higher due to the indexing requirements of .

We have also performed testing on other DNN models such as DBN, and found that CirCNN can achieve similar or even higher compression ratio, demonstrating the wide application of block-circulant matrices. Moreover, a 5$\times$ to 9$\times$ acceleration in training can be observed for DBNs, which is less phenomenal than the model reduction ratio. This is because GPUs are less optimized for FFT operation than matrix-vector multiplications.

# CirCNN Architecture

Based on block-circulant matrix-based algorithms, we propose CirCNN architecture, — a universal DNN inference engine that can be implemented in various hardware/software platforms with configurable network architecture (e.g., layer type, size, scales, etc.).

Applying CirCNN to neural network accelerators enables notable architectural innovations. *1)* Due to its recursive property and its intrinsic role in CirCNN, FFT is implemented as the *basic computing block* (Section ). It ensures universal and small-footprint implementations. *2)* *Pipelining and parallelism* optimizations (Section ). Taking advantage of the compressed but regular network structures, we aggressively apply inter-level and intra-level pipelining in the basic computing block. Moreover, we can conduct joint-optimizations considering parallelization degree, performance and power consumption. *3)* *Platform-specific optimizations* focusing on weight storage and memory management.(Section ).

## Recursive Property of FFT: the Key to Universal and Small Footprint Design

In CirCNN, the "FFT$\rightarrow$component-wise multiplication $\rightarrow$IFFT" in Fig. is a universal procedure used in both FC and CONV layers, for both inference and training processes, and for different DNN models. We consider *FFT as the key computing kernel* in CirCNN architecture due to its **recursive property**. It is known that FFT can be highly efficient with O($n\log n$) computational complexity, and hardware implementation of FFT has been investigated in . The recursive property states that the calculation of a size-$n$ FFT (with $n$ inputs and $n$ outputs) can be implemented using two FFTs with size $n/2$ plus one additional level of butterfly calculation, as shown in Fig. . It can be further decomposed to four FFTs with size $n/4$ with two additional levels.

The recursive property of FFT is the key to ensure *a universal and reconfigurable* design which could handle different DNN types, sizes, scales, etc. It is because: *1)* A large-scale FFT can be calculated by recursively executing on the same computing block and some additional calculations; and *2)* IFFT can be implemented using the same structure as FFT with different preprocessing procedure and parameters . It also ensures the design with small footprint, because: *1)* Multiple small-scale FFT blocks can be multiplexed and calculate a large-scale FFT with certain parallelism degree; and *2)* The additional component-wise multiplication has O($n$) complexity and relatively small hardware footprint.

Actual hardware systems, such as FPGA or ASIC designs, pose constraints on parallel implementation due to hardware footprint and logic block/interconnect resource limitations. As a result, we define the *basic computing block* with a parallelization degree $p$ and depth $d$ (of butterfly computations), as shown in Fig. . A *butterfly computation* in FFT comprises cascade connection of complex number-based multiplications and additions . The basic computing block is responsible for implementing the major computational tasks (FFT and IFFTs). An FFT operation (with reconfigurable size) is done by decomposition and iterative execution on the basic computing blocks.

Compared with conventional FFT calculation, we simplify the FFT computing based on the following observation: Our inputs of the deep learning system are from actual applications and are real values without imaginary parts. Therefore, the FFT result of each level will be a symmetric sequence except for the base component . As an example shown in the basic computing block shown in Fig. , the partial FFT outcomes at each layer of butterfly computations will be symmetric, and therefore, the outcomes in the red circles do not need to be calculated and stored as partial outcomes. This observation can significantly reduce the amount of computations, storage of partial results, and memory traffic.

## Overall Architecture

The overall CirCNN architecture is shown in Fig. , which includes the basic computing block, the peripheral computing block, the control subsystem, the memory subsystem, and I/O subsystem (I/O buffers). The basic computing block is responsible for the major FFT and IFFT computations. The peripheral computing block is responsible for performing component-wise multiplication, ReLU activation, pooling etc., which require lower (linear) computational complexity and hardware footprint. The implementations of ReLU activation and pooling are through comparators and have no inherent difference compared with prior work . The control subsystem orchestrates the actual FFT/IFFT calculations on the basic computing block and peripheral computing block. Due to the different sizes of CONV layer, FC layer and different types of deep learning applications, the different setting of FFT/IFFT calculations is configured by the control subsystem. The memory subsystem is composed of ROM, which is utilized to store the coefficients in FFT/IFFT calculations (i.e., the $W_n^i$ values including both real and imaginary parts); and RAM, which is used to store weights, e.g., the FFT results $\text{FFT}(\mathbf{w}_{ij})$. For ASIC design, a memory hierarchy may be utilized and carefully designed to ensure good performance.

We use 16-bit fixed point numbers for input and weight representations, which is common and widely accepted to be enough accurate for DNNs . Furthermore, it is pointed out that inaccuracy caused by quantization is largely independent of inaccuracy caused by compression and the quantization inaccuracy will not accumulate significantly for deep layers.

## Pipelining and Parallelism

Thanks to the regular block-circulant matrix structure, effective pipelining can be utilized to achieve the optimal tradeoff between energy efficiency and performance (throughput). CirCNN architecture considers two pipelining techniques as shown in Fig. . In *inter-level pipelining*, each pipeline stage corresponds to one level in the basic computing block. In *intra-level pipelining*, additional pipeline stage(s) will be added within each butterfly computation unit, i.e., by deriving the optimal stage division in the cascade connection of complex number-based multiplication and additions. The proper selection of pipelining scheme highly depends on the target operating frequency and memory subsystem organization. In the experimental prototype, we target at a clock frequency around 200MHz (close to state-of-the-art ASIC tapeouts of DCNNs ), and therefore the inter-level pipelining with a simpler structure will be sufficient for efficient implementations.

Based on the definition of $p$ (parallelization degree) and $d$ (parallelization depth), larger $p$ and $d$ values indicate higher level of parallelism and therefore will lead to higher performance and throughput, but also with higher hardware cost/footprint. A larger $d$ value would also result in less memory accesses at the cost of higher control complexity. We derive the optimal $p$ and $d$ values by optimizing an overall metric $M\big(Perf(p,d),$ $Power(p,d)\big)$ as a suitable function of (average) performance and power consumption. The performance $Perf(p,d)$ will be an increasing function of $p$ and $d$ but also depends on the platform specifications, and the target type and size of DNN models (averaged over a set of learning models). The power consumption $Power(p,d)$ is a close-to-linear function of $pd$ accounting for both static and dynamic components of power dissipation. The optimization of $p$ and $d$ is constrained by the compute and memory hardware resource limits as well as memory and I/O bandwidth constraints. The proposed algorithm for design optimization is illustrated in Algorithm , which sets $p$ as optimization priority in order not to increase control complexity. This algorithm depends on the accurate estimation of performance and power consumption at each design configuration.

We provide an example of design optimization and effects assuming a block size of 128 for FPGA-based implementation (Cyclone V). Because of the low operating frequency, increasing the $p$ value from 16 to 32 while maintaining $d=1$ only increases power consumption by less than 10%. However, the performance can be increased by 53.8% with a simple pipelining control. Increasing $d$ from 1 to 2 results in even less increase in power of 7.8%, with performance increase of 62.2%. The results seem to show that increasing $d$ is slightly more beneficial because of the reduced memory access overheads. However, a $d$ value higher than 3 will result in high control difficulty and pipelining bubbles, whereas $p$ can be increased with the same control complexity thanks to the high bandwidth of block memory in FPGAs. As a result, we put $p$ as the optimization priority in Algorithm 3.

## Platform-Specific Optimizations

Based on the generic CirCNN architecture, this section describes platform-specific optimizations on the FPGA-based and ASIC-based hardware platforms. We focus on weight storage and memory management, in order to simplify the design and achieve higher energy efficiency and performance.

**FPGA Platform**. The *key observation* is that the weight storage requirement of representative DNN applications can be (potentially) met by the on-chip block memory in state-of-the-art FPGAs. As a representative large-scale DCNN model for ImageNet application, the whole AlexNet results in only *around 4MB storage* requirement after (i) applying block-circulant matrices only to FC layers, and (ii) using 16-bit fixed point numbers that results in negligible accuracy loss. Such storage requirement can be fulfilled by the on-chip block memory of state-of-the-art FPGAs such as Intel (former Altera) Stratix, Xilinx Virtex-7, etc., which consist of up to tens of MBs on-chip memory . Moreover, when applying block-circulant matrices also to CONV layers, the storage requirement can be further *reduced to 2MB or even less* (depending on the block size and tolerable accuracy degradation). Then, the storage requirement becomes comparable with the input size and can be potentially supported by low-power and high energy-efficiency FPGAs such as Intel (Altera) Cyclone V or Xilinx Kintex-7 FPGAs. A similar observation holds for other applications (e.g., the MNIST data-set ), or different network models like DBN or RNN. Even for future larger-scale applications, the full model after compression will (likely) fit in an FPGA SoC leveraging the storage space of the integrated ARM core and DDR memory . This observation would make the FPGA-based design significantly more efficient.

In state-of-the-art FPGAs, on-chip memory is organized in the form of memory blocks, each with certain capacity and bandwidth limit. The number of on-chip memory blocks represents a proper tradeoff between the lower control complexity (with more memory blocks) and the higher energy efficiency (with fewer memory blocks), and thus should become an additional knob for design optimizations. State-of-the-art FPGAs are equipped with comprehensive DSP resources such as $18\times 18$ or variable-size multipliers , which are effectively exploited for performance and energy efficiency improvements.

**ASIC platform**. We mainly investigate two aspects in the memory subsystem: *1)* the potential memory hierarchy and *2)* the memory bandwidth and aspect ratio. The representative deep learning applications require hundreds of KBs to multiple MBs memory storage depending on different compression levels, and we assume a conservative value of multiple MBs due to the universal and reconfigurable property of CirCNN architecture. The potential memory hierarchy structure depends strongly on the target clock frequency of the proposed system. Specifically, if we target at a clock frequency around 200MHz (close to state-of-the-art ASIC tapeouts of DCNNs ), then the memory hierarchy is not necessary because a single-level memory system can support such operating frequency. Rather, memory/cache reconfiguration techniques can be employed when executing different types and sizes of applications for performance enhancement and static power reduction. If we target at a higher clock frequency, say 800MHz, an effective memory hierarchy with at least two levels (L1 cache and main memory) becomes necessary because a single-level memory cannot accommodate such high operating frequency in this case. Please note that the cache-based memory hierarchy is highly efficient and results in very low cache miss rate because, *prefetching* , the key technique to improve performance, will be highly effective due to the regular weight access patterns. The effectiveness of prefetching is due to the *regularity* in the proposed block-circulant matrix-based neural networks, showing another advantage over prior compression schemes. In our experimental results in the next section, we target at a lower clock frequency of 200MHz and therefore the memory hierarchy structure is not needed.

Besides the memory hierarchy structure, the memory bandwidth is determined by the parallelization degree $p$ in the basic computing block. Based on such configuration, the aspect ratio of the memory subsystem is determined. Because of the relatively high memory bandwidth requirement compared with the total memory capacity (after compression using block-circulant matrices), column decoders can be eliminated in general , thereby resulting in simpler layout and lower routing requirements.

# Evaluation

In this section, we provide detailed experimental setups and results of the proposed universal inference framework on different platforms including FPGAs, ASIC designs, and embedded processors. Experimental results on representative benchmarks such as MNIST, CIFAR-10, SVHN, and ImageNet have been provided and we have conducted a comprehensive comparison with state-of-the-art works on hardware deep learning systems. Order(s) of magnitude in energy efficiency and performance improvements can be observed using the proposed universal inference framework.

## FPGA-Based Testing

First, we illustrate our FPGA-based testing results using a low-power and low-cost Intel (Altera) Cyclone V 5CEA9 FPGA. The Cyclone V FPGA exhibits a low static power consumption less than 0.35W and highest operating frequency between 227MHz and 250MHz (but actual implementations typically have less than 100MHz frequency), making it a good choice for energy efficiency optimization of FPGA-based deep learning systems.

Fig.  illustrates the comparison of performance (in giga operations per second, GOPS) and energy efficiency (in giga operations per Joule, GOPS/W) between the proposed and reference FPGA-based implementations. The FPGA implementation uses the AlexNet structure, a representative DCNN model with five CONV layers and three FC layers for the ImageNet applications . The reference FPGA-based implementations are state-of-the-arts represented by \[FPGA16\] , \[ICCAD16\] , \[FPGA17, Han\] , and \[FPGA17, Zhao\] . The reference works implement large-scale AlexNet, VGG-16, medium-scale DNN for CIFAR-10, or a custom-designed recurrent neural network . Note that we use equivalent GOPS and GOPS/W for all methods with weight storage compression, including ours. Although those references focus on different DNN models and structures, both GOPS and GOPS/W are general metrics that are independent of model differences. It is widely accepted in the hardware deep learning research to compare the GOPS and GOPS/W metrics between their proposed designs and those reported in the reference work, as shown in . Please note that this is not entirely fair comparison because our implementation is layerwise implementation (some reference works implement end-to-end networks) and we extracts on-chip FPGA power consumptions.

In Fig. , we can observe the significant improvement achieved by the proposed FPGA-based implementations compared with prior arts in terms of energy efficiency, even achieving 11$\times$-16$\times$ improvement when comparing with prior work with heuristic model size reduction techniques (reference uses the heuristic weight pruning method, and uses a binary-weighted neural network XOR-Net). When comparing with prior arts with a uncompressed (or partially compressed) deep learning system , the energy efficiency improvement can reach 60-70$\times$. These results demonstrate a clear advantage of CirCNN using block-circulant matrices on energy efficiency. The performance and energy efficiency improvements are due to: *1)* algorithm complexity reduction and *2)* efficient hardware design, weight reduction, and elimination of weight accessing to the off-chip storage. The first source results in 10$\times$-20$\times$ improvement and the second results in 2$\times$-5$\times$. Please note that CirCNN architecture does not yield the highest throughput because we use a single low-power FPGA, while reference uses a high-performance FPGA together with large off-chip DRAM on a custom-designed recurrent neural network. If needed, we can increase the number of FPGAs to process multiple neural networks in parallel, thereby improving the throughput without incurring any degradation in the energy efficiency.

Besides comparison with large-scale DCNNs with CirCNN architecture with state-of-the-art FPGA implementations, we also compare it with IBM TrueNorth neurosynaptic processor on a set of benchmark data sets including MNIST, CIFAR-10, and SVHN, which are all supported by IBM TrueNorth[^1]. This is a fair comparison because both the proposed system and IBM TrueNorth are end-to-end implementations. IBM TrueNorth is a neuromorphic CMOS chip fabricated in 28nm technology, with 4096 cores each simulating 256 programmable silicon neurons in a time-multiplexed manner. It implements *Spiking Neural Networks*, which is a bio-inspired type of neural networks and benefits from the ability of globally asynchronous implementations, but is widely perceived to achieve a lower accuracy compared with state-of-the-art DNN models. IBM TrueNorth exhibits the advantages of reconfigurability and programmability. Nevertheless, reconfigurability also applies to CirCNN architecture.

Fig. compares the throughput and energy efficiency of FPGA-based implementations of CirCNN architecture and IBM TrueNorth on different benchmark data sets. The throughput and energy efficiency of IBM TrueNorth results are from  ( for MNIST and for CIFAR-10 and SVHN), and we choose results from the low-power mapping mode using only a single TrueNorth chip for high energy efficiency. We can observe the improvement on the throughput for MNIST and SVHN data sets and energy efficiency on the same level of magnitude. The throughput of CIFAR-10 using the FPGA implementation of CirCNN is lower because *1)* TrueNorth requires specific preprocessing of CIFAR-10   before performing inference, and *2)* the DNN model we chose uses small-scale FFTs, which limits the degree of improvements. Besides, CirCNN architecture achieves the higher test accuracy in general: it results in *very minor accuracy degradation compared with software DCNNs* (cf. Fig. ), whereas the low-power mode of IBM TrueNorth incurs higher accuracy degradation . These results demonstrate the high effectiveness of CirCNN architecture because it is widely perceived that FPGA-based implementations will result in lower performance and energy efficiency compared with ASIC implementations, with benefits of a short development round and higher flexibility.

## ASIC Designs Synthesis Results

We derive ASIC synthesis results of CirCNN architecture for large-scale DCNN implementations (AlexNet) and compare with state-of-the-art ASIC developments and synthesis results. The delay, power, and energy of our ASIC designs are obtained from synthesized RTL under Nangate 45nm process using Synopsys Design Compiler. The memories are SRAM based and estimated using CACTI 5.3 . Fig.  illustrates the comparison results on ASIC-based implementations/synthesis with state-of-the-arts, which also target at large-scale deep learning systems. The reference ASIC implementations and synthesis results are represented by \[EIE\] , \[Eyeriss\] , \[ISSCC16, KAIST\] , \[ISSCC17, ST\] , and \[ISSCC17, KULeuvin\] .

In Fig. , we can observe that our synthesis results achieve both the highest throughput and energy efficiency, more than 6 times compared with the highest energy efficiency in the best state-of-the-art implementations. It is also striking that even our FPGA implementation could achieve the same order of energy efficiency and higher throughput compared with the best state-of-the-art ASICs. It is worth noting that the best state-of-the-art ASIC implementations report the highest energy efficiency in the near-threshold regime with an aggressively reduced bit-length (say 4 bits, with a significant accuracy reduction in this case). When using 4-bit input and weight representations and near-threshold computing of 0.55V $V_{dd}$ voltage level, another 17$\times$ improvement on energy efficiency can be achieved in the synthesis results compared with our super-threshold implementation, as shown in Fig. . This makes it a total of 102$\times$ improvement compared with the best state-of-the-art. Moreover, in our systems (in the super-threshold implementation), memory in fact consumes slightly less power consumption compared with computing blocks, which demonstrates that weight storage is no longer the system bottleneck. Please note that the overall accuracy when using 4-bit representation is low in CirCNN architecture (e.g., less than 20% for AlexNet). Hence, 4-bit representation is only utilized to provide a fair comparison with the baseline methods using the same number of bits for representations.

We also perform comparison on performance and energy efficiency with the most energy-efficient NVIDIA Jetson TX1 embedded GPU toolkit, which is optimized for deep learning applications. It can be observed that an energy efficiency improvement of 570$\times$ can be achieved using our implementation, and the improvement reaches 9,690$\times$ when incorporating near-threshold computing and 4-bit weight and input representations.

## Embedded ARM-based Processors

Because ARM-based embedded processors are the most widely used embedded processors in smartphones, embedded and IoT devices, we implement the proposed block-circulant matrix-based DNN inference framework on a smartphone using ARM Cortex A9 processor cores, and provide some sample results. The aim is to demonstrate the potential of real-time implementation of deep learning systems on embedded processors, thereby significantly enhancing the wide adoption of (large-scale) deep learning systems in personal, embedded, and IoT devices. In the implementation of LeNet-5 DCNN model on the MNIST data set, the proposed embedded processor-based implementation achieves a performance of 0.9ms/image with 96% accuracy, which is slightly faster compared with IBM TrueNorth in the high-accuracy mode (1000 Images/s). The energy efficiency is slightly lower but at the same level due to the peripheral devices in a smartphone. When comparing with a GPU-based implementation using NVIDIA Tesla C2075 GPU with 2,333 Images/s, the energy efficiency is significantly higher because the GPU consumes 202.5W power consumption, while the embedded processor only consumes around 1W. It is very interesting that when comparing on the fully-connected layer of AlexNet, our smartphone-based implementation of CirCNN even achieves higher throughput (667 Layers/s vs. 573 Layers/s) compared with NVIDIA Tesla GPU. This is because the benefits of computational complexity reduction become more significant when the model size becomes larger.

## Summary and Discussions

***Energy Efficiency and Performance:*** Overall, CirCNN architecture achieves a significant gain in energy efficiency and performance compared with the best state-of-the-arts on different platforms including FPGAs, ASIC designs, and embedded processors. The key reasons of such improvements include the fundamental algorithmic improvements, weight storage reduction, a significant reduction of off-chip DRAM accessing, and the highly efficient implementation of the basic computing block for FFT/IFFT calculations. The fundamental algorithmic improvements accounts for the most significant portion of energy efficiency and performance improvements around 10$\times$-20$\times$, and the rest accounts for 2$\times$-5$\times$. In particular, we emphasize that: *1)* the hardware resources and power/energy consumptions associated with memory storage will be at the same order as the computing blocks and will not be the absolute dominating factor of the overall hardware deep learning system; *2)* medium to large-scale DNN models can be implemented in small footprint thanks to the recursive property of FFT/IFFT calculations. These characteristics are the key to enable highly efficient implementations of CirCNN architecture in low-power FPGAs/ASICs and the elimination of complex control logics and high-power-consumption clock networks.

***Reconfigurability:*** It is a key property of CirCNN architecture, allowing it be applied to a wide set of deep learning systems. It resembles IBM TrueNorth and could significantly reduce the development round and promote the wide application of deep learning systems. Unlike IBM TrueNorth, CirCNN*1)* does not need a specialized offline training framework and specific preprocessing procedures for certain data sets like CIFAR ; and *2)* does not result in any hardware resource waste for small-scale neural networks and additional chips for large-scale ones. The former property is because the proposed training algorithms are general, and the latter is because different scales of DNN models can be conducted on the same basic computing block using different control signals thanks to the recursive property of FFT/IFFT. The software interface of reconfigurability is under development and will be released for public testing.

***Online Learning Capability:*** The CirCNN architecture described mainly focuses on the inference process of deep learning systems, although its algorithmic framework applies to both inference and training. We focus on inference because it is difficult to perform online training in hardware embedded deep learning systems due to the limited computing power and data set they can encounter.

# Conclusion

This paper proposes CirCNN, a principled approach to represent weights and process neural networks using *block-circulant* matrices. CirCNN utilizes the *Fast Fourier Transform (FFT)*-based fast multiplication, *simultaneously* reducing the computational complexity (both in inference and training) from O($n^2$) to O($n\log n$) and the storage complexity from O($n^2$) to O($n$), with negligible accuracy loss. We propose the CirCNN architecture, a universal DNN inference engine that can be implemented in various hardware/software platforms with configurable network architecture (e.g., layer type, size, scales, etc.). To demonstrate the performance and energy efficiency, we test CirCNN architecture in FPGA, ASIC and embedded processors. Our results show that CirCNN architecture achieves very high energy efficiency and performance with a small hardware footprint. Based on the FPGA implementation and ASIC synthesis results, CirCNN achieves 6 - 102$\times$ energy efficiency improvements compared with the best state-of-the-art results.

# Acknowledgement

This work is funded by the National Science Foundation Awards CNS-1739748, CNS-1704662, CNS-1337300, CRII-1657333, CCF-1717754, CNS-1717984, Algorithm-in-the-Field program, DARPA SAGA program, and CASE Center at Syracuse University.

[^1]: Please note that AlexNet is not currently supported by IBM TrueNorth due to the high-degree neural connections.


author: Alessandro De Angelis[^1]
title: Galileo Galilei and a forgotten poem on the 1604 supernova

On the evening of October 9, 1604, curious and enthusiasts from all over the world were observing a rare conjunction between Mars, Jupiter and Saturn, full of astrological implications. Suddenly, near that place in the sky, which was located in the Ophiucus constellation, near the foot of the Serpent bearer (Fig. ), a new light appeared, brighter than all the planets with the exception of Venus. It remained there for a year and a half and then, as it had appeared, it disappeared. That "new star", as it was called, changed the history of astronomy and cosmology: it was believed at the time that stars were fixed, immutable and ungenerable, but obviously this was not true. Scientists with different conceptions of the Universe competed and collaborated to explain the nature and origin, including Galilei and Kepler, and also Chinese, Korean and Arab astronomers reported observations . Today we know that that new star was a supernova, the last of the seven supernovae observed with the naked eye in the Milky Way – the six previous ones of which there is documentation were recorded in the years 185, 393, 1006, 1054, 1181, 1572. So not a new star, but a star that dies and explodes. What remains today is a celestial object still full of mysteries: the cloud projected by the explosion is expanding at very high speed, in some places ten thousand kilometers per second, one thirtieth of the speed of light, and the shock waves of this expansion accelerate cosmic particles up to very high energies. We astrophysicists of today still study it carefully , always with an eye to the first analysis of our illustrious predecessors of four hundred years ago.

In October 1604, Galileo Galilei was professor of mathematics and astronomy at the University of Padua, teaching the mechanics of planets. When the "stella nova" appeared in the sky, he was therefore the figure of reference to whom all the doubts and questions that such an apparition brought with it were addressed.

That bright and pulsating sphere had generated wonder, but also terror and curiosity. The University of Padua asked Galilei to outline the situation by exposing in three public conferences his point of view, in order to answer to the many questions that raged among the academic community and the common people. Three conferences that Galilei held almost immediately, between November and December, in the Aula Magna of the Bo, the central building of the University – fragments of these lectures are in the Italian national edition of the Opere di Galileo Galilei (Works of Galileo Galilei), edited by Antonio Favaro , vol. II. Even if today it seems difficult to believe it, Galilei reported that that a thousand people attended each lecture (vol. II of , ). Galilei demonstrated through the method of parallax that the new star was beyond the Moon, and therefore in that Heaven that according to Aristotle was immutable. Parallax is the apparent displacement of an object due to a change in the observer's point of view: looking at the tip of your nose with the left eye closed, and then with the right eye closed, your nose will appear to move; knowing the distance between eyes one can determine the length of his/her nose. The same thing could be done from different points on the Earth to measure the position of planets and stars, and Galilei did it in collaboration with Neapolitan and Spanish correspondents. The new star was shown to all observers to be at the same location with respect to the stars of Sagittarius and Scorpio, and therefore it had to be farther away than the Moon and the planets: precisely between the fixed stars. Today only a few fragments of Galilei's lecture notes remain (vol. II of ), and they are fascinating – Galilei was a person of all-round culture, a man of letters as well as a musician, painter and of course a scientist.

A month after Galilei's lectures, a treatise on the supernova appeared in Padua. The unknown Antonio Lorenzini, behind whose name it is easy to see the inspiration of Cesare Cremonini, an Aristotelian professor of natural philosophy in Padua and friend-rival of his colleague Galilei, published a booklet entitled *Discourse around the new star* which debunked the conclusions of Galilei. In order to justify Aristotle of Stagira's concept of immutability of the sky, which seemed to be contradicted by the new star, Lorenzini affirmed that looking at the stars to measure their distance was useless: the principles of terrestrial physics did not apply to the sky.

The response to Lorenzini's speech was rapid and original, but also very detailed. On February 28, 1605 was published in Padua, by the same publisher Tozzi who had published the book of Lorenzini, the *Dialogo de Cecco di Ronchitti da Bruzene in perpuosito de la stella nuova (Dialogue by Cecco di Ronchitti from Bruzene about the new star)* , a pseudonymous booklet in Paduan dialect (a local form of Venetian dialect) written according to Antonio Favaro and to many critics by Galilei together with his student and Benedictine monk Girolamo Spinelli (who had a deeper knowledge of the Paduan dialect). It is not known how many copies of the book (whose cover page is shown in Fig. ) were published; certainly a second edition was printed the same year in Verona by the publisher Merlo , with minimal changes (not substantial but with some improvements from the point of view of readability) to the text and a substantial change in the appendix, which contained *Alcune ottave d'incerto, per la medesima stella, contra Aristotele (Some octaves by unknown author, for the same star, against Aristotle)*, in Florentine vernacular, which were substantially modified.

The *Dialogo* was an explicit mockery of what had been published by Lorenzini; it defamed the entire Aristotelian dogma, which explains why the author(s) preferred to hide under the pseudonym of Cecco di Ronchitti. Cecco di Ronchitti from Brugine (a small village of peasants near Padua), an unknown self-proclaimed farmer and land surveyor, gave in the book many indications of being in reality an astronomer from Padua. The fact that the book was written in Paduan dialect instead of Latin, which was the noble language used to deal with relevant subjects, was a sign that the topic around which the booklet was built was not worthy of consideration,

In addition to the fact that the dialect was perfect for light-hearted teasing, the dialect of Padua was in vogue thanks to the writings of Angelo Beolco, a sort of *ante litteram* "blogger" who had published in the first half of the sixteenth century under the name of "Ruzante" or "Ruzzante", which in Paduan means "scratcher" or "mumbler", about the events that took place in Padua.

The *Dialogue* takes place between two peasants, Matthio (Matteo) and Nale (Natale), who confront each other after Natale read Lorenzini's book and summarizes to Matteo its contents. The booklet analyzes point by point the contradictions and distortions expressed by Lorenzini in his *Discourse.* It is like Galilei did not want to react personally to what Lorenzini had written, but did so through the dialectal, intelligent and frank voices of two peasants.

It is easy to recognize in the content of the *Dialogo* many characteristics of Galilei, so that already at the time of its publication the treatise was attributed to him. In addition to this, no family named Ronchitti is recorded in the birth registers of the parish of Brugine or in the neighboring towns; moreover, Cecco is also a Tuscan nickname. Galilei had good knowledge of the Paduan vernacular: Niccolò Gherardini, in his *Vita del signor Galileo Galilei (Life of Galileo Galilei)* , writes: "he was still very familiar with a book entitled *Il Ruzzante,* written in the rustic Paduan language, taking pleasure in those crude tales and ridiculous incidents" (Gherardini confuses the author with the title). Among the *Miscellanea galileiana inedita (Unpublished Galilei's works)* published by Favaro in Venice in 1887 , one can read a thought of Galilei that contains words of the rustic Paduan dialect. Finally, in his letter to Paolo Gualdo of June 16, 1612, published in volume XI of the *Opere* , Galilei writes whole sentences using the same dialect, and Galilei's Venetian correspondents often wrote to him in various flavors of the Venetian dialect. In a letter to Paolo Gualdo dated August 16, 1614, Girolamo Spinelli was called by Galilei "one of my students, monk of Santa Giustina, companion of Cecco de' Ronchetti". A focus on the form of the text caused Galilei's contribution to be greatly diminished in editions following Favaro's (especially from Paduan critics), in favor of Girolamo Spinelli. This gradual deauthorization has certainly also depended on an excessive attention to form, admittedly so particular, which has unfortunately come at the expense of attention to specific content, which is equally (if not even more) important .

In the mouth of one of the two interlocutors, Matteo, there are Galilean concepts; we can say that Galilei's ideas about the new star are all in Matteo's words. The Copernican idea of the author of the *Dialogue* is in the explanation of the concept of parallax made in the booklet, which also contemplates the seasonal displacement of the Earth linked to its revolution around the Sun. Two drawings reproduced from Galilean manuscripts kept at the Biblioteca Nazionale Centrale di Firenze, presumably contemporary to the publication of the *Dialogue,* and included in a section entitled by Galilei *De stella anni 1604 (About the star of 1604)*, clarify even better the meaning of parallax applied to the Earth's revolution.[^2]

Note the Copernican essence of the drawings, and the outer circle of the Zodiac with the signs. In the first drawing the Earth, third planet from the Sun which is the black point in the center, is indicated by a sign badly written, as if Galilei was afraid that his notes would fall into the wrong hands, but it is still between Mars and Venus and is the center of the observations – a drawing similar will be reproduced in the *Dialogue concerning the two chief world systems* in 1632. In the second you can see the seasonal variation of parallax. The drawings are not in the original first edition of the *Opere* , but were added in an appendix to the second volume in later editions.

In addition to being more compelling than a treatise, as Plato had already shown and as prescribed in numerous courses on rhetoric of Galilei's times, a dialogue allows for circumventing formalism in certain conclusions that Galilei was probably not able to develop rigorously: a dialogue allows its participants to forego certain rigorous demonstrations and replace them with assumptions of sufficient plausibility, made stronger by the use of common sense and of humour. In this sense the technique of persuasion in this dialogue anticipates by three decades the two great Galilei's dialogues: the *Dialogue concerning the two chief world systems* (1632) and the *Discourses and mathematical demonstrations relating to two new sciences* (1638).

The present *Dialogue* appears as Galilei's work in the second volume of the Italian national edition of the *Opere* . In this edition it does not appear instead a poem in octaves (i.e., groups or "stanzas" of eight verses), by an unknown author, published as an appendix to the first edition of 1605, and immediately replaced in the second edition. An English translation of the *Dialogue* appears in a book by Drake ; a commented translation in modern Italian is published in . Some (see , also for notes on the litterary production by Galilei) possibly attribute it to Antonio Querengo[^3], to whom the *Dialogue* is dedicated.

The following elements strengthen the attribution of the poem to Galilei:

-   Galilei had likely a strong participation to the *Dialogo de Cecco di Ronchitti*; being a poet (the full volume IX of the national edition of his *Opere* is dedicated to his litterary activity), it appears strange that the poem in the appendix of the *Dialogo* itself was written by somebody else, or at least that Galilei did not heavily rework it;

-   the style, including the verbal violence, is coherent with Galilei, and there is quite a strong attack to the Aristotelians;

-   as Milani observes in her 1992 translation of the *Dialogo* (the translation of the text is done from the second edition ), some rhymes are common with those of Galilei, and one stanza begins with the imperative "Dunque", as in Galilei's reworking of Andrea Salvadori's *Canzone per le Stelle Medicee* (vol. IX of Galilei's *Opere* );

-   a description of the method of parallax is appropriate for an astronomer, and a Copernican statement is present in the fourth stanza of the poem.

In any case, attribution always contains some degree of arbitrariness.

Here follows the poem published in the appendix of the *Dialogue,* in the first edition of 1605.

cm

STANZE D'INCERTO CONTRA ARISTOTELE
PER LA STELLA NUOVAMENTE APPARSA

> Che più vaneggi, o Stagirita stolto:
> e puro il Cielo e ingenerabil credi?
> Stella nuova, in lui fissa, il chiaro volto
> discopre scintillando, e non la vedi?
> O più che mai ne' primi errori involto
> il senso neghi ed altre prove chiedi?
> Il senso neghi, onde i principii certi
> dicesti avere de le scienze aperti?
>
> Ma che nel Cielo sia la nata stella
> stimerai forse non verace assunto;
> e chiaro è pur che a questa gente e a quella
> si mostra fissa in un medesmo punto.
> Vede ciascun che da la punta bella
> di Sagittario ha il suo splendor disgiunto
> e quasi fugga di Scorpion la coda
> s'erge tremante e sovra lei s'inchioda.
>
> Or s'ella è sotto il ciel, sotto la Luna,
> sì lungi a quelle del sublime tetto,
> come di un altro clima in parte alcuna
> non si nasconde o cangia almen l'aspetto,
> e al variar de' siti or sembra in una
> or in altra figura aver ricetto,
> anzi pur, come a noi torna mostrarse
> nel mattutino ove la sera apparse?
>
> Né cessan qui le vere mie parole,
> che a ragion ferme e nove ecco ritorno.
> Dimmi, se volge la terrestre mole
> ventidue mila miglia intorno intorno
> e questa nova luce a par del Sole
> tutta la gira in una notte e un giorno,
> come tre ore e più sull'orizzonte
> nostro nel tramontar mostrò la fronte?
>
> Esser non può che, s'ella umile è tanto,
> così il passo allentar da noi sia vista,
> ché in breve spazio di mirarla il vanto
> la retta cederebbe umana vista;
> né patria sempre assimigliar (se, quanto
> da noi s'allunga, mole non acquista
> e quella perde poi mentre s'appressa)
> a l'occhio nostro la grandezza stessa.
>
> Ma poi ch'ella è fissa nel ciel sublime,
> provan l'altezza sua sì ferme prove,
> che come posta infra le stelle prime
> a ragion seco si mostra e move;
> né d'esser può che in altre parti o in ime
> varia di mole o sito, ella si trove,
> ché nulla face il mutar loco in terra
> all'ampiezza del Cielo, in cui si serra.
>
> Dunque di cecità squarciando il panno
> omai la lingua sciogli in vere note
> e se credesti con tuo scorno e danno
> dianzi immutabil le soperne rote,
> or che fiamma novella apre l'inganno
> conosci il Ciel che generar si puote,
> e grazie rendi a la Natura madre,
> ond'hai luci del ver tanto leggiadre.

cm

This is my translation:

cm

STANZAS BY AN UNKNOWN AUTHOR TO THE NEWLY APPEARED STAR
AGAINST ARISTOTLE

> What raving, o foolish Stagirite,
> your belief that creations are forbidden in Heaven!
> A newborn star, in it fixed, its clear face
> shows glittering, and you don't see it?
> Or you deny sense more than ever
> getting stuck in your first mistakes, and ask for other proofs?
> But how to deny this simple evidence
> when all your science was based on sense?
>
> That in Heaven the star was born
> maybe you think not to be true;
> though every nation sees it
> fixed in the same point up on the sky.
> Everybody sees it in the beautiful location
> where Sagittarius has its splendor separated
> and as if from Scorpion's tail escaping
> rises trembling, and it is nailed over it.
>
> Now if it be under the Heavens, under the Moon,
> so far from the stars of the sublime roof,
> as of another climate, how it does not partly hide,
> or at least changes its aspect,
> and in a variety of sites seems now
> in the same constellation to have shelter?
> Indeed, how does it return to us in the morning
> where it appeared in the evening?
>
> Nor here cease my true words,
> which firm and new now return.
> Tell me, if the Earth
> turns twenty-two thousand miles around
> and this new light, like for the Sun
> turns it all round in a night and a day,
> how could it for three hours and more
> on our horizon show its face at sunset?
>
> It cannot be that, if it is so humble,
> it sets its pace as to be seen,
> since in a short time
> it would become invisible;
> nor it could look always similar, unless, as it lengthens from us,
> it does acquire mass
> and then loses it while it draws near
> to look the same in bulk at every place.
>
> But since it is fixed in the sublime sky,
> Its height proves so firmly, that
> as placed between the first stars
> naturally shows itself there and moves accordingly;
> nor it could be found in lower parts
> varying in size or site,
> since its place on Earth does not change
> given the enormous circumference of the Heavens.
>
> Therefore, ripping the cloth blinding you,
> now let your tongue melt into true notes:
> if you believed, at your scorn and harm
> the upper wheels to be immutable,
> now that this new flame shows your mistake,
> you know that Heaven can be generated,
> and thank Mother Nature,
> who sent you such graceful light of truth.

A second edition of the *Dialogo de Cecco di Ronchitti* was published later the same year in Verona . The poem was largely amended, with some uncertainties in the poetic language. In addition, the dangerous Copernican allusion to Earth rotation in the fourth stanza was removed, and some criticisms to Aristotle were softened and became "politically correct" (for example, in the first verse the sentence "stolto Stagirita", where "stolto" literally means "fool", "foolish", or even "stupid" , became "per altro Stagirista saggio", i.e., "otherwise sage Stagirite"). This new version, which I leave here untranslated, when compared to other poems by Galilei, lacks the usual verbal aggressivity.

> Dunque, o per altro Stagirita saggio,
> vaneggi ancora, e l'alte rote credi
> immutabili e pure? Un novo raggio
> tra quei zaffiri è nato, e non lo vedi?
> O pur facendo a te medesmo oltraggio
> il senso neghi, ed altre prove chiedi?
> Il senso neghi, onde i principii certi
> dicesti aver de le scienze aperti?
>
> Ma che in Ciel sia la generata stella
> stimerai forse tu pensier fallace;
> e chiaro è pur che questa gente e quella
> un solo osserva in lei sito verace;
> vede tra' segni luminosi ch'ella
> di Sagittario al dardo aggiunge face,
> e quasi fugga di Scorpion la coda,
> scintilla e s'alza e sovra lei s'inchioda.
>
> Or quando, al tuo parer, sotto la Luna
> resti l'obliquo suo camin ristretto,
> come non mai s'asconde, e non s'imbruna
> a' varii Climi opposta, o cangia aspetto?
> E non rassembra a noi dal Cielo in una,
> ad altri in altra parte haver ricetto?
> Anzi come il mattin torna mostrarse
> fissa nel punto ove la sera apparse?
>
> Aggiungo: e come all'or, che parte o riede
> e si mostra or d'appresso, et or distante
> al Zenit nostro, non mai muta o cede
> la figura, e la luce ogn'or tremante?
> Come in lungo passaggio non la vede,
> e meno o più distinta, occhio costante?
> E non cresce e non scema, al guardo esposta,
> qual'or, girando, s'avvicina e scosta?
>
> Meglio dirò: s'ella il suo corso stende
> pel vicino sentier de gli Elementi,
> come sopra di noi tant'ore splende
> quante del sommo Giro i lumi ardenti?
> Ed essi regolato il moto apprende,
> né vien che più s'affretti, o più s'allenti,
> come l'infimo raggio col supremo
> ne va del par da l'uno a l'altro estremo?
>
> Esser non può che in cerchio angusto e basso
> de' corpi eccelsi al gran viaggio assista,
> e mentre tien su l'Orizonte il passo
> di mille aspetti al variar resista,
> ché d'umil Cerchio una gran parte il crasso
> globo ben toglie a la terrena vista;
> ma la spera stellata ei non ingombra,
> in cui riguardo egli d'un punto è l'ombra.
>
> Or questi effetti indubitati e veri
> son del nostro saper fondate prove,
> poiché la stella unita a i segni alteri
> tra quei scintilla, e si dimostra, e move;
> né da gl'Indi scorrendo a i lidi Iberi,
> fia chi di mole o forma altra la trove,
> ché simil varietà non cape il senso,
> né la distanza di quell'Orbe immenso.
>
> Dunque, o Maestro di color che sanno,
> sgombra le nubi omai da l'intelletto,
> e se credesti, con tuo scorno e danno,
> dianzi immutabil quel soperno oggetto,
> or che fiamma novella apre l'inganno
> fatto il confessa, e a corruttion soggetto,
> e rendi gratie a la Natura Madre,
> ond'hai luci del ver tanto leggiadre.

## Acknowlegedments

I thank Selenia Broccio and Marianna Giannicolo for their help, and Sandro Bettini, Michele Camerota, Luisa Cifarelli, Ivano Paccagnella and Bill Shea for their comments and suggestions (Bill also for many pleasant discussions).

This article is dedicated to the 800th anniversary of the University of Padova, and has been written in the framework of the celebrations. *Universa universis patavina libertas.*

[^1]: E-mail: email@example.com

[^2]: *Manoscritti galileiani* 70, c. 16v e 47, c. 10v. Images reproduced by permission of the Italian Ministero della Cultura, Biblioteca Nazionale Centrale di Firenze.

[^3]: The Paduan Antonio Querengo (or Querenghi, or Quarenghi, or Antuogno Squerengo in the dialectal mispronunciation) was a very learned diplomat and poet in Latin, a friend of Torquato Tasso. After the death in 1601 of Giovanni Vincenzo Pinelli, a good friend and sponsor of Galilei since his arrival in Padua, Querengo's house replaced Pinelli's as a mecca for all important visitors and a meeting place for local scholars and men of letters. Among these were Paolo Gualdo and Lorenzo Pignoria, who were also good friends of Galilei. Querengo continued to follow with interest the career of Galilei after he left Padua, and it is from his pen that we have the most amusing letters concerning the unfortunate campaign of Galilei in Rome in 1615-1616 to support the Copernican point of view.