Tartan: Accelerating Fully-Connected and Convolutional Layers in Deep Learning Networks by Exploiting Numerical Precision Variability
Alberto Delmas Sayeh Sharify Patrick Judd Andreas Moshovos
cs.NE



[NO \title GIVEN]
    [NO \author GIVEN]
    December 30, 2023
======================



 



 



 



  


 (), a hardware accelerator for inference with Deep Neural Networks (DNNs), is presented and evaluated on Convolutional Neural Networks. exploits the variable per layer precision requirements of DNNs to deliver execution time that is proportional to the precision p in bits used per layer for convolutional and fully-connected layers. Prior art has demonstrated an accelerator with the same execution performance only for convolutional layers<cit.>. Experiments on image classification CNNs show that on average across all networks studied,  outperforms a state-of-the-art bit-parallel accelerator <cit.> by without any loss in accuracy while it is more energy efficient. requires no network retraining while it enables trading off accuracy for additional improvements in execution performance and energy efficiency. For example, if a 1% relative loss in accuracy is acceptable, is on average faster and more energy efficient than a conventional bit-parallel accelerator. A Tartan configuration that processes 2-bits at time, requires less area than the 1-bit configuration, improves efficiency to 1.24× over the bit-parallel baseline while being 73% faster for convolutional layers and 60% faster for fully-connected layers is also presented.




§ INTRODUCTION


It is only recently that commodity computing hardware in the form of graphics processors  delivered the performance necessary for practical, large scale Deep Neural Network applications <cit.>. At the same time, the end of Dennard Scaling in semiconductor technology <cit.> makes it difficult to deliver further advances in hardware performance using existing general purpose designs. It seems that further advances in DNN sophistication would have to rely mostly on algorithmic and in general innovations at the software level which can be helped by innovations in hardware design. Accordingly, hardware DNN accelerators have emerged. The DianNao accelerator family was the first to use a wide single-instruction single-data (SISD) architecture to process up to 4K operations in parallel on a single chip  <cit.> outperforming graphics processors by two orders of magnitude. Development in hardware accelerators  has since proceeded in two directions: either toward more general purpose accelerators that can support more machine learning algorithms while keeping performance mostly on par with DaDianNao () <cit.>, or toward further specialization on  specific layers or classes of DNNs with the goal of outperforming in execution time and/or energy efficiency, e.g., <cit.>. This work is along the second direction. While an as general purpose as possible DNN accelerator is desirable further improving performance and energy efficiency for specific machine learning algorithms will provides us with the additional experience that is needed for developing the next generation of more general purpose machine learning accelerators.  
Section <ref> reviews several other accelerator designs.

While 's functional units process 16-bit fixed-point values, DNNs exhibit varying precision requirements across and within layers, e.g., <cit.>. Accordingly, it is possible to use shorter, per layer representations for activations and/or weights. However, with existing bit-parallel functional units doing so does not translate into a performance nor an energy advantage as the values are expanded into the native hardware precision inside the unit. Some designs opt to hardwire the whole network on-chip by using tailored datapaths per layer, e.g., <cit.>. Such hardwired implementations are of limited appeal for  many modern DNNs  whose footprint ranges several 10s or 100s of megabytes of weights and activations.  Accordingly, this work targets accelerators that can translate any precision reduction into performance and that do not require that the precisions are hardwired at implementation time.

This work presents (), a massively parallel hardware accelerator whose execution time for fully-connected () and convolutional () layers scales with the precision p used to represent the input values. uses hybrid bit-serial/bit-parallel functional units and exploits the abundant parallelism of typical DNN layers with the following  goals: 1) exceeding 's execution time performance and energy efficiency, 2) maintaining the same activation and weight memory interface and wire counts, 3) maintaining wide, highly efficient accesses to weight and activation memories. Ideally, improves execution time over by 16/p where p is the precision used for the activations  in and for the activations and weights in . Every single bit of precision that can be eliminated ideally reduces execution time and increases energy efficiency. For example, decreasing precision from 13 to 12 bits in an can ideally boost the performance improvement over to 33% from 23% respectively. builds upon the () accelerator <cit.> which improves execution time and energy efficiency on  only. While  matches the performance of a bit-parallel accelerator on its energy efficiency suffers considerably.  improves  performance and energy efficiency over a bit-parallel accelerator for both and . 

This work evaluates on a set of convolutional neural networks (CNNs) for image classification. On average reduces inference time by ,  and  over for the fully-connected, the convolutional, and all layers  respectively. Energy efficiency compared to with is ,  and  respectively.
 By comparison, efficiency with compared to is 0.73×,  1.21× and 1.14× respectively. Additionally, enables trading off accuracy for improving execution time and energy efficiency. For example, on average on , accepting a 1% loss in relative accuracy improves performance to  and energy efficiency to  compared to .

In detail this work makes the following contributions:



  * Extends the accelerator offering performance improvements on . Not only does not improve performance on , but its energy efficiency suffers compared to . 


  * incorporates  cascading multiple serial inner-product (SIP) units improving utilization when the number or filters or the dimensions of the filters is not a multiple of the datapath lane count.


  * It uses the methodology of Judd et al., <cit.> to determine per layer weight and activation precisions for the fully-connected layers of several modern image classification CNNs.

 
  * It evaluates a configuration of which trades off some of the performance improvement for enhancing energy and area efficiency.
The evaluated configuration processes two activation bits per cycle and requires half the parallelism and the SIPs than the bit-serial configuration. 


  * Reports energy efficiency and area measurements derived from a layout of the accelerator demonstrating its benefits over the preciously proposed and accelerators.


 
The rest of this document is organized as follows: Section <ref> motivates . Section <ref> illustrates the key concepts behind via an example. Section <ref> reviews the architecture and presents an equivalent configuration.  Section <ref> presents the experimental results. Section <ref> reviews related work and discusses the limitations of this study and the potential challenges with . Section <ref> concludes.





§ MOTIVATION


This section motivates by showing that: 1)  the precisions needed for the of several modern image classification CNNs are far below the fixed 16-bit precision used by , and 2) the energy efficiency of is below that of for . Combined these results motivate which improves performance and energy efficiency for both  and compared to .


 §.§ Numerical Representation Requirements Analysis


The experiments of this section corroborate past results that the precisions needed vary per layer for several modern image classification CNNs and during inference.  The section also shows that there is significant potential to improve performance if it were possible to exploit per layer precisions even for the .  The per layer precision profiles presented here were found via the methodology of Judd et al. <cit.>.  Caffe <cit.> was used to measure how reducing the precision of each affects the network's overall top-1 prediction accuracy over 5000 images. The network definitions and pre-trained synaptic weights are taken from the Caffe Model Zoo <cit.>. The networks are used as-is without retraining. Further reductions in precisions may be possible with retraining.
As Section <ref> will explain, 's performance on an  layer L is bound by the maximum of the weight (P_w^L) and activation (P_a^L) precisions. Accordingly, precision exploration was limited to cases where both P^L_w and P^L_a are equal. The search procedure is a gradient descent where a given layer's precision is iteratively decremented one bit at a time, until the network's accuracy drops. For weights, the fixed-point numbers are set to represent values between -1 and +1. For activations, the number of fractional bits is fixed to a previously-determined value known not to hurt accuracy, as per Judd et al.<cit.>. While both activations and weights use the same number of bits, their precisions and ranges differ. For only the activation precision is adjusted as with the design there is no benefit in adjusting the weight precisions as well. Weights remain at 16-bits for . While, reducing the weight precision for can reduce their memory footprint <cit.>, an option we do not explore further in this work.

Table <ref> reports the resulting per layer precisions separately for and . The ideal speedup columns report the performance improvement that would be possible if execution time could be reduced proportionally with  precision compared to a 16-bit bit-parallel baseline. For the , the precisions required range from 8 to 10 bits and the potential for performance improvement is 1.64× on average and ranges from 1.63× to 1.66×. If a 1% relative reduction in accuracy is acceptable then the performance improvement potential increases to 1.75× on average and ranges from 1.63× to as much as 1.85×. Given that the precision variability for is relatively low (ranges from 8 to 11 bits) one may be tempted to conclude that a bit-parallel architecture with 11 bits may be an appropriate compromise. However, note that the precision variability is much larger for the (range is 5 to 13 bits) and thus performance with a fixed precision datapath would be far below the ideal. For example, speedup with a 13-bit datapath would be just 1.23× vs. the 2× that is be possible with an 8-bit precision. A key motivation for is that  its  incremental cost  over that already supports variable per layer precisions for is well justified given the benefits. Section <ref> quantifies this cost and the resulting performance and energy benefits.









 §.§ Energy Efficiency with 



() uses hybrid bit-serial/bit-parallel inner-product units for processing activations and weights respectively exploiting the per layer precision variability of modern CNNs <cit.>. However,  exploits precision reductions only for  as it relies on weight reuse across multiple windows to maintain the width of the weight memory the same as in  (there is no weight reuse in ). Figure <ref> reports the energy efficiency of  over that of for (Section <ref> details the experimental methodology).
While performance is virtually identical to , energy efficiency is on average 0.73× compared to . This result combined with the reduced precision requirements of serves as motivation for extending to improve performance and energy efficiency compared to  on both and .





 §.§ Motivation Summary

This section showed that: 1) The per layer precisions for on several modern CNNs for image classification vary significantly and exploiting them has the potential to improve performance by 1.64× on average. 2) that exploits variable precision requirements only for achieves only 0.73× the energy efficiency of a bit-parallel baseline. Accordingly, an architecture that would exploit precisions for as well as is worth investigating in hope that it will eliminate this energy efficiency deficit resulting in an accelerator that is higher performing and more energy efficient for both layer types. Combined and account for more than 99% of the execution time in .









§ : A SIMPLIFIED EXAMPLE


This section illustrates at a high-level the way operates by showing how it would process two purposely trivial cases: 1) a fully-connected layer  with a single input activation producing two output activations, and 2) a convolutional layer  with two input activations and one single-weight filter producing two output activations. The per layer calculations are:




    Fully-Connected:        Convolutional:
    
    f_1 = w_1 × a        c_1 = w × a_1 
    
    f_2 = w_2 × a        c_2 = w × a_2




Where f_1, f_2, c_1 and c_2 are output activations, w_1, w_2, and w are weights, and a_1, a_2 and a are input activations. For clarity all values are assumed to be represented in 2 bits of precision. 



 §.§ Conventional Bit-Parallel Processing

Figure <ref>a shows a bit-parallel processing engine representative of . Every cycle, the engine can calculate the product of two 2-bit inputs, i (weight) and v (activation) and accumulate or store it into the output register OR. Parts (b) and (c) of the figure show how this unit can calculate the example CVL over two cycles. In part (b) and during cycle 1, the unit accepts along the v input bits 0 and 1 of a_1 (noted as a_1/0 and a_1/1 respectively on the figure), and along the i input bits 0 and 1 of w and produces both bits of output c_1. Similarly, during cycle 2 (part (c)), the unit processes a_2 and w to produce c_2. In total, over two cycles, the engine produced two 2b× 2b products. Processing the example FCL also takes two cycles: In the first cycle, w_1 and a produce f_1, and in the second cycle w_2 and a produce f_2. This process is not shown in the interest of space.










 §.§ 's Approach


Figure <ref> shows how a -like engine would process the example CVL. Figure <ref> shows the engine's structure which comprises two subunits. The two subunits accept each one bit of an activation per cycle through inputs v_0 and v_1 respectively and as before, there is a common 2-bit weight input (i_1,i_0). In total, the number of input bits is 4, the same as in the bit-parallel engine.

Each subunit contains three 2-bit registers: a shift-register AR, a parallel load register BR, and an parallel load output register OR. Each cycle each subunit can calculate the product of its single bit v_i input with BR which it can write or accumulate into its OR.  There is no bit-parallel multiplier since the subunits process a single activation bit per cycle. Instead, two AND gates, a shift-and-add functional unit, and OR form a shift-and-add multiplier/accumulator. Each AR can load a single bit per cycle from one of the i wires, and BR can be parallel-loaded from AR or from the i wires.

Convolutional Layer:
Figure <ref> through Figure <ref> show how processes the CVL. The figures abstract away the unit details showing only the register contents. As Figure <ref> shows, during cycle 1, the w synapse is loaded in parallel to the BRs of both subunits via the i_1 and i_0 inputs. During cycle 2, bits 0 of a_1 and of a_2 are sent via the v_0 and v_1 inputs respectively to the first and second subunit. The subunits calculate concurrently a_1/0× w and a_2/0× w and accumulate these results into their ORs. Finally, in cycle 3, bit 1 of a_1 and a_2 appear respectively on v_0 and v_1. The subunits calculate respectively a_1/1× w and a_2/1× w accumulating the final output activations c_1 and c_2 into their ORs. 

In total, it took 3 cycles to process the layer. However, at the end of the third cycle, another w could have been loaded into the BRs (the i inputs are idle) allowing a new set of outputs to commence computation during cycle 4. That is, loading a new weight can be hidden during the processing of the current output activation for all but the first time. In the steady state, when the input activations are represented in two bits, this engine will be producing two 2b × 2b terms every two cycles thus matching the bandwidth of the bit-parallel engine. 

If the activations a_1 and a_2 could be represented in just one bit, then this engine would be producing two output activations per cycle, twice the bandwidth of the bit-parallel engine. The latter is incapable of exploiting the reduced precision for reducing execution time. In general, if the bit-parallel hardware was using P_BASE bits to represent the activations while only P^L_a bits were enough, would outperform the bit-parallel engine by P_BASE/P_a^L.


Fully-Connected Layer: Figure <ref> shows how a -like unit would process the example FCL. As Figure <ref> shows, in cycle 1, bit 1 of w_1 and of w_2 appear respectively on lines i_1 and i_0. The left subunit's AR is connected to i_1 while the right subunit's AR is connected to i_0. The ARs shift in the corresponding bits into their least significant bit sign-extending to the vacant position (shown as a 0 bit on the example).
During cycle 2, as Figure <ref> shows, bits 0 of w_1 and of w_2 appear on the respective i lines and the respective ARs shift them in. At the end of the cycle, the left subunit's AR contains the full 2-bit w_1 and the right subunit's AR the full 2-bit w_2. 
In cycle 3, Figure <ref> shows that each subunit copies the contents of AR into its BR. From the next cycle, calculating the products can now proceed similarly to what was done for the CVL. In this case, however, each BR contains a different weight whereas when processing the CVL in the previous section, all BRs held the same w value. The shift capability of the ARs coupled with having each subunit connect to a different  i wire allowed to load a different weight bit-serially over two cycles. Figure <ref> and Figure <ref> show cycles 4 and 5 respectively. During cycle 4, bit 0 of a1 appears on both v inputs and is multiplied with the BR in each subunit. In cycle 5, bit 1 of a1 appears on both v inputs and the subunits complete the calculation of f_1 and f_2. It takes two cycles to produce the two 2b × 2b products once the correct inputs appear into the BRs. 

While in our example no additional inputs nor outputs are shown, it would have been possible to overlap the loading of a new set of w inputs into the ARs while processing the current weights stored into the BRs. That is the loading into ARs, copying into BRs, and the bit-serial multiplication of the BRs with the activations is a 3-stage pipeline where each stage can take multiple cycles. In general, assuming that both activations and weights are represented using 2 bits, this engine would match the performance of the bit-parallel engine in the steady state. When both set of inputs i and v can be represented with fewer bits (1 in this example) the engine would produce two terms per cycle, twice the bandwidth of the bit-parallel engine of the previous section.



Summary: In general, if P_BASE the precision of the bit-parallel engine, and P_a^L and P_w^L the precisions that can be used respectively for activations and weights for layer L, a engine can ideally outperform an equivalent bit parallel engine by P_BASE/P_a^L for CVLs, and by P_BASE/max(P_a^L, P_w^L) for FCLs. This example used the simplest engine configuration. Since typical layers exhibit massive parallelism, can be configured with many more subunits while exploiting weight reuse for CVLs and activation reuse for FCLs. The next section describes the baseline state-of-the-art DNNs accelerator and presents an equivalent configuration.





§ ARCHITECTURE



This work presents as a modification 
of the state-of-the-art DaDianNao accelerator. Accordingly, Section <ref> reviews 's design and how it can process FCLs and CVLs. For clarity, in what follows the term brick refers to a set of 16 elements of a 3D activation or weight array input which are contiguous along the i dimension, e.g., a(x,y,i)...a(x,y,i+15). Bricks will be denoted by their origin element with a B subscript, e.g., a_B(x,y,i).  The size of a brick is a design parameter.
Furthermore, an FCL can be thought of as a CVL where the input activation  array has unit x and y dimensions, and there are as many filters as output activations, and where the filter dimensions are identical to the input activation array.










 §.§ Baseline System: DaDianNao




Figure <ref> shows a which processes 16 filters concurrently calculating 16 activation and weight products per filter for a total of 256 products per cycle <cit.>. Each cycle the accepts 16 weights per filter for total of 256 weight and 16 input activations. The multiplies each weight with only one activation whereas each activation is multiplied with 16 weights, one per filter.  The reduces the 16 products per filter into a single partial output activation, for a total of 16 partial output activations for the . Each chip comprises 16 such , each processing a different set of 16 filters per cycle. Accordingly, each cycle, the whole chip processes 16 activations and 256× 16=4K weights producing 16× 16=256 partial output activations, 16 per .

Internally, each has: 1) a synapse buffer (SB) that provides 256 weights per cycle one per weight lane, 2) an input neuron buffer (NBin) which provides 16 activations per cycle through 16 neuron lanes, and 3) a neuron output buffer (NBout) which accepts 16 partial output activations per cycle. In the 's datapath each activation lane is paired with 16 weight lanes one from each filter. 
Each  weight and activation lane pair feeds a multiplier, and an  adder tree per filter lane reduces the 16 per filter 32-bit products into a partial sum. 
In all, the filter lanes produce each a partial sum per cycle, for a total of 16 partial output activations per tile.
Once a full window is processed, the 16 resulting sums are fed through a non-linear activation function, f, to produce the 16 final output activations.  
The multiplications and reductions needed per cycle are implemented via 256 multipliers one per weight lane and sixteen 17-input (16 products plus the partial sum from NBout) 32-bit adder trees one per filter lane. 

Figure <ref>a shows an overview of the chip. There are 16 processing tiles connected via an interconnect to a shared 2MB central eDRAM Neuron Memory (NM). 's main goal was minimizing off-chip bandwidth while maximizing on-chip compute utilization. To avoid fetching weights from off-chip, uses a 2MB eDRAM Synapse Buffer (SB) for weights per for a total of 32MB eDRAM for weight storage. All inter-layer activation outputs except for the initial input and the final output are stored in NM which is connected via a broadcast interconnect to the 16 Input Neuron Buffers (NBin) buffers. All values are 16-bit fixed-point, hence a 256-bit wide interconnect can broadcast a full activation brick in one step. Off-chip accesses are needed only for reading: 1) the input image, 2) the weights once per layer, and 3) for writing the final output.


Processing starts by reading from external memory the first layer's filter weights, and the input image. The weights are distributed over the SBs and the input is stored into NM. Each cycle an input activation brick is broadcast to all units. Each units reads 16 weight bricks from its SB and produces a partial output activation brick which it stores in its NBout. Once computed, the output activations are stored through NBout to NM and then fed back through the NBins when processing the next layer. Loading the next set of weights from external memory can be overlapped with the processing of the current layer as necessary.  



 §.§ 



As Section <ref> explained, processes activations bit-serially multiplying a single activation bit with a full weight per cycle. Each tile multiplies 16 16-bit activations with 256 weights each cycle. To match 's computation bandwidth, needs to multiply 256 1-bit activations with 256 weights per cycle. Figure <ref> shows the tile. It comprises 256 Serial Inner-Product Units (SIPs) organized in a 16× 16 grid. Similar to each SIP  multiplies 16 weights with 16 activations and reduces these products into a partial output activation. Unlike , each SIP accepts 16 single-bit activation inputs. Each SIP has two registers, each a vector of 16 16-bit subregisters: 1) the Serial Weight Register (SWR), and 2) the Weight Register (WR). These correspond to AR and BR of the example of Section <ref>. NBout remains as in , however, it is distributed along the SIPs as shown.

Convolutional Layers: Processing starts by reading in parallel 256 weights from the SB as in , and loading the 16 per SIP row weights in parallel to all SWRs in the row. Over the next P_a^L cycles, the weights are multiplied by the bits of an input activation brick per column. exploits weight reuse across 16 windows sending a different input activation brick to each column. For example, for a CVL with a stride of 4 a will processes 16 activation bricks a_B(x,y,i), a_B(x+4,y,i) through a_B(x+63,y,i)
in parallel a bit per cycle.  
Assuming that the processes filters f_i though f_i+15, after P_a^L cycles it would produce the following 256 partial output activations:
o_B(x/4,y/4,f_i), through o_B(x/4+15,y/4,f_i), 
that is 16 contiguous on the x dimension output activation bricks. Whereas would process 16 activations bricks over 16 cycles, processes them concurrently but bit-serially over P_a^L cycles. If P_a^L is less than 16, will outperform by 16/P_a^L, and when P_a^L is 16, will match 's performance. 

Fully-Connected Layers: Processing starts by loading bit-serially and in parallel over P_w^L cycles, 4K weights into the 256 SWRs, 16 per SIP. Each SWR per row gets a different set of 16 weights as each subregister is connected to one out of the 256 wires of the SB output bus for the SIP row (is in  there are  256× 16=4K wires).
Once the weights have been loaded, each SIP copies its SWR to its SW and multiplication with the input activations can then proceed bit-serially over P_a^L cycles. Assuming that there are enough output activations so that a different output activation can be assigned to each SIP, the same input activation brick can be broadcast to all SIP columns. For example, for an FCL a will process one activation brick a_B(i) bit-serially to produce 16 output activation bricks o_B(i) through o_B(i× 16) one per SIP column. Loading the next set of weights can be done in parallel with processing the current set, thus execution time is constrained by P_max^L = max(P_a^L,P_w^L). Thus, a produces 256 partial output activations every P_max^L cycles, a speedup of 16/P_max over since a always needs 16 cycles to do the same.

Cascade Mode: For to be fully utilized an FCL must have at least 4K output activations. Some of the networks studied have a layer with as little as 2K output activations. To avoid underutilization, the SIPs along each row are cascaded into a daisy-chain, where the output of one can feed into an input of the next via a multiplexer. This way, the computation of an output activation can be sliced over the SIPs along the same row. In this case, each SIP processes only a portion of the input activations resulting into several partial output activations along the SIPs on the same row. Over the next np cycles, where np the number of slices used, the np partial outputs can be reduced into the final output activation. The user can chose any number of slices up to 16, so that can be fully utilized even with fully-connected layers of just 256 outputs. 
This cascade mode can be useful in other Deep Learning networks such as in NeuralTalk <cit.> where the smallest can have 600 outputs or fewer. 

Other Layers: like can process the additional layers needed by the studied networks. For this purpose the includes additional hardware support for max pooling similar to . An activation function unit is present at the output of NBout in order to apply nonlinear activations before the output neurons are written back to NM.



 §.§ SIP and Other Components


SIP: Bit-Serial Inner-Product Units: 
Figure <ref> shows 's Bit-Serial Inner-Product Unit (SIP). Each SIP multiplies 16 activation bits, one bit per activation, by 16 weights to produce an output activation. Each SIP has two registers, a Serial Weight Register (SWR) and a Weight Register (WR), each containing 16 16-bit subregisters. Each SWR subregister is a shift register with a single bit connection to one of the weight bus wires that is used to read weights bit-serially for FCLs. Each WR subregister can be parallel loaded from either the weight bus or the corresponding SWR subregister, to process CVLs or FCLs respectively. Each SIP includes 256 2-input AND gates that multiply the weights in the WR with the incoming activation bits, and a 16× 16b adder tree that sums the partial products. A final adder plus a shifter accumulate the adder tree results into the output register OR.
In each SIP, a multiplexer at the first input of the adder tree implements the cascade mode supporting slicing the output activation computation along the SIPs of a single row. To support signed 2's complement neurons, the SIP can subtract the weight corresponding to the most significant bit (MSB) from the partial sum when the MSB is 1. This is done with negation blocks for each weight before the adder tree. Each SIP also includes a comparator (max) to support max pooling layers. 



Dispatcher and Reducers: Figure <ref>b shows an overview of the full system. As in there is a central NM and 16 tiles. A Dispatcher unit is tasked with reading input activations from NM always performing eDRAM-friendly wide accesses. It transposes each activation and communicates each a bit a time over the global interconnect. For CVLs the dispatcher has to maintain a pool of multiple activation bricks, each from different window, which may require fetching multiple rows from NM. However, since a new set of windows is only needed every P_a^L cycles, the dispatcher can keep up for the layers studied. For FCLs one activation brick is sufficient. A Reducer per title is tasked with collecting the output activations and writing them to NM. Since output activations take multiple cycles to produce, there is sufficient bandwidth to sustain all 16 tiles.





 §.§ Processing Several Activation Bits at Once


In order to improve 's area and power efficiency, the number of activation bits processed at once can be adjusted at design time. The chief advantage of these designs is that less SIPs are needed in order to achieve the same throughput – for example, processing two activation bits at once reduces the number of SIP columns from 16 to 8 and their total number to half. Although the total number of bus wires is similar, the distance they have to cover is significantly reduced. Likewise, the total number of adders required stays similar, but they are clustered closer together. A drawback of these configurations is they forgo some of the performance potential as they force the activation precisions to be multiple of the number of bits that they process per cycle. A designer can chose the configuration that best meets their area, energy efficiency and performance target.

  

In these configurations the weights are multiplied with several activation bits at once, and the multiplication results are partially shifted before they are inserted into their corresponding adder tree.
In order to load the weights on time, the SWR subregister has to be modified so it can load several bits in parallel, and shift that number of positions every cycle. The negation block (for 2's complement support) will operate only over the most significant product result.





§ EVALUATION



This section evaluates 's performance, energy and area compared to . It also explores the trade-off between accuracy and performance for .
Section <ref> described the experimental methodology. Section <ref> reports the performance improvements with . Section <ref> reports energy efficiency and Section <ref> reports 's area overhead. Finally, Section <ref> studies a configuration that processes two activation bits per cycle.


 §.§ Methodology





, and were modeled using the same methodology for consistency. A custom cycle-accurate simulator models execution time. Computation was scheduled as described by <cit.> to maximize energy efficiency for . 

The logic components of the both systems were synthesized with the Synopsys Design Compiler <cit.> for a TSMC 65nm library to report power and area. The circuit is clocked at 980 MHz. The NBin and NBout SRAM buffers were modelled using CACTI <cit.>. The eDRAM area and energy were modelled with Destiny <cit.>. Three design corners were considered as shown in Table <ref>, and the typical case was chosen for layout. 











 









 §.§ Execution Time
 

Table <ref> reports 's performance and energy efficiency relative to for the precision profiles in Table <ref> separately for , , and the whole network. 
For the 100% profile, where no accuracy is lost, yields, on average, a speedup of  over on FCLs. With the 99% profile, it improves to .

There are two main reasons the ideal speedup can't be reached in practice: dispatch overhead and under-utilization. Dispatch overhead occurs on the initial P^L_w cycles of execution, where the serial weight loading process prevents any useful products to be performed. In practice, this overhead is less than 2% for any given network, although it can be as high as 6% for the smallest layers. Underutilization can happen when the number of output neurons is not a power of two, or lower than 256. The last classifier layers of networks designed to perform recognition of ImageNet  categories <cit.> all provide 1000 output neurons, which leads to 2.3% of the SIPs being idle.

Compared to , matches its performance improvements on while offering  performance improvements on . We do not report the detailed results for since they would have been identical to for and within 1% of for . 

We have also evaluated on NeuralTalk LSTM <cit.> which uses long short-term memory to automatically generate image captions. Precision can be reduced down to 11 bits without affecting the accuracy of the predictions (measured as the BLEU score when compared to the ground truth) resulting in a ideal performance improvement of 1.45×translating into a 1.38× speedup with .
 We do not include these results in Table <ref> since we did not study the nor did we explore reducing precision further to obtain a 99% accuracy profile.


 §.§ Energy Efficiency


 This section compares the Energy Efficiency or simply efficiency of and . Energy Efficiency is the inverse of the relative energy consumption of the two designs. As Table <ref> reports, the average efficiency improvement with across all networks and layers for the 100% profile is . In FCLs, is more efficient than .

Overall, efficiency primarily comes from the reduction in effective computation following the use of reduced precision arithmetic for the inner product operations. Furthermore, the amount of data that has to be transmitted from the SB and the traffic between the central eDRAM and the SIPs is decreased proportionally with the chosen precision.
































 §.§ Area
 

Table <ref> reports the area breakdown of and . Over the full chip, needs 1.49× the area compared to while delivering on average a improvement in speed. Generally, performance would scale sublinearly with area for due to underutilization. The 2-bit variant, which has a lower area overhead, is described in detail in the next section. 














 §.§ TRT2b


This section evaluates the performance, energy efficiency and area for a multi-bit design as described in Section <ref>, where 2 bits are processed every cycle in as half as many total SIPs. The precisions used are the same as indicated in Table <ref> for the 100% accuracy profile rounded up to the next multiple of two. Table <ref> reports the resulting performance. The 2-bit always improves performance compared to as the “vs. ” columns show. Compared to the 1-bit performance is slightly lower however given that the area of the 2-bit is much lower, this can be a good trade-off. Overall, there are two forces at work that shape performance relative to the 1-bit . There is performance potential lost due to rounding all precisions to an even number, and there is performance benefit by requiring less parallelism. The time needed to serially load the first bundle of weights is also reduced. In VGG_19 the performance benefit due to the lower parallelism requirement outweighs the performance loss due to precision rounding. In all other cases, the reverse is true.



A hardware synthesis and layout of both and 's 2-bit variant using TSMC's 65nm typical case libraries shows that the total area overhead can be as low as 24.9% (Table <ref>), with an improved energy efficiency in fully connected layers of 1.24× on average (Table <ref>).








§ RELATED WORK AND LIMITATIONS



The recent success of Deep Learning has led to several proposals for hardware acceleration of DNNs. This section reviews some of these recent efforts. However, specialized hardware designs for neural networks is a field with a relatively long history. Relevant to , bit-serial processing hardware for neural networks has been proposed several decades ago, e.g., <cit.>. While the performance of these designs scales with precision it would be lower than that of an equivalently configured bit-parallel engine. For example, Svensson et al., uses an interesting bit-serial multiplier which requires O(4× p) cycles, where p the precision in bits <cit.>. Furthermore, as semiconductor technology has progressed the number of resources that can be put on chip and the trade offs (e.g., relative speed of memory vs. transistors vs. wires) are today vastly different facilitating different designs. However, truly bit-serial processing such as that used in the aforementioned proposals needs to be revisited with today's technology constraints due to its potentially high compute density (compute bandwidth delivered per area).


In general, hardware acceleration for DNNs has recently progressed in two directions: 1) considering more general purpose accelerators that can support additional machine learning algorithms, and 2) considering further improvements primarily for convolutional neural networks and the two most dominant in terms of execution time layer types: convolutional and fully-connected. In the first category there are accelerators such as Cambricon <cit.> and Cambricon-X <cit.>. While targeting support for more machine learning algorithms is desirable, work on further optimizing performance for specific algorithms such as is valuable and needs to be pursued as it will affect future iterations of such general purpose accelerators.

is closely related to  <cit.> whose execution time scales with precision but only for CVLs. does not improve performance for FCLs. improves upon by enabling: 1)  performance improvements for FCLs, and 2) slicing the activation computation across multiple SIPs thus preventing under-utilization for layers with fewer than 4K outputs. Pragmatic uses a similar in spirit organization to but its performance on CVLs depends only on the number of activation bits that are 1 <cit.>. It should be possible to apply the extensions to Pragmatic, however, performance in FCLs will still be dictated by weight precision. The area and energy overheads would need to be amortized by a commensurate performance improvement necessitating a dedicated evaluation study.

The Efficient Inference Engine (EIE) uses synapse pruning, weight compression, zero activation elimination, and network retraining to drastically reduce the amount of computation and data communication when processing fully-connected layers <cit.>. An appropriately configured EIE will outperform for FCLs, provided that the network is pruned and retrained. However, the two approaches attack a different component of FCL processing and there should be synergy between them. Specifically, EIE currently does not exploit the per layer precision variability of DNNs and relies on retraining the network. It would be interesting to study how EIE would benefit from a -like compute engine where EIE's data compression and pruning is used to create vectors of weights and activations to be processed in parallel. EIE uses single-lane units whereas uses a coarser-grain lane arrangement and thus would be prone to more imbalance. A middle ground may be able to offer some performance improvement while compensating for cross-lane imbalance.

Eyeriss uses a systolic array like organization and gates off computations for zero activations <cit.> and targets primarily high-energy efficiency. An actual prototype has been built and is in full operation. Cnvlutin is a SIMD accelerator that skips on-the-fly ineffectual activations such as those that are zero or close to zero <cit.>. Minerva is a DNN hardware generator which also takes advantage of zero activations and that targets high-energy efficiency <cit.>. Layer fusion can further reduce off-chip communication and create additional parallelism <cit.>. As multiple layers are processed concurrently, a straightforward combination with would use the maximum of the precisions when layers are fused. 


Google's Tensor Processing Unit uses quantization to represent values using 8 bits <cit.> to support TensorFlow <cit.>. As Table <ref> shows, some layers can use  lower than 8 bits of precision which suggests that even with quantization it may be possible to use fewer levels and to potentially benefit from an engine such as .



 §.§ Limitations
 As in this work assumed that each layer  fits on-chip. However, as networks evolve it is likely that they will increase in size thus requiring multiple nodes as was suggested in . However, some newer networks tend to use more but smaller layers. Regardless, it would be desirable to reduce the area cost of most of which is due to the eDRAM buffers. We have not explored this possibility in this work. Proteus <cit.> is directly compatible with and can reduce memory footprint by about 60% for both convolutional and fully-connected layers. Ideally, compression, quantization and pruning similar in spirit to EIE <cit.> would be used to reduce computation, communication and footprint. General memory compression <cit.> techniques offer additional opportunities for reducing footprint and communication.

We evaluated only on CNNs for image classification. Other network architectures are important and the layer configurations and their relative importance varies. enables performance improvements for two of the most dominant layer types. We have also provided some preliminary evidence that works well for NeuralTalk LSTM <cit.>. Moreover, by enabling output activation computation slicing it can accommodate relatively small layers as well. 

Applying some of the concepts that underlie the design to other more general purpose accelerators such as Cambricon <cit.> or graphics processors would certainly be more preferable than a dedicated accelerator in most application scenarios. However, these techniques are best first investigated into specific designs and then can be generalized appropriately.

We have evaluated only for inference only. Using an engine whose performance scales with precision would provide another degree of freedom for network training as well. However, needs to be modified accordingly to support all the operations necessary during training and the training algorithms need to be modified to take advantage of precision adjustments.

This section commented only on related work on digital hardware accelerators for DNNs. Advances at the algorithmic level would impact as well or may even render it obsolete. For example, work on using binary weights <cit.> would obviate the need for an accelerator whose performance scales with weight precision. Investigating 's interaction with other network types and architectures and other machine learning algorithms is left for future work. 



§ CONCLUSION


This work presented , an accelerator for inference with Convolutional Neural Networks whose performance scales inversely linearly with the number of bits used to represent values in fully-connected and convolutional layers. also enables on-the-fly accuracy vs. performance and energy efficiency trade offs and its benefits were demonstrated over a set of popular image classification networks. The new key ideas in are: 1) Supporting both the bit-parallel and the bit-serial loading of weights into processing units to facilitate the processing of either convolutional or fully-connected layers, and 2) cascading the adder trees of various subunits (SIPs) to enable slicing the output computation thus reducing or eliminating cross-lane imbalance for relatively small layers. 

opens up a new direction for research in inference and training by enabling precision adjustments to translate into performance and energy savings. These precisions adjustments can be done statically prior to execution or dynamically during execution. While we demonstrated for inference only, we believe that , especially if combined with Pragmatic, opens up a new direction for research in training as well. For systems level research and development, with its ability to trade off accuracy for performance and energy efficiency enables a new degree of adaptivity for operating systems and applications.



ieeetr





