Under review at the 45th International Symposium on Computer Architecture (ISCA), 2018

Bit Fusion: Bit-Level Dynamically Composable
Architecture for Accelerating Deep Neural Networks
Hardik Sharma Jongse Park ∗ Naveen Suda
Benson Chau Joon Kyung Kim ∗ Vikas Chandra

∗ Liangzhen Lai
† Hadi Esmaeilzadeh

Alternative Computing Technologies (ACT) Lab

{hsharma, jspark, ben.chau, jkkim}@gatech.edu
† hadi@eng.ucsd.edu
lianghzhen.lai, vikas.chandra}@arm.com

INTRODUCTION

The growth in computing power has been a major driver
for the rapid advances in Deep Neural Networks (DNN) due
to their demand for copious amount of calculations. Naturally, both the research community [1–28] as well the industry [29–31] have turned to accelerators to accommodate

1bit/1bit

2bit/2bit

4bit/4bit

8bit/8bit

4bit/1bit
1bit

100%

100%

80%

80%

60%

60%

40%

40%

4bit

8bit

(a) Mul(ply-Add Bitwidth (Input/Weight)

Avg

VGG-7

RNN

SVHN

RESNET-18

LSTM

LeNet-5

AlexNet

CIFAR10

Avg

VGG-7

RNN

SVHN

0%
RESNET-18

0%

2bit

20%

20%
LSTM

Hardware acceleration of Deep Neural Networks (DNNs)
aims to tame their enormous compute intensity. Fully realizing the potential of acceleration in this domain requires understanding and leveraging algorithmic properties of DNNs.
This paper builds upon the algorithmic insight that bitwidth
of operations in DNNs can be reduced without compromising their classification accuracy. However, to prevent loss of
accuracy, the bitwidth varies significantly across DNNs and
it may even be adjusted for each layer individually. Thus,
a fixed-bitwidth accelerator would either offer limited benefits to accommodate the worst-case bitwidth requirements,
or inevitably lead to a degradation in final accuracy. To alleviate these deficiencies, this work introduces dynamic bitlevel fusion/decomposition as a new dimension in the design
of DNN accelerators. We explore this dimension by designing Bit Fusion, a bit-flexible accelerator, that constitutes an
array of bit-level processing elements that dynamically fuse
to match the bitwidth of individual DNN layers. This flexibility in the architecture enables minimizing the computation
and the communication at the finest granularity possible with
no loss in accuracy. We evaluate the benefits of Bit Fusion using eight real-world feed-forward and recurrent DNNs. The
proposed microarchitecture is implemented in Verilog and
synthesized in 45 nm technology. Using the synthesis results
and cycle accurate simulation, we compare the benefits of Bit
Fusion to two state-of-the-art DNN accelerators, Eyeriss [1]
and Stripes [2]. In the same area, frequency, and technology node, Bit Fusion offers 4.3× speedup and 9.6× energy
savings over Eyeriss. Bit Fusion provides 2.4× speedup and
4.1× energy reduction over Stripes at 45 nm node when Bit
Fusion area and frequency are set to those of Stripes. Compared to Jetson-TX2, Bit Fusion offers 4.3× speedup and almost matches the performance of Titan X, which is 4.6×
faster than TX2.

AlexNet CIFAR10 LSTM LeNet-5 RESNET-18 RNN SVHN VGG-7
DNN
% Mul(ply-Add 99.8 99.9 99.9 99.4
99.7 99.9 99.8 99.5

LeNet-5

ABSTRACT

1.

† University of California San Diego

AlexNet

∗ {naveen.suda,

∗ Arm, Inc.

CIFAR10

arXiv:1712.01507v1 [] 5 Dec 2017

Georgia Institute of Technology

(b) Weights Bitwidth Distribu(on

Figure 1: Bitwidth variation across real-world DNNs.

modern DNN computation. However, the algorithmic properties of DNNs have not fully been utilized to push the envelope on their acceleration efficiency and performance.
To that end, we leverage the following three algorithmic
properties of DNNs to introduce a novel acceleration architecture, called Bit Fusion. (1) DNN computation is mostly
a collection of massively parallel multiply-add operations.
(2) The bitwidth of these operations can be reduced with no
loss in accuracy [32–35]. (3) However, to preserve accuracy,
the bitwidth varies significantly across DNNs and it may
even be adjusted for each layer individually. Thus, a fixedbitwidth accelerator design would either yield limited benefits to accommodate the worst-case bitwidth requirements,
or inevitably lead to a degradation in final accuracy. To alleviate these deficiencies, Bit Fusion introduces the concept of
runtime bit-level fusion/decomposition as a new dimension
in the design of DNN accelerators. We explore this dimension by designing a bit-flexible accelerator, which comprises
an array of processing engines that fuse at the bit-level granularity to match the bitwidth of the individual DNN layers.
The bit-level flexibility in the architecture enables minimizing the computation and the communication at the finest
granularity possible with no loss in accuracy. As such, the
following three insights both motivate and guide Bit Fusion.
First, the number of bit-level operations required for
the multiply operator is proportional to the product of the
operands’ bitwidths and scales linearly for the addition op-

erator. Therefore, matching the bitwidth of the multiplyadd units to the reduced bitwidth of the DNN layers, almost
quadratically reduces the bit-level computations. This strategy will significantly affect the acceleration since the large
majority of DNN operations (> 99%) are multiply-adds as
shown in the table included in Figure 1. For instance, each
single image classification with AlexNet requires a total of
726 million operations, of which 99.76% (724 million) are
multiply-adds [36]. To this end, the processing engines of
Bit Fusion can dynamically fuse or decompose to match the
bitwidth of each individual multiply-add operand without requiring the operands to be encoded in the same bitwidth.
Second, energy consumption for DNN acceleration is usually dominated by data accesses to the on-chip storage and
off-chip memory [1, 3, 4]. Therefore, Bit Fusion comes with
encoding and memory access logic that store and retrieve the
values in the lowest required bitwidth. This logic reduces the
overall number of bits read or written to the on-chip and offchip memory, proportionally reducing the energy dissipation
of memory accesses and the acceleration. Furthermore, this
strategy increases the effective on-chip storage capacity.
Third, Bit Fusion builds upon the extensive prior work
that shows DNNs can operate with reduced bitwidth without degradation in classification accuracy [2, 32–35, 37].
This opportunity exists across different classes of real-world
DNNs, as shown in Figure 1. One category is Convolutional
Neural Networks (CNNs) that usually use convolution and
pooling layers followed by a stack of fully-connected layers. AlexNet, Cifar-10, LeNet-5, RESNET-18, SVHN, and
VGG-7 in Figure 1 belong to this category. Recurrent Neural Networks (RNN) are another sub-class of DNNs that use
special recurrent layers including Long Short Term Memory (LSTM), vanilla RNN layers to extract temporal features from time-varying data. The RNN and LSTM benchmark DNNs in Figure 1 represent these categories, respectively. Furthermore, as the table in Figure 1 shows, most
operations in DNNs (> 99%), regardless of their categories,
are multiply-adds. As Figure 1(a) illustrates, on average,
97.3% of multiply-adds require four or fewer bits and even
in some DNNs a large fraction of the operations can be done
with bitwidth equal to one. More interestingly, the bitwidths
vary within and across DNNs to guarantee no loss of accuracy. Such a variation is not limited to the intermediate
operands and exists in trained weights as illustrated in Figure 1(b). There is even asymmetry in the bitwidth of the
operands (AlexNet) as depicted in Figure 1(a). To exploit
this property, a programmable accelerator needs to offer bitlevel flexibility at runtime, which leads us to Bit Fusion.
To harvest the aforementioned opportunities, this paper
makes the following contributions and realizes a new dimension in the design of DNN accelerators.
1. Dynamic bit-level fusion and decomposition. The paper introduces and explores the dimension of bit-level
flexible DNN accelerator architectures, Bit Fusion, that
dynamically matches bit-level composable processing engines to the varying bitwidths required by DNN layers.
By offering this flexibility, Bit Fusion aims to minimize
the computation and communication required by a DNN
at the bit granularity on a per layer basis.
2. Microarchitecture design for bit-level composability.

To explore Bit Fusion, we design and implement a DNN
accelerator using a novel bit-flexible computation unit,
called BitBricks. The accelerator supports both feedforward (CNN) and recurrent (LSTM and RNN) layers.
A 2D array of BitBricks constructs a fusible processing
engine that can perform the DNN computation at various
bitwidths. The microarchitecture also comes with a storage logic that allows feeding the BitBricks with different
bitwidth operands.
3. Hardware-software abstractions for bit-flexible acceleration. To enable DNN applications to take advantage of these unique bit-level fusion capabilities, we propose a block-structured instruction set architecture, called
Fusion-ISA. To amortize the cost of programmability,
Fusion-ISA expresses operations of DNN layers as bitflexible instruction blocks with iterative semantics.
These three contributions define the novel architecture of
Bit Fusion, a possible microarchitecture implementation, and
the hardware-software abstractions to offer bit-level flexibility. Other complementary and inspiring works have explored
bit serial computation [2, 6] without exploring the fusion dimension. In contrast, Bit Fusion spatially fuses a group of
BitBricks together, to collectively execute operations at different bitwidths. Using eight real-world feed-forward and
recurrent real-world DNNs, we evaluate the benefits of Bit
Fusion. We implemented the proposed microarchitecture in
Verilog and synthesized in 45 nm technology. Using the synthesis results and cycle accurate simulation, we compare the
benefits of Bit Fusion to two state-of-the-art DNN accelerators, Eyeriss [1] and Stripes [2]. The latter is an optimized
bit-serial architecture. In the same area, frequency, and technology node, Bit Fusion offers 4.3× speedup and 9.6× energy savings over Eyeriss. Bit Fusion is 2.4× faster and 4.1×
lower energy than Stripes at 45 nm node when Bit Fusion
area and frequency are set to those of Stripes. Compared
to Jetson-TX2, Bit Fusion offers 4.3× speedup and almost
matches the performance of Titan X, which is 4.6× faster
than TX2.

2. BIT FUSION ARCHITECTURE
To minimize the computation and communication at the
finest granularity, Bit Fusion dynamically matches the architecture of the accelerator to the bitwidth required for the
DNN without any loss in accuracy, which may vary layer
by layer. As such, Bit Fusion is a collection of bit-level
computational elements, called BitBricks, that dynamically
compose to logically construct Fused Processing Engines
(Fused-PE) that execute DNN operations with the required
bitwidth. Specifically, Fused-PEs provide bit-level flexibility for multiply-adds, which are the dominant operations
across all types of DNNs. Below, we discuss how BitBricks
can be dynamically fused together to support a range of
bitwidths, yet provide a significant increase in parallelism
when operating at lower bitwidths.

2.1

Bit-Level Flexibility via Dynamic Fusion

As depicted in Figure 2, Bit Fusion arranges the BitBricks
in a 2-dimensional physical grouping, called Fusion Units.
Each BitBrick in the Fusion Unit can perform individual binary
(0, +1) and ternary (-1, 0, +1) operations of the multiply-add.

BB
BB

+

+

BB
BB

+

BB

BB
BB
BB

+

+

BB

BB

BB

BB

BB

BB

BB

BB

IBUF (Shared)

WBUF

BB
BB
BB
BB

+

+

+

+

Fusion Unit
BB
BB
BB
BB

+

BB

BB
BB
BB

WBUF
+

+

BB
BB

BB

BB

BB

BB
BB

+

BB

BB
BB
BB

+

+

BB

BB

BB

BB

BB

BB

BB

BB

+

+

Ac.va.on Unit

Fusion Unit
BB
BB
BB
BB

+

BB

BB
BB
BB

+

+

BB
BB
BB
BB

+
Pooling Unit

OBUF

Ac.va.on Unit

OBUF

Figure 2: Bit Fusion systolic architecture comprising a collection of BitBricks (BBs) that can fuse to form FPEs.

As Figure 3 shows, the BitBricks logically fuse together at
run-time to form Fused Processing Engines (Fused-PEs) that
match the bitwidths required by the multiply-add operations.
The BitBricks in a Fusion Unit multiply an incoming variablebitwidth input (input forward) to a variable-bitwidth weight
(from WBUF) to generate the product. The Fusion Unit then
adds the product to an incoming partial sum to generate an
outgoing partial sum (Psum forward in Figure 3(a)).
Figures 3(b), 3(c), and 3(d) show three different ways of
logically fusing BitBricks to form (b) 16 Fused-PEs that support ternary (binary); (c) four Fused-PEs that support mixedbitwidths (2-bits for weights and 8-bits for inputs), (d) one
Fused-PE that supports 8-bit operands, respectively. For binary or ternary operations (Figures 3(b)), each Fused-PE
contains a single BitBrick, offering the highest parallelism.
The Fusion Unit then adds the results from all Fused-PEs
and the incoming partial sum to generate a single outgoing partial sum. Figure 3(c) shows four BitBricks fused together in a column to form a Fused-PE that can multiply
2-bit weights with 8-bit inputs. The bitwidths of operands
supported by a Fused-PE depend on the spatial arrangement
of BitBricks fused together. Alternatively, by varying the spatial arrangement of the four fused BitBricks, the Fused-PE
can support 8-bit/2-bit, 4-bit/4-bit, and 2-bit/8-bit configurations for inputs/weights. Finally, up to 16 BitBricks can
fuse together to construct a single Fused-PE that can operate on 8-bit operands for the multiply-add operations (Figure 3(d)). The BitBricks fuse together in powers of 2. That
is, a single Fusion Unit with 16 BitBricks can offer 1, 2, 4, 8,
and 16 Fused-PEs with varying operand bitwidths. Dynamic
composability of the Fusion Units at the bit level enables the
architecture to expose the maximum possible level of parallelism with the finest granularity that matches the bitwidth
of the DNN operands.

2.2

+

BB
BB

BB
+

BB
BB

+

+

BB

Accelerator Organization

Two insights guide the architecture design of Bit Fusion.
First, DNNs offer high degrees of parallelism and benefit significantly from increasing the number of Fusion Units

F-PE F-PE

BB

Input
+
forward F-PE F-PE

BB

F-PE F-PE

+

+

+

F-PE F-PE
F-PE F-PE
+

F-PE F-PE
Psum forward

(b) 16x Parallelism, Binary (1-bit)
or Ternary (2-bit)

WBUF
Input
forward

F-PE F-PE

F-PE F-PE

BB
BB
Psum forward

(a) Fusion Unit with 16 BitBricks

+
Pooling Unit

WBUF

BB

Input
+
forward BB
BB

BB

WBUF

Fusion Unit
BB
BB

BB

WBUF

F-PE

BB

Fusion Unit
BB
BB

F-PE

BB

WBUF

F-PE

WBUF

F-PE

IBUF (Shared)

Control

Psum forward
(c) 4x Parallelism, Mixed-Bitwidth
(2-bit weights, 8-bit inputs)

Input
forward

F-PE
+

+

+

+

+

Psum forward
(d) No Parallelism, 8-bits

Figure 3: Dynamic composition of BitBricks (BBs) to construct Fused Processing Engines (FPEs).

available within the accelerator’s area budget. Therefore,
it is essential to minimize the overhead of control in the
accelerator by not only maximizing the number of Fusion
Units but also minimizing the overhead of dynamically constructing Fused-PEs, thereby integrating the maximum number of BitBricks in the area budget. Second, on-chip SRAM
and register-file accesses dominate the energy consumption
when accelerating DNNs [1, 3, 4]. Therefore, it is essential
to reduce the number of bits exchanged with on-chip and
off-chip memory while maximizing data reuse.
Systolic array. With these insights, we employ a 2dimensional systolic array of Fusion Units as the architecture
for Bit Fusion, as shown in Figure 2. The systolic organization reduces the overhead of control by sharing the control
logic across the entire systolic array. More importantly, systolic execution alleviates the need for provisioning control
for each Fused-PE as a dataflow architecture would have required. As such, the systolic architectures fit the most number of BitBricks in a given area budget. Thus, the entire systolic array composed of Fused-PEs acts as a single compute
unit that can execute, for example, a single matrix-vector
multiplication operation with various bitwidths, which also
sets the level of parallelism. In addition, the systolic organization of Fusion Units enforces sharing of input data across
columns of the array and accumulates partial results across
rows of the array to minimize access to on-chip memory. As
depicted in Figure 2, the input buffers (IBUFs) only located
at the borders and feed the rows simultaneously. Similarly,
the outputs buffers (OBUFs) reside on the bottom and collect
the flowing results, which is accumulated by each column’s
accumulator. As shown in Figure 2, each column harbors a
pooling and an activation unit before its output buffer. Finally, the systolic organization also eliminates the need for
local buffers for input, output, or partial results within Fusion Units. As such, each Fusion Unit is accompanied by
only a weight buffer (WBUF) Using Fused-PEs as the building blocks, the performance of the systolic array maximally
matches the bitwidths, with the highest performance at binary and ternary settings.

WBUF
WBUF

WBUF
WBUF

WBUF

WBUF

WBUF

2b
2b
2b
2b

2b
2b
2b
2b

2b
2b
2b
2b

2b
2b
2b
2b

2b
2b
2b
2b

2b
2b
2b
2b
2b
2b
2b
2b

2b
2b
2b
2b

y2 y1 y0

x2 x1 x0

32b

32b

32b

M
OBUF

OBUF

OBUF

Figure 4: Bit-Flexible matrix-vector multiplication.

Memory organization. Depending on the number of FusedPEs and their organization, the buffers must supply different
numbers of operands with various bitwidths. As such, we
augment the input and the weight buffers with a register that
holds a row of data that is gradually fed to the Fused-PEs
according to their bitwidth. As illustrated in Figure 2, a series of multiplexers after the register make this data infusion
possible. The benefit of this design is avoiding multiple accesses to the data array of the buffer which conserves energy.
With this design, at each cycle, the systolic array consumes
a vector of inputs and matrix of weights to produce a vector
of outputs with the fewest accesses to the data arrays and the
minimal bitwidth possible. To finish the computation, the
Fused-PEs within each column forward partial results downwards to the Fused-PEs in the adjacent row. The results are
accumulated and written once to the output buffer (OBUF).
Before being written to the buffer, the results pass through
the pooling or activation units.

2.3 Bit Fusion Execution Model
Figure 6 illustrates the Bit Fusion systolic execution in the
mixed-bitwidth mode using when an input vector is multiplied to a weight matrix. The input vector has 4 × N 8-bit elements that are being multiplied to a matrix with 4 × N × M
2-bit elements. As such, the 16-BitBrick Fusion Units are broken down to four 8×2 Fused-PEs. Both the input and weight
buffers provide 32 bits per each access. The read values are
split into 8-bit input values and 2-bit weight values in the
output register of each buffer using its accompanying multiplexers as mentioned before. The input values are shared
across the Fusion Units of each row and weight values are
specific to each Fused-PE. As such, all of the 4 × N × M
Fused-PEs work in parallel while only a single 32-bit value
is read from the input and weight buffers. Exploiting the

3

Figure 5: A single BitBrick.

F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE

F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE
F-PE

IBUF

8b 8b 8b 8b

N

IBUF
IBUF

8b 8b 8b 8b

s0 x1 x0

6
m5 m4 m3 m2 m1 m0

M

8b 8b 8b 8b

s1 y1 y0

3
2b
2b
2b
2b

N

WBUF
WBUF

lower bitwidth of weights, Bit Fusion increases the level of
parallelism by 4× while reducing the number of accesses
to the weight buffer data arrays by the same factor of four.
As discussed above, each Fusion Unit adds the results of its
Fused-PEs with its incoming partials results and forwards
the partial output to the Fusion Unit underneath it. As shown
in Figure 6, we support 32-bit bitwidth for the partial and
final results to avoid any inaccuracies.

3.

BITFUSION MICROARCHITECTURE

Given the overall organization of Bit Fusion and its bitflexible systolic execution model, this section delves into
the details of BitBricks and Fusion Units. First, we present
the BitBrick design. Then, we define the Bit Fusion operator,
that executes a bit-flexible multiply operation using the BitBricks. In our microarchitecture, the Fusion Unit is an implementation of the Bit Fusion operator that uses 16 BitBricks to
operate on data with flexible bitwidths, up to 16-bits. To enable flexible bitwidths for operands, the Bit Fusion-operator
breaks down the operands into a fixed bitwidth of 2-bits and
generates partial products using BitBricks. Below, we first
discuss the microarchitecture of a single BitBrick, followed
by a mathematical formulation of how to combine/fuse together the results from multiple BitBricks to enable flexible
bitwidths for operands.

3.1 BitBricks Microarchitecture
Figure 5 shows the microarchitecture of a single BitBrick.
As shown, a BitBrick takes as input two 2-bit operands–
{x1 , x0 } and {y1 , y0 } and two sign-bits–{s1 , s0 }. The signbit s0 and s1 define if the 2-bit operand {x1 , x0 } is signed
(between -2 to 1) or unsigned (between 0 to 3). According to the sign-bit, the BitBricks first extends the 2-bit input
operands to create 3-bit sign extended operands. Finally, the
BitBricks employ a 3-bit signed multiplier (shown with an encircled × in Figure 5) to generate a 6-bit output{m5 , .., m0 }.
Thus, a BitBrick supports both signed and unsigned numbers
as its inputs. Below, we discuss how the Bit Fusion-operator
uses the BitBricks to support variable bitwidth for operands.

3.2

The Bit Fusion Operator
We define the Bit Fusion operator as fuse_multiply2→n , that
supports the variable bitwidths in the range of 2 and n in
powers of two. We explain this operator using the following example. The fuse_multiply2→n operator breaks down
the variable bitwidth operands (inputs/weights) into a fixedlength of 2-bits, that can be fed into BitBricks to generate partial products. The following equation shows a 4-bit signed

2n-bit

n-bit

n-bit

2n-bit

n-bit

22n
+ 2n

2n-bit

2n-bit

BitBricks to support 16, 8, 4 and 2-bit operands as follows.

Opera/on

Bitwidth Mode
(A2n )hi

(B2n )hi + 2n

(A2n )hi

(B2n )lo
(B2n )lo

(A2n )lo

(B2n )hi + 20

(A2n )lo

n

(A2n )hi

Bn

Bn

(C2n )hi

Dn

+ 2
+ 20

(A2n )lo

+ 2n

(C2n )lo

Dn

(B2n )hi + 2n
(B2n )lo + 20

Cn

(D2n )hi

Cn

(D2n )lo

+ 20
+ 20

Cn

Dn

Gn

Hn

2

2n

An

+ 20

An

0

An

Bn

En

Fn

2
+ 20

n-bit

0

Figure 6: Supported modes for a fuse_multiplyn→2n

number A4 ({a3 , a2 , a1 , a0 }) and then divides the 4-bit number into two parts (A4 )hi and (A4 )lo , each of which are 2-bit.
2

A4 = −23 × a3 + ∑ ai × 2i

fuse_multiply2→2n

=
+
+
+

2s0 × fuse_multiply2→n (x0 , y0 )
2s1 × fuse_multiply2→n (x1 , y1 )
2s2 × fuse_multiply2→n (x2 , y2 )
2s3 × fuse_multiply2→n (x3 , y3 )
(5)

Thus, a fuse_multiply2→8 operator can be broken into four
fuse_multiply2→4 operators, each of which requires four multiplications using BitBricks as shown in Equation (4). Therefore, a fuse_multiply2→8 requires 16 multiplications using
BitBricks. Using another level of recursion to support up
to 16 − bit operands, we need 64 multiplications using BitBricks. The Bit Fusion architecture uses 16 BitBricks to construct a Fusion Unit that executes the fuse_multiply2→16 operator. Thus, the Fusion Unit takes four cycles to execute
the fuse_multiply2→16 operator and support from 2-bit to 16-

= (−2 × a3 + a2 × 2 ) + (2 × a1 + a0 × 2 )

bit operands (inputs/weights). The next subsection discusses
the microarchitecture of the Fusion Unit.

= (−21 × a3 + a2 × 20 ) × 22 + (21 × a1 + a0 × 20 )

3.3 Fusion Unit Micro-Architecture

i=0

3

2

1

0

2

= (A4 )hi × 2 + (A4 )lo

(1)

In Equation (1), (A4 )hi is a 2-bit signed number that consists
of the two most significant bits in A4 , while (A4 )lo is a 2bit unsigned number that consists of the two least significant
bits. Applying Equation (1) to a multiply operation between
two 4-bit operands, A4 and B4 , we get:
A4 ×B4 = 24 × (A4 )hi × (B4 )hi + 22 × (A4 )hi × (B4 )lo
+ 22 × (A4 )lo × (B4 )hi + 20 × (A4 )lo × (B4 )lo

(2)

Similarly, we can express a 4-bit × 2-bit and 2-bit × 4-bit
multiplication as follows:
A4 × B2 =
A2 × B4 =

20 × (A4 )lo × B2 + 22 × (A4 )hi × B2
0

2

2 × A2 × (B4 )lo + 2 × A2 × (B4 )hi

(3a)
(3b)

Note that the Equation (2) requires four 2-bit multiplications using BitBricks and Equations (3a) and (3b) require two
2-bit multiplications. By generalizing Equations (2), (3a),
and (3b), we define the fuse_multiply2→4 operator that supports 2-bit and 4-bit operands as follows:
fuse_multiply2→4 =

2s0 × x0 × y0
+ 2s2 × x2 × y2

+ 2s1 × x1 × y1
+ 2s3 × x3 × y3 (4)

In Equation (4), fuse_multiply2→4 represents the Bit Fusion
operator that supports 4 and 2-bits by varying the xi , yi , and
si ∀i ∈ [0, 3] according to Figure 6. Multiplication with 2si is
a shift operation. Note that the operands xi and yi ∀i ∈ [0, 3]
can be both signed and unsigned operands, which the BitBricks support. In the 4 × 4-bit mode, the equation represents a scalar multiplication between two 4-bit operands. In
the 4 × 2-bit, 2 × 4-bit, and 2 × 2-bit modes, the equation
represents a dot-product between vectors of size 2, 2, and 4,
respectively.
Supporting arbitrary bitwidths. We use the following recursion to express a Bit Fusion multiply operation that uses

We explore two techniques to realize the Bit Fusion operator (fuse_multiply2→16 ) within the Fusion Unit using BitBricks:
For clarity, we first compare the two fusion techniques in the
context of a fuse_multiply2→4 operator.
1. Temporal Fusion, that uses each BitBrick independently
over 4 cycles to realize the fuse_multiply2→4 operator.
2. Spatial Fusion, that uses 4 BitBricks over a single clock
cycle to realize the fuse_multiply2→4 operator.
Temporal fusion. Figure 7 shows the design of an example
temporal multiplier that implements fuse_multiply2→4 operator and supports variable bitwidths–from 2-bits up to 4-bits.
The variable-bitwidths multiply operation for the temporal
multiplier consists of three steps: (1) 2-bit multiplication using BitBricks to generate partial products, (2) shift operation
to multiply with the appropriate power of 2, and (3) accumulation in a register. The temporal multiplier requires 4
cycles to implement the fuse_multiply2→4 operator. The shift
operation is simply a 4-input multiplexer (mux). Compared
to a fixed 4-bit multiplier, the temporal multiplier uses much
smaller BitBricks for 2-bit operands, which requires significantly less area. However, the number of gates required for
the shifter and the accumulator depend on the highest supported bitwidth (4-bit in case of fuse_multiply2→4 operator).
For instance, to support temporal fusion up to 16-bits using BitBricks, the shifter and the accumulator use up around
90% of the area, which limits the benefits provided by temporal multiplier. Nevertheless, the temporal fusion reduces
area consumption over a fixed-bitwidth multiplier designed
to support the highest required bitwidth.
Spatial fusion. The spatial multiplier implements the
fuse_multiply2→4 operator spatially, by combining the results
of four BitBricks over a single clock cycle. Figure 8 illustrates the design of a spatial multiplier that supports up to
4 bits for either of the two operands using BitBricks. Similar to the temporal multiplier, the spatial multiplier requires
three steps: (1) multiplication using BitBricks, (2) shift-add
using the shift-add tree, and (3) accumulation of results in a
register. The spatial multiplier improves upon the temporal

c₃ d₃

e₃ f₃

g₃ h₃

a₂ b ₂

c₂ d₂

e₂ f₂

g₂ h₂

a₁ b ₁

c₁ d₁

e₁ f₁

g₁ h₁

a₀ b ₀

c₀ d₀

e₀ f₀

g₀ h₀

1
2
3

Inputs over
time

Inputs over
time

a₃ b ₃

g₃ h₃

g₂ h₂

g₁ h₁

g₀ h₀

e₃ f₃

e₂ f₂

e₁ f₁

e₀ f₀

c₃ d₃

c₂ d₂

c₁ d₁

c₀ d₀

a₃ b ₃

a₂ b ₂

a₁ b ₁

a₀ b ₀

<<

Out

<<

Out

<<

Out

<<

Out

2

ShiftRegister
Add
2989
1454
934
91

Total
Area
4905
1394

ShiftTotal
Register
Add
Power
Temporal
60
550
1103
1712
Fusion Unit
46
424
69
538
Synthesized using a commercial 45 nm technology

Power (nW) BitBricks

1

Figure 7: Temporal fusion.

<<

3

<<

<<

<<

Figure 9: Area and Power comparison of
the Fusion Unit. Temporal fusion provided as reference.

Out

Figure 8: Spatial fusion.

multiplier by using a shift-add tree and a single shared accumulator to reduce the number of gates required. Each level
of the shift-add tree consists of three shift-units and a fourinput adder that together implement Equation 5. Compared
to a 4-bit fixed bitwidth multiplier the spatial multiplier that
implements fuse_multiply2→4 requires more area but delivers
4× higher performance for 2-bit operations. Overall, spaormance
tial fusion provides higher per farea
compared to temporal
fusion by packing more BitBricks in the same area.
Fusion Unit using spatial and temporal fusion.
As
discussed, a Fusion Unit is an implementation of the
fuse_multiply2→16 operator that supports 2-bit to 16-bit
operands. Using Equation (4) recursively, we can implement
the fuse_multiply2→16 operator circuit with temporal fusion,
spatial fusion, or a combination of both. For a fixed area
budget, using spatial fusion with 64 BitBrick to implement
the fuse_multiply2→16 operator would pack the highest number of BitBricks. At the same time, feeding the 64 BitBricks
for spatial fusion would require 128-bit wide accesses to the
SRAM buffers (IBUF and WBUF in Figure 2) per Fusion
Unit. Increasing the width of SRAMs increases the area required by the IBUF and WBUF. Therefore, we make a tradeoff wherein we use spatial fusion to combine 16 BitBricks
spatially to realize fuse_multiply2→8 operator, and then combine it with temporal fusion to deliver the fuse_multiply2→16
over four cycles. With this hybrid approach we balance both
bit-level flexibility and the corresponding area overhead due
to increased SRAM sizes. Figure 9 compares the area and
power requirements for a Fusion Unit with 16 BitBricks that
uses the hybrid approach. The area and power consumption
of using temporal fusion is provided as a point of reference.
As shown, for 16 BitBricks, the hybrid Fusion Unit has 2.8×
less area and 2.6× less power compared to temporal fusion
with the same number of BitBricks.
The next section discusses the Bit Fusion-ISA, that exposes the bit-level flexibility of Bit Fusion to the software.

4.

Area
BitBricks
(um^2)
Temporal
463
Fusion Unit
369

INSTRUCTION SET ARCHITECTURE

To leverage the unique bit-level flexibility of Bit Fusion,
we need to design a new hardware-software interface that
exposes those capabilities in an abstract manner. Furthermore, the abstraction must be flexible to enable a wide range
of DNN models so as to exploit bit-level fusion. The fol-

Table 1: Bit Fusion Instruction Set.
OpCode
5-bits
setup
ld-mem
st-mem
rd-buf
wr-buf

Operand Specification
6-bits
op0.bitwidth

op1.bitwidth

X

X

X

num-words

loop-id

ld/st

compute
block-end

Immediate
16-bits

mem.bitwidth
scratchpadtype

gen-addr
loop

Loop Identifier
5-bits

fn
X

X
stride
X

loop-level

num-iterations
Address of next instruction

lowing lists the requirements for an ISA that provides this
abstraction and enables productive use of Bit Fusion for various categories of DNNs.
1. Amortize the cost of bit-level fusion by grouping operation. The operations in a DNN are organized into
groups, called layers, wherein the same mathematical operation repeats a large number of times (often hundreds of
thousands). To avoid the overhead of fine-grained control
over the operations at such a scale, the abstraction needs
to amortize the cost of bit-level fusion across blocks of
instruction that implement the layers.
2. Enable a flexible data-path for Bit Fusion. Both the
number of words and the bitwidth of each word that feeds
the Fused-PEs varies depending on how the BitBricks are
composed as discussed in Section 2. Thus, the semantics
of instructions for data accesses must vary according to
the fusion configuration to enable a flexible data-path.
3. Provide a concise expression for a wide range of DNN
layers. As research in DNNs is still volatile, it is necessary to devise an ISA that is general enough to express a
wide range of DNN operations/layers. At the same time,
the ISA should minimize the von Neumann overhead of
instruction handling and require a small footprint.

4.1 Fusion-ISA for Bit-Flexible Acceleration
Table 1 summarizes the Bit Fusion-ISA that aims to satisfy
these requirements. The rest of this section discusses the
instruction formats and provide the insight that drives them.
Block-structured ISA for DNN layers. To leverage the
commonalities in the operations of a layer, the Bit Fusion
ISA is block structured. As such, the fusion configuration

of the BitBricks is fixed across each block of instructions that
implement a specific layer. The setup instruction marks the
beginning of an instruction block and configures the Fusion
Units and its data delivery logic to the specified bitwidth for
the operands. This instruction effectively defines the logical
fusion of the BitBricks into Fused-PEs for all the instructions
in the block. The block-end instruction signifies the end
of a block and provides the address to the next instruction in
the next-inst field.
Concise expression of DNN layers. DNNs consist of a
large number of simple operations like multiply-accumulate
and min/max, repeated over a large number of neurons (over
700 million multiply-add operations in AlexNet. See Table 2). Thus, the von Neumann overhead of instruction fetch
and decode can limit performance due to the large number
of operations required by a DNN. To minimize the number
of instruction fetches and decodes required for DNN execution, we leverage the following insight. Each layer in
a DNN is a series of simple mathematical operations over
hyper-dimensional arrays. How the operations walk through
the array elements and the type of mathematical operation
(multiply-add/max) uniquely defines a layer. As such, the
ISA provides loop instructions that enable a concise way of
defining the walks and the operations in a DNN layer. Each
loop instruction has a unique ID in the block. As shown in
Table 1, the num-iterations field in the loop instruction
defines iteration count. On the hand, the compute instruction specifies the type of operation, while the gen-addr instruction dictates how to walk through the elements of the
input/output hyper-dimensional arrays. The stride field
in the gen-addr instruction specifies how to walk through
the array elements in the loop, which is identified by the
loop-id field. The words after the setup instruction define the memory base address for the data that fills the three
buffers of input, output, and weights. The gen-addr instruction uses the following formula to generate the addresses that
walk through the memory data and fill the buffers.
Address = base + ∑(loop_iterator[id] × stride[id]) (6)
id

In Equation (6), id is the loop-id field of all the
gen-addr instruction in the block and the loop_iterator
is the current iteration of the corresponding loops and
their strides. The fundamental assumption is that multiple gen-addr instructions repeated by corresponding loop
instructions define the complex multi-dimensional walks
that expresses various kinds of DNN layers from LSTM to
CNN. In the evaluated benchmarks, blocks with 30-86 instructions are enough to cover LSTM, CNN, pooling, and
fully connected. These blocks use a combination of loop,
compute, and gen-addr instructions to define these DNN
layers nested loops. These statistics shows that our ISA
can concisely express various DNN layers while providing
bit-level fusion capabilities. Note that these instructions are
fetched and decoded once at the beginning of an instruction
block, amortizing the von Neumann overhead over the entire
execution of the block.
Managing memory accesses for Fused-PEs.
The
ld-mem/st-mem instructions exchange data between the onchip buffers (IBUF, OBUF, and WBUF) in Figure 2–and the off-

chip memory. Similarly, the rd-buf/wr-buf instructions
read/write data from the on-chip buffers specified by the
scratchpad-type field as shown in Table 1. In these four
instructions, the size of the operands, which are variablebitwidth arrays, depends on the number of array elements
and their bitwidth. These parameters, which control the
logic that feeds the Fused-PEs, is dependent on the bitlevel fusion configuration (number of Fused-PEs in each
Fusion Unit) and the type of data (input/weights). To capture this variation in the size of data, the semantics of
rd-buf/wr-buf and ld-mem/st-mem instructions for accessing on-chip and off-chip memory vary according to the
fusion configuration of their instruction block, set apriori. In
particular, the sizes of memory accesses by ld-mem/st-mem
instructions depend on both its num-words field and the fusion configuration defined by the corresponding setup instruction. Additionally, the ISA assumes a microarchitectural mechanism that provides the bit-flexible accesses that
vary according to the fusion configuration.
Decoupling on-chip and off-chip memory accesses. The
data required by DNNs, and subsequently, the number of
memory accesses are large. Hence, the latency due to offchip memory accesses can be a performance bottleneck. To
hide the latency of off-chip accesses, the ISA decouples the
on-chip memory accesses with off-chip accesses to enable
data streaming. Furthermore, decoupling the two types of
memory accesses allows the accelerator to reuse on-chip
data using simple scratchpad buffers, instead of hardwaremanaged caches.

4.2

Code Optimizations

As discussed in Section 4, the Fusion-ISA uses simple instructions combined with explicit loop instructions to express neural networks. The use of simpler instructions
makes the ISA flexible to express a large range of DNNs.
Nonetheless, the flexibility in the ISA enables incorporating layer-specific optimizations to improve the performance
and energy gains. For brevity, we use an example fullyconnected layer to discuss the code optimizations. Figure 10
shows the matrix-matrix multiplication associated with this
example. We perform the following three optimizations as
depicted in Figure 11.
Loop ordering. Loop-ordering optimizes the order of the
outer loops and memory instructions to further reduce offchip accesses. Recall that Bit Fusion-ISA uses loop indices
to generate memory addresses (Section 4). When the address
for a memory instruction does not depend on the index of the
previous loop instruction, their order can be exchanged. The
optimized code in Figure 11(b) uses Output-Stationary
for executing the fully-connected layer, to reduce read/write accesses to the output buffer. Changing the order
allows Bit Fusion to switch between Input-Stationary,
Output-Stationary, and Weight-Stationary to minimize off-chip and on-chip accesses.
Loop tiling. Loop-tiling refers to partitioning of a loop instruction in the Bit Fusion-ISA into smaller tiles such that
the data required by a loop operation fits inside the on-chip
scratchpads. The smaller tiles are accessed using a single
LD/ST instruction and are reused in the inner-loop to reduce
off-chip accesses. Compared to the original code in Fig-

Table 2: The evaluated CNN/RNN benchmarks.

OC

IC
IC

B

OC
B

Figure 10: A single Fully-Connected Layer. The × symbol represents matrix-matrix multiplication.
Compute
Loop

Memory
Buﬀer

loop: oc -> (OC)
loop: ic -> (IC)
ld-mem, IBUF, 1
ld-mem, WBUF, 1
ld-mem, OBUF, 1
rd-buf, IBUF -> in
rd-buf, WBUF -> wgt
rd-buf, OBUF -> out
out += in * wgt
wr-buf: out -> OBUF
st-mem: OBUF, 1

(a) Ini5al code

loop: toc -> (1, #tileoc)
ld-mem, OBUF, tileoc
loop: tic -> (1, #tileic)
ld-mem, IBUF, tileic
ld-mem, WBUF, tilewgt
loop: oc -> (1, tileoc)
rd-buf, OBUF -> out
loop: ic -> (1, tileic)
rd-buf, IBUF -> in
rd-buf, WBUF -> wgt
out += in * wgt
wr-buf: out -> OBUF
st-mem, OBUF, tileoc

(b) Op5mized code

Figure 11: (a) The equivalent code for the Fully Connected Layer.
(b) Optimized code using loop tiling and ordering. setup and
gen-addr instructions omitted for clarity.

ure 11(a), the tiled version in Figure 11(b) reduces off-chip
accesses for output buffer by a factor of IC×, and on-chip
accesses for output buffer by a factor of tileic . Note that
IC is the a dimension in the matrix multiplication operation
as depicted in Figure 10. Convolution layers typically require six loop instructions, which increases to 12 after tiling
optimizations. The overhead of increasing the number of
instructions on performance is negligible since the cost of
fetch and decode is amortized throughout the execution of
the layer.
Layer fusion. The Bit Fusion architecture consists of a 2D systolic array of multipliers, along with a 1-D array of
pooling/activation units. The example layer in Figure 10,
the fully-connected layer uses the 2-D systolic array. If the
next layer is activation, then we can fuse the layers and create one block of instruction for computing the both layers.
In general, when two consecutive layers use mutually exclusive on-chip resources, the instructions for the two layers are
combined such that the data produced by the first layer is directly fed into the subsequent layer, avoiding costly off-chip
accesses.

5.

EVALUATION

5.1

Methodology

Benchmarks. Table 2 shows the list of 8 CNN and RNN
benchmarks from diverse domains including image classification, object and optical character recognition, and language modeling. The selected DNN benchmarks use a diverse size of input data, which allows us to evaluate the
effect of input data size on the Bit Fusion architecture.

DNN

Type

AlexNet
Cifar-10
LSTM
LeNet-5
ResNet-18
RNN
SVHN
VGG-7

CNN
CNN
RNN
CNN
CNN
RNN
CNN
CNN

Domain

Dataset

Multiply-Add
Operations

Model
Weights

Image Classification
ImageNet
724 Mops 10.7 MBytes
Object Recognition
CIFAR-10
617 Mops 3.3 MBytes
Language Modeling
Penn TreeBank
13 Mops 6.2 MBytes
Optical Character Recognition
MNIST
16 Mops 0.5 MBytes
Image Classification
ImageNet
1,126 Mops 1.6 MBytes
Language Modeling
Penn TreeBank
17 Mops 8.0 MBytes
Optical Character Recognition
SVHN
158 Mops 0.8 MBytes
Object Recognition
CIFAR-10
317 Mops 2.7 MBytes

Table 3: The evaluated ASIC and GPU platforms. *Stripes entries
are per-tile.
Chip
Cores
(1.1 mm^2)

BitFusion

ASIC
Eyeriss

Stripes*

Chip

512 FU

168 Pes

4096 SIPs

Cores

On-chip Memory

112 KB

Chip Area (mm^2)
Frequency
Technology

5.87
500 MHz
45

GPU
Titan X Tegra X2
3,584

256

2 MB eDRAM
Memory
12 GB
8 GB
16 KB SRAM
5.87
3.62
TDP
250 W
7.5 W
500 MHz
980 MHz
Frequency 1,531 MHz 875 MHz
45
45
Technology
16 nm
16 nm
108 KB

AlexNet [32, 36], SVHN [35, 38], CIFAR10 [35, 39], LeNet5 [34, 40], VGG-7 [34, 41], RESNET-18 [34, 42] are popular and widely-used CNN models. Among them, AlexNet
and RESNET-18 benchmarks are image classification ap-

plications that have different network topologies that use
the ImageNet dataset. The SVHN and LeNet-5 benchmarks
are optical character recognition applications that recognize the house numbers from the house view photos and
handwritten/machine-printed characters, respectively. CIFAR10 and VGG-7 are object recognition applications based
on the CIFAR-10 and ImageNet dataset, respectively. The
RNN and LSTM [43] are recurrent networks that perform language modeling on the Penn TreeBank dataset [44]. In Table 2, the “Multiply-Add Operations” column shows the required number of Multiply-Add operations for each model
and the “Model Weights” column shows the size of model
parameter. Note that the multiply-add operations and model
weights have variable bitwidths as presented in Figure 1.
Reduced bitwidth DNN models. Bit Fusion aims to accelerate the inference of a wide range of DNN models with varying bitwidth requirements, with no loss in classification accuracy. The benchmarks, listed in Table 2, employ the model
topologies proposed in prior literature [32, 34, 35] that train
low bitwidth DNNs and achieve the same accuracy as the
32-bit floating-point models.
Accelerator development and synthesis. We use RTLVerilog to implement the configuration of the Bit Fusion
architecture and verify the design through extensive RTLsimulations. We synthesize Bit Fusion at 45nm technology
node using Synopsys Design Compiler (L-2016.03-SP5) and
a commercial 45-nm standard-cell library. Design Compiler
provides the chip area, achievable frequency, and dynamic/static power, which we use to estimate the performance
and energy-efficiency of the Bit Fusion accelerator.
Simulation infrastructure for Bit Fusion. We compile
each DNN benchmark to the instructions of the Fusion-ISA
(Section 4). We develop a cycle-accurate simulator that takes
the Fusion-ISA instructions for the given DNN and simulates
the execution to calculate the cycle counts as well as the
number of accesses to on-chip buffers (IBUF, OBUF, and
WBUF in Figure 2) and off-chip memory. We verify the
cycle counts of the simulator against our Verilog implemen-

Energy Breakdown
PlaIorm

Benchmark

BitFusion

AlexNet

Eyeriss

AlexNet

0.15568319

BitFusion

Cifar-10

0.089193394 0.172484584

Eyeriss

Cifar-10

0.162973985 0.00859988 0.577250157 0.251175979

BitFusion

LSTM

0.092911858 0.232506951

Eyeriss

LSTM

0.171110733 0.006774956 0.615672168 0.206442143

LeNet-5

0.135613633 0.015006519 0.461452446 0.387927402

Register File

DRAM

0 0.678064451

0.01101809 0.558872748 0.274425972
0 0.738322021
0 0.674581191

19.6

Speedup over
Batch Size = 1

.42

.44

.14

04

97

32

.25

53

.68

cle

Speedup over BW
= 32 bits/cycle

.22

.46

.00

2.5

87
4

.56

77

.86

TX2

9.9
5.2

TitanX

5.2.1

Net Cifar-10

8×

6×

4×

2×

0×

15.1
21.4
21.4

14.8
21.1
21.1

=

tation of the Bit Fusion architecture. Using the frequency defined in Table 3 and the cycle counts, the simulator measures
the execution time of the Bit Fusion architecture. To evalu6×
ate
the energy
we
model
Batchefficiency,
Size = 1
Batch
Size =the
4 energy
Batch consumption
Size = 16
Batch Size = 32
Batch Size = 64
for
4× on-chip buffers for the Bit Fusion accelerator using the
results from CACTI-P [45].
2×
Comparison
with Eyeriss. To measure the performance
and
0× energy dissipation of our comparison point, Eyeriss, we
AlexNet Cifar-10 LSTM LeNet-5 ResNet-18 RNN
SVHN
VGG-7 geomean
use their
own open-source simulation infrastructure
[4]. The
resulting area and energy metrics are shown in Table 3. As
mentioned, we use the same area budgets as Eyeriss to synthesize Bit Fusion, the details of which are presented in Table 3. Eyeriss operates on the 16-bit operands and Bit Fusion
supports flexible bitwidths from 2, 4, 8, to 16 bits.
Comparison with Stripes. The authors of Stripes gra6×
ciously 32
shared
their simulator
[2]. Their
power estimation
bits/cycle
64 bits/cycle
128 bits/cycle
256 in
bits/cycle
512 bits/cycle
tools were
65 nm node,
which we scaled to 45 nm. Stripes
4×
operates on 16-bit inputs and variable-bitwidth weights (1
through 16), using Serial Inner-Product units (SIPs). Stripes
2×
is organized into 16 tiles each of which has 4096 SIPs. For
0×
a fair
comparison,
replace
theRNN
4096SVHN
SIPsVGG-7
in each
tile of
AlexNet
Cifar-10 LSTMwe
LeNet-5
ResNet-18
geomean
Stripes with our proposed Bit Fusion systolic array with 512
Fusion Units, each with 16 BitBricks to match the same budget of 1.1mm2 for compute, which is the area after scaling to
45 nm.
GPU. We use two GPUs (Titan X and Tegra X2) based
on Nvidia’s Pascal architecture to compare the performance
benefits of Bit Fusion. Table 3 shows the details of the
two GPUs. We use the popular Caffe-2 library for high
performance GPU implementations of the DNNs described
in Table 2. Across GPU platforms, we use 1,000 warmup batches, followed by 10,000 batches to measure performance and use the average.

AlexNet

Bit Fusion

Experimental Results
Comparison to Eyeriss

Performance and energy improvement. To evaluate the
performance
and efficiency
from the Bit Fusion arLeNet-5 ResNet-18 RNN
SVHN
VGG-7 benefits
geomean
chitecture, we compare with a state-of-the-art accelerator
Eyeriss [1] that proposes an optimized dataflow architecture
for DNNs. We match the same area budget of 1.1mm2 for
computational logic across both architectures: systolic array
in Bit Fusion and PEs in Eyeriss, and match the total SRAM
capacity.
configurations for both Eyeriss
PerformanceTable 3 details
Energythe
Reduction
21.6
and Bit Fusion
. For a fair comparison, we scale the area and
energy consumption of the PEs, register-files, on-chip network, and DRAM in Eyeriss to 45nm technology according
to the methodology proposed in [4]. For a fair comparison
between the two architectures,
we use the same frequency
Cifar-10 LSTM LeNet-5 ResNet-18 RNN
SVHN
VGG-7 geomean

LSTM

AlexNet per-layer improvement over Eyeriss
Layers
Conv 8/8-bit 105MOps

Performance

Energy
Reduc3on

1.6694367456.502816144

et

xN

Ale

10

arCif

TM

LS

t-5

Ne

Le

8

1
et-

sN
Re

N
RN

HN

SV

Eyeriss

Eyeriss

0%

BitFusion

Figure 12: Bit Fusion performance and energy improvements
over Eyeriss.

25%

BitFusion

0.068346381 0.020760414 0.218879419 0.692013787
0.094100809
0.248371721
LeNet-5
ResNet-18
RNN
SVHN0 0.65752747
VGG-7 geomean
0.068623546 0.029393022 0.217938506 0.684044926

50%

Eyeriss

4.3

0 0.670126181

Eyeriss

SVHN
VGG-7
LSTM
VGG-7

0.575854454 0.25985111

4.5

DRAM

75%

BitFusion

Eyeriss
BitFusion
AlexNet
Cifar-10
Eyeriss

7.8

9.6

BitFusion

RNN
0.156498902
4.9 0.007795533
2.8
2.8 2.5
2.8
SVHN
0.096532044 0.233341775

0 0.741993357

Register File

Eyeriss

0.066867658
8.2

Eyeriss

RNN

10.1
0.191138985

BitFusion

9.6
BitFusion
10 ×
6.1
5.4 Eyeriss
5×
BitFusion

0 0.721858658

10.4

Eyeriss

12.0
0.165314865 0.009843179 0.565539943 0.259302012

BitFusion

0.079166591 0.198974751

ResNet-18

Eyeriss

ResNet-18

Eyeriss

Buﬀers

100%

BitFusion

BitFusion

13.8

Compute

Energy Reduction
0 0.753640678

0.133855151

BitFusion

Improvement
over Eyeriss

15 ×

0×

Performance
LeNet-5
0.112504171

Eyeriss

Buﬀers

Energy Breakdown

BitFusion

20 ×

Compute

0.111405113 0.210530435

7
G-

VG

Figure 13: Energy breakdown of Bit Fusion and Eyeriss.

of 500MHz reported in the paper [4] for both Eyeriss and
Bit Fusion. Figure 12 presents the performance and energy
benefits of Bit Fusion in comparison with Eyeriss. On average, Bit Fusion delivers 4.3× speedup since the Bit Fusion
architecture can perform more DNN operations with lower
bitwidth in a given area compared to Eyeriss. Depending
on the types of DNN operations and the required bitwidths,
the benchmarks see different performance gains. The CNN
benchmarks (AlexNet, SVHN, CIFAR10, LeNet-5, VGG-7, and
RESNET-18) see higher performance gains than the recurrent networks (RNN and LSTM) since the convolution operations are more amenable for data reuse in systolic architecture of Bit Fusion. CIFAR10 sees the highest benefits of 6.1×
speedup since most of its operations can be computed with
the smallest bitwidth (2-bit input and 2-bit weight) and its
operations provide a large degree of parallelism that can exploit the increased number of Fused-PEs. In contrast, LeNet5 achieves the lowest speedup, 2.5×, because the existing
parallelism in this benchmark is significantly less than others; thus, it is unable to utilize the increased hardware parallelism by lowering the operand bitwidths. Using variable
bitwidth improves performance and energy efficiency, since
it increases compute capacity and reduces active hardware
components. Figure 12 shows the energy reduction. The average improvement is 9.6×, with the largest of 19.6× from
SVHN and the smallest of 2.8× from LSTM. The significant
energy reduction attributes to both Fusion Unit organizations
and memory access reductions, which we discuss below in
more detail.
Energy breakdown. To understand the sources of the energy reduction, we break down the energy consumptions for
each hardware component (compute units, on-chip SRAM
buffers, register file, and off-chip DRAM memory). Figure 13 shows the per-component energy dissipation for Bit
Fusion and Eyeriss. This figure should be considered with
the energy reduction results from Figure 12. Both accelerators consume more than 80% of energy for the on-chip and
off-chip memory accesses. The bit-level flexibility for memory accesses in Bit Fusion significantly reduces energy consumption for both on-chip buffers (IBUF, OBUF, and WBUF
in Figure 2) and off-chip DRAM. Furthermore, with bit-level
flexibility, our buffers can hold more data at lower-bitwidths,
effectively giving Bit Fusion more on-chip storage capacity,
which leads fewer off-chip memory accesses. Eyeriss employs local register files within each PEs, which constitutes
a significant portion of the energy consumption. Bit Fusion’s

Performance

2 bits/cycle

6×

2.22
2.46

10 ×

4×

4.00
2.5

2×

2.87
4

0×

2.56

5×
0×

19.6

Performance compared to GPUs

13.8

15 ×

Energy Reduction

Benchmark

9.6

TX2

10.4

AlexNet
8.2
64 bits/cycle

32 bits/cycle
6.1
5.4256 bits/cycle

Cifar-10
512 bits/cycle
4.9

2.8 2.8 LSTM
2.5

AlexNet Cifar-10

AlexNet Cifar-10

LeNet-5

Bit Fusion
12.0

TitanX

10.1

1 128

3.64 6.04
4.5

1

2.8
1

3.91

2.77

4.3

3.35

9.95351829 8.37298514

1

LSTM
LeNet-5 ResNet-18 1 RNN
SVHN
ResNet-18
5.094291057

LSTM

2.67 9.6

4.79
bits/cycle
7.8

RNN

1 4.884339406

SVHN

1

VGG-7

1

VGG-7 geomean
5.094291057

3.740452613

3.47VGG-7 geomean
3.69
3.73
3.84

LeNet-5 ResNet-18 RNN

SVHN

Speedup over
Batch Size = 1

1.44
21.14

4910704

1171497

1541032

2×
0×

1.25

3189853

AlexNet Cifar-10

LSTM

Performance

Energy
Reduc4on

AlexNet

1.25

19.30

2.17

Cifar-10

1.82

46.99

5.28

LSTM

2.83

56.59

6.36

LeNet-5

4.90

192.50

21.63

ResNet-18
LeNet-5
ResNet-18 RNN 2.24SVHN

40.22
4.52
VGG-7 geomean

SVHN

2.00

11.78

1.32

VGG-7

2.09

12.25

1.38

geomean
2.44
36.77
4.13
TitanX
Bit Fusion
architecture
avoids
the
need for register
files
and en9.9
forces explicit data sharing for inputs and partial results, as
shown in Figure 2. Therefore, Bit Fusion saves on Register
File energy, but requires more SRAM accesses. The combined
6× effect of bit-level flexibility and the systolic organi32 bits/cycle
64 bits/cycle
128 bits/cycle
zationLeNet-5
of BitBricks
in the
Bit bits/cycle
Fusion
architecture provides an
LSTM
ResNet-18
RNN
SVHN
VGG-7 geomean
256 bits/cycle
512
4×
average energy savings of 9.6×. Off-chip DRAM accesses,
however,
are still a significant portion of Bit Fusion’s energy
2×
consumption and its share AlexNet
growsper-layer
due toimprovement
the significant
reover Eyeriss
0×
Layers
Performance
Energy
duction
of
compute
and
on-chip
storage
energy.
AlexNet Cifar-10 LSTM LeNet-5 ResNet-18 RNN
SVHN
VGG-7 geomean

Speedup over BW
= 32 bits/cycle

AlexNet
Cifar-10
2.46

2.5
2.87
4
2.56

Reduc4on

2.77

Conv 8/8-bit 105MOps

5.2.2Performance
Sensitivity Study
Energy
ConvReduction
4/1-bit 560MOps

2.86

21.6

1.6694367456.502816144
6.39389677916.83670656
3.31034047330.73860155
1.00523431710.28666077

over Stripes

Improvement
over Stripes

FC 4/1-bits 54MOps
Sensitivity to memory bandwidth.
Our default off-chip
FC 8/8-bit 4MOps
6×
memory bandwidth is 128 bits per cycle. However, depend4×
ing on the characteristics of model topology and DNN op2×
erations, the optimal bandwidth
that maximizes
the perforResNet-18
per-layer improvement
over
0×
Layers
Performance
Energy
varies.
ToResNet-18
understand
the
correlation
the
AlexNet mance
Cifar-10 LSTM
LeNet-5
RNN
SVHN
VGG-7 geomean between
Reduc4on
bandwidth and performance, we perform a sensitivity study
for bandwidth. Figure 14 shows the performance improvements with Bit Fusion as we change the bandwidth from
0.25× to 4× of the default value. The baseline in this study
the Bit Fusion with the default bandwidth of 128 bits per cyaverage, when Bit
weFusion
scale
bandwidth
upover
to Eyeriss-2
4×, Bit
TX2 cle. On TitanX
LSTM the
per-layer
improvement
9.9
Layers
Fusion
provides 1.6× speedup compared
to Performance
the defaultEnergy
setReduc4on
ting, while with 0.25× bandwidth, the performance degrades
60%. Since CNN benchmarks see more opportunities for
data reuse, they have less sensitivity to the bandwidth compared to the RNN benchmarks.
The two RNN benchmarks,
AlexNet Cifar-10 LSTM LeNet-5 ResNet-18 RNN
SVHN
VGG-7 geomean
LSTM and RNN, provide almost linearly-scaling speedup as
the bandwidth increases since their fully-connected layers
are bottlenecked by the bandwidth.
Sensitivity to batch size. To amortize the overhead of the
weight read, batching is considered as a standard way for
DNN
execution. The Bit
Fusion
architecture also supports
Performance
Energy
Reduction
21.6
batching to maximize
the weight reuse and minimize the off8×
chip memory accesses. Figure 15 shows how performance
6×
changes as we increase the batch size from 1 through 256
4×
2×
with the batch size 1 as the baseline (no batching). Our
0×
default batch size is 16. On
average,
Bit Fusion with the
AlexNet Cifar-10 LSTM LeNet-5 ResNet-18 RNN
SVHN
VGG-7 geomean
batch size of 256 engenders 2.7× speedup with the highest speedup of 21.4× from RNN. Since batching is effective when the bandwidth is limited and the performance is

8×

LeNet-5 ResNet-18 RNN

SVHN

VGG-7 geomean

Figure 16: Performance comparison to GPUs.

Energy
Reduc4on

TX2systolic

its/cycle

LSTM

Batch Size = 16

RNN as the batch
4.20 size58.32
6.55
Figure 15: Bit Fusion performance
increases.

2.68

Bit Fusion

Improvement over Stripes
Benchmark

4×

AlexNet Cifar-10

Performance
Improvement
over Stripes

6×

1.42

TitanX
9.9

15.1
21.4
21.4

h Size =
256

Batch Size = 4
Batch Size = 256

14.8
21.1
21.1

Batch Size = 1
Batch Size = 64

4.00

10×
8×
6×
4×
2×
0×

geomean
4.65 changes.
4.32
Figure 14: Bit Fusion performance
as the1.00
bandwidth

2.86

2.22

TX2
Speedup over
TX2

Speedup over BW
= 32 bits/cycle
Improvement
over Eyeriss

20 ×

Energy Reduction

21.6

8×
6×

4× Fusion performance and energy improvements
Figure 17: Bit
over Stripes. 2×
0×

AlexNet Cifar-10

LSTM

LeNet-5 ResNet-18 RNN

SVHN

VGG-7 geomean

bandwidth-bound, the trends are similar to the bandwidth
sensitivity results presented in Figure 14. However, there is
a marginal gain across all the benchmarks when the batch
is increased from 64 to 256, since the number of Fused-PEs
in Bit Fusion, even with the low bitwidth, is not enough to
parallelize the computation for 256 inputs.

5.2.3

Comparison to GPUs

Performance comparison to GPUs. GPUs are the most
widely-used general-purpose processors for DNNs. We
compare the performance of Bit Fusion accelerators with two
GPUs: (1) Tegra X2 (TX2), and (2) Titan X based on the
Pascal architecture (TitanX), details of which are presented
in Table 3. Figure 16 shows the speedup of TitanX and Bit
Fusion using the TX2 as the baseline. In the comparisons
to TX2, TitanX and Bit Fusion provide on average 4.6× and
4.3× speedup, respectively. The LeNet-5 benchmark sees the
maximum gains of 9.9× and 8.4× performance from TitanX
and Bit Fusion, respectively. These benefits come from the
small memory footprint of LeNet-5’s input and weight data
and the low bitwidth requirements of its operations.

5.2.4

Comparison to Stripes

Performance compared to Stripes. Figure 17 presents the
performance and energy benefits of Bit Fusion in comparison
with Stripes. On average, Bit Fusion provides 2.4× speedup
over Stripes. Stripes uses bit-serial computations to support
variable bitwidths just for DNN weights. As opposed to
Stripes, the Bit Fusion architecture offers dynamically composable BitBricks to support flexible bitwidths for both inputs and weights in DNNs. Bit Fusion achieves the highest
speedup of 4.9× and lowest speedup of 1.3× over Stripes for
benchmarks LeNet-5 and AlexNet, respectively. ResNet-18
which is the most recent and the biggest of the benchmarks
sees 4.2× performance benefits as it can use low bitwidth
on both operands. AlexNet uses 8-bit inputs/weights for the
first convolution layer and the last fully-connected layer. The
two 8-bit layers limit the benefits of Bit Fusion over Stripes.
Benchmark LeNet-5, on the other hand, uses low bitwidths
for both inputs and weights, resulting in the highest performance benefits with Bit Fusion.
Energy reduction compared to Stripes. Figure 17 also depicts the improvement in energy when Bit Fusion is com-

pared to Stripes. As mentioned, Bit Fusion benefits from
reduction in both computation and memory access at lower
bitwidths for both inputs and weights. On average, Bit Fusion
reduces energy consumption by 4.1× over Stripes. LeNet-5
sees the highest energy reduction of 21.6×, while benchmark RNN sees the least energy reduction of 1.3× over
Stripes. Benchmarks RNN and LSTM require a large number
of accesses to the on-chip storage, which limits the energy
reduction benefits from Bit Fusion. For ResNet-18, the energy is reduced by a factor of 6.6× and the energy benefits
for AlexNet is 2.2×.
Bit Fusion offers a fundamentally different approach from
Stripes and explores the dimension of bit-level dynamic
composablilty, which significantly improves performance
and energy.

6.

RELATED WORK

A growing body of related works develop DNN accelerators. Bit Fusion fundamentally differs from prior work
as it introduces and explores a new dimension of bit-level
composable architectures that can dynamically match the
bitwidth required by DNN operations. Bit Fusion aims to
minimize both computations and communications in the
finest granularity possible without compromising on the
DNN accuracy. Below, we discuss the most related work.
Precision flexibility in DNNs. Stripes [2] and Tartan [6]
use bit-serial compute units to provide precision flexibility
for weights at the cost of additional area overhead. Both
works provide performance and efficiency benefits that are
proportional to the precision reduction for weight (model parameters). We directly compare the benefits of Bit Fusion to
Stripes in Section 5. Moons et al. propose aggressive voltage scaling techniques when operating at low precision for
increased energy efficiency at constant throughput by turning off parts of the multiplier [37, 46]. As such, they do
not offer fusion capabilities. TPU [30] propose a systolic
architecture for DNNs and supports 8-bit and 16-bit precision. This work, on the other hand, proposes an architecture that dynamically composes low-bitwidth compute units
(BitBricks) to match the bitwidth requirements of DNN layers
at runtime.
Binary DNN accelerators. Several inspiring works have
explored ASIC and FPGA accelerators optimized for Binary DNNs. FINN [47] use FPGAs for accelerating Binary
DNNs, while YodaNN [48] and BRein [49] propose an ASIC
accelerator for Binary DNNs. Kim, et al. [50] decompose
the convolution weights for binary CNNs to improve performance and energy efficiency. The above works focus solely
on binary DNNs to achieve high performance at the cost of
classification accuracy. Bit Fusion, on the other hand, flexibly matches the bitwidths of DNN operations for performance/energy benefits without losing accuracy.
Sparse Accelerators for DNNs. EIE [5], CambriconX [15], Cnvlutin [13], and SCNN [51] explore the sparsity in the DNN layers and use zero-skipping to provide
performance and energy-efficiency benefits. Orthogonal to
the works above, Bit Fusion explores the dimension of bitflexible accelerators for DNNs.
Other ASIC accelerators for DNNs. DaDianNao [7] uses

eDRAM to eliminate off-chip accesses and provide high performance and efficiency for DNNs. PuDianNao [9] is an
accelerator designed for machine learning, but does not support CNNs. Minerva [12] propose operation pruning and
data quantization techniques to reduce power consumption
for ASIC acceleration. Eyeriss [1, 3] present an optimized
row-stationary dataflow for DNNs to improve efficiency.
Tetris [4] and Neurocube [11] propose 3-D stacked DNN
accelerators to provide high bandwidth for DNN operations.
ISAAC [26], PipeLayer [28], and Prime [27] use resistive
RAM (ReRAM) for accelerating DNNs.
Instruction Sets for DNNs. Cambricon [14] provides an
ISA to express the different computations in a DNN using
vector and matrix operations without significant loss in efficiency over DaDianNao. DnnWeaver [22] proposes a coarse
grained ISA to express layers of DNNs, which are first translated to micro-codes for FPGA acceleration. Unlike prior
work, the Fusion-ISA proposed in the work is designed to
enable bit-level flexibility for accelerating DNNs. Further,
the Fusion-ISA uses loop instructions with iterative semantics to significantly reduce instruction footprint.
Code optimization techniques. Alwani, et. al [52] propose
layer-fusion, that combines multiple convolutional layers to
save off-chip accesses for FPGA acceleration of CNNs. Escher [53] propose a CNN FPGA accelerator using flexible
buffering that balances the off-chip accesses for inputs and
weights in CNNs. The above works have inspired the codeoptimizations explored in this paper, however, the key contribution of this work is a bit-level flexible DNN accelerator.
Software techniques for Binary/XNOR DNNs. QNN [35]
shows that efficient GPU kernels for XNOR-based binary
DNNs can provide up to 3.4× improvement in performance. XNOR-Net [54] show that specialized libraries for
Binary/XNOR-nets can achieve 58× performance on CPUs.
In contrast, Bit Fusion is an ASIC accelerator architecture
that supports a wide range of bitwidths (binary to 16-bits)
for DNNs with no accuracy loss.
Core Fusion and CLPs. Core Fusion [55] and CLPs [56]
are dynamically configurable chip multiprocessors that a
group of independent processors can fuse and form a more
capable CPU. In contrast to these inspiring works, Bit Fusion
performs the composition in the bit level rather than at the
level of full-fledged cores.

7.

CONCLUSION

Deep neural networks use abundant computation, but can
withstand very low bitwidth operations without any loss in
accuracy. Leveraging this property of DNNs, we develop Bit
Fusion, a bit-level dynamically composable architecture, for
their efficient acceleration. The architecture comes with an
ISA that enables the software to utilize this bit-level fusion
capability to maximize the parallelism in computations and
minimize the data transfer in the finest granularity possible.
We evaluate the benefits of Bit Fusion by synthesizing the
Verilog implementation of the proposed microarchitecture in
45 nm technology node and using cycle accurate simulations
with eight real-world DNNs that require different bitwidths
in their layers. Bit Fusion achieves significant speedup and
energy benefits compared to state-of-the-art accelerators.

8.

ACKNOWLEDGMENTS

We thank Amir Yazdanbakhsh, Jacob Sacks, and Payal
Preet Bagga for insightful discussions.
This work
was in part supported by NSF awards CNS#1703812,
ECCS#1609823, CCF#1553192, Air Force Office of Scientific Research (AFOSR) Young Investigator Program (YIP)
award #FA9550-17-1-0274, and gifts from Google, Microsoft, and Qualcomm.

[15]
[16]
[17]

References
[1] Y.-H. Chen, J. Emer, and V. Sze, “Eyeriss: A spatial
architecture for energy-efficient dataflow for convolutional neural networks,” in ISCA, 2016.
[2] P. Judd, J. Albericio, T. Hetherington, T. M. Aamodt,
and A. Moshovos, “Stripes: Bit-serial deep neural network computing,” in MICRO, 2016.
[3] Y.-H. Chen, T. Krishna, J. S. Emer, and V. Sze, “Eyeriss: An energy-efficient reconfigurable accelerator for
deep convolutional neural networks,” IEEE Journal of
Solid-State Circuits, vol. 52, no. 1, pp. 127–138, 2017.
[4] M. Gao, J. Pu, X. Yang, M. Horowitz, and
C. Kozyrakis, “Tetris: Scalable and efficient neural network acceleration with 3d memory,” in ASPLOS, 2017.
[5] S. Han, X. Liu, H. Mao, J. Pu, A. Pedram, M. A.
Horowitz, and W. J. Dally, “Eie: efficient inference
engine on compressed deep neural network,” in ISCA,
2016.
[6] A. Delmas, S. Sharify, P. Judd, and A. Moshovos, “Tartan: Accelerating fully-connected and convolutional
layers in deep learning networks by exploiting numerical precision variability,” CoRR, vol. abs/1707.09068,
2017.
[7] Y. Chen, T. Luo, S. Liu, S. Zhang, L. He, J. Wang,
L. Li, T. Chen, Z. Xu, N. Sun, et al., “Dadiannao: A
machine-learning supercomputer,” in MICRO, 2014.
[8] T. Chen, Z. Du, N. Sun, J. Wang, C. Wu,
Y. Chen, and O. Temam, “Diannao: a small-footprint
high-throughput accelerator for ubiquitous machinelearning,” in ASPLOS, 2014.
[9] D. Liu, T. Chen, S. Liu, J. Zhou, S. Zhou, O. Teman,
X. Feng, X. Zhou, and Y. Chen, “Pudiannao: A polyvalent machine learning accelerator,” in ASPLOS, 2015.
[10] Z. Du, R. Fasthuber, T. Chen, P. Ienne, L. Li, T. Luo,
X. Feng, Y. Chen, and O. Temam, “Shidiannao: shifting vision processing closer to the sensor,” in ISCA,
2015.
[11] D. Kim, J. Kung, S. Chai, S. Yalamanchili, and
S. Mukhopadhyay, “Neurocube: A programmable digital neuromorphic architecture with high-density 3d
memory,” in ISCA, 2016.
[12] B. Reagen, P. Whatmough, R. Adolf, S. Rama, H. Lee,
S. K. Lee, J. M. Hernández-Lobato, G.-Y. Wei, and
D. Brooks, “Minerva: Enabling low-power, highlyaccurate deep neural network accelerators,” in ISCA,
2016.
[13] J. Albericio, P. Judd, T. Hetherington, T. Aamodt,
N. E. Jerger, and A. Moshovos, “Cnvlutin: ineffectualneuron-free deep neural network computing,” in ISCA,
2016.
[14] S. Liu, Z. Du, J. Tao, D. Han, T. Luo, Y. Xie, Y. Chen,

[18]
[19]

[20]

[21]
[22]
[23]
[24]

[25]

[26]

[27]

[28]
[29]

and T. Chen, “Cambricon: An instruction set architecture for neural networks,” in ISCA, 2016.
S. Zhang, Z. Du, L. Zhang, H. Lan, S. Liu, L. Li,
Q. Guo, T. Chen, and Y. Chen, “Cambricon-x: An accelerator for sparse neural networks,” in MICRO, 2016.
V. Gokhale, J. Jin, A. Dundar, B. Martini, and E. Culurciello, “A 240 g-ops/s mobile coprocessor for deep
neural networks,” in CVPRW, 2014.
J. Sim, J. S. Park, M. Kim, D. Bae, Y. Choi, and L. S.
Kim, “14.6 a 1.42tops/w deep convolutional neural network recognition processor for intelligent ioe systems,”
in ISSCC, 2016.
F. Conti and L. Benini, “A ultra-low-energy convolution engine for fast brain-inspired vision in multicore
clusters,” in DATE, 2015.
Y. Wang, J. Xu, Y. Han, H. Li, and X. Li, “Deepburning: Automatic generation of fpga-based learning
accelerators for the neural network family,” in DAC,
2016.
L. Song, Y. Wang, Y. Han, X. Zhao, B. Liu, and X. Li,
“C-brain: A deep learning accelerator that tames the diversity of cnns through adaptive data-level parallelization,” in DAC, 2016.
C. Zhang, P. Li, G. Sun, Y. Guan, B. Xiao, and J. Cong,
“Optimizing fpga-based accelerator design for deep
convolutional neural networks,” in FPGA, 2015.
H. Sharma, J. Park, D. Mahajan, E. Amaro, J. K. Kim,
C. Shao, A. Misra, and H. Esmaeilzadeh, “From highlevel deep neural models to fpgas,” in MICRO, 2016.
M. Alwani, H. Chen, M. Ferdman, and P. Milder,
“Fused-layer cnn accelerators,” in MICRO, 2016.
N. Suda, V. Chandra, G. Dasika, A. Mohanty, Y. Ma,
S. Vrudhula, J.-s. Seo, and Y. Cao, “Throughputoptimized opencl-based fpga accelerator for large-scale
convolutional neural networks,” in FPGA, 2016.
J. Qiu, J. Wang, S. Yao, K. Guo, B. Li, E. Zhou, J. Yu,
T. Tang, N. Xu, S. Song, et al., “Going deeper with
embedded fpga platform for convolutional neural network,” in FPGA, 2016.
A. Shafiee, A. Nag, N. Muralimanohar, R. Balasubramonian, J. P. Strachan, M. Hu, R. S. Williams, and
V. Srikumar, “Isaac: A convolutional neural network
accelerator with in-situ analog arithmetic in crossbars,”
in ISCA, 2016.
P. Chi, S. Li, C. Xu, T. Zhang, J. Zhao, Y. Liu, Y. Wang,
and Y. Xie, “Prime: A novel processing-in-memory
architecture for neural network computation in rerambased main memory,” in ISCA, 2016.
L. Song, X. Qian, H. Li, and Y. Chen, “Pipelayer: A
pipelined reram-based accelerator for deep learning,”
in HPCA, 2017.
E. Chung, J. Fowers, K. Ovtcharov, M. Papamichael,
A. Caulfield, T. Massengil, M. Liu, D. Lo, S. Alkalay,
M. Haselman, C. Boehn, O. Firestein, A. Forin, K. S.
Gatlin, M. Ghandi, S. Heil, K. Holohan, T. Juhasz,
R. K. Kovvuri, S. Lanka, F. van Megen, D. Mukhortov,
P. Patel, S. Reinhardt, A. Sapek, R. Seera, B. Sridharan,
L. Woods, P. Yi-Xiao, R. Zhao, and D. Burger, “Accelerating persistent neural networks at datacenter scale,”
in HotChips, 2017.

[30] N. P. Jouppi, C. Young, N. Patil, D. Patterson,
G. Agrawal, R. Bajwa, S. Bates, S. Bhatia, N. Boden,
A. Borchers, et al., “In-datacenter performance analysis of a tensor processing unit,” in ISCA, 2017.
[31] “Apple a11-bionic neural engine.” https://en.wikipedia.
org/wiki/Apple_A11.
[32] S. Zhou, Z. Ni, X. Zhou, H. Wen, Y. Wu, and
Y. Zou, “Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients,”
CoRR, vol. abs/1606.06160, 2016.
[33] C. Zhu, S. Han, H. Mao, and W. J. Dally,
“Trained ternary quantization,” arXiv preprint
arXiv:1612.01064, 2016.
[34] F. Li, B. Zhang, and B. Liu, “Ternary weight networks,”
arXiv preprint arXiv:1605.04711, 2016.
[35] I. Hubara, M. Courbariaux, D. Soudry, R. El-Yaniv,
and Y. Bengio, “Quantized neural networks: Training
neural networks with low precision weights and activations,” arXiv preprint arXiv:1609.07061, 2016.
[36] A. Krizhevsky, “One weird trick for parallelizing convolutional neural networks,” arXiv preprint
arXiv:1404.5997, 2014.
[37] B. Moons, R. Uytterhoeven, W. Dehaene, and M. Verhelst, “Dvafs: Trading computational accuracy for
energy through dynamic-voltage-accuracy-frequencyscaling,” in DATE, 2017.
[38] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and
A. Y. Ng, “Reading digits in natural images with unsupervised feature learning,” in NIPS workshop on deep
learning and unsupervised feature learning, vol. 2011,
p. 5, 2011.
[39] A. Krizhevsky and G. Hinton, “Learning multiple layers of features from tiny images,” Computer Science
Department, University of Toronto, Tech. Rep, vol. 1,
no. 4, p. 7, 2009.
[40] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner,
“Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11,
pp. 2278–2324, 1998.
[41] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,”
arXiv preprint arXiv:1409.1556, 2014.
[42] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual
learning for image recognition,” in CVPR, 2016.
[43] S. Hochreiter and J. Schmidhuber, “Long short-term
memory,” Neural computation, vol. 9, no. 8, pp. 1735–
1780, 1997.
[44] M. P. Marcus, M. A. Marcinkiewicz, and B. Santorini, “Building a large annotated corpus of english:
The penn treebank,” Computational linguistics, vol. 19,
no. 2, pp. 313–330, 1993.
[45] S. Li, K. Chen, J. H. Ahn, J. B. Brockman, and N. P.
Jouppi, “CACTI-P: Architecture-level Modeling for
SRAM-based Structures with Advanced Leakage Reduction Techniques,” in ICCAD, 2011.
[46] B. Moons and M. Verhelst, “A 0.3–2.6 tops/w
precision-scalable processor for real-time large-scale
convnets,” in VLSI-Circuits, 2016.
[47] Y. Umuroglu, N. J. Fraser, G. Gambardella, M. Blott,
P. Leong, M. Jahre, and K. Vissers, “Finn: A frame-

[48]

[49]

[50]
[51]

[52]
[53]
[54]

[55]
[56]

work for fast, scalable binarized neural network inference,” in FPGA, 2017.
R. Andri, L. Cavigelli, D. Rossi, and L. Benini, “Yodann: An ultra-low power convolutional neural network accelerator based on binary weights,” CoRR,
vol. abs/1606.05487, 2016.
K. Ando, K. Ueyoshi, K. Orimo, H. Yonekawa, S. Sato,
H. Nakahara, M. Ikebe, T. Asai, S. TakamaedaYamazaki, T. Kuroda, et al., “Brein memory: A 13layer 4.2 k neuron/0.8 m synapse binary/ternary reconfigurable in-memory deep neural network accelerator
in 65 nm cmos,” in VLSI Circuits, 2017 Symposium on,
pp. C24–C25, IEEE, 2017.
H. Kim, J. Sim, Y. Choi, and L.-S. Kim, “A kernel
decomposition architecture for binary-weight convolutional neural networks,” in DAC, 2017.
A. Parashar, M. Rhu, A. Mukkara, A. Puglielli,
R. Venkatesan, B. Khailany, J. Emer, S. W. Keckler, and W. J. Dally, “SCNN: An Accelerator for
Compressed-sparse Convolutional Neural Networks,”
in ISCA, 2017.
M. Alwani, H. Chen, M. Ferdman, and P. Milder,
“Fused-layer cnn accelerator,” in MICRO, 2016.
Y. Shen, M. Ferdman, and P. Milder, “Escher: A cnn
accelerator with flexible buffering to minimize off-chip
transfer,” in FCCM, 2017.
M. Rastegari, V. Ordonez, J. Redmon, and A. Farhadi,
“Xnor-net: Imagenet classification using binary convolutional neural networks,” CoRR, vol. abs/1603.05279,
2016.
E. Ipek, M. Kirman, N. Kirman, and J. F. Martinez,
“Core fusion: accommodating software diversity in
chip multiprocessors,” in ISCA, 2007.
C. Kim, S. Sethumadhavan, M. Govindan, N. Ranganathan, D. Gulati, D. Burger, and S. W. Keckler, “Composable lightweight processors,” in MICRO,
2007.

