Improving Latency Tolerance of Multithreading through Decoupling.
AbstractThe increasing hardware complexity of dynamically scheduled superscalar processors may compromise the scalability of this organization to make an efficient use of future increases in transistor budget. SMT processors, designed over a superscalar core, are therefore directly concerned by this problem. This work presents and evaluates a novel processor microarchitecture which combines two paradigms: simultaneous multithreading and access/execute decoupling. Since its decoupled units issue instructions in-order, this architecture is significantly less complex, in terms of critical path delays, than a centralized out-of-order design, and it is more effective for future growth in issue-width and clock speed. We investigate how both techniques complement each other. Since decoupling features an excellent memory latency hiding efficiency, the large amount of parallelism exploited by multithreading may be used to hide the latency of functional units and keep them fully utilized. Our study shows that, by adding decoupling to a multithreaded architecture, fewer threads are needed to achieve maximum throughput. Therefore, in addition to the obvious hardware complexity reduction, it places lower demands on the memory system. Since one of the problems of multithreading is the degradation of the memory system performance, both in terms of miss latency and bandwidth requirements, this improvement becomes critical for high miss latencies, where bandwidth might become a bottleneck. Finally, although it may seem rather surprising, our study reveals that multithreading by itself exhibits little memory latency tolerance. Our results suggest that most of the latency hiding effectiveness of SMT architectures comes from the dynamic scheduling. On the other hand, decoupling is very effective at hiding memory latency. An increase in the cache miss penalty from 1 to cycles reduces the performance of a 4-context multithreaded decoupled processor by less than 2 percent. For the nondecoupled multithreaded processor, the loss of performance is about 23 percent.
Introduction
The gap between the speeds of processors and memories has kept increasing in the past decade
and it is expected to sustain the same trend in the near future. This divergence implies, in terms of
clock cycles, an increasing latency of those memory operations that cross the chip boundaries. In
addition, processors keep growing their capabilities to exploit parallelism by means of greater
issue widths and deeper pipelines, which makes even higher the negative impact of memory
latencies on the performance. To alleviate this problem, most current processors devote a high
fraction of their transistors to on-chip caches, in order to reduce the average memory access time.
Several prefetching techniques have also been developed, both hardware and software [3].
Some processors, commonly known as out-of-order processors [40, 20, 18, 8, 9], include
dynamic scheduling techniques, most of them based on Tomasulo's algorithm [34] or variations of
it, that allow them to tolerate both memory and functional unit latency, by overlapping it with
useful computations of other independent instructions. To implement it, the processor is capable of
filling issue slots with independent instructions by looking forward in the instruction stream, into a
limited instruction window. This is a general mechanism that aggressively extracts the instruction
parallelism available in the instruction window.
As memory latencies continue to grow in the future, out-of-order processors will need larger
instruction windows to find independent instructions to fill the increasing number of empty issue
slots, and this number will grow even faster with greater issue widths. The increase in the
instruction window size will have an obvious influence on the chip area, but its major negative
impact will strike at the processor clock cycle time. As reported recently [21], the networks
involved in the issue wake-up and bypass mechanisms, and also - although to a less extent - those
of the renaming stage, are in the critical path that determines the clock cycle time. In their
analysis, the authors of that study state that the delay function of these networks has a component
that increases quadratically with the window length. And, although linearly, it also depends
strongly on the issue width. Moreover, higher density technologies only accelerate the increase in
these latencies. Their analysis suggest that out-of-order architectures could find in the future a
serious boundary on their clock speeds. Different kinds of architectures have been proposed
recently, either in-order or out-of-order, which address the clock cycle problem by partitioning
critical components of the architecture and/or providing less complex scheduling mechanisms [30,
6, 16, 21, 41]. They follow different partitioning strategies. One of them is the access/execute
paradigm, which was first proposed for early scalar architectures to provide them with dual issue
and a limited form of dynamic scheduling that is especially oriented to tolerate memory latency.
We believe that decoupled access/execute architectures can regain progressively interest as far as
issue widths and memory latencies keep growing and demanding larger instruction windows,
because these trends will make it worth trading issue complexity for clock speed.
Typically, a decoupled access/execute architecture [26, 27, 7, 39, 38, 23, 2, 12] splits, either
statically or dynamically, the instruction stream into two. The access stream is composed of all
those instructions involved in the fetch of data from memory, and it runs asynchronously with
respect to the execute stream, which is formed by the instructions that process these data. Both
streams are executed on independent processing units (called AP and EP respectively, in this
paper). The AP is expected to execute in advance of the EP and to prefetch data from memory into
the appropriate buffering structures, so that the EP can consume them without any delay. This
anticipation or slippage, may involve multiple conditional branches. However, the amount of
slippage between the AP and the EP highly depends on the program ILP, because data and control
dependences can force both units to synchronize - the so called Loss of Decoupling events [2, 35]
- producing a serious performance degradation.
The decoupling model presented in this paper performs dynamic code partitioning, as in [27,
12], by following a simple scheme which is based on the instruction data types, i.e. integer or fp.
Although this rather simplistic scheme mostly benefits to numerical programs, it still provides a
basis for our study which is mainly focused on the latency hiding potential of decoupling and its
synergy with multithreading. Recent studies [22, 24] have proposed other alternative compiler-assisted
partitioning schemes that address the partitioning of integer codes. Since one of the main
arguments for the decoupled approach is the reduced issue logic complexity, it has been chosen to
issue instructions in-order within each processing unit. Such a decoupled architecture adapts to
higher memory latencies by scaling much simpler structures than an out-of-order, i.e. scaling at a
lower hardware cost, or conversely scaling at a higher degree with similar cost. It may be argued
that in-order processors have a limited potential to exploit ILP. However, current compiling
techniques can extract much ILP and thus, the compiler can pass this information to the hardware
instead of using run-time schemes. This is the approach that emerging EPIC (Explicitly Parallel
Instruction Computing) architectures take [10].
We propose a new decoupled architecture which provides both the AP and the EP with a
powerful dynamic scheduling mechanism: simultaneous multithreading [37, 36]. Each processing
unit has several contexts, each issuing instructions in the above mentioned decoupled mode,
which are active simultaneously and compete for the issue slots, so that instructions from different
contexts can be issued in the same cycle. We show in this study that the combination of
decoupling and mulithreading takes advantage of their best features: while decoupling is a simple
but effective technique for hiding high memory latencies with a reduced issue complexity,
multithreading provides enough parallelism to hide functional unit latencies and keep functional
units busy. In addition, multithreading also helps to hide memory latency when a program
decouples badly. However, as far as decoupling succeeds in hiding memory latency, few threads
are needed to keep the functional units busy and achieve a near-peak issue rate. This is an
important result, since having few threads reduces the memory pressure, which has been reported
to be the major bottleneck in multithreading architectures, and reduces the hardware cost and
complexity.
The rest of this paper is organized as follows. Section 2 describes the base decoupled
architecture. It is then analyzed in Section 3, providing justification for multithreading. Section 4
describes and evaluates the proposed multithreaded decoupled architecture. Finally, we
summarize our conclusions in Section 5.
2. The basic decoupled architecture model
The baseline decoupled architecture considered in this paper (Figure 1) consists of two superscalar
decoupled processing units: the Address Processing unit (AP) and the Execute Processing unit
(EP). The decoupled processor executes a single instruction stream, based on the DEC-alpha ISA
[5], by splitting it dynamically and dispatching the instructions to either the AP or the EP. There
are two separate physical register files, one in the AP with 64 integer registers, the other in the EP
with 96 FP registers. Both units share a common fetch and dispatch stage, while they have
separate issue, execute and write-back stage pipelines. Next, there is a brief description of each
stage:
Memory Subsystem
Store
Addres
s

Figure

1: Scheme of the base decoupled processor
Fetch Decode & Rename
Instruction
Reg.
File
Reg.
File
Map

Table

Register
The fetch stage reads up to 4 consecutive instructions per cycle (but less than 4 if there is a
taken branch among them) from an infinite I-cache. Notice that I-cache miss ratios for SPEC FP95
are usually very low, so this approximation introduces a small perturbation. It is also provided
with a conditional branch prediction scheme based on a 2K entry Branch History Table, with a 2-
bit saturating counter per entry [25].
The dispatch stage decodes and renames up to 4 instructions per cycle and sends them to either
the AP or to the instruction queue IQ (48 entries) of the EP, depending on whether they are integer
or floating point instructions. All memory instructions are dispatched to the AP. The IQ allows the
AP to execute ahead of the EP, providing the necessary slippage between them to hide the memory
latency. Exceptions are kept precise by means of a reorder buffer, a graduation mechanism, and
the register renaming map table [13, 28]. Other decoupled architectures [27] had chosen to steer
memory instructions to both units to allow copying data from the load queue to registers. Since
preliminary studies showed that such code expansion would significantly reduce the performance,
we implemented dynamic register renaming, which avoids any duplication. That is, data fetched
from memory is written into a physical register rather than into a data queue, eliminating the need
for copying. It is also a convenient way to manage the disordered completion of loads when a
lockup-free cache is present. Duplication of conditional branch instructions, also used in [27], may
be avoided by incorporating similar speculation and recovery mechanisms as it uses the MIPS
R10000 to identify the instructions to squash in case of a misprediction.
Both the AP and the EP are provided with 2 general purpose, fully pipelined functional units
whose latencies are 1 cycle (AP) and 4 cycles (EP), respectively. Each processing unit can read
and issue up to 2 instructions per cycle. To better exploit the parallelism between the AP and the
EP, the instructions can issue and execute speculatively beyond up to four unresolved branches (as
the MIPS R10000 [40] or the PowerPC 620 [20]). This feature may become sometimes a key
factor to enable the AP to slip ahead of the EP. Store addresses are held in the SAQ queue (32
entries) until the stores graduate. Loads are issued to the cache after being disambiguated against
all the addresses held in the SAQ. Whenever a dependence is encountered, the data from the
pending store is immediately bypassed to the register if it is available. Otherwise, the load is put
aside until this data is forwarded to it.
The primary data cache is on-chip, 2-ported [31], direct-mapped, 64 KB sized, with a
block length, and it implements a write-back policy to minimize off-chip bus traffic. It is a
lockup-free cache [17], modelled similarly to the MAF of the Alpha 21164 [5]. It can hold up to
outstanding (primary) misses to different lines, each capable to merge up to 4 (secondary)
per pending line. We assume that L1 cache misses always hit in an infinite multibanked
off-chip L2 cache, and they have a 16 cycle latency plus any penalty due to bus contention. The
L1-L2 interface consists of a fast 128-bit wide data bus, capable to deliver 16 bytes per cycle, like
that of the R10000 (the bus is busy during 2 cycles for each line that is fetched or copied back).
3. Quantitative evaluation of a decoupled processor
In this section it is first characterized the major sources of wasted cycles in a typical single-threaded
decoupled processor. Next, the latency hiding effectiveness of this architecture is
evaluated, identifying the main factors that influence the latency tolerance of the architecture.
Other studies on decoupled machines have been carried out before [1, 26, 7, 29, 27, 39, 38, 19,
14], but they did not incorporate techniques like store-load forwarding, control speculation or
lockup-free caches. This section also provides the motivation for the multithreaded decoupled
architecture that is analyzed in Section 4.
3.1. Experimental framework
The experiments were carried out with a trace driven simulator. The binary code was obtained by
compiling the SPEC FP95 benchmark suite [33], for a DEC AlphaStation 600 5/266, with the
DEC compiler applying full optimizations. The trace was generated by running this code
previously instrumented with the ATOM tool [32]. The simulator modelled, cycle-by-cycle, the
architecture described in the previous section, and run the SPEC FP95 benchmarks, fed with their
largest available input data sets. Since it is very slow, due to the detail of the simulations, we run
only a portion of 100M instructions of each benchmark., after skipping an initial start-up phase. To
determine the appropriate initial discarded offset we compared the instruction-type frequencies of
such a fragment starting at different points, with the full run frequencies. We found that this phase
has not the same length for all the benchmarks: about 5000 M instructions for 101.tomcatv and
1000 M for 104.hydro2d and 146.wave5; and just 100 M for the rest of the
benchmarks.
3.2. Sources of wasted cycles

Figure

2 shows the throughput of the issue stage in terms of the percentage of committed
instructions over the total issue slot count (i.e. percent of issue slots where it is really doing useful
work) for the AP and the EP. The wasted throughput is also characterized, by identifying the cause
for each empty issue slot. Four different configurations have been evaluated, which differ in
whether lockup-free cache is included and whether the store-load forwarding mechanism is
enabled. To stress the memory system, in this section we assume an 8 KB L1 data cache.
As shown in Figure 2, when a lockup-free cache is not present (first and second bars), the AP
is stalled by load misses and the EP is starved for most of the time. Miss latency increases the AP
cycle count far above the EP cycle count. The AP execution time becomes the bounding limit of
the global performance, and decoupling can hardly hide memory latencies. The nature of these

Figure

2: Issue slot breakdown for several decoupled architectures, that show the effects of a lockup-free
cache and a store-load forwarding mechanism (8 KB L1 cache size).
none forwd l-free l-free
+forwd
none forwd l-free l-free
+forwd
Configuration1030507090%of
Issue
slots
wrong-path
instr. or idle
wait operand
from FU
wait operand
from memory
blocking miss
st/ld hazard
other
useful work
stalls is a structural hazard. When a lockup-free cache is used, this kind of stalls are almost
eliminated (third and fourth bars). Of course, this uncovers other overlapped causes, but the
overall improvement in performance achieves an impressive 2.3 speed-up (from 0.98 to 2.32 IPC).
A memory data hazard can occur between a store and a FP load, and it is detected during
memory disambiguation. When store-load forwarding is not enabled (first and third bars), a
memory hazard produces a stall on the AP until the store is issued to the cache. In addition, it
causes a slippage reduction between the two units - we call this event a loss of decoupling, or LOD
[2, 35] - that may expose the EP to be penalized by the memory latency in case of a subsequent
load miss. The amount of slippage reduction between the AP and the EP caused by a memory
hazard depends on how close the load is scheduled after the matching store. The results depicted
in

Figure

2 show that the AP stalls (labelled st/ld hazards) are almost completely removed when
the store-load forwarding is enabled. However, the average improvement on the EP performance
is almost negligible (overall IPC increases just by 1.8%). This latter fact suggests that either the
stores are scheduled enough in advance of the matching loads, or there is little probability to get a
subsequent miss.
Finally, for a full featured configuration (fourth bar in the graph), it can be observed that the
major source of wasted slots in the EP are true data dependences between register operands
(labelled wait operand from FU), and that these stalls are less than those caused by misses
(labelled wait operand from memory). Notice that although there are many more loads to the EP
registers than loads to the AP registers, the stalls caused by misses are similar on both processor
units because each integer load miss produces a higher penalty, as this will be more clearly
illustrated in the next section.
3.3. Latency hiding effectiveness
The interest of a decoupled architecture is closely related to its ability to hide high memory
latencies without resorting to other more complex issue mechanisms. The latency hiding potential
of a decoupled processor depends strongly on the decoupling behaviour of the programs being
tested. For some programs, the scheduling ability of the compiler to remove LOD events, which
force the AP and the EP to synchronize, is also a key factor. However, the compiler we have used
(Digital f77) is not especially tailored to a decoupled processor. Therefore, since the latency
hiding effectiveness of decoupling provides the basis for our proposed multithreaded decoupled
architecture, in order to validate our conclusions, we are interested in having an assessment of it in
our base architecture, without any specific compiler support. For this purpose, we have run the 10
benchmarks with the external L2 cache latency varying from 1 to 256 cycles. The simulations
assume all the architectural parameters described in Section 2 except that all the architectural51525
L2 Latency (cycles)51525
Average
Perceived
FP-Load
Miss
Latency
(cycles) tomcatv
su2cor
hydro2d
mgrid
applu
turb3d
apsi
tomcat swim su2cor hydro mgrid applu turb3d apsi fpppp wave5
Benchmark2060100
Miss
latency
stores
loads20601001401
L2 Latency (cycles)2060100140Average
Perceived
I-Load
Miss
Latency
(cycles) tomcatv
su2cor
hydro2d
mgrid
applu
turb3d
apsi
L2 Latency (cycles)
loss
su2cor
hydro2d
mgrid
applu
turb3d
apsi

Figure

3-a: Perceived miss latency of FP loads. Figure 3-b: Perceived miss latency of Integer loads.

Figure

3-c: Miss Ratios of Loads and Stores, when
L2 latency is 256 cycles.

Figure

3-d: Impact of latency on performance (loss
relative to the 1-cycle L2 latency case).
queues and physical register files are scaled up proportionally to the L2 latency. In addition to the
performance, we have also measured separately the average "perceived" latency of integer and FP
load misses. Since we are interested in the particular benefit of decoupling, independently of the
cache miss ratio, this average does not include load hits.
The perceived latency of FP load misses measures the EP stalls caused by misses, and reveals
the "decoupled behavior" of a program, i.e. the amount of slippage of the AP with respect to the
EP. As shown in Figure 3-a, except for fpppp, more than 96% of the FP load miss latency is
always hidden. The perceived latency of integer load misses measures the AP stalls caused by
misses, and it depends on the ability of the compiler to schedule integer loads ahead of other
dependent instructions. As shown in Figure 3-b, fpppp, su2cor, turb3d and wave5 are the
programs that experience the largest integer load miss stalls.
Regarding the impact of the L2 latency on performance (see Figure 3-d), although programs
like fpppp or turb3d have quite high perceived load miss latencies, they are hardly performance
degraded due to their extremely low miss ratios (depicted in Figure 3-c). The most performance
degraded programs are those with both high perceived miss latencies and significant miss ratios:
hydro2d, wave5 and su2cor.
To summarize, performance is little affected by the L2 latency when either it can be hidden
efficiently (tomcatv, swim, mgrid, applu and apsi), or when the miss ratio is low (fpppp and
turb3d), but it is seriously degraded for programs that lack both features (su2cor, wave5 and
hydro2d). The hidden miss latency of FP loads depends on the good decoupling behavior of the
programs, while that of integer loads relies exclusively on the static instruction scheduling.
4. A multithreaded decoupled architecture
As shown in the previous section, most of the stalls of a decoupled processor may be removed,
except those caused by true data dependences between register operands in the EP (Figure 2 right,
labelled wait operand from FU), because of the restricted ability of the in-order issue model to
exploit ILP. If both the AP and the EP were provided with some dynamic scheduling capability,
most of these stalls could also be removed. Simultaneous multithreading (SMT) is a dynamic
scheduling technique that increases processor throughput by exploiting thread level parallelism.
Multiple contexts simultaneously active compete for issue slots and functional units. Previous
studies of SMT focused on several dynamic instruction scheduling mechanisms [4, 11, 37, 36,
among others] other than decoupling. In this paper, we analyze its potential when implemented on
a decoupled processor. We still refer to it as simultaneous although there are obvious substantial
differences from the original SMT, because it retains the key concept of issuing from different
threads during a single cycle. Since decoupling provides excellent memory latency tolerance, and
multithreading supplies enough amounts of parallelism to remove the remaining stalls, we expect
important synergistic effects in a new microarchitecture which combines these two techniques. In
this section we present and evaluate the performance and memory latency tolerance of the
multithreaded decoupled access/execute architecture, and we analyze the mutual benefits of both
techniques, especially when the miss latency is large.
4.1. Architecture overview
Our proposal is a multithreaded decoupled architecture (Figure 4). That is, each thread executes in
a decoupled mode, sharing the functional units and the data cache with other threads. The base
Memory Subsystem
Store
Addres
s

Figure

4: Scheme of the multithreaded decoupled processor
Instruction
Reg.
Files
Reg.
Files
Map

Tables

Register Fetch Dispatch & Rename
multithreaded decoupled architecture is based on the decoupled design of the previous section
with some extensions: it can run up to 6 threads and issue up to 8 instructions per cycle (4 at the
AP and 4 at the EP) to 8 functional units. The L1 lockup-free data cache is augmented to 4 ports.
The fetch and dispatch stages - including branch prediction and register map tables - and the
register files and queues are replicated for each context. The issue logic, functional units and the
data cache are shared by all the threads.
In our model, all the threads are allowed to compete for each of the 8 issue slots each cycle,
and priorities among them are determined in pure round-robin order (similar to the full
simultaneous issue scheme reported in [37]). Each cycle, only two threads have access to the I-
cache, and each of them can fetch up to 8 consecutive instructions (up to the first taken branch).
The chosen threads are those with less instructions pending to be dispatched (similar to the RR-2.8
with I-COUNT schemes, reported in [36]).
4.2. Experimental evaluation
The multithreaded decoupled simulator is fed with t different traces, corresponding to t
independent threads. The trace of every thread is built by concatenating the first 10 million
instructions of the 10 traces used in the previous section - each thread using a different
permutation - thus totalling 100 million instructions per thread. In this way, all threads have
different traces but balanced workloads, similar miss-ratios, etc. Figure 5 shows the wasted issue
slots when varying the number of threads from 1 to 6. Since different threads may be candidates
for the same slot, and each can lose it because of a different cause, in order to characterize the loss
of performance, we have classified the wasted issue slots proportionally to the causes that prevent
individual threads from issuing.
4.3. Wasted issue slots in the multithreaded decoupled architecture
The first column in Figure 5 represents the case with a single thread, and it reveals, as expected,
that the major bottleneck is caused by the EP functional units latency (caused by the lack of
parallelism of the in-order issue policy, as discussed in Section 3). When two more contexts are
added, the multithreading mechanism reduces drastically these stalls in both units, and produces a
2.31 speed-up (from 2.68 IPC to 6.19 IPC). Since with 3 threads the AP functional units are nearly
saturated (90.7%), negligible additional speed-ups are obtained by adding more contexts (6.65
IPC is achieved with 4 threads).
Notice that although the AP almost achieves its maximum throughput, the EP functional units
do not saturate due to the load imbalance between the AP and the EP. Therefore, the effective peak
performance is reduced by 17%, from 8 to 6.65 IPC. This problem could be addressed with a
different choice of the number of functional units in each processor unit, but this is beyond the
scope of this study.
Another important remark is that when the number of threads is increased, the combined
working set is larger, and the miss ratios increase progressively, putting greater demands on the
external bus bandwidth. On average, there are more pending misses, thus increasing the effective
load miss latency, and increasing the EP stalls caused by waiting operands from memory (see
rightmost graph of Figure 5). On the other hand, the AP stalls due to integer load misses (see
operands from memory in the leftmost graph of Figure 5) are almost eliminated by multithreading
since these loads do not benefit from decoupling.
Number of threads1030507090%of
Issue
Cycles
idle
wait operand
from FU
wait operand
from memory
other
useful work
Number of threads1030507090%of
Issue
Cycles
empy i-queue
wait operand
from FU
wait operand
from memory
other
useful work

Figure

5: AP (left) and EP (right) issue slots breakdown for the multithreaded decoupled architecture.
4.4. Latency hiding effectiveness
Multithreading and decoupling are two different approaches to tolerate high memory latencies. We
have run some experiments, similar to those of Section 3.3, for a multithreaded decoupled
processor having from 1 to 4 contexts to quantify its latency tolerance. In addition, some other
experiments are also carried out to reveal the contribution of each mechanism to the latency hiding
effect. They consist of a set of identical runs on a degenerated version of our multithreaded
architecture where the instruction queues are disabled (i.e. a non-decoupled multithreaded
architecture).

Figure

6-a shows the average perceived load miss latency from the point of view of each
individual thread, for the 8 configurations mentioned above, by varying L2 latency from 1 to 256
cycles. This metric expresses the average number of times an instruction of a scheduled thread
cannot issue because its operand depends on a pending load miss. Figure 6-b shows the
corresponding relative performance loss (with respect to the 1-cycle L2 latency) of each of the 8
configurations. Notice that this metric compares the tolerance of these architectures to memory
latency, rather than their absolute performance. Several conclusions can be drawn from these
graphs.
First, we can observe in Figure 6-a that the average load miss latency perceived by an
individual thread is quite low when decoupling is enabled (less than 6 cycles, for a L2 latency of
256 cycles) but it is much higher when decoupling is disabled. Second, the load miss latency
perceived by an individual thread is slightly longer when more threads are running. Although
having more threads effectively reduces the number of stall cycles of each thread, it also increases
the miss ratio (due to the larger combined working set) and produces longer bus contention delays,
which becomes the - slightly - dominant effect.
Third, it is shown in Figure 6-b that when the L2 memory latency is increased from 1 cycle to
cycles, the decoupled multithreaded architecture experiences performance drops of less than
3.6% (less than 1.5%, with 4 threads), while the performance degradation observed in all non-
-decoupled configurations is greater than 23%. Even for a huge memory latency of 256 cycles, the
performance loss of all the decoupled configurations is lower than 39% while it is greater than
79% for the non-decoupled configurations. Fourth, multithreading provides some additional
latency tolerance improvements, especially in the non-decoupled configurations, but it is much
lower than the latency tolerance provided by decoupling.
Some other conclusions can be drawn from Figure 6-c. While multithreading raises the
performance curves, decoupling makes them flatter. In other words, while the main effect of
L2 Latency (cycles)1030507090110
Perceived
Load
Miss
L2 Latency (cycles)
loss
(relative
tocycle
latency)

Figure

6-a: Average perceived load miss latency of
individual threads.

Figure

6-b: Latency tolerance: performance loss is
relative to the 1-cycle L2 latency case

Figure

6-c: Contribution of decoupling and
multithreading to performance.13571
L2 Latency (cycles)1357IPC
4 T, decoupled
3 T, decoupled
T, decoupled
4 T, non-decoupled
3 T, non-decoupled
T, non-decoupled
non-decoupled
multithreading is to provide more throughput by exploiting thread level parallelism, the major
contribution to memory latency tolerance, which is related to the slope of the curves, comes from
decoupling, and this is precisely the specific role that decoupling plays in this hybrid architecture.
4.5. Hardware context reduction and the external bus bandwidth bottleneck
Multithreading is a powerful mechanism that highly improves the processor throughput, but it has
a cost: it needs a considerable amount of hardware resources. We have run some experiments that
illustrate how decoupling reduces the hardware context requirements. We have measured the
performance of several configurations having from 1 to 8 contexts, both with a decoupled
multithreaded architecture and a non-decoupled multithreaded architecture (see Figure 7-a). While
the decoupled configuration achieves the maximum performance with just 3 or 4 threads, the non-
decoupled configuration needs 6 threads to achieve similar IPC ratios.
One of the traditional claims of the multithreading approach is its ability to sustain a high
processor throughput even in systems with a high memory latency. Since hiding a longer latency
may require a higher number of contexts and, as it is well known, this has a strong negative impact
on the memory performance, the reduction in hardware context requirements obtained by
decoupling may become a key factor when L2 memory latency is high. To illustrate this, we have
run the previous experiment having a L2 memory latency of 64 cycles. As shown in Figure 7-b,13571 2 3 4 5 6 7 8
Number of Threads1357IPC
decoupled
non-decoupled
Number of Threads1357IPC
decoupled
non-decoupled

Figure

7-a: Decoupling reduces the
number of hardware contexts

Figure

7-b: Maximum performance without decoupling cannot
be reached due to external bus saturation.
while the decoupled architecture achieves the maximum performance with just 4 or 5 threads, the
non-decoupled architecture cannot reach similar performance with any number of threads,
because it would need so many that would they saturate the external L2 bus: the average bus
utilization is 89% with 12 threads, and 98% for threads. Moreover, notice that the decoupled
architecture requires just 3 threads to achieve about the same performance as the non-decoupled
architecture with 12 threads. Thus, decoupling significantly reduces the amount of parallelism
required to reach a certain level of performance.
The previous result suggests that the external L2 bus bandwidth is a potential bottleneck in this
kind of architectures. To further describe its impact, we have measured the performance and bus
utilization of several configurations having from 1 to 6 hardware contexts, for three different
external bus bandwidths of 8, 16 and 32 bytes/cycle. Results are shown in Figure 8-a and

Figure

8-b. For an 8 bytes/cycle bandwidth, the bus becomes saturated when more than 3 threads
are running, and performance is degraded beyond this point.
To summarize, decoupling and multithreading complement each other to hide memory latency
and increase ILP with reduced amounts of thread-level parallelism and low issue logic complexity.

Figure

8-a: IPC, for several bus bandwidths Figure 8-b: External L2 bus utilization, for several
bus bandwidths2060100
Number of Threads2060100
External
Bus
Utilization
cycles)
8 bytes/cycle
Number of Threads1357IPC
8 bytes/cycle
5. Summary and conclusions
In this paper we have analized the synergy of multithreading and access/execute decoupling. A
multithreaded decoupled architecture aims at taking advantage of the latency hiding effectiveness
of decoupling, and the potential of multithreading to exploit ILP. We have analyzed the most
important factors that determine its performance and the synergistic effect of both paradigms.
A multithreaded decoupled architecture hides efficiently the memory latency: the average load
miss latency perceived by an individual thread is less than 6 cycles in the worst case (with 4
threads and a L2 latency of 256 cycles). We have also found that, for L2 latencies lower than
cycles, their impact on the performance is quite low: less than 3.5% IPC loss, relative to the 1-
cycle latency scenario, and it is quite independent of the number of threads. However, this impact
is greater than a 23% IPC loss if decoupling is disabled. This latter fact shows that the main
contribution to the memory latency tolerance corresponds to the decoupling mechanism.
The architecture reaches maximum performance with very few threads, significantly less than
in a non-decoupled architecture. The number of simultaneously active threads supported by the
architecture has a significant impact on the hardware chip area (e.g. number of registers,
instruction queues) and complexity (e.g. the instruction fetch and issue mechanisms) and
consequently in clock cycle.
Reducing the number of threads also reduces the cache conflicts and the required memory
bandwidth, which is usually one of the potential bottlenecks of a multithreaded architecture. We
have shown how the external L2 bus bandwidth becomes a bottleneck when the miss latency is 64
cycles, if decoupling is disabled, preventing it from achieving the maximum performance with
any number of threads.
In summary, we can conclude that decoupling and multithreading techniques complement each
other to exploit instruction level parallelism and to hide memory latency. This particular
combination obtains its maximum performance with few threads, has a reduced issue logic
complexity, and it is hardly performance degraded by a wide range of L2 latencies. All of these
features make it a promising alternative for future increases in clock speed and issue width.
6.



--R

A Decoupled Access/Execute Architecture for Efficient Access of Structured Data.
The Effectiveness of Decoupling.
A performance study of software and hardware data prefetching schemes.
The Concurrent Execution of Multiple Execution Streams on Super-scalar Processors
Alpha 21164 Microprocessor Hardware Reference Manual
The Multicluster Architecture: Reducing Cycle Time Through Partitioning.
PIPE: A VLSI Decoupled Architecture.
's P6 Uses Decoupled Superscalar Design.
Digital 21264 Sets New Standard.
HP Make EPIC Disclosure.
An Elementary Processor Architecture with Simultaneous Instruction Issuing from Multiple Threads.
Designing the TFP Microprocessor.
Superscalar Microprocessor Design.
A Limitation Study into Access Decoupling.
Improving Direct-Mapped Cache Performance by the Addition of a Small Fully-Associative Cache and Prefetch Buffers
PEWs: A Decentralized Dynamic Scheduler for ILP Processing.


Memory Latency Effects in Decoupled Architectures.
The PowerPC 620

Decoupling Integer Execution in Superscalar Processors.
Structured Memory Access Architecture.
Exploiting Idle Floating-Point Resources For Integer Execution
A Study of Branch Prediction Strategies.
Decoupled Access/Execute Computer Architectures.

Implementation of Precise Interrupts in Pipelined Processors.
A Simulation Study of Decoupled Architecture Computers.
Multiscalar Processors.

ATOM: A System for Building Customized Program Analysis Tools.
Standard Performance Evaluation Corporation.
An Efficient Algorithm for Exploiting Multiple Arithmetic Units.
Compiling and Optimizing for Decoupled Architectures.
Exploiting Choice: Instruction Fetch and Issue on an Implementable Simultaneous Multithreading Processor.
Simultaneous Multithreading: Maximizing On-Chip Par- allelism
MISC: A Multiple Instruction Stream Computer.
An Evaluation of the WM Architecture.
The Mips R10000 Superscalar Microprocessor.

--TR
A simulation study of decoupled architecture computers
The ZS-1 central processor
High-bandwidth data memory systems for superscalar processors
An elementary processor architecture with simultaneous instruction issuing from multiple threads
Evaluation of the WM architecture
MISC
The effectiveness of decoupling
ATOM
Designing the TFP Microprocessor
Compiling and optimizing for decoupled architectures
Simultaneous multithreading
Multiscalar processors
Decoupling integer execution in superscalar processors
Exploiting choice
Complexity-effective superscalar processors
Trace processors
The multicluster architecture
Exploiting idle floating-point resources for integer execution
Performance modeling and code partitioning for the DS architecture
Improving direct-mapped cache performance by the addition of a small fully-associative cache and prefetch buffers
Implementation of precise interrupts in pipelined processors
Decoupled access/execute computer architectures
The MIPS R10000 Superscalar Microprocessor
Memory Latency Effects in Decoupled Architectures
A Limitation Study into Access Decoupling
The PowerPC 620 microprocessor
Lockup-free instruction fetch/prefetch cache organization
A study of branch prediction strategies
A Cost-Effective Clustered Architecture
The Latency Hiding Effectiveness of Decoupled Access/Execute Processors
simultaneous multithreading;instruction-level parallelism;hardware complexity;latency hiding;access/execute decoupling