A Class of Highly Scalable Optical Crossbar-Connected Interconnection Networks (SOCNs) for Parallel Computing Systems.
AbstractA class of highly scalable interconnect topologies called the Scalable Optical Crossbar-Connected Interconnection Networks (SOCNs) is proposed. This proposed class of networks combines the use of tunable Vertical Cavity Surface Emitting Lasers (VCSEL's), Wavelength Division Multiplexing (WDM) and a scalable, hierarchical network architecture to implement large-scale optical crossbar based networks. A free-space and optical waveguide-based crossbar interconnect utilizing tunable VCSEL arrays is proposed for interconnecting processor elements within a local cluster. A similar WDM optical crossbar using optical fibers is proposed for implementing intercluster crossbar links. The combination of the two technologies produces large-scale optical fan-out switches that could be used to implement relatively low cost, large scale, high bandwidth, low latency, fully connected crossbar clusters supporting up to hundreds of processors. An extension of the crossbar network architecture is also proposed that implements a hybrid network architecture that is much more scalable. This could be used to connect thousands of processors in a multiprocessor configuration while maintaining a low latency and high bandwidth. Such an architecture could be very suitable for constructing relatively inexpensive, highly scalable, high bandwidth, and fault-tolerant interconnects for large-scale, massively parallel computer systems. This paper presents a thorough analysis of two example topologies, including a comparison of the two topologies to other popular networks. In addition, an overview of a proposed optical implementation and power budget is presented, along with analysis of proposed media access control protocols and corresponding optical implementation.
as opposed to the L  N2 log2Nlinks required for a
standard binary hypercube. This multiplexing greatly
reduces the link complexity of the entire network, reducing
implementation costs proportionately.
4.2 Network Diameter
The diameter of a network is defined as the minimum
distance between the two most distant processors in the
network. Since each processor in an OHC2N cluster can
communicate directly with every processor in each directly
connected cluster, the diameter of a OHC2N containing
NH  n 2d processors is:

KH  d  log2 ; 9
which is dependent only on the degree of the hypercube
(the diameter and the degree of a hypercube network are
the same).
4.3 Bisection Width
The bisection width of a network is defined as the minimum
number of links in the network that must be broken to
partition the network into two equal sized halves. The
bisection width of a d-dimensional binary hypercube is
2d1; since that many links are connected between two
d  1-dimensional hypercubes to form a d-dimensional
hypercube. Since each link in an OHC2N contains
n channels, the bisection width of the OHC2N is:
which increases linearly with the number of processors.
A major benefit of such a topology is that a very large
number of processors can be connected with a relatively
small diameter and relatively fewer intercluster connec-
tions. For example, with n  processors per cluster and
fiber links per cluster, 1,024 processors can be
connected with a high degree of connectivity and a high
bandwidth. The diameter of such a network is 6, which
implies a low network latency for such a large system, and
only 192 bidirectional intercluster links are required. If a
system containing the same number of processors is
constructed using a pure binary hypercube topology, it
would require a network diameter of 10, and 5,120
interprocessor links.
WEBB AND LOURI: A CLASS OF HIGHLY SCALABLE OPTICAL CROSSBAR-CONNECTED INTERCONNECTION NETWORKS (SOCNS) FOR. 449
4.4 Average Message Distance cluster in the network  1. This would increase
The average message distance for a network is defined as the size of the network by c:
the average number of links that a message should traverse
through the network. This is a slightly better measure of
network latency than the diameter, because it aggregates and would not effect the cluster node degree of the
distances over the entire network rather than just looking at network. This is very similar to the fixed-c case for the
the maximum distance. The average message distance l can OC2N configuration, and the granularity of size scaling in
be calculated as [27]: this case is also the number of clusters c. Again, this is the
easiest method for scaling because it does not require the
addition of any network hardware, and it more fully utilizes
l  iNi; 11
the inherently high bandwidth of the WDM optical links.
The two topologies presented in this paper are by no
where Ni represents the number of processors at a distance i means the only two topologies that could be utilized to
from the reference processor, N is the total number of construct an SOCN class network. As an example, assume
processors in the network, and K is the diameter of the that an SOCN network exists that is configured in a torus
network. configuration, and the addition of some number of
Since the OHC2N is a hybrid of a binary hypercube and processors is required. The total number of processors
a crossbar network, the equation for the number of required in the final network may be the number supported
processors at a given distance in an OHC2N can be derived by an OHC2N configuration. In this case, the network
from the equation for a binary hypercube: could be reconfigured into an OHC2N configuration by
simply changing the routing of the intercluster links and
changing the routing algorithms. This reconfigurability
makes it conceivable to reconfigure an SOCN class network
and since each cluster in the OHC2N hypercube topology with a relatively arbitrary granularity of size scaling.
contains n processors, the number of processors at a
4.6 Fault Tolerance and Congestion Avoidance
distance i for an OHC2N can be calculated as:
Since the OHC2N architecture combines the edges of a

K hypercube network with the edges a crossbar network, the
tolerance and congestion avoidance schemes of both
architectures can be combined into an even more powerful
with the addition of n  1 for i  1 to account for the
congestion avoidance scheme. Hypercube routers typically
processors within the local cluster. Substituting into (11)
scan the bits of the destination address looking for a
and computing the summation gives the equation for the
difference between the bits of the destination address and
average messages distance for the OHC2N:
the routers address. When a difference is found, the
KN message is routed along that dimension. If there are
multiple bits that differ, the router may choose any of those
dimensions along which to route the message. The number
Substituting in the diameter of the OHC2N produces: of redundant links available from a source processor along
an optimum path to the destination processor is equal to the
Hamming distance between the addresses of the two
respective processors. If one of the links are down, or if
4.5 Granularity of Size Scaling of the OHC2N one of the links is congested due to other traffic being
routed through the connecting router, the message can be
For an OHC2N hypercube connected crossbar network
routed along one of the other dimensions.
containing a fixed number of processors per cluster n,we In addition, the crossbar network connections between
can increase the network size by increasing the size of the clusters greatly increases the routing choices of the routers.
second level hypercube topology. Since the granularity of The message only must be transmitted using the wave-
size scaling for an c-processor hypercube is c, it would length of the destination processor when it is transmitted
require the addition of c clusters to increase the size of the over the last link in the transmission (the link that is directly
OHC2N in the fixed-n case c2  2c1. Increasing the size of connected to the destination processor). A message can be
the OHC2N in the fixed-n case would also require adding transmitted on any channel over any other link along the
routing path. This means that each router along the path of
another intercluster link to each cluster in the network,
the message traversal not only has a choice of links based on
increasing the intercluster node degree by one. In this case,
the hypercube routing algorithm, but also a choice of n
the granularity of size scaling is: different channels along each of those links. The router may
d choose any of the n links that connect the local cluster to the
remote cluster. This feature greatly increases the fault
If we assume, instead, the fixed-c case, then we can increase tolerance of the network as well as the link load balancing
the network size by adding another processor to each and congestion avoidance properties of the network.
450 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 11, NO. 5, MAY 2000

Comparison of Size, Degree, Diameter, and Number of Links of Several Popular Networks
number of processors per cluster, number of clusters, number of processors per bus/ring/multichannel link, and
total number of processors.
As an example, if the Hamming distance between the
cluster address of the current router and the destination
processor cluster address is equal to b, the router will have a
choice b n different channels with which to choose to
route to. Even if all the links along all the routing
dimensions for the given message are down or are
congested, the message can still be routed around the
failure/congestion via other links along nonoptimal paths,
as long as the network has not been partitioned. In addition,
if nonshortest path routing algorithms are used to further
reduce network congestion, many more route choices are
made available.
5COMPARISON TO OTHER POPULAR NETWORKS
In this section, we present an analysis of the scalability of
the SOCN architecture with respect to several scalability
parameters. Bisection width is used as a measure of the
bandwidth of the network, and diameter and average
message distance are used as measures of the latency of the
network. Common measures of the cost or complexity of an
interconnection network are the node degree of the network
and the number of interconnection links. The node degree
and number of links in the network relates to the number of
parts required to construct the network. Cost, though, is
also determined by the technology, routing algorithms, and
communication protocols used to construct the network.
Traditionally, optical interconnects have been considered a
more costly alternative to electrical interconnects, but recent
advances in highly integrated, low power arrays of emitters
(e.g., VCSELs and tunable VCSELs) and detectors, inexpensive
polymer waveguides, and low cost microoptical
components can reduce the cost and increase the scalability
of high performance computer networks, and can make
higher node degrees possible and also cost effective.
Both the OC3N and OHC2N configurations are compared
with several well-known network topologies that
have been shown to be implementable in optics. These
network topologies include: a traditional Crossbar network
(CB), the Binary Hypercube (BHC) [27], the Cube Connected
Cycles (CCC) [28], the Torus [29], the Spanning Bus
Hypercube (SBH) [30], and the Spanning Multichannel
Linked Hypercube (SMLH) [25]. Each of these networks
will be compared with respect to degree, diameter, number
of links, bisection bandwidth, and average message dis-
tance. There are tradeoffs between the OC3N and OHC2N
configuration, and other configurations of a SOCN class
network might be considered for various applications, but it
will be shown that the OC3N and OHC2N provide some
distinct advantages for medium sized to very large-scale
parallel computing architectures.
Various topological characteristics of the compared
networks are shown in Tables 1 and 2. The notation
OC3Nn  16;c implies that the number of processors
per cluster n is fixed and the number of clusters c is
changed in order to vary the number of processors N. The
notation OHC2Nn  16;d implies that the number of
processors per cluster n is fixed, and the dimensionality of

Comparison Bisection Bandwidth and Average Message Distance for Several Popular Networks
WEBB AND LOURI: A CLASS OF HIGHLY SCALABLE OPTICAL CROSSBAR-CONNECTED INTERCONNECTION NETWORKS (SOCNS) FOR. 451
the hypercube d is varied. The number of processors is the
only variable for a standard crossbar, so CBN implies a
crossbar containing N processors. For the binary hypercube,
the dimensionality of the hypercube d varies with the size of
the network. The notation CCCd implies that the number
of dimensions of the Cube Connected Cycles d varies. The
notation Torusw; d  3 implies that the dimensionality d is
fixed and the size of the rings n varies with the number of
processors. The notation SBHw  3;d implies that the
size of the buses in the SBH network w remains constant
while the dimensionality d changes. The notation
SMLHw  32;d denotes that the number of multichannel
links w is kept constant and the dimensionality of the
hypercube d is varied.
5.1 Network Degree
Fig. 4 shows a comparison of the node degree of various
networks with respect to system size (number of processing
elements). It can be seen that for medium size networks
containing 128 processors or less, the two examples OC3N
networks provide a respectable cluster degree of 4 for a
OC3Nn  16;c configuration, and 8 for a OC3Nn  32;c
configuration. This implies that a fully connected crossbar
network can be constructed for a system containing 128
processors with a node degree as low as 4. A traditional
crossbar would, of course, require a node degree of 127 for
the same size system.
The node degrees of the OHC2Nn  16;d and
OHC2Nn  32;d configurations are very respectable for
much larger system sizes. For a system containing on the
order of 10; 000 processor, both the OHC2Nn  16;d and
the OHC2Nn  32;d configurations would require a node
degree of around 7-8, which is comparable to most of the
other networks, and much better than some.
5.2 Network Diameter
Fig. 5 shows a comparison of the diameter of various
networks with respect to the system size. The network
diameter is a good measure of the maximum latency of the
network because it is the length of the shortest path
between the two most distant nodes in the network. Of
course, the diameter of the OC3N network is the best
because each node is directly connected to every other
node, so the diameter of the OC3N network is identically 1.
As expected, the diameter of the various OHC2N
networks scale the same as the BHC network, with a fixed
negative bias due to the number of channels in each
crossbar. The SMLHw; d networks also scale the same as
the BHC network, with a larger fixed bias. For a 10; 000
processor configuration, the various OHC2N networks are
comparable or better than most of the comparison net-
works, although the SMLHw; d networks are better
because of their larger inherent fixed bias.
5.3 Number of Network Links
The number of links (along with the degree of the network)
is a good measure of the overall cost of implementing the
network. Ultimately, each link must translate into some sort
of wire(s), waveguide(s), optical fiber(s), or at least some set
of optical components (lenses, gratings, etc. It should be
noted that this is a comparison of the number of
interprocessor/intercluster links in the network and a link
could consist of multiple physical data paths. For example,
an electrical interface would likely consist of multiple wires.
The proposed optical implementation of a SOCN crossbar
consists of an optical fiber pair (send and receive) per
intercluster link.
Fig. 6 shows a plot of the number of network links with
respect to the number of processors in the system. The
OC3N network compares very well for small to medium
sized systems, although the number of links could become
prohibitive when the number of processors gets very large.
The OHC2N network configurations show a much better
scalability in the number of links for very large-scale
systems. For the case of around 10; 000 processors, the
OHC2Nn  32;d network shows greater than an order of
magnitude less links than any other network architecture.
5.4 Bisection Width
The bisection width of a network is a good measure of the
overall bandwidth of the network. The bisection width of a
network should scale close to linearly with the number of
processors for a scalable network. If the bisection width
does not scale well, the interconnection network will
become a bottleneck as the number of processors is
increased.
Fig. 7 shows a plot of the bisection width of various
network architectures with respect to the number of
processors in the system. Of course, the OC3N clearly
provides the best bisection width because the number of
interprocessor links in an OC3N increases as a factor of
ON2with respect to the number of processors. the
OHC2N configurations are very comparable to the best of
the remaining networks, and are much better than some of
the less scalable networks.
5.5 Average Message Distance
The average message distance within a network is a good
measure of the overall network latency. The average
message distance can be a better measure of network
latency than the diameter of the network because the
average message distance is aggregated over the entire
network and provides an average latency rather than the
maximum latency.
Fig. 8 shows a plot of the average message distance with
respect to the number of processors in the system. Of
course, the OC3N provides the best possible average
message distance of 1 because each processor is connected
to every other processor. The OHC2N network configurations
displays a good average message distance for medium
to very large-scale configurations, which is not as good as
the average message distance of the SMLH networks, but is
much better than the remaining networks.
6OPTICAL IMPLEMENTATION OF THE SOCN
Tunable VCSELs provide a basis for designing compact all-optical
crossbars for high speed multiprocessor intercon-
nects. An overview of a compact all-optical crossbar can be
seen in Fig. 9. A single tunable VCSEL and a single fixed-
frequency optical receiver are integrated onto each processor
in the network. This tight coupling between the optical
452 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 11, NO. 5, MAY 2000
Fig. 4. Comparison of network degree with respect to system size for various networks.
Fig. 5. Comparison of network diameter with respect to system size for various networks.
Fig. 6. Comparison of the total number of network interconnection links with respect to system size for various networks.
WEBB AND LOURI: A CLASS OF HIGHLY SCALABLE OPTICAL CROSSBAR-CONNECTED INTERCONNECTION NETWORKS (SOCNS) FOR. 453
Fig. 7. Comparison of the bisection width with respect to system size for various networks.
Fig. 8. Comparison of the average message distance within the network with respect to system size for various networks.
transceivers and the processor electronics provides an all-optical
path directly from processor to processor, taking full
advantage of the bandwidth and latency advantages of
optics in the network.
The optical signal from each processor is directly
coupled into polymer waveguides that route the signal
around the PC board to a waveguide based optical
combiner network. Polymer waveguides are used for this
design because they provide a potentially low cost, all-optical
signal path that can be constructed using relatively
standard manufacturing techniques. It has been shown that
polymer waveguides can be constructed with relatively
small losses and greater than 30dB crosstalk isolation with
waveguide dimensions on the order of 50m 50m and
with a 60m pitch [31], implying that a relatively large-scale
crossbar and optical combiner network could be constructed
within an area of just a few square centimeters.
The combined optical signal from the optical combiner is
routed to a free-space optical demultiplexer/crossbar.
Within the optical demultiplexer, passive free-space optics
is utilized to direct the beam to the appropriate destination
waveguide. As can be seen in the inset in Fig. 9, the beam
emitted from the input optical waveguide shines on a
concave, reflective diffraction grating that diffracts the
beam through a diffraction angle that is dependent on the
wavelength of the beam, and focuses the beam on the
appropriate destination waveguide. The diffraction angle
varies with the wavelength of the beam, so the wavelength
of the beam will define which destination waveguide, and
hence, which processor receives the transmitted signal.
Each processor is assigned a particular wavelength that it
will receive based on the location of its waveguide in the
output waveguide array. For example, for processor 1 to
transmit to processor 3, processor 1 would simply transmit
on the wavelength assigned to processor 3 (e.g., 3). If each
processor is transmitting on a different wavelength, each
signal will be routed simultaneously to the appropriate
destination processor. Ensuring that no two processors are
transmitting on the same wavelength is a function of the
454 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 11, NO. 5, MAY 2000
Fig. 9. A proposed compact optical crossbar consisting of polymer waveguides directly coupled to processor mounted VCSELs, a polymer
waveguide based optical combiner, and a compact free-space optical crossbar/demultiplexer. The proposed optical crossbar can be connected to
remote processors using a single optical fiber or connected locally by eliminating the optical fiber.
media access control (MAC) protocol (detailed in a later
section).
After routing through the free-space optical demultiplex-
er, the separate optical signals are routed to the appropriate
destination processor via additional integrated optical
waveguides. As can be seen in Fig. 9, the combined optical
signal between the optical combiner network and the
demultiplexer can be coupled into a single optical fiber to
route to a remote PC board to implement an intercluster
optical crossbar, or a short length of polymer waveguide
could replace the optical fiber to implement a local
(intracluster) optical crossbar.
A power budget and signal-to-noise ratio (SNR) analysis
have been conducted for the intracluster and intercluster
optical crossbars [32], [33], [34]. The result of the power
budget analysis is shown in Table 3. Assuming a necessary
receiver power of 30dBm, a VCSEL power of 2dBm [35]
and a required bit-error rate (BER) of 1015,itwas
determined that with current research level technology,
processors could be supported by such a network with
processors very nearly possible.
Details of the optical implementation of the SOCN
crossbar interconnect and a thorough analysis of the optical
implementation can be found in references [32], [33], [34],
and [36].
7MEDIA ACCESS IN THE SOCN
An SOCN network contains a local intracluster WDM
subnetwork and multiple intercluster WDM subnetworks at
each processing node. Each of these intracluster and
intercluster subnetworks has its own medium that is shared
by all processors connected to the subnetwork. Each of
these subnetworks are optically isolated, so the media
access can be handled independently for each subnetwork.
One advantage of an SOCN network is that each subnetwork
connects processors in the same cluster to processors
in a single remote cluster. The optical media are shared only
among processors in the same cluster. This implies that
media access control interaction is only required between
processors on the same cluster. Processors on different
clusters can transmit to the same remote processor at the
same time, but they will be transmitting on different media.
This could cause conflicts and contention at the receiving
processor, but these conflicts are an issue of flow control,
which is not in the scope of this paper.
7.1 SOCN MAC Overview
In a SOCN network, the processors cannot directly sense
the state of all communication channels that they have
access to, so there must be some other method for
processors to coordinate access to the shared media. One
method of accomplishing this is to have a secondary
broadcast control/reservation channel. This is particularly
advantageousinaSOCNclasnetworkbecausethe
coordination need only happen among processors local to
the same cluster. This implies that the control channel can
be local to the cluster, saving the cost of running more
intercluster cabling, and ensuring that it can be constructed
with the least latency possible. For control-channel based
networks, the latency of the control channel is particularly
critical because a channel must be reserved on the control
channel before a message is transmitted on the data
network, so the latency of the control channel adds directly
to the data transfer latency when determining the overall
network latency.
Since there are multiple physical channels at each cluster
(the local intracluster network and the various intercluster
network connections), it is conceivable that each physical
data channel could require a dedicated control channel.
WEBB AND LOURI: A CLASS OF HIGHLY SCALABLE OPTICAL CROSSBAR-CONNECTED INTERCONNECTION NETWORKS (SOCNS) FOR. 455

(in dB) for Each Component of the Optical Crossbar
Fortunately, each physical data channel on a given cluster is
shared by the same set of processors, so it is possible to
control access to all data channels on a cluster using a single
control channel at each cluster. Each WDM channel on each
physical channel is treated as a shared channel, and MAC
arbitration is controlled globally over the same control
channel.
7.2 A Carrier Sense Multiple Access with Collision
Detection (CSMA/CD) MAC Protocol
If we assume that a control channel is required, one possible
implementation of a MAC protocol would be to allow
processors to broadcast channel allocation requests on the
control channel prior to transmitting on the data channel. In
this case, some protocol would need to be devised to resolve
conflicts on the control channel. One candidate might be the
Carrier Sense Multiple Access/Collision Detection (CSMA/
CD) protocol.
Running CSMA/CD over the control channel to request
access to the shared data channels is similar to standard
CSMA/CD protocols, such as that used in Ethernet net-
works, except that Ethernet is a broadcast network, where
each node can see everything that is transmit, so the
CSMA/CD used within ethernet is run over the data
network and a separate control channel is not required.
There are some advantages to using CSMA/CD as a media
access control protocol. The primary advantage is that the
minimum latency for accessing the control channel is zero.
The primary disadvantage to using such a protocol for a
SOCN based system is that it requires that state information
be maintained at each node in the network. Each processing
node must monitor the control channel and track which
channels have been requested. When a channel is re-
quested, each processor must remember the request so that
it will know if the channel is busy when it wishes to
transmit. There is also a question about when a data
channel becomes available after being requested. A node
could be required to relinquish the data channel when it is
finished with it by transmitting a data channel available
message on the control channel, but this would double the
utilization of the control channel, increasing the chances of
conflicts and increasing latency. The requirement that a
large amount of state information be maintained at each
node also increases that chances that a node could get out-
of-sync, creating conflicts and errors in the data network.
7.2.1 A THORN-Based Media Access Control Protocol
Another very promising control channel based media
access control protocol was proposed for the HORN
network [24]. This protocol, referred to as the Token
Hierarchical Optical Ring Network protocol (THORN) is a
token based protocol based on the Decoupled Multichannel
Optical Network (DMON) protocol [37]. In the THORN
protocol, tokens are passed on the control channel in a
virtual token ring. As can be seen in Fig. 10, THORN tokens
contain a bit field containing the active/inactive state of
each of the data channels. There is also a bit field in the
token that is used to request access to a channel that is
currently busy. In addition, there is an optional payload
field that can be used to transmit small, high priority data
packets directly over the control channel. All state information
is maintained in the token, so local state information is
not be required at the processing nodes in the network,
although processors may store the previous token state in
the eventuality that a token might be lost by a processor
going down or other network error. In this eventuality, the
previous token state could be used to regenerate the token.
This still requires that processors maintain a small amount
of state information, but this state information would be
constantly refreshed and would seldom be used, so the
chances of the state becoming out-of-sync is minimal.
As can be seen in Fig. 11, there is a single control channel
for any number of data channels, and tokens are continuously
passed on the control channel that hold the entire
state of the data channels. If a processing node wishes to
transmit on a particular data channel, it must wait for the
token to be received over the control channel. It then checks
Fig. 10. The layout of a THORN-based token request packet. Each token packet contains one bit per channel for busy status and one bit per channel
for the channel requests. The token packet also contains an optional payload for small, low latency messages.
456 IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS, VOL. 11, NO. 5, MAY 2000
Fig. 11. A timing diagram for control tokens and data transfers in a SOCN architecture using a form of the THORN protocol. A node may transmit on
a data channel as soon as it acquires the appropriate token bit. Setting the request bit forces the relinquishing of the data channel.
the the busy bit of the requested data channel to see if it is
set. If the busy bit is not set, then the data channel is not
currently active, and the processing node can immediately
begin transmitting on the data channel. It must also
broadcast the token, setting the busy bit in the data channel
that it is transmitting on. If the busy bit is already set, it
implies that some other transmitter is currently using the
requested data channel. If this is the case, then the
processing node must set the request bit of the desired
data channel, which indicates to the processing node that is
currently transmitting on the desired channel that another
transmitter is requesting the channel.
A disadvantage of a token-ring based media access
control protocol is that the average latency for requesting
channels will likely be higher that with a CSMA/CD
protocol. If we assume a single control channel per cluster,
with a cluster containing n processors and m physical data
channels (one intracluster subnetwork and m  1 intercluster
subnetworks), the control token would contain n m
bits busy bits and n m request bits. For example, if a
system contains n  processors per cluster and m  8
WDM subnetwork links, the control token would require
128 busy bits and 128 request bits. If we assume a control
channel bandwidth of 2Gbps, and if we ignore the
possibility of a token payload, we can achieve a maximum
token rotation time (TRT) of 128ns. This is assuming that a
node starts retransmitting the token as soon at it starts
receiving the token, eliminating any token holding latency.
This would imply a minimum latency for requesting a
channel of close to zero (assuming the token is just about to
arrive at the requesting processing node) and up to a
maximum of 128ns, which would give an average control
channel imposed latency of approximately 64ns. If a lower
latency is required, a CSMA/CD protocol could be
implemented, or multiple control channels could be
constructed that would reduce the latency proportionally.
7.3 Control Channel Optical Implementation
Irrespective of the media access control protocol, a
dedicated control channel is required that is broadcast to
each processor sharing transmit access to each data channel.
Since each physical data channel is shared among only
processors within the same cluster, the control channel can
be implemented local to the cluster. This will simplify the
design and implementation of the control channel because it
will not require routing extra optical fibers between
clusters, and will not impose the optical loss penalties
associated with routing the optical signals off the local
cluster.
An implementation of a broadcast optical control
channel is depicted in Fig. 12. The optical signal from a
dedicated VCSEL on each processor is routed through a
polymer waveguide based star coupler that combines all the
signals from all the processors in the cluster and broadcasts
the combined signals back to each processor, creating
essentially an optical bus. The primary limitation of a
broadcast based optical network is the optical splitting
losses encountered in the star coupler. Using a similar
system as a basis for a power budget estimation [38] yields
an estimated optical loss in the control network of
approximately 8dB  3dB log2n (Table 4), which
would support approximately 128 processor per cluster
on the control channel if we assume a minimum required
receiver power of 30dBm and a VCSEL power of 2dBm.
Again, the optical implementation of the SOCN MAC
network has been throughly analyzed, but due to page
limitation the analysis could not be included in this article.
Fig. 12. An optical implementation of a dedicated optical control bus
using an integrated polymer waveguide-based optical star coupler.
WEBB AND LOURI: A CLASS OF HIGHLY SCALABLE OPTICAL CROSSBAR-CONNECTED INTERCONNECTION NETWORKS (SOCNS) FOR. 457

(in dB) for Each Component of the Optical Crossbar
This paper presents the design of a proposed optical
[10]
network that utilizes dense wavelength division multi-
plexing for both intracluster and intercluster communication
links. This novel architecture fully utilizes the benefits
of wavelength division multiplexing to produce a highly
scalable, high bandwidth network with a low overall
latency that could be very cost effective to produce. A
[13]
design for the intracluster links, utilizing a simple grating
multiplexer/demultiplexer to implement a local free space [14]
crossbar switch was presented. A very cost effective
implementation of the intercluster fiber optic links was [15]
also presented that utilizes wavelength division multiplexing
to greatly reduce the number of fibers required for
interconnecting the clusters, with wavelength reuse being [16]
utilized over multiple fibers to provide a very high degree
of scalability. The fiber-based intercluster interconnects
[17]
presented could be configured to produce a fully connected
crossbar network consisting of tens to hundreds of [18]
processors. They could also be configured to produce a
hybrid network of interconnected crossbars that could be [19]
scalable to thousands of processors. Such a network
architecture could provide the high bandwidth, low latency
communications required to produce large distributed
shared memory parallel processing systems.


--R

Interconnection Networks and Engineering Approach

High Performance Computing: Challenges for Future

The Stanford Dash




Homogeneous Hierarchical Interconnection Structures

Cray Research Inc.

Hierarchical Multiprocessor Interconnection Networks with Area



Interconnection Network of Hypercubes
Parallel and Distributed Systems



for Multicomputer Systems

Scalable Photonic Architectures for High Performance Processor


Newsletters of the Computer Architecture Technical Committee
Computer Architecture: A
Quantitative Approach.








Optical Information Processing.
Optical Computer Architectures: The Application of
Optical Concepts to Next Generation Computers.
An Introduction to Photonic Switching Fabrics.


and Multicomputers

A Gradually Scalable Optical Interconnection
Network for Massively Parallel Computing
and Distributed Systems

and Choices





Versatile Network for Parallel Computation



IEEE TRANSACTIONS ON PARALLEL AND DISTRIBUTED SYSTEMS









projects involved with parallel and distributed processing

He is a current member of the IEEE.
He has published numerous journal and
conference articles on the above topics.
Article of

he was the recipient of the Advanced Telecommunications Organization
of Japan Fellowship

Scientifique (CNRS)
of the Japanese Society for the Promotion of Science Fellowship.
with the Computer Research Institute at the University of Southern


served as a member of the Technical Program Committee of several

OSA/IEEE Conference on Massively Parallel Processors using Optical

edu/department/ocppl.
--TR

--CTR
Lachlan L. H. Andrew, Fast simulation of wavelength continuous WDM networks, IEEE/ACM Transactions on Networking (TON), v.12 n.4, p.759-765, August 2004
Roger Chamberlain , Mark Franklin , Praveen Krishnamurthy , Abhijit Mahajan, VLSI Photonic Ring Multicomputer Interconnect: Architecture and Signal Processing Performance, Journal of VLSI Signal Processing Systems, v.40 n.1, p.57-72, May       2005
David Er-el , Dror G. Feitelson, Communication Models for a Free-Space Optical Cross-Connect Switch, The Journal of Supercomputing, v.27 n.1, p.19-48, January 2004
Ahmed Louri , Avinash Karanth Kodi, An Optical Interconnection Network and a Modified Snooping Protocol for the Design of Large-Scale Symmetric Multiprocessors (SMPs), IEEE Transactions on Parallel and Distributed Systems, v.15 n.12, p.1093-1104, December 2004
Peter K. K. Loh , W. J. Hsu, Fault-tolerant routing for complete Josephus cubes, Parallel Computing, v.30 n.9-10, p.1151-1167, September/October 2004
Nevin Kirman , Meyrem Kirman , Rajeev K. Dokania , Jose F. Martinez , Alyssa B. Apsel , Matthew A. Watkins , David H. Albonesi, Leveraging Optical Technology in Future Bus-based Chip Multiprocessors, Proceedings of the 39th Annual IEEE/ACM International Symposium on Microarchitecture, p.492-503, December 09-13, 2006
multiprocessor interconnection;parallel architectures;scalability;wavelength division multiplexing;crossbars;optical interconnections;networks;hypercubes