\cutname{gen.html}

\section{Preamble} \label{preamble}

We wrote~\diy{} as part of our empirical approach to studying relaxed memory
models: developing in tandem testing tools and models of multiprocessor
behaviour.  In this tutorial, we attempt an independent tool presentation.
Readers interested by the companion formalism are invited
to refer to our CAV~2010 publication~\cite{ams10}.

The distribution includes additional test generators:
\ahrefloc{diyone:intro}{\diyone} for generating one test
and \ahrefloc{diycross:intro}{\diycross}
for generating simple variations on one test.

\subsection{Relaxation of Sequential Consistency}

Relaxation is one of the key concepts behind simple analysis of weak memory
models.  We define a candidate relaxation by reference to
the most natural model of
parallel execution in shared memory: Sequential Consistency (SC), as defined by
L.~Lamport \cite{lam79}. A parallel program running on a sequentially
consistent machine behaves as an interleaving of its sequential threads.

Consider once more the
example~\ahref{SB-labelled.litmus}{\file{SB.litmus}}:
\verbatiminput{SB-labelled.litmus} To focus on interaction through shared
memory, let us consider memory accesses, or \emph{memory events}.  A memory
event will hold a direction (write, written W, or read, written R), a memory
location (written x, y) a value and a unique label. In any run of the simple
example above, four memory events occur: two writes \lb{c}{W}{x}{1}
and~\lb{a}{W}{y}{1} and two reads \lb{b}{R}{x}{$v_1$} with a certain value $v_1$
and \lb{d}{R}{y}{$v_2$} with a certain value $v_2$.

If the program's
behaviour is modelled by the interleaving of its events, the first event must
be a write of value~\verb+1+ to location~\verb+x+ or \verb+y+ and at least one
of the loads must see a~\verb+1+. Thus, a SC machine would exhibit only three
possible outcomes for this test:

\begin{center}
\begin{tabular}{|l|}
\hline
Allowed: \mbox{0:EAX} $\mathord{=}$ \mbox{0} $\wedge$ \mbox{1:EAX} $\mathord{=}$ \mbox{1}\\\hline
Allowed: \mbox{0:EAX} $\mathord{=}$ \mbox{1} $\wedge$ \mbox{1:EAX} $\mathord{=}$ \mbox{0}\\\hline
Allowed: \mbox{0:EAX} $\mathord{=}$ \mbox{1} $\wedge$ \mbox{1:EAX} $\mathord{=}$ \mbox{1}\\\hline
\end{tabular}
\end{center}

However, running (see Sec.~\ref{x86:classic})
this test on a x86 machine yields an additional result:
\begin{center}
\begin{tabular}{|l|}
\hline
Allowed: \mbox{0:EAX} $\mathord{=}$ \mbox{0} $\wedge$ \mbox{1:EAX} $\mathord{=}$ \mbox{0}\\\hline
\end{tabular}
\end{center}

And indeed, x86 allows each write-read pair on both processors to be
reordered \cite{intel:white}: thus the write-read pair in program order is relaxed on each of these
architectures. We cannot use SC as an accurate memory model for modern
architectures.
Instead we analyse memory models as  \emph{relaxing} the
ordering constraints of the SC~memory model.

\subsection{Introduction\label{candidate:intro} to candidate relaxations}
Consider again our classical example,
from a~SC perspective.
We briefly argued that the outcome
``\mbox{0:EAX} $\mathord{=}$ \mbox{0} $\wedge$
\mbox{1:EAX} $\mathord{=}$ \mbox{0}''
is forbidden by~SC.
We now present a more complete reasoning:
\begin{itemize}
\item From the condition on outcome, we get the values in
read events: \lb{b}{R}{x}{0} and~\lb{d}{R}{y}{0}.
\item Because of these values,
\lb{b}{R}{x}{0} must precede the write \lb{c}{W}{x}{1}
in the final interleaving of~SC.
Similarly,  \lb{d}{R}{y}{0} must precede the write \lb{a}{W}{y}{1}.
This we note $(b) \fr (c)$ and $(d) \fr (a)$.

\item Because of sequential execution order on one single processor
(a.k.a. \emph{program order}),
\lb{a}Wy1 must precede \lb{b}Rx0 (first processor);
while \lb{c}Wx1 must precede~\lb{d}Ry0 (second processor).
This we note  $(a) \po (b)$ and $(c) \po (d)$.

\item We synthesise the four constraints  above as the following graph:
\begin{center}
\cycle{classic-sc}
\end{center}
Constraint arrows or \emph{global} arrows are shown in brown colour.
As the graph contains a cycle of brown arrows, the events cannot be ordered.
Hence the execution presented is not allowed by~SC.
\end{itemize}

The key idea of \diy{} resides in producing programs from similar
cycles. To that aim, the edges in cycles must convey additional
information:
\begin{itemize}
\item For $\po$ edges, we consider whether the locations
of the events on both sides of the edge are the same or not
('s' or 'd'); and the direction of these events (W or~R).
For instance the two $\po$ edges in the example are PodWR.
(program order edge between a write and a read whose locations are
different).
\item For $\fr$ edges, we consider whether the processor
of the events on both sides of the edge are the same or not
('i' for internal, or 'e' for external).
For instance the two $\fr$ edges in the example are Fre.
\end{itemize}

So far so good, but our x86 machine produced the outcome
\(\mbox{0:EAX} \mathord{=} \mbox{0} \wedge
\mbox{1:EAX} \mathord{=} \mbox{0}\).
The Intel Memory Ordering White Paper~\cite{intel:white}
specifies: ``Loads may be reordered with older stores to different locations'',
which we rephrase as: PodWR is relaxed.
Considering Fre to be safe, we have the graph:
\begin{center}
\cycle{classic}
\end{center}
And the brown sub-graph becomes acyclic.

We shall see later why we choose to relax PodWR and not Fre.
At the moment, we observe that we can assume PodWR to be relaxed
and Fre not to be (\emph{i.e.} to be \emph{safe}) and test
our assumptions, by producing and running more litmus tests.
The \diy{} suite precisely provides tools for this approach.

\label{diyone:intro}As a first example,
\afile{SB.litmus}
can be created as follows:
\begin{verbatim}
% diyone7 -arch X86 -name SB Fre PodWR Fre PodWR
\end{verbatim}

As a second example, we can produce several similar tests
as follows:
\begin{verbatim}
% diy7 -arch X86 -safe Fre -relax PodWR -name SB
Generator produced 2 tests
Relaxations tested: {PodWR}
\end{verbatim}
\diy{} produces two litmus tests, \file{SB000.litmus}
and \file{SB001.litmus}, plus one index file~\file{@all}.
One of the litmus tests generated is the same as above, while
the new test is:
\begin{verbatim}
% cat SB001.litmus
X86 SB001
"Fre PodWR Fre PodWR Fre PodWR"
Cycle=Fre PodWR Fre PodWR Fre PodWR
Relax=PodWR
Safe=Fre
{ }
 P0          | P1          | P2          ;
 MOV [z],$1  | MOV [x],$1  | MOV [y],$1  ;
 MOV EAX,[x] | MOV EAX,[y] | MOV EAX,[z] ;
exists (0:EAX=0 /\ 1:EAX=0 /\ 2:EAX=0)
% cat @all
# diy -arch X86 -safe Fre -relax PodWR -name SB
# Revision: 3333
SB000.litmus
SB001.litmus
\end{verbatim}

\diy{} first generates cycles from the candidate relaxations
given as arguments, up to a limited size, and then generates
litmus tests from these cycles.

\subsection{More candidate relaxations}

We assume the memory to be \emph{coherent}.
Coherence implies that, in a given execution,
the writes to a given location are performed by following a sequence,
or \emph{coherence order},
and that all processors see the same sequence.

\label{sec:ws}In \diy{}, the coherence orders are specified indirectly.
For instance, the candidate relaxation Wse (resp. Wsi) specifies two writes,
performed by different processors (resp. the same processor),
to the same location~$\ell$, the first write preceding the second in
the coherence order of~$\ell$.
The condition of the produced test then selects the specified coherence orders.
Consider for instance:
\begin{verbatim}
% diyone7 -arch X86 -name x86-2+2W Wse PodWW Wse PodWW
\end{verbatim}
The cycle that reveals a violation of the SC memory model is:
\begin{center}
\cycle[.2\linewidth]{ws-sc}
\end{center}
So the coherence order is $0$ (initial store, not depicted),
$1$, $2$ for both locations x and~y.
While the produced test is:
\verbatiminput{x86-2+2W.litmus}
By the coherence hypothesis, checking the final
value of locations suffices to characterise those two coherence orders,
as expressed by the final condition of~\ahref{x86-2+2W.litmus}{\ltest{x86-2+2W}}:
\begin{verbatim}
exists (x=2 /\ y=2)
\end{verbatim}
See Sec.~\ref{observers} for alternative means
to identify coherence orders.

\label{iriw}Candidate relaxations Rfe and Rfi relate
writes to reads that load their value.
We are now equipped to generate the famous iriw test
(independent reads of independent writes):
\begin{verbatim}
% diyone7 -arch X86 Rfe PodRR Fre Rfe PodRR Fre -name iriw
\end{verbatim}
We generate its internal variation (\emph{i.e.} where all Rfe are replaced by Rfi) as easily:
\begin{verbatim}
% diyone7 -arch X86 Rfi PodRR Fre Rfi PodRR Fre -name iriw-internal
\end{verbatim}
We get the cycles of \myfig{\ref{fig:iriw}},
\begin{figure}[p]
\caption{\label{fig:iriw} Cycles for iriw and iriw-internal}
\begin{center}
\begin{tabular}{m{.40\linewidth}@{\quad}@{\quad}m{.40\linewidth}}
\begin{center}\cycle[.6\linewidth]{iriw-small-caps}\end{center}
&
\begin{center}\cycle[.6\linewidth]{iriw-internal}\end{center}
\end{tabular}
\end{center}
\end{figure}
and the litmus tests of~\myfig{\ref{fig:iriw:test}}.
\begin{figure}[p]
\caption{\label{fig:iriw:test} Litmus tests iriw and iriw-internal}
\begin{center}\small
\begin{tabular}{p{.55\linewidth}@{\quad}|@{\quad}p{.40\linewidth}}
\verbatiminput{iriw.litmus} & \verbatiminput{iriw-internal.litmus} \\
\end{tabular}
\end{center}
\end{figure}

Candidate relaxations given as arguments really are a
``concise specification''. As an example,
we get iriw for Power, simply by changing \opt{-arch X86} into
\opt{-arch PPC}.
\begin{verbatim}
% diyone7 -arch PPC Rfe PodRR Fre Rfe PodRR Fre
PPC A
"Rfe PodRR Fre Rfe PodRR Fre"
{
0:r2=y; 0:r4=x;
1:r2=x;
2:r2=x; 2:r4=y;
3:r2=y;
}
 P0           | P1           | P2           | P3           ;
 lwz r1,0(r2) | li r1,1      | lwz r1,0(r2) | li r1,1      ;
 lwz r3,0(r4) | stw r1,0(r2) | lwz r3,0(r4) | stw r1,0(r2) ;
exists (0:r1=1 /\ 0:r3=0 /\ 2:r1=1 /\ 2:r3=0)
\end{verbatim}
Also notice that without the \opt{-name} option, \diyone{} writes
its result to standard output.

\subsection{Summary of simple candidate relaxations}
We summarise the candidate relaxations available on all architectures.

\subsubsection{Communication\label{communication:cr} candidate relaxations}
We call communication candidate relaxations the relations between two events
communicating through memory, though they could belong to the same processor.
Thus, these events operate on the same memory location.
\begin{center}
\begin{tabular}{c|c|c|c|p{.4\linewidth}}
\diy{} syntax & Source & Target & Processor &
\multicolumn{1}{c}{Additional property}
\\\hline
\tt Rfi      & W      & R      & Same      &
Target reads its value from source \\\hline
\tt Rfe      & W      & R      & Different &
Target reads its value from source  \\\hline
\tt Wsi      & W      & W      & Same      &
Source precedes target in coherence order
\\\hline
\tt Wse      & W      & W      & Different &
Source precedes target in coherence order \\\hline
\tt Fri     & R      & W      & Same      &
Source reads a value from a write that precedes target in coherence order
  \\\hline
\tt Fre  & R      & W      & Different &
Source reads a value from a write that precedes target in coherence order
\\\hline
\end{tabular}
\end{center}

\subsubsection{Program order candidate relaxations}

We call program order candidate relaxations each relation
between two events in the
program order. These events are on the same processor, since they are in
program order.
As regards code output,
\diy{} interprets a program order candidate relaxation by generating
two memory instructions (load or store) following one another.

Program order candidate relaxations have the following syntax:
\begin{center}
Po(s\vbar{}d)(R\vbar{}W)(R\vbar{}W)
\end{center}
where:
\begin{itemize}
\item s (resp. d) indicates that the two events are to the same (resp. different)
location(s);
\item R (resp. W) indicates an event to be a  read (resp. a write);
\end{itemize}

In practice, we have:
\begin{center}
\begin{tabular}{c|c|c|c}
\diy{} syntax  &  Source & Target  & Location \\ \hline
\tt PosRR & R & R & Same \\ \hline
\tt PodRR & R & R & Diff  \\ \hline
\tt PosRW & R & W & Same \\ \hline
\tt PodRW & R & W & Diff \\ \hline
\tt PosWW & W & W & Same \\ \hline
\tt PodWW & W & W & Diff \\ \hline
\tt PosWR & W & R & Same \\ \hline
\tt PodWR & W & R & Diff \\ \hline
\end{tabular}
\end{center}
It is to be noticed
that PosWR, PosWW and PosRW are similar to Rfi, Wsi and~Fri, respectively.
More precisely, \diy{} is unable to consider a PosWR (or PosWW, or PosRW)
candidate relaxation as not being also a Rfi (or Wsi, or Fri) candidate
relaxation.
However, litmus tests conditions may be more informative in the case of
Rfi and~Fri.

\subsubsection{Fence candidate relaxations}
Relaxed architectures provide specific instructions, namely \emph{barriers}
or \emph{fences}, to enforce order of memory accesses.
In \diy{} the presence of a fence instruction is specified with
fence candidate relaxations, similar to program order candidate relaxations,
except that a fence instruction is inserted.
Hence we have FencesRR, FencedRR. etc.
The inserted fence is the strongest fence provided by
the architecture --- that is, \texttt{mfence} for x86 and
\texttt{sync}~for~Power.

Fences can also be specified by using specific names.
For instance, we have  MFence for x86;
while on PPC we have Sync, LwSync, Eieio and~ISync.
Hence, to yield two reads to different locations
and separated by the lightweight PPC barrier~\texttt{lwsync},
we specify LwSyncdRR.

The table in figure~\ref{fence:relax:fig} lists all fence prefixes
for all supported architectures:
\begin{figure}[h]
\caption{\label{fence:relax:fig}Fence prefixes per architecture}
\begin{center}
\begin{minipage}{0.9\linewidth}
\begin{center}
\begin{tabular}{l>{\tt}p{0.7\linewidth}}
Arch & \multicolumn{1}{c}{Fence prefixes} \\ \hline
X86 & MFence \\
PPC & Sync LwSync Eieio ISync \\
ARM & DSB DMB DMB.ST DSB.ST ISB \\
AArch64\footnote{More fences available through command line option \texttt{-moreedges true}}\qquad\qquad & DMB.SY DMB.ST DMB.LD ISB \\
MIPS & Fence \\
RISCV & Fence.rw.rw Fence.rw.r Fence.rw.w  Fence.r.rw Fence.r.r Fence.r.w
Fence.w.rw Fence.w.r Fence.w.w\\
C & FenceSc FenceAR FenceAcq FenceRel FenceCons FenceRlx\\
LISA\footnote{Fence annotations~\emph{an} are defined in the bell file as
\texttt{instructions~F[{\ldots,'\emph{an},\ldots}]}.}
 & Fence\emph{An}\ldots\\
\end{tabular}
\end{center}
\end{minipage}
\end{center}
\end{figure}
Also notice that, for a given architecture~\emph{A},
recognized fence prefixes can be listed with the following command:
``\texttt{diyone7 -arch \emph{A} -show fences}''.

\subsection{Annotations}

Most architectures, and more specifically LISA, support annotations that
decorate events. The most common such annotation is ``\texttt{X}'' (or ``\texttt{A}'' depending on the architecture)
for ``atomic'' accesses that will generate atomic accesses, \emph{i.e.}
load-reserve/store-conditional pairs (lr/sc) or read-modify write instructions.

As annotations apply to events, they are  pseudo-candidates that
appear in-between actual candidate relaxations. For instance, consider
the AArch64 architecture that features primitive store release
(annotation~``\texttt{L}'')
and load acquire (annotation~``\texttt{A}'') instructions.
One may specify the well known message-passing
release-acquire idiom as follows:
\begin{verbatim}
% diyone7 -arch AArch64 PodWW L Rfe A PodRR Fre
AArch64 A
...
{
0:X1=x; 0:X3=y;
1:X1=y; 1:X3=x;
}
 P0           | P1           ;
 MOV W0,#1    | LDAR W0,[X1] ;
 STR W0,[X1]  | LDR W2,[X3]  ;
 MOV W2,#1    |              ;
 STLR W2,[X3] |              ;
exists (1:X0=1 /\ 1:X2=0)
\end{verbatim}
In the ARMv8 assembler code above, one notices the \texttt{STLR} (store release)
and \texttt{LDAR} (load acquire) instructions.

In AArch64, the ``atomic'' annotation reads~``\texttt{X}'', as ``\texttt{A}''
stands for ``acquire''. This applies to both stores and loads and generate
code snippets that implement ```atomic'' accesses using
lr/sc pairs.
\begin{verbatim}
% diyone7 -arch AArch64 PodWW XL Rfe XA PodRR Fre
AArch64 A
...
{
0:X1=x; 0:X2=y;
1:X0=y; 1:X4=x;
}
 P0               | P1              ;
 MOV W0,#1        | Loop01:         ;
 STR W0,[X1]      | LDAXR W1,[X0]   ;
 MOV W3,#1        | STXR W2,W1,[X0] ;
 Loop00:          | CBNZ W2,Loop01  ;
 LDXR W4,[X2]     | LDR W3,[X4]     ;
 STLXR W5,W3,[X2] |                 ;
 CBNZ W5,Loop00   |                 ;
exists
(y=1 /\ 0:X4=0 /\ 1:X1=1 /\ 1:X3=0)
\end{verbatim}
Observe that the ``\texttt{L}'' and ``\texttt{A}'' annotations can
be combined, yielding ``atomic release'' store and ``atomic acquire'' load
operations.

\label{unroll:example}Loop unrolling of lr/sc idioms is controlled
through the \opt{-ua <$n$>} (abbreviation of \opt{-unrollatonic <$n$>}) command line option. Of some interest is the setting \opt{-ua 0} that commands unrolling the loops once and
testing store conditional  success in the final condition:
\begin{verbatim}
% diyone7 -ua 0 -arch AArch64 PodWW XL Rfe XA PodRR Fre
AArch64 A
"PodWWPXL RfeXLXA PodRRXAP Fre"
Generator=diyone7 (version 7.49+01(dev))
Prefetch=0:x=F,0:y=W,1:y=F,1:x=T
Com=Rf Fr
Orig=PodWWPXL RfeXLXA PodRRXAP Fre
{
0:X1=x; 0:X2=y;
1:X0=y; 1:X4=x;
}
 P0               | P1              ;
 MOV W0,#1        | LDAXR W1,[X0]   ;
 STR W0,[X1]      | STXR W2,W1,[X0] ;
 MOV W3,#1        | LDR W3,[X4]     ;
 LDXR W4,[X2]     |                 ;
 STLXR W5,W3,[X2] |                 ;
exists
(y=1 /\ 0:X5=0 /\ 0:X4=0 /\ 1:X2=0 /\ 1:X1=1 /\ 1:X3=0)
\end{verbatim}

As another example of atomic accesses, the X86 architecture use
atomic exchange instructions to implement atomic store operations:
\begin{verbatim}
% diyone7 -arch X86  A PodWR Fre A PodWR Fre
X86 A
...
{
}
 P0           | P1           ;
 MOV EAX,$1   | MOV EAX,$1   ;
 XCHG [x],EAX | XCHG [y],EAX ;
 MOV EBX,[y]  | MOV EBX,[x]  ;
exists
(x=1 /\ y=1 /\ 0:EBX=0 /\ 1:EBX=0)
\end{verbatim}

The table in figure~\ref{annot:relax:fig} lists all annotations
for all supported architectures. Notice that all annotations  do not apply to all accesses. For instance, the \texttt{X86} ``\texttt{A}'' applies to stores only,
while  the \texttt{PPC} ``\texttt{R}'' (Reserve) applies to loads only,~etc.
\begin{figure}[h]
\caption{\label{annot:relax:fig}Annotation pseudo-candidates}
\begin{center}
\begin{minipage}{0.9\linewidth}
\begin{center}
\begin{tabular}{l>{\tt}p{0.7\linewidth}}
Arch & \multicolumn{1}{c}{Pseudo-candidates} \\ \hline
X86 &  A \\
PPC & R A\\
ARM & R A\\
AArch64\footnote{Annotation~``\texttt{A}'' is acquire, annotation~``\texttt{X}'' is atomic}\qquad\qquad &  A Q L X XL XA XAL \\
MIPS & R A\\
RISCV\footnote{Some annotations are experimental and are not included in ISA} & XARAR XARRl XARAq XARP XRlAR XRlRl XRlAq XRlP XAqAR XAqRl XAqAq XAqP XPAR XPRl XPAq X AR Rl Aq\\
C & Con Rlx Sc AR Rel Acq\\
LISA\footnote{Fence annotations~\emph{An} are defined in the bell file (option \opt{-bell <\emph{bell file}>}) as
\texttt{instructions~R[{\ldots,'\emph{an},\ldots}]} and
\texttt{instructions~W[{\ldots,'\emph{an},\ldots}]}.}& \emph{An}\ldots\\
\end{tabular}
\end{center}
\end{minipage}
\end{center}
\end{figure}
Also notice that, for a given architecture~\emph{A},
recognized annotation pseudo relaxations can be listed with the
following command: ``\texttt{diyone7 -arch \emph{A} -show annot}.

\section{Testing\label{diy:intro} candidate relaxations with \diy}

The tool~\diy{} can probably be used in various, creative, ways;
but the tool first stems from our technique for testing relaxed memory
models.
The \opt{-safe} and \opt{-relax} options are crucial here.
We describe our technique by the means of an example: X86-TSO.

%Notice that this style of model exploration is mechanised
%by~\dont\prog{ (diy)} --- see Part~\ref{part:auto}.

\subsection{\aname{test2}{Principle}}

Before engaging in testing it is important to categorise
candidate relaxations as safe or relaxed.

This can done by interpretation of vendor's documentation.
For instance, the iriw test of~\mysec{\ref{iriw}} is the example 7.7 of
\cite{intel:white}
``Stores Are Seen in a Consistent Order by Other Processors'',
with a Forbid specification.
Hence we deduce that Fre, Rfe and PodRR are safe.
Then, from test~iriw-internal of~\mysec{\ref{iriw}},
which is Intel's test~7.5 ``Intra-Processor Forwarding Is Allowed''
with an allow specification,
we deduce that Rfi is relaxed.
Namely, the cycle of iriw-internal is
``Fre Rfi PodRR Fre Rfi PodRR''. Therefore, the only possibility is for
Rfi to be relaxed.

Overall, we deduce:
\begin{itemize}
\item Candidate relaxations PosWR (Rfi) and PodWR are relaxed
\item The remaining candidate relaxations PosRR, PodRR, PosWW (Wsi),  PodWW,
PosRW (Fri), Fre and Wse are safe.
Fence relaxations FencedsWR and FenceddWR are also safe
and worth testing.
\end{itemize}

Of course these remain assumptions to be tested.
To do so, we perform one series of tests per relaxed candidate relaxation,
and one series of tests for confirming safe candidate relaxations
as much as possible. Let $S$~be all safe candidate relaxations.
\begin{itemize}
\item Let $r$ be a relaxed candidate relaxation.
We produce tests for confirming $r$~being relaxed by
\texttt{diy -relax}~$r$~\texttt{-safe}~$S$.
We run these tests with \litmus{}. If one of the tests yields~\texttt{Ok},
then $r$~is confirmed to be relaxed, provided the experiments on~$S$ below
do not  fail.
\item For confirming the safe set, we produce tests
by \texttt{diy -safe}~$S$.
We run these tests as much as possible and expect never to see~\texttt{Ok}.
\end{itemize}
\label{diy:one:relax}Namely, \diy{} builds cycles as follows:
\begin{itemize}
\item \texttt{diy -relax}~$r$~\texttt{-safe}~$S$ build cycles
with at least one~$r$ taking other candidate relaxations from~$S$.
\item \texttt{diy -safe}~$S$ build cycles from the candidate relaxations in~$S$.
\end{itemize}
For the purpose of confirming relaxed
candidate relaxations, $S$ can be replaced by a subset.

\subsection{Testing~x86}
Repeating command line options is painful and error prone.
Besides, configuration parameters may get lost.
Thus, we regroup those in configuration files
that simply list the options to be passed to \diy, one option per line.
For instance here is the configuration file for testing the safe relaxations
of~x86, \ahref{x86-safe.conf}{\file{x86-safe.conf}}.
\verbatiminput{x86-safe.conf}
Observe that the syntax of candidate relaxations allows one shortcut:
the wildcard \texttt{*} stands for \texttt{W} and~\texttt{R}.
Thus \texttt{PodR*} gets expanded  to the two candidate
relaxations \texttt{PodRR} and~\texttt{PodRW}.

\label{safe:test:sec}%
We get safe tests by issuing the following command, preferably in a
specific directory, say \texttt{safe}.
\begin{verbatim}
% diy7 -conf x86-safe.conf
Generator produced 38 tests
Relaxations tested: {}
\end{verbatim}

Here are the configuration files for confirming
that Rfi and PodWR are relaxed, \afile{x86-rfi.conf}
and~\afile{x86-podwr.conf}.
\begin{center}\small
\begin{tabular}{p{0.7\linewidth}@{\quad}|@{\quad}p{0.2\linewidth}}
\verbatiminput{x86-rfi.conf} &
\verbatiminput{x86-podwr.conf}
\end{tabular}
\end{center}
Notice that we used the complete safe list in
\file{x86-rfi.conf} and a reduced list in~\file{x86-podwr.conf}.
Tests are to be generated in specific directories.
%HEVEA To that aim, we provide a convenient archive~\ahref{x86.tar}{\texttt{x86.tar}}.
\begin{verbatim}
% cd rfi
% diy7 -conf x86-rfi.conf
Generator produced 11 tests
Relaxations tested: {Rfi}
% cd ../podwr
% diy7 -conf x86-podwr.conf
Generator produced 2 tests
Relaxations tested: {PodWR}
% cd ..
\end{verbatim}

Now, \ahref{litmus.html}{let us run} all tests at once, with the
parameters of machine \texttt{saumur} (4 physical cores with hyper-threading):
\begin{verbatim}
% litmus7 -mach saumur rfi/@all > rfi/saumur.rfi.00
% litmus7 -mach saumur podwr/@all > podwr/saumur.podwr.00
% litmus7 -mach saumur safe/@all > safe/saumur.safe.00
\end{verbatim}
If your machine has 2 cores only, try \verb+litmus -a 2 -limit true+\ldots

\label{readRelax:intro}We now look for
the tests that have validated their condition
in the result files of~\litmus.
A simple tool, \readRelax, does the job:
\begin{verbatim}
% readRelax7 rfi/saumur.rfi.00 podwr/saumur.podwr.00 safe/saumur.safe.00
   .
   .
   .
** Relaxation summary **
{Rfi} With {Rfe, Fre, Wse, PodRW, PodRR} {Rfe, Fre, PodRR}\
{Fre, Wse, PodWW, PodRR} {Fre, PosWW, PodRR, MFencedWR}\
{Fre, PodWW, PodRR, MFencedWR} {Fre, PodRR} {Fre, PodRR, MFencedWR}
{PodWR} With {Fre}
\end{verbatim}
The tool \readRelax{} first lists the result of all tests
(which is omitted above), and then dumps a summary of the
relaxations it found.
The sets of the candidate relaxations  that need to be safe for the tests to
indeed reveal  a relaxed candidate relaxation are also given.
Here, Rfi and~PodWR are confirmed to be relaxed, while no candidate relaxation
in the safe set is found to be relaxed.
Had it been the case, a line \verb+{} With {...}+ would have occurred
in the relaxation summary.
The safe tests need to be run a lot of times, to increase our
confidence in the safe set.

\section{Additional relaxations}
We introduce some additional candidate relaxations
that are specific to the Power architecture.
We shall not detail here our experiments on Power machines.
See our experience report~\ahrefurl{http://diy.inria.fr/phat/}
for more details.

\subsection{Intra-processor dependencies}
In a very relaxed architecture such as Power,
\emph{intra-processor dependencies} becomes significant.
Roughly, intra-processor dependencies fall into two categories:
\begin{description}
\item[Data dependencies] occur when a memory access instruction
reads a register whose contents depends upon a previous (in program order)
load. In \diy{} we specify such a dependency as:
\begin{center}Dp(s\vbar{}d)(R\vbar{}W)\end{center}
where, as usual,
s (resp. d) indicates that the source and target events
are to the same (resp. different) location(s);
and  R (resp. W) indicates that the target event is a read (resp. a write).
As a matter of fact, we do not need to specify the direction of
the source event, since it always is a read.

Finally, one may control the nature of the dependency:
address dependency (DpAddr(s\vbar{}d)(R\vbar{}W) or
data dependency (DpData(s\vbar{}d)W).

\item[Control dependencies] occur when the execution of a memory access
is conditioned by the contents of a previous load.
Their syntax is similar to the one of Dp~relaxations, with a Ctrl~tag:
\begin{center}Ctrl(s\vbar{}d)(R\vbar{}W)\end{center}
This default syntax expands to control dependencies as
guaranteed by the Power documentation.
For read to write, conditioning execution is enough
(expanded syntax, DpCtrl(s\vbar{}d)W).
But for read to read, an extra instruction, \texttt{isync}, is needed
(expanded syntax  DpCtrlIsync(s\vbar{}d)R, see \ahrefloc{ctrlisync}{below}).
The syntax DpCtrl(s\vbar{}d)R also exists,
it expresses the conditional execution of a load instruction and
does \emph{not} create ordering.

ARM has similar candidate relaxations, Isync being replaced by ISB.
\end{description}
In the produced code, \diy{} expresses
a data dependency by a \emph{false dependency} (or \emph{dummy dependency})
that operates on the address of the target memory access.
For instance:
\begin{verbatim}
% diyone7 DpdW Rfe DpdW Rfe
PPC A
"DpAddrdW Rfe DpAddrdW Rfe"
{
0:r2=y; 0:r5=x;
1:r2=x; 1:r5=y;
}
 P0            | P1            ;
 lwz r1,0(r2)  | lwz r1,0(r2)  ;
 xor r3,r1,r1  | xor r3,r1,r1  ;
 li r4,1       | li r4,1       ;
 stwx r4,r3,r5 | stwx r4,r3,r5 ;
exists (0:r1=1 /\ 1:r1=1)
\end{verbatim}
On \P{0}, the effective address of the indexed store \verb+stwx r4,r3,r5+
depends on the contents of the index register~\verb+r3+, which itself
depends on the contents of~\verb+r1+.
The dependency is a ``false'' one, since the contents of~\verb+r3+
always is zero, regardless of the contents of~\verb+r1+.
One may observe that DpdW is changed into DpAddrdW in the comment
field of the test.
As a matter of fact, DpdW is a macro for the address dependency
DpAddrW. We could have specified data dependency instead:
\begin{verbatim}
% diyone7 DpDatadW Rfe DpAddrdW Rfe
PPC A
"DpDatadW Rfe DpAddrdW Rfe"
{
0:r2=y; 0:r4=x;
1:r2=x; 1:r5=y;
}
 P0           | P1            ;
 lwz r1,0(r2) | lwz r1,0(r2)  ;
 xor r3,r1,r1 | xor r3,r1,r1  ;
 addi r3,r3,1 | li r4,1       ;
 stw r3,0(r4) | stwx r4,r3,r5 ;
exists
(0:r1=1 /\ 1:r1=1)
\end{verbatim}
On \P{0}, the value stored by the last (store) instruction
\verb+stw r3,0(r4)+ is now computed from the
value read by the first (load) instruction \verb+lwz r1,0(r2)+.
Again, this is a ``false'' dependency.

\aname{ctrlisync}{A control}
dependency is implemented by the means of an useless compare
and branch sequence, plus the \texttt{isync} instruction when the target event
is a load. For instance
\begin{verbatim}
% diyone7 CtrldR Fre SyncdWW Rfe
PPC A
"DpCtrlIsyncdR Fre SyncdWW Rfe"
{
0:r2=y; 0:r4=x;
1:r2=x; 1:r4=y;
}
 P0           | P1           ;
 lwz r1,0(r2) | li r1,1      ;
 cmpw r1,r1   | stw r1,0(r2) ;
 beq  LC00    | sync         ;
 LC00:        | li r3,1      ;
 isync        | stw r3,0(r4) ;
 lwz r3,0(r4) |              ;
exists
(0:r1=1 /\ 0:r3=0)
\end{verbatim}
Also notice that CtrldR is interpreted as DpCtrlIsyncR in the comment
field of the test.

Of course, in all cases, we assume that ``false'' dependencies are not
``optimised out'' by the assembler or the hardware.

\subsection{\aname{composite}{Composite} relaxations and cumulativity}
Users may specify a small sequence of single candidate relaxations
as behaving as a single candidate relaxation to~\diy. The~syntax~is:
\begin{center}
[$r1$, $r2$, \ldots]
\end{center}
The main usage of the feature is to specify
\emph{cumulativity candidate relaxations},
that is, the sequence of Rfe and of a fence candidate relaxation
(A-cumulativity),
the sequence of a fence candidate relaxation and of~Rfe (B-cumulativity),
or both (AB-cumulativity).

Cumulativity candidate relaxations are best expressed by the following
syntactical shortcuts:
let $r$ be a fence candidate relaxation, then
\texttt{AC}$r$ stands for~\texttt{[Rfe,}$r$\texttt{]},
\texttt{BC}$r$ stands for~\texttt{[}$r$\texttt{,Rfe]},
while \texttt{ABC}$r$ stands for~\texttt{[Rfe,}$r$\texttt{,Rfe]},

Hence, a simple way to generate iriw-like (see~\mysec{\ref{iriw}})
litmus tests  with \texttt{lwsync} is as follows:
\begin{verbatim}
% diy7 -name iriw-lwsync -nprocs 8 -size 8 -relax ACLwSyncdRR -safe Fre
Generator produced 3 tests
Relaxations tested: {ACLwSyncdRR}
\end{verbatim}
where we have for instance:
\begin{verbatim}
% cat iriw-lwsync001.litmus
PPC iriw-lwsync001
"Fre Rfe LwSyncdRR Fre Rfe LwSyncdRR Fre Rfe LwSyncdRR"
Cycle=Fre Rfe LwSyncdRR Fre Rfe LwSyncdRR Fre Rfe LwSyncdRR
Relax=ACLwSyncdRR
Safe=Fre
{
0:r2=z; 0:r4=x; 1:r2=x;
2:r2=x; 2:r4=y; 3:r2=y;
4:r2=y; 4:r4=z; 5:r2=z;
}
 P0           | P1           | P2           | P3           | P4           | P5           ;
 lwz r1,0(r2) | li r1,1      | lwz r1,0(r2) | li r1,1      | lwz r1,0(r2) | li r1,1      ;
 lwsync       | stw r1,0(r2) | lwsync       | stw r1,0(r2) | lwsync       | stw r1,0(r2) ;
 lwz r3,0(r4) |              | lwz r3,0(r4) |              | lwz r3,0(r4) |              ;
exists (0:r1=1 /\ 0:r3=0 /\ 2:r1=1 /\ 2:r3=0 /\ 4:r1=1 /\ 4:r3=0)
\end{verbatim}

\subsection{Detour \label{detour:def}candidate relaxations}
Detours combine a Pos candidate relaxation and a sequence
of two \emph{external} communication candidate relaxations.
More precisely detours are some constrained Pos candidate relaxations:
the source and target events must be related by a sequence of
two communication candidate relaxations, whose target and source
are a common event whose processor is new.
\begin{center}
\begin{tabular}{c|c|c|c}
\diy{} syntax  &  Source & Target  & Detour \\ \hline
\tt DetourR & R & R & Fre; Rfe \\ \hline
\tt DetourW & W & R & Wse; Rfe  \\ \hline
\tt DetourRW & R & W & Fre;Wse \\ \hline
\tt DetourWW & W & W & Wse;Wse \\ \hline
\end{tabular}
\end{center}
DetourRR and DetourWR are accepted as synonyms for
DetourR and DetourW respectively.

Graphically, we have:
\begin{center}
\img{DetourR}\quad
\img{DetourW}\quad
\img{DetourRW}\quad
\img{DetourWW}
\end{center}
Finally notice that ``internal'' detours need no special treatement
as they can be expressed by the sequences ``Fri; Rfi'', ``Wsi;Rfi'', etc.

\section{Test\label{diycross:intro} variations with \diycross{}}
The tool \diycross{} has an interface similar to \diyone,
except it accepts list of candidate relaxations where \diyone{} accepts
single candidate relaxations.
The new tool produces the test resulting by ``cross producing'' the lists.
For instance, one can generate all variations on the IRIW test
(see Sec.~\ref{iriw}) that
involve data dependencies and the lightweight barrier~\texttt{lwsync}
as follows:
\begin{verbatim}
% diycross7 -arch PPC -name IRIW Rfe DpdR,LwSyncdRR Fre Rfe DpdR,LwSyncdRR Fre
Generator produced 3 tests
% ls
@all  IRIW+addrs.litmus  IRIW+lwsync+addr.litmus  IRIW+lwsyncs.litmus
\end{verbatim}
\diycross{} outputs the index file \texttt{@all}
that lists the test source files, and three tests, with
names we believe to be self-explanatory:
\begin{verbatim}
% cat IRIW+lwsync+addr.litmus
PPC IRIW+lwsync+addr
"Rfe LwSyncdRR Fre Rfe DpAddrdR Fre"
Cycle=Rfe LwSyncdRR Fre Rfe DpAddrdR Fre
{
0:r2=y;
1:r2=y; 1:r4=x;
2:r2=x;
3:r2=x; 3:r5=y;
}
 P0           | P1           | P2           | P3            ;
 li r1,1      | lwz r1,0(r2) | li r1,1      | lwz r1,0(r2)  ;
 stw r1,0(r2) | lwsync       | stw r1,0(r2) | xor r3,r1,r1  ;
              | lwz r3,0(r4) |              | lwzx r4,r3,r5 ;
exists (1:r1=1 /\ 1:r3=0 /\ 3:r1=1 /\ 3:r4=0)
\end{verbatim}

Users may use the special keywords allRR, allRW, allWR and allWW to
specify the set of all existing program order candidate relaxations
between the specified ``R'' or ``W''. For instance, we get the complete
variations on IRIW by:
\begin{verbatim}
% diycross7 -arch PPC -name IRIW Rfe allRR Fre Rfe allRR Fre
Generator produced 28 tests
% ls
@all
IRIW.litmus
IRIW+addr+po.litmus
IRIW+lwsync+addr.litmus
...
IRIW+isyncs.litmus
\end{verbatim}

\section{Identifying\label{observers} coherence orders with observers}
We first produce the ``\emph{four writes}'' test~\ahref{2+2W.litmus}{\file{2+2W}}
for Power:
\begin{verbatim}
% diyone7 -name 2+2W -arch PPC PodWW Wse PodWW Wse
% cat 2+2W.litmus
PPC 2+2W
"PodWW Wse PodWW Wse"
{ 0:r2=x; 0:r4=y; 1:r2=y; 1:r4=x; }
 P0           | P1           ;
 li r1,2      | li r1,2      ;
 stw r1,0(r2) | stw r1,0(r2) ;
 li r3,1      | li r3,1      ;
 stw r3,0(r4) | stw r3,0(r4) ;
exists (x=2 /\ y=2)
\end{verbatim}
Test \ltest{2+2W} is the Power version of the x86 test
\atest{x86-2+2W} of Sec.~\ref{sec:ws}.
In that section, we argued that the final condition \verb+exists (x=2 /\ y=2)+
suffices to identify the coherence orders $0$, $1$, $2$
for locations \texttt{x} and~\texttt{y}.
As a consequence, a positive final condition reveals the occurrence
of the specified cycle: Wse PodWW Wse PodWW.

\subsection{Simple observers}
\emph{Observers} provide an alternative, perhaps more intuitive,
means to identify coherence orders: an observer simply is an additional thread
that performs several loads from the same location in sequence.
Here, loading value~$1$ and then value~$2$ from location~\texttt{x}
identifies the coherence order  $0$, $1$, $2$.
The command line switch \opt{-obs force} commands the production
of observers (test \ahref{2+2WObs.litmus}{\file{2+2WObs}}):
\begin{verbatim}
% diyone7 -name 2+2WObs -obs force -obstype straight -arch PPC PodWW Wse PodWW Wse
% cat 2+2WObs.litmus
PPC 2+2WObs
"PodWW Wse PodWW Wse"
{ 0:r2=x; 1:r2=y; 2:r2=x; 2:r4=y; 3:r2=y; 3:r4=x; }
 P0           | P1           | P2           | P3           ;
 lwz r1,0(r2) | lwz r1,0(r2) | li r1,2      | li r1,2      ;
 lwz r3,0(r2) | lwz r3,0(r2) | stw r1,0(r2) | stw r1,0(r2) ;
              |              | li r3,1      | li r3,1      ;
              |              | stw r3,0(r4) | stw r3,0(r4) ;
exists (0:r1=1 /\ 0:r3=2 /\ 1:r1=1 /\ 1:r3=2)
\end{verbatim}
Thread \texttt{P0} observes location~\texttt{x}, while
thread \texttt{P1} observes location~\texttt{y}.
With respect to \texttt{2+2W}, final condition has changed, the direct
observation of the final contents of locations \texttt{x} and~\texttt{y}
being replaced by two successive observations of the contents of
\texttt{x} and~\texttt{y}.

It should first be noticed that the reasoning above assumes
that having the same thread to read $1$ from say~\texttt{x}
and then $2$ implies that $1$ takes place before $2$
in the coherence order of~\texttt{x}.
This may not be the case in general --- although it holds for Power.
Moreover, running \file{2+2W} and~\file{2+2WObs}
yields contrasted results. While a positive conclusion is
immediate for \file{2+2W}, we were not able to reach a similar conclusion
for \file{2+2WObs}.
As a matter of fact, \file{2+2WObs} yielding Ok stems from the
still-to-be-observed coincidence
of several events: \emph{both} observers threads must run at the right pace to
observe the change from~$1$ to~$2$, while the cycle must indeed occur.

\subsection{More\label{sec:obstype} observers}

\subsubsection{Fences and loops in observers}
A simple observer consisting of loads performed in sequence is a
\emph{straight} observer. We define two additional sorts of observers:
\emph{fenced} observers, where loads are separated by the strongest fence
available, and \emph{loop} observers, which poll on location contents change.
Those are selected by the homonymous tags given as arguments to the command
line switch \opt{-obstype}. For instance, we get the test
\ahref{2+2WObsFenced.litmus}{\file{2+2WObsFenced}} by:
\begin{verbatim}
% diyone7 -name 2+2WObsFenced -obs force -obstype fenced -arch PPC PodWW Wse PodWW Wse
% cat 2+2WObsFenced.litmus
PPC 2+2WObsFenced
"PodWW Wse PodWW Wse"
{ 0:r2=x; 1:r2=y; 2:r2=x; 2:r4=y; 3:r2=y; 3:r4=x; }
 P0           | P1           | P2           | P3           ;
 lwz r1,0(r2) | lwz r1,0(r2) | li r1,2      | li r1,2      ;
 sync         | sync         | stw r1,0(r2) | stw r1,0(r2) ;
 lwz r3,0(r2) | lwz r3,0(r2) | li r3,1      | li r3,1      ;
              |              | stw r3,0(r4) | stw r3,0(r4) ;
exists (0:r1=1 /\ 0:r3=2 /\ 1:r1=1 /\ 1:r3=2)
\end{verbatim}
Invoking \diyone{} as ``\verb+diyone -obs force -obstype loop ...+''
yields the additional  test \ahref{2+2WObsLoop.litmus}{\file{2+2WObsLoop}}\ifhevea:
\verbatiminput{2+2WObsLoop.litmus}
A loop observer first busily waits for the observed location not
to hold its initial contents~$0$, and then busily waits for another change
of location contents.
The second loop is performed at most a finite number of times
(here~$200$), in order to ensure termination.
\else.
The \textsc{html}~version of this document provides details.
\fi

\subsubsection{Local observers}
With local observers, coherence order is observed by the test threads.
This implies changing the tests, and some care must be exercised when
interpreting results.

The idea is as follows: when two threads are connected by a Wse candidate
relaxation, meaning that the first thread ends by writing $v$ to some location~$\ell$ and that the second threads starts by writing $v+1$ to the same location~$\ell$, we add an observing read of location~$\ell$ at the end of
the first thread. Then, reading $v+1$ means that the write by the first thread
precedes the write by the second thread in $\ell$ coherence order.
More concretely, we instruct \diy{} generators to emit such local observers
with option \opt{-obs local}:
\begin{verbatim}
% diyone7 -name 2+2WLocal -obs local -obstype straight -arch PPC PodWW Wse PodWW Wse
% cat 2+2WLocal.litmus
PPC 2+2WLocal
"PodWW Wse PodWW Wse"
{
0:r2=x; 0:r4=y;
1:r2=y; 1:r4=x;
}
 P0           | P1           ;
 li r1,2      | li r1,2      ;
 stw r1,0(r2) | stw r1,0(r2) ;
 li r3,1      | li r3,1      ;
 stw r3,0(r4) | stw r3,0(r4) ;
 lwz r5,0(r4) | lwz r5,0(r4) ;
exists
(0:r5=2 /\ 1:r5=2)
\end{verbatim}
With respect to \texttt{2+2W}, final condition has changed, the direct
observation of the final contents of locations \texttt{y} and~\texttt{x}
being replaced local observation of \texttt{y} by thread~0 and
local observation of~\texttt{x} by thread~1.

Based for instance on the test execution witness, whose only SC-violation
cycle is the same as as  for \ltest{2+2W},
\begin{center}
\cycle{2+2WLocal+DOT}
\end{center}
one may argue that tests
\ltest{2+2W} and~\ltest{2+2WLocal} are equivalent, in the sense that
both are allowed or both are forbidden by a model or machine.

Local observers can also be fenced or looping.
For instance, one produces
\ahref{2+2WLocalFenced.litmus}{\ltest{2+2WLocalFenced}},
the fenced local observer version of \ltest{2+2W}
as follows:
\begin{verbatim}
% diyone7 -name 2+2WLocalFenced -obs local -obstype fenced -arch PPC PodWW Wse PodWW Wse
% cat 2+2WLocalFenced.litmus
PPC 2+2WLocalFenced
"PodWW Wse PodWW Wse"
{
0:r2=x; 0:r4=y;
1:r2=y; 1:r4=x;
}
 P0           | P1           ;
 li r1,2      | li r1,2      ;
 stw r1,0(r2) | stw r1,0(r2) ;
 li r3,1      | li r3,1      ;
 stw r3,0(r4) | stw r3,0(r4) ;
 sync         | sync         ;
 lwz r5,0(r4) | lwz r5,0(r4) ;
exists
(0:r5=2 /\ 1:r5=2)
\end{verbatim}
While one produces
\ahref{2+2WLocalLoop.litmus}{\ltest{2+2WLocalLoop}}, the looping local observer version of~\ltest{2+2W} as follows:
\begin{verbatim}
% diyone7 -name 2+2WLocalLoop -obs local -obstype loop -arch PPC PodWW Wse PodWW Wse
% cat 2+2WLocalLoop.litmus
PPC 2+2WLocalLoop
"PodWW Wse PodWW Wse"
{
0:r2=x; 0:r4=y;
1:r2=y; 1:r4=x;
}
 P0            | P1            ;
 li r1,2       | li r1,2       ;
 stw r1,0(r2)  | stw r1,0(r2)  ;
 li r3,1       | li r3,1       ;
 stw r3,0(r4)  | stw r3,0(r4)  ;
 li r6,200     | li r6,200     ;
 L00:          | L02:          ;
 lwz r5,0(r4)  | lwz r5,0(r4)  ;
 cmpwi r5,1    | cmpwi r5,1    ;
 bne  L01      | bne  L03      ;
 addi r6,r6,-1 | addi r6,r6,-1 ;
 cmpwi r6,0    | cmpwi r6,0    ;
 bne  L00      | bne  L02      ;
 L01:          | L03:          ;
exists (0:r5=2 /\ 1:r5=2)
\end{verbatim}
In the code above,
observing loads are attempted at most 200~time or until a value different
from~$1$ is read.

\subsubsection{Performance of observers}
As an indication of the performance of the various sorts of observers,
the following table summarises a \litmus{} experiment performed on
a 8-cores 4-ways SMT Power7 machine
machine\ifhevea --- \ahref{W.00}{complete \litmus{} log}\fi.
\begin{center}
\begin{tabular}{l|*{7}{c}}
 & \ltest{2+2W} & \ltest{2+2WObs} & \ltest{2+2WObsFenced} & \ltest{2+2WObsLoop} &
\ltest{2+2WLocal} & \ltest{2+2WLocalFenced} & \ltest{2+2WLocalLoop} \\
\hline
Positive & 2.2M/160M & 0/80M & 326/80M & 25k/80M &
2/160M & 34k/160M & 111k/160M\\
\end{tabular}
\end{center}
The row ``Positive'' shows the number of observed positive outcomes/total
number of outcomes  produced.
For instance, in the case of \ltest{2+2W}, we observed the positive outcome
\verb+x=2 /\ y=2+
more than 2 millions times out of a total of 160 millions  outcomes.
As a conclusion, all techniques achieve decent results, except straight
observers.

\subsection{Three \label{sec:obs}stores or more}
In test \ltest{2+2W} the coherence orders sequence two writes.
If there are three writes or more to the same location,
it is no longer possible to
identify a coherence order by observing the final contents of the
memory location involved. In other words, observers are mandatory.

The argument to the \opt{-obs} switch commands
the production of observers. It can take four values:
\begin{description}
\item[{\tt accept}] Produce observers when absolutely needed.
More precisely, given memory location~\texttt{x}, no equality
on~\texttt{x} appears
in the final condition for zero or one write to~\texttt{x},
one such appears for two writes, and observers are produced for
three writes or more.
\item[{\tt avoid}] Never produce observers, \emph{i.e.} fail when there
are three writes to the same location.
\item[{\tt force}] Produce observers for two writes or more.
\item[{\tt local}] Always produce local observers.
\end{description}

With \diyone, one easily build a three writes test as for instance
the following~\ahref{W5.litmus}{\file{W5}}:
\begin{verbatim}
% diyone7 -obs accept -obstype fenced -arch PPC -name W5 Wse Wse PodWW Wse PodWW
% cat W5.litmus
PPC W5
"Wse Wse PodWW Wse PodWW"
{ 0:r2=y; 1:r2=y; 1:r4=x; 2:r2=x; 2:r4=y; 3:r2=y; }
 P0           | P1           | P2           | P3           ;
 lwz r1,0(r2) | li r1,3      | li r1,2      | li r1,2      ;
 sync         | stw r1,0(r2) | stw r1,0(r2) | stw r1,0(r2) ;
 lwz r3,0(r2) | li r3,1      | li r3,1      |              ;
 sync         | stw r3,0(r4) | stw r3,0(r4) |              ;
 lwz r4,0(r2) |              |              |              ;
exists (x=2 /\ 0:r1=1 /\ 0:r3=2 /\ 0:r4=3)
\end{verbatim}
As apparent from the code above, we have a fenced observer thread
on~\texttt{y} (\verb+P0+),
while the final state of~\texttt{x} is observed directly
(\verb+x=2+).
The command line switch \opt{-obs force} would yield two observers,
while \opt{-obs avoid} would lead to failure.

With command line switch \opt{-obs local} we get three local observations of coherence,
which suffice to reconstruct the complete coherence orders:
\begin{verbatim}
% diyone7 -obs local -obstype fenced -arch PPC -name W5Local Wse Wse PodWW Wse PodWW
chi% cat W5Local.litmus
PPC W5Local
"Wse Wse PodWW Wse PodWW"
{
0:r2=x; 0:r4=y;
1:r2=y; 1:r4=x;
2:r2=x;
}
 P0           | P1           | P2           ;
 li r1,3      | li r1,2      | li r1,2      ;
 stw r1,0(r2) | stw r1,0(r2) | stw r1,0(r2) ;
 li r3,1      | li r3,1      | sync         ;
 stw r3,0(r4) | stw r3,0(r4) | lwz r3,0(r2) ;
 sync         | sync         |              ;
 lwz r5,0(r4) | lwz r5,0(r4) |              ;
exists (0:r5=2 /\ 1:r5=2 /\ 2:r3=3)
\end{verbatim}

\section{Command usage}

The~\diy{} suite consists in four main tools:
\begin{description}
\item[\diyone] generates one litmus test from the specification
of a violation of the sequential consistency memory model as a cycle
--- see~\mysec{\ref{diyone:intro}}.
\item[\diycross] generates variations of \diyone{} style tests
--- see~\mysec{\ref{diycross:intro}}.
\item[\diy] generates several tests, aimed at confirming
that candidate relaxations are relaxed or safe---see~\mysec{\ref{diy:intro}}.
\item[\readRelax] Extract relevant information from
the results of tests---see~\mysec{\ref{readRelax:intro}}.
\end{description}

\subsection{A note \label{sec:names}\label{naming}on test names}
We have designed a simple naming scheme for tests.
A normalised test name decomposes first as a family name,
and second as a description of program-order (or internal)
candidate relaxations.

\subsubsection{Family names}
Cycles (and thus tests) are first grouped by families.
Family names describe test structure,
based upon external communication candidates relaxations.
More specifically, external communication candidates relaxations
suffice to settle the directions (\texttt{W} or~\texttt{R}) of
first and last events of threads,
considering the case when those two events are the same.
For instance, consider the cycle ``PodWW Rfe PodRR Fre'':
there are two threads
in the corresponding test (as there are two external communication candidate
relaxations), one thread starts and ends with a write (written \texttt{WW}),
while the other thread starts and ends with a read (written \texttt{RR}).
The family name is thus
\texttt{WW+RR}, (or \texttt{RR+WW}, but we choose the former).
For reference, a normalised family name is the minimal
amongst the representations of a given cycle, following the lexical order
derived from the order
$\texttt{W} < \texttt{WW} <
\texttt{RR} < \texttt{RW} < \texttt{WR} < \texttt{R}$.

The most common families have nicknames, which are defined
by \footahref{\urlfamilies}{this document}.
For instance, consider the test whose cycle is
``\ahref{SB.litmus}{PodWR  Fre PodWR Fre}''.
The family name is \texttt{WR+WR}, as
this is a two-thread test, both threads starting with
a write and ending with a read.
The nickname for this family is, as we already know, SB (store-buffering).
Here is the list of nicknames and family names for two thread tests:
\begin{center}
\begin{tabular}{ll|l}
\hline
2+2W & \texttt{WW+WW} & PodWW Wse PodWW Wse \\
LB  & \texttt{RW+RW} & PodRW Rfe PodRW Rfe \\
MP & \texttt{WW+RR} & PodWW Rfe PodRR Fre \\
R & \texttt{WW+WR} & PodWW Wse PodWR Fre \\
S & \texttt{WW+RW} & PodWW Rfe PodRW Wse \\
SB & \texttt{WR+WR} & PodWR Fre PodWR Fre \\
\hline
\end{tabular}
\end{center}

Isolated writes (and reads) originate from the combinations of
communication relaxations,
for instance [Fre,Rfe].
They appear as ``\texttt{W}'' (and~\texttt{R}) in family names.
For instance,  ``\ahref{tst-ppc/IRIW.litmus}{Rfe PodRR Fre Rfe PodRR Fre}''
contains two such isolated
writes, its name is thus \texttt{W+RR+W+RR} and its nickname is,
as we know, IRIW (Independent reads of independent writes).
The test ``\ahref{WRC.litmus}{Rfe PodRW Rfe PodRR Fre}''
contains one isolated write, as apparent from
this diagram:
\begin{center}\img{WRC}\end{center}
The family name is thus \texttt{W+RW+RR} and the nickname is
WRC (Write to Read Causality).

\subsubsection{Descriptive names for variants}
Every family has a prototype, homonymous test where every thread code
consists in one (for \texttt{W} or~\texttt{R}) or two memory accesses
to different locations (for \texttt{WW}, \texttt{WR} etc.).
For instance, the \ahref{MP.litmus}{\ltest{MP}} test
is derived from the cycle ``PodWW Rfe PodRR Fre''.
Variants are described by tags that illustrates the various
program-order relaxations: they appear after the family name, still with ``\texttt{+}'' as a separation.
For instance the test derived from ``LwSyncdWW Rfe DpAddrdR Fre''
is named \ahref{MP+lwsync+addr.litmus}{\ltest{MP+lwsync+addr}}.

When all threads have the same tag \emph{tag}, the test name is abbreviated
as \texttt{\emph{Family}+\emph{tag}s}. For instance,
the test \ltest{MP+lwsync+lwsync} (``LwSyncdWW Rfe LwSyncdRR Fre'') is in fact
\ahref{MP+lwsyncs.litmus}{\ltest{MP+lwsyncs}}.
Additionally, the tag \texttt{pos} (all \texttt{po}'s) is omitted,
in order to yield family names for the prototype tests --- cf.
 \atest{\ltest{MP}} whose name would have been
\texttt{MP+pos} otherwise.

For the sake of terseness, tags do not describe program-order relaxations
completely. For instance both DpAddrdR and DpAddrdW (address dependency to
read and write, respectively) have the same tag, \texttt{addr}.
It does not harm for simple tests, as the missing direction can be inferred from
the family name. Consider for instance
\ahref{MP+lwsync+addr.litmus}{\ltest{MP+lwsync+addr}} and
\ahref{LB+lwsync+addr.litmus}{\ltest{LB+lwsync+addr}}.
\begin{center}
\cycle{MP+lwsync+addr}\qquad\cycle{LB+lwsync+addr}
\end{center}

The naming scheme extends to cycles with consecutive program-order relaxations,
by separating tags with ``\texttt{-}'' when they follow one another:
for instance ``LwSyncdWW Rfe DpAddrdR PodRR Fre'' is named
\ltest{MP+lwsync+addr-po}. Unfortunately, the current naming scheme falls short
in supplying non-ambiguous names to all tests.
For instance,
``LwSyncdWW Rfe DpAddrdW PodWR Fre'' is also named \ltest{MP+lwsync+addr-po}.
In that situation tools will either fail or silently add a numeric suffix,
depending on the boolean \texttt{-addnum} option.
\begin{verbatim}
% diycross7 -addnum false LwSyncdWW Rfe [DpAddrdR,PodRR],[DpAddrdW,PodWR] Fre
Fatal error: Duplicate name MP+lwsync+addr-po
% diycross7 -addnum true LwSyncdWW Rfe [DpAddrdR,PodRR],[DpAddrdW,PodWR] Fre
Generator produced 2 tests
% cat @all
# diycross7 -addnum true LwSyncdWW Rfe [DpAddrdR,PodRR],[DpAddrdW,PodWR] Fre
MP+lwsync+addr-po.litmus
MP+lwsync+addr-po001.litmus
\end{verbatim}
As a result, we get the two tests:
\atest{MP+lwsync+addr-po} and~\atest{MP+lwsync+addr-po001}.
\begin{center}
\cycle{MP+lwsync+addr-po}\qquad\cycle{MP+lwsync+addr-po001}
\end{center}

Future versions of \diy{} may solve this issue in a more satisfying manner.
At the moment, users are advised not to rely too much
on the automatic naming scheme.
Users may name tests in a non-ambiguous fashion by (1) specifying
an explicit family name  (\texttt{-name \emph{name}}) and~(2)
selecting the numeric scheme (\texttt{-num true}):
\begin{verbatim}
% diycross7 -name MP+X -num true LwSyncdWW Rfe [DpAddrdR,PodRR],[DpAddrdW,PodWR] Fre
Generator produced 2 tests
\end{verbatim}
The \diycross{} generator outputs the same tests as above, with
names \atest{MP+X000} and~\atest{MP+X001}.
\ifhevea
\begin{center}
\cycle{MP+X000}\qquad\cycle{MP+X001}
\end{center}
\fi

\subsection{Common options}
All test generators accept the following documented
command-line options:
\begin{description}
\item[{\tt -v}] Be verbose, repeat to increase verbosity.
\item[{\tt -version}]  Show version number and exit.
\item[{\tt -arch (\allarch)}] Set architecture. Default is \texttt{PPC}.
\item[{\tt -bell <name>}]  Read bell file \opt{<name>},
implies \opt{-arch LISA}.
\item[{\tt -o <dest>}]
Redirect output to \opt{<dest>}.
This option applies when tools generate a set of tests and
an index file \texttt{@all}, \emph{.i.e.} in all situations
except for \diyone{} simplest operating mode.

If  argument \opt{<dest>} is an archive (extension \opt{.tar})
or a compressed archive (extension \opt{.tgz}),
the tool builds an archive.
Otherwise, \opt{<dest>} is interpreted as the name of an
existing directory.
Default is ``\opt{.}'', that is tool output goes into the
current~directory.
\item[{\tt -obs (accept|avoid|force|local)}] Management of observers,
default is \opt{avoid}. See Sec.~\ref{sec:obs}.
\item[{\tt -obstype (fenced|loop|straight)}]
Style of observers, default is \opt{fenced}. See Sec.~\ref{sec:obstype}.
\item[{\tt -cond (cycle|uni|observe)}]
Control final condition of tests, default is \opt{cycle}.
In mode \opt{cycle}, the final condition identifies executions
that correspond to the generating cycle.
In mode \opt{unicond}, the final condition identifies executions
that are valid w.r.t. the uniproc model (see Sec.~\ref{defuniproc}).
In mode  \opt{observe} there is no final condition:
the \litmus{} and \herd{} tools will simply list the final
values of locations.
\item[{\tt -unrollatomic <int>}] Unroll atomic pairs (\emph{i.e.} lr/sc idioms)
according to the given number. Zero has the special meaning to unroll once and
to test the success of teh store conditional in the test final condition. See~section~\ref{unroll:example} for an example.
\item[{\tt -ua <int>}] Shorthand for \opt{-unrollatomic <int>}.
\item[{\tt -oneloc}] Do not reject tests that acts on unique location.
Notice that conditions on such tests usually are not significant
and should probaly be edited by hand. See also options
\texttt{-cond unicond} and~ \texttt{-cond observe} above.
\item [{\tt -optcond}] Optimise conditions by disregarding the values
of loads that are neither the target of~Rf, nor the source of~Fr. This is
the default.
\item [{\tt -nooptcond}] Do not optimise conditions.
\item [{\tt -optcoherence}] Optimise conditions assuming that
the tested system (at least) follows the uniproc model
(see Sec.~\ref{defuniproc}).
\item [{\tt -nooptcoherence}] Do not optimise conditions assuming that
the tested system (at least) follows the uniproc model. This is the default.
\item [{\tt -neg <bool>}] Negate final condition, default is~\texttt{false}.
\item[{\tt -c <bool>}] Avoid equivalent cycles. Default is~\opt{true}.
Setting \opt{-c true} is intended for debug.
\end{description}

The naming of tests is controlled by the following options:
\begin{description}
\item[{\tt -name <name>}] Use name for naming tests, the exact consequences
depend on the generator. By default the generator has no name available.
\item[{\tt -num <bool>}] Use numeric names, \emph{i.e.} from a base name
<base> the generator will name tests as \texttt{<base>000}, \texttt{<base>001} etc. The default depends upon the generator.
\item[{\tt -addnum <bool>}] If true, when faced with tests whose name
\texttt{<name>} has already been given, use names
\texttt{<name>001}, \texttt{<name>002}, etc.
Otherwise fail in the same situation.
The default depends upon the generator.
\item[{\tt -fmt <n>}] Size of numerical suffixes, default is 3.
\end{description}

The following option permits the listing for valid candidate relaxations,
given the selected architecture.
\begin{description}
\item[{\tt -show (edges|fences|annotations)}]
List all candidate relaxations, all fence prefixes or all annotations, respectively.
\end{description}

\subsection{Usage\label{diyone:usage} of \diyone}
The tool \aname{diyone}{\diyone}
has two operating modes.
The selected mode depends on the presence of command-line arguments,

In the first operating mode, \diyone{}
takes a non-empty list of candidate relaxations as arguments
and outputs a litmus test.
Note that \diyone{} may fail to produce the test, with a message
that briefly details the failure.
\begin{verbatim}
% diyone7 Rfe Rfe PodRR
Test a [Rfe Rfe PodRR] failed:
Impossible direction PodRR Rfe
\end{verbatim}

In this mode, {\tt -name <name>} sets the name of the test to \texttt{<name>}
and output it into file \texttt{<name>.litmus}.
If absent, the test name is \texttt{A} and output goes to standard output.

Otherwise, \emph{i.e.} when there are no command-line arguments,
\diyone{} reads the standard input and generates the tests described
by the lines it reads.
Each line starts with a test name~\textit{name},
followed by ``\texttt{:}'', followed
by a list of candidate relaxations~\textit{RS}.
Then, \diyone{} acts as if invoked as \texttt{diyone}~\textit{opts}
\texttt{-name}~\textit{name}~\textit{RS}.

\aname{diyone:norm}{The tool}
\diyone{} accepts the following documented option:
\begin{description}
\item[{\tt -norm}] Normalise tests and give them normalised names.
In the first operating mode (when a cycle is explicitly given) the test
will be named with a family name and a descriptive name.
In the second operating mode,
numeric names are used, base being either given explicitly
(with option \texttt{-name <base>}) or being a normalised family name.
\end{description}

\subsection{Usage of \diycross}
\diycross{} produces several tests by ``cross~producing'' lists
of candidate relaxations given as arguments, see
Sec~\ref{diycross:intro}.
\diycross{} also produces an index file \texttt{@all} that lists all produced
litmus source files.

If option \opt{-name <name>} is given, it sets the family name of
generated tests, otherwise standard family names are used
(cf. Sec.~\ref{sec:names}).
By default descriptive names are used (\emph{i.e.} \texttt{-num false})
and \diycross{} will fail if two different tests have the same name
(\emph{i.e.} \texttt{-addnum false}):
\begin{verbatim}
% diycross7 PodWW Rfe [DpAddrdR,PodRR],[DpAddrdW,PodWR] Fre
Fatal error: Duplicate name MP+po+addr-po
\end{verbatim}
Should this happen users can resort either to numeric names,
\begin{verbatim}
%diycross7 -num true PodWW Rfe [DpAddrdR,PodRR],[DpAddrdW,PodWR] Fre
Generator produced 2 tests
con% ls
@all  MP000.litmus  MP001.litmus
\end{verbatim}
or to disambiguating numeric suffixes.
\begin{verbatim}
%diycross7 -addnum true PodWW Rfe [DpAddrdR,PodRR],[DpAddrdW,PodWR] Fre
Generator produced 2 tests
con% ls
@all  MP+po+addr-po001.litmus  MP+po+addr-po.litmus
\end{verbatim}

\subsection{Usage\label{diy:usage} of {\diy}}
As \diycross, \diy{} produce several files, hence naming issues are critical.
By default, \diy{} uses family names and the numeric naming scheme
(\texttt{-num true}).
Users can specify a family name~\emph{family}
for all tests with \texttt{-name \emph{family}},
or attempt using the descriptive names of Sec~\ref{sec:names}
with \texttt{-num false}.
Moreover, \diy{} produces an index file \file{@all} that lists
the file names of all tests produced.

The tool \diy{} also accepts the following, additional, documented options.
\begin{description}
\item[{\tt -conf <file>}] Read configuration file \opt{<file>}.
A configuration file consists in a list of options, one option per line.
Lines introduced by \verb+#+ are comments and are thus ignored.
\item[{\tt -size <n>}] Set the maximal size of cycles. Default is~$6$.
\item[{\tt -exact}] Produce cycles of size exactly~\opt{<n>},
in place of size up to~\opt{<n>}.
\item[{\tt -nprocs <n>}] Reject tests with more than \opt{<n>} threads.
Default is~$4$.
\item[{\tt -eprocs}] Produce tests with exactly \opt{<n>}
threads, where \opt{<n>} is set above.
\item[{\tt -ins <n>}] Reject tests as soon as the code of one thread
originates from \opt{<n>} edges or more. Default is~$4$.
\item[{\tt -relax <relax-list>}] Set relax list. Default is empty.
The syntax of \opt{<relax-list>} is a comma (or space)
separated list of candidate relaxations.
\item[{\tt -mix <bool>}] Mix the elements of the relax list
(see below), default \opt{false}.
\item[{\tt -maxrelax <n>}]
In mix mode, upper bound on the number of different candidate
relaxations tested together.
Default is~$100$
\item[{\tt -safe <relax-list>}] Set safe list. Default is empty.
\item[{\tt -mode (critical|sc|free|uni)}]
Control generation  of cycles, default~\opt{sc}.
Those tags command the activation of some constraints over cycle
generation, see \ahrefloc{mode:describe}{below}.

\item[{\tt -cumul <bool>}]
Permit implicit cumulativity,
\emph{i.e.} authorise building up the sequence Rfe followed by a fence,
or the reverse. Default is~\opt{true}.
\end{description}
The relax and safe lists command the generation of
cycles as follows:
\begin{enumerate}
\item When the relax list is empty,
cycles are built from the candidate relaxations of the safe list.
\item When the relax list is of size~$1$,
cycles are built from its single element~$r$ and from the elements of
the safe list. Additionally, the cycle produced contains~$r$ at least once.
\item
When the relax list is of size~$n$, with~$n > 1$,
the behaviour of~\diy{} depends on the mix mode:
\begin{enumerate}
\item
By default (\opt{-mix false}),
\diy{} generates~$n$ independent sets of cycles,
each set being built with one relaxation from the relax list and all
the relaxations in the safe list.
In other words, \diy{} on a relax list of size~$n$ behaves similarly
to $n$~runs of~\diy{} on each candidate relaxation in the list.
\item Otherwise (\opt{-mix true}), \diy{} generates cycles that contains
at least one element from the relax list, including some cycles
that contain different relaxations from the relax list.
The cycles will contain at most~$m$ different elements from the relax list,
where $m$ is specified with option~``\opt{-maxrelax}~$m$''.
\end{enumerate}
\end{enumerate}

\label{mode:describe}Generally
speaking, \diy{} generates ``some'' cycles and does not generate
``all'' cycles (up to a certain size \emph{e.g.}).
In (default) sc mode, \diy{} performs some optimisation,
most of which we leave unspecified.
\label{compose:com}As an exception to
this non-specification, \diy{} in \opt{sc} (default) mode
is guaranteed not to
generate redundant elementary communication relaxation in the following sense:
let us call Com the union of Ws, Rf and Fr (the e\vbar{}i specification
is irrelevant here).
Ws being transitive and by definition of Fr,
one easily shows that the transitive closure Com+ of Com is the union
of Com plus [Ws,Rf] (Ws followed by Rf) plus [Fr,Rf].
As a consequence, maximal subsequences of communication
relaxations in \diy{} cycles are limited
to  single relaxations (\emph{i.e.} Ws, Rf and Fr)
and to the above mentioned two sequences
(\emph{i.e.} [Ws,Rf] and  [Fr,Rf]).
For instance, [Ws,Ws] and [Fr,Ws] should never appear in \diy{} generated
cycles.
However, such subsequences can be generated on an individual basis with
\diyone, see the example of \file{W5} in Sec~\ref{sec:obs}.

\label{critical:def}%
In critical mode (\opt{-mode critical}),  cycles are strictly specified as
follows:
\begin{enumerate}
\item Communication candidate relaxations sequences are
limited to Rf,Fr,Ws,[Ws,Rf] and~[Fr,Rf], as in sc mode.
\item No two internal\footnote{That is, the source and target
accesses are by the same processor.} candidate relaxations follow one another.
\item If the option \opt{-cumul false} is specified,
\diy{} will not construct
the sequence of~Rfe followed by a fence (or B-cumulativity) candidate
relaxation or
of a fence (or A-cumulativity) candidate relaxation followed by Rfe.
\item Cycles that access one single memory location are rejected.
\item None of the rules above applies to the internal sequences of
composite candidate relaxations. For instance, if [Rfi,PodRR]
is given as a candidate relaxation, the sequence ``Rfi,PodRR'' appears
in cycles.
\end{enumerate}
The cycles described above are the \emph{critical} cycles of~\cite{ss88}.

\label{free:def}%
In free mode (\opt{-mode free}), cycles are strictly specified as
follows:
\begin{enumerate}
\item Communication candidate relaxations sequences are
limited to Rf,Fr,Ws,[Ws,Rf] and~[Fr,Rf]. However, arbitrary sequences
of communication candidates are accepted when they are internal and external
or external and internal.
\item Cycles that access one single memory location are rejected.
\end{enumerate}

\label{uni:def}Finally, the \opt{uni} mode enforces the following constraints
on cycles:
\begin{enumerate}
\item Sequences of communication  candidate relaxations
are restricted in the same manner as for
\opt{free} mode (see \ahrefloc{free:def}{above}).
\item Sequences of Po candidate relaxation are rejected.
\end{enumerate}

\subsection{\aname{readRelax}{Usage} of \readRelax}
\readRelax{} is a simple tool to extract relevant information
out of \litmus{} run logs of tests produced by the \diy{} generator.
For a given run of a given litmus test, the relevant information is:
\begin{itemize}
\item Whether the test yielded \texttt{Ok} or not,
\item An optional candidate relaxation, which is the one
given as argument to \diy~option
\opt{-relax} at test build time, or none.
\item The safe list relevant to the given test, \emph{i.e.}
the safe candidate relaxations that appear in the tested cycle.
\end{itemize}
See \mysec{\ref{readRelax:intro}} for an example.

The tool \readRelax{} takes file names as arguments.
If no argument is present, it reads a list
of file names on standard input, one name per line.

\section{Additional tools: extracting cycles and classification}
When non-standard family names or numeric names are used, it proves convenient
to rename tests with the standard naming scheme.
We provide two tools to do so: \mcycles{} that extracts cycles from
litmus source files and \classify{} that normalises and renames cycles.

For instance, one can use \diy{} to generate all simple, critical, tests up
to three threads for X86 with the following configuration file~\afile{X.conf}
\verbatiminput{X.conf}
\begin{verbatim}
% diy7 -conf X.conf
Generator produced 23 tests
% ls
@all         X003.litmus  X007.litmus  X011.litmus  X015.litmus  X019.litmus  X.conf
X000.litmus  X004.litmus  X008.litmus  X012.litmus  X016.litmus  X020.litmus
X001.litmus  X005.litmus  X009.litmus  X013.litmus  X017.litmus  X021.litmus
X002.litmus  X006.litmus  X010.litmus  X014.litmus  X018.litmus  X022.litmus
\end{verbatim}
Cycles are extracted with \mcycles, which takes the index file \texttt{@all} as argument:
\begin{verbatim}
% mcycles7 @all
X000: Wse PodWR Fre PodWR Fre PodWW
X001: Rfe PodRR Fre PodWR Fre PodWW
X002: Wse PodWR Fre PodWW
X003: Wse PodWW Wse PodWR Fre PodWW
X004: Rfe PodRW Wse PodWR Fre PodWW
X005: Rfe PodRR Fre PodWW
X006: Wse PodWW Rfe PodRR Fre PodWW
X007: Rfe PodRW Rfe PodRR Fre PodWW
X008: Wse Rfe PodRR Fre PodWW
X009: Wse PodWW Wse PodWW
...
\end{verbatim}
The output of \mcycles{} can be piped into \classify{} for family
classification:
\begin{verbatim}
% mcycles7 @all | classify7 -arch X86
2+2W
  X009 -> 2+2W : PodWW Wse PodWW Wse
3.2W
  X010 -> 3.2W : PodWW Wse PodWW Wse PodWW Wse
3.LB
  X020 -> 3.LB : PodRW Rfe PodRW Rfe PodRW Rfe
3.SB
  X016 -> 3.SB : PodWR Fre PodWR Fre PodWR Fre
ISA2
  X007 -> ISA2 : PodWW Rfe PodRW Rfe PodRR Fre
LB
  X019 -> LB : PodRW Rfe PodRW Rfe
MP
  X005 -> MP : PodWW Rfe PodRR Fre
...
\end{verbatim}
Notice that \classify{} accepts the \opt{arch} option, as it needs
to parse cycles.

Finally, one can normalise tests, using normalised names by piping
\mcycles{} output into \diyone{} with options
\ahrefloc{diyone:norm}{\opt{-norm -num false}}:
\begin{verbatim}
% mkdir src
% mcycles7 @all | diyone7 -arch X86 -norm -num false -o src
Generator produced 23 tests
% ls src
2+2W.litmus  @all         R.litmus    WRC.litmus     WRW+WR.litmus  Z6.2.litmus
3.2W.litmus  ISA2.litmus  RWC.litmus  WRR+2W.litmus  WWC.litmus     Z6.3.litmus
3.LB.litmus  LB.litmus    SB.litmus   WRW+2W.litmus  Z6.0.litmus    Z6.4.litmus
3.SB.litmus  MP.litmus    S.litmus    W+RWC.litmus   Z6.1.litmus    Z6.5.litmus
\end{verbatim}
Alternatively, one may instruct \classify{} to produce output for
\diyone. In that case one should pass option \opt{-diyone} to \classify{}
so as to instruct it to produce output that is parsable by \diyone:
\begin{verbatim}
% rm -rf src && mkdir src
% mcycles7 @all | classify7 -arch X86 -diyone | diyone7 -arch X86 -o src
Generator produced 23 tests
% ls src
2+2W.litmus  @all         R.litmus    WRC.litmus     WRW+WR.litmus  Z6.2.litmus
3.2W.litmus  ISA2.litmus  RWC.litmus  WRR+2W.litmus  WWC.litmus     Z6.3.litmus
3.LB.litmus  LB.litmus    SB.litmus   WRW+2W.litmus  Z6.0.litmus    Z6.4.litmus
3.SB.litmus  MP.litmus    S.litmus    W+RWC.litmus   Z6.1.litmus    Z6.5.litmus
\end{verbatim}

\subsection{Usage of \mcycles{}}
The tool \mcycles{} has no options and takes litmus source files or
index files as arguments.
It outputs a list of lines to standard output.
Each line starts with a test name, suffixed by ``\texttt{:}'', then
the cycle of the named test. Notice that this format is the input format
to \diyone{} in its second operating mode --- see Sec.~\ref{diyone:usage}.

It is important to notice that, for \mcycles{} to extract cycles, those
must be present as meta-information in source files.
In practice, this means that  \mcycles{} operates normally on
sources produced by \diyone, \diycross{} and~\diy.
Moreover only one instance of a given cycle will be output.

\subsection{Usage\label{classify:usage}of \classify}
The tool \classify{} reads its standard input, interpreting is as
a list of cycles in the output format of \mcycles.
It normalises and classifies those cycles.
The tool \classify{} accepts the following documented options:
\begin{description}
\item[{\tt -arch (\allarch)}] Set architecture. Default is \texttt{PPC}.
\item[{\tt -bell <name>}]  Read bell file \opt{<name>},
implies  \opt{-arch LISA}.
\item[{\tt -lowercase <bool>}] Use lowercase familly names, default false.
\item[{\tt -u}] Instruct \classify{} to fail when two tests have the same
normalised name. Otherwise \classify{} will output one line per test,
regardless of duplicate names.
\item[{\tt -diyone}] Output a normalised list of names and cycles,
which is legal input for \diyone.
\end{description}

\subsection{Usage\label{norm:usage} of \norm}
The tool \norm{} is a simplified \classify{} tool: it takes
a cycle on the command line and outputs a normalized cycles, with name.
This output can serve as standard input to \diyone.

\begin{description}
\item[{\tt -arch (\allarch)}] Set architecture. Default is \texttt{PPC}.
\item[{\tt -bell <name>}]  Read bell file \opt{<name>}, implies \opt{-arch LISA}.
\item[{\tt -lowercase <bool>}] Use lowercase familly names, default false.
\end{description}

For instance, here is hwo one can find the systematic mane of a variation
on the store-buffer ``\textsf{SB}'' test:
\begin{verbatim}
% norm7 -arch AArch64 Rfi DMB.LDdRR Fre Rfi DpAddrdR Fre
SB+rfi-dmb.ld+rfi-addr: Rfi DMB.LDdRR Fre Rfi DpAddrdR Fre
\end{verbatim}
Notice that the option \opt{-arch} is mandatory (or defaults to \texttt{PPC}),
as  candidate relaxations depends on the selected architecture, as illustrated
by the fence candidate \texttt{DMB.LD}\ldots{} above.

\endinput


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage{mathtools}
\usepackage{pgfplots}
\usepackage{natbib}
\usepackage{graphicx}
\usepackage{amsmath}
\usepackage{amssymb}
\usepackage{amsfonts}
\usepackage{natbib}
\usepackage{minted}
\usepackage[a4paper,margin=1in,footskip=0.25in]{geometry}
\allowdisplaybreaks

\title{CSE546 Machine Learning HW1 - A}
\author{Bobby Deng | 1663039 | dengy7 }
\date{15 April 2020}

\begin{document}
\maketitle

\section{A.0}
\subsection{In your own words, describe what bias and variance are? What is bias-variance trade-off?}

In class, we have derived that the total variability is composed by three parts: irreducible error, bias and variance. Irreducible error is the lowest error that our model can get. Bias is how much our model is deviated from the true model. When the model is two simple, it is more likely to deviate largely from the model. When the model is too complex and fits all training data points. There would not be any bias at all. Variance is basically the variance of the model, when model is a horizontal line, there is no variance at all. But if the model is too complex, the geometry representation of the model becoming too zigzag, and this is where the variance tends to be very big.
The bias-variance trade-off is saying that we need to find a balance point where testing error is at the lowest point. Testing error is presented as an u shape, so ideally we want to find the model that produces the testing error at the bottom of the u shape curve.

\subsection{What happens to bias and variance when the model complexity increases/decreases?}
When complexity increase, bias decrease, variance increase. When complexity decrease, bias increase, variance decrease.
Bias will be monotonically decrease and finally fits all training data points. Variance will be monotonically increase.

\subsection{True or False: The bias of a model increases as the amount of training data available increases.}

False, bias dose not depend on the amount of training data points.

\subsection{True or False: The variance of a model decreases as the amount of training data available increases.}

True, the variance do depend on the amount of training data.

\subsection{True or False: A learning algorithm will generalize better if we use less features to represent our data.}

False, we want enough useful and good predictors in the modes. The quality of predictor and coefficients determined how good our model might be. And less feature will not be better to present the ground truth model.

\subsection{To get better generalization, should we use the train set or the test set to tune our hyper-parameters?}

We should use train set to train the model and get hyper-parameters. We must not use any testing data in getting the hyper-parameters.
\subsection{True or False: The training error of a function on the training set provides an overestimate of the true error of that function.}

False, since we saw in both practice and theory that the training error is always lower than the testing error, in expectation. And when we train out model, our based model are generated by minimizing the error on training data. So it is an optimistic/underestimated estimation of the true function.

\section*{A.1}
Use MLE to estimate $\lambda$ in Poisson Distribution.
\subsection*{a.}

\begin{subequations}
\begin{align*}
P(x|\lambda) & = \frac{e^{-\lambda}\lambda^{x}}{x!} \\
P(x_1, x_2, x_3, x_4, x_5|\lambda)
& = \prod^{n}_{i=1}\frac{e^{-\lambda}\lambda^{x_i}}{x_i!} \text{ (Where n = 5.)} \\
\hat{\lambda}_{MLE} & = \arg\max_{\lambda} (P(x_1, x_2, x_3, x_4, x_5|\lambda) ) \\
& = \arg\max_{\lambda} (\prod^{n}_{i=1}\frac{e^{-\lambda}\lambda^{x_i}}{x_i!}) \\
& = \arg\max_{\lambda} ln(\prod^{n}_{i=1}\frac{e^{-\lambda}\lambda^{x_i}}{x_i!}) \\
& = \arg\max_{\lambda} \sum_{i=1}^{n} ln(\frac{e^{-\lambda}\lambda^{x_i}}{x_i!} ) \\
& = \arg\max_{\lambda} \sum_{i=1}^{n} (lne^{-\lambda}+ln\lambda^{xi}-ln(xi!)) \\
& = \arg\max_{\lambda}  (-n\lambda + ln\lambda*\sum_{i=1}^{n}xi + \sum_{i=1}^{n}ln(xi!)) \\
& \text{Then set the derivative to 0.} \\
0 & = \frac{d}{d\lambda} (-n\lambda + ln\lambda*\sum_{i=1}^{n}xi + \sum_{i=1}^{n}ln(xi!)) \\
& = -n + \frac{1}{\lambda} \sum_{i=1}^{n} xi + 0 \\
n & = \frac{1}{\lambda} \sum_{i=1}^{n} xi \\
\hat{\lambda}_{MLE} & = \frac{1}{n} \sum_{i=1}^{n} xi , n = 5.\\
\end{align*}
\end{subequations}

\subsection*{b.}
\begin{subequations}
\begin{align*}
	\hat{\lambda}_{MLE} & = \arg\max_{\lambda} (P(\mathcal{D}|\lambda))                   \\
	                    & =  \arg\max_{\lambda} (P(x_1, x_2, x_3, x_4, x_5, x_6|\lambda)) \\
	                    & = \frac{1}{n} \sum_{i=1}^{n} xi , n = 6.
\end{align*}
\end{subequations}

\subsection*{c.}

For $\lambda $ after 5, $\hat{\lambda}_{MLE} = (2+0+1+1+2)/5 = \frac{6}{5}$. \newline
For $\lambda$ after 6, $\hat{\lambda}_{MLE} = (2+0+1+1+2+4)/6 = \frac{5}{3}$. \newline

So, we can see the result that the outlier has a big effect on the estimation.

\section*{A.2 German tank problem}
From the problem, we know that the serial of number came from a uniform distribution with $[0,\theta]$.

\[
f(x_i) = \begin{cases}
\frac{1}{\theta}  & \text{for $0\le x_i \le \theta$.}  \\
0 & \text{Otherwise.}
\end{cases}
\]

We can derive the MLE with uniform distribution.

\begin{subequations}
\begin{align*}
\hat{\theta}_{MLE} & = \arg\max_{\theta} (P(\mathcal{D}|\theta))                   \\
& =  \arg\max_{\theta} (\prod^{n} \frac{1}{\theta}) \\
& = \arg\max_{\theta} ln(\prod^{n} \frac{1}{\theta}) \\
& = \arg\max_{\theta} ln( \frac{1}{\theta^n}) \\
& = \arg\max_{\theta} -n ln(\theta) \\
\frac{d(ln(L(\theta)))}{d\theta} & = -n/\theta
\end{align*}
\end{subequations}

We can not set the derivative to zero in this particular problem since $\theta$ is impossible to be infinity. So here, the only thing we can do is let $\theta$ to be as large as it could be. We can not arbitrages assign value to $\theta$. What we can do is to assign the biggest X we have observed to $\theta$.  \newline

\[\hat{\theta} = max(x_1, \dots, x_n)\]

\section*{A.3}
\subsection*{a.}

\[ E_{train}[\hat{\epsilon}_{train}(f)] = E[ \frac{1}{N_{train}} \sum_{(x,y)\in S_{train}} (f(x)-y)^2 ] \]

\[ E_{train}[\hat{\epsilon}_{train}(f)] = \frac{N_{train}}{N_{train}} E[(f(x)-y)^2]\]

\[ E_{train}[\hat{\epsilon}_{train}(f)] = E[(f(x)-y)^2] =  \epsilon (f)\]
Since data from train and test are draw iid, the expectation for $(f(x)-y)^2$ should be the same.
\[E_{test}[\hat{\epsilon}_{test}(f)]  = \frac{N_{test}}{N_{test}} E[(f(x)-y)^2] = E[(f(x)-y)^2] =  \epsilon (f)\]
So,
\[E_{train}[\hat{\epsilon}_{train}(f)] = E_{test}[\hat{\epsilon}_{test}(f)] = \epsilon (f)\]

\subsection*{b.}
No, the equation is not generally true, because the estimator we got from training is by minimizing the training error. It is not able to be general enough to describe the true model.
\subsection*{c.}
From the hint we know that,
\[
E_{train,test}[\hat{\epsilon}_{test}(\hat{f}_{train})]
= \sum_{f\in \mathcal{F}} E_{test}[\hat{\epsilon}_{test}(f)]  \mathbb{P}_{train}(\hat{f}_{train}=f)
\]
From a., we know that $ E_{train}[\hat{\epsilon}_{train}(f)] =  E_{test}[\hat{\epsilon}_{test}(f)]$. \newline
So, \[ = \sum_{f\in \mathcal{F}} E_{train}[\hat{\epsilon}_{train}(f)]  \mathbb{P}_{train}(\hat{f}_{train}=f) \]
Since $\hat{\epsilon}_{train}(\hat{f}) \le \hat{\epsilon}_{train}(f)$:

\[ \sum_{f\in \mathcal{F}} E_{train}[\hat{\epsilon}_{train}(\hat{f})]  \mathbb{P}_{train}(\hat{f}_{train}=f) \le \sum_{f\in \mathcal{F}} E_{train}[\hat{\epsilon}_{train}(f)]  \mathbb{P}_{train}(\hat{f}_{train}=f) \]

\[ \sum_{f\in \mathcal{F}} E_{train}[\hat{\epsilon}_{train}(\hat{f})]  \mathbb{P}_{train}(\hat{f}_{train}=f) \le  E_{train,test}[\hat{\epsilon}_{test}(\hat{f}_{train})] \]
Because $\sum_{f\in \mathcal{F}} \mathbb{P}_{train}(\hat{f}_{train}=f) = 1$:
\[ E_{train}[\hat{\epsilon}_{train}(\hat{f})] \sum_{f\in \mathcal{F}} \mathbb{P}_{train}(\hat{f}_{train}=f) \le  E_{train,test}[\hat{\epsilon}_{test}(\hat{f}_{train})] \]

\[
E_{train}[\hat{\epsilon}_{train}(\hat{f}_{train})] \le
E_{train,test}[\hat{\epsilon}_{test}(\hat{f}_{train})]
\]

\section*{A.4}

\begin{minted}[mathescape,linenos,obeytabs=true,tabsize=2]{python}

import numpy as np

#-----------------------------------------------------------------
#  Class PolynomialRegression
#-----------------------------------------------------------------

class PolynomialRegression:

def __init__(self, degree=1, reg_lambda=1E-8):
	"""
	Constructor
	"""
	self.theta = None
	self.regLambda = reg_lambda
	self.degree = degree
	self.mean = None
	self.std = None

def polyfeatures(self, X, degree):
	"""
	Expands the given X into an n * d array of polynomial features of
	degree d.

	Returns:
	A n-by-d numpy array, with each row comprising of
	X, X * X, X ** 3, ... up to the dth power of X.
	Note that the returned matrix will not include the zero-th power.

	Arguments:
	X is an n-by-1 column numpy array
	degree is a positive integer
	"""
	outputX = X[:]
	for i in range(2, degree + 1):
		outputX = np.hstack((outputX,X**i))
	return outputX

def fit(self, X, y):
	"""
	Trains the model
	Arguments:
	X is a n-by-1 array
	y is an n-by-1 array
	Returns:
	No return value
	Note:
	You need to apply polynomial expansion and scaling
	at first
	"""

	X = self.polyfeatures(X, self.degree)
	# standardization
	self.mean = np.mean(X, axis=0)
	self.std = np.std(X, axis=0)
	X = (X - self.mean) / self.std
	n = len(X)
	# add 1s column
	X_ = np.c_[np.ones([n, 1]), X]
	n, d = X_.shape
	reg_matrix = self.regLambda * np.identity(d )
	reg_matrix[0, 0] = 0
	self.theta = np.linalg.pinv((X_.T @ X_) + reg_matrix) @ (X_.T @ y)
	print(self.theta)

def predict(self, X):
	"""
	Use the trained model to predict values for each instance in X
	Arguments:
	X is a n-by-1 numpy array
	Returns:
	an n-by-1 numpy array of the predictions
	"""

	n = len(X)
	X = self.polyfeatures(X, self.degree)
	X = (X - self.mean) / self.std
	# add 1s column
	X_ = np.c_[np.ones([n, 1]), X]
	# predict
	return X_ @ self.theta

#-----------------------------------------------------------------
#  End of Class PolynomialRegression
#-----------------------------------------------------------------

\end{minted}

When lambda is 0:

\includegraphics[width=5cm, height=4cm]{A4_0.png}

When lambda is 0.0001:

\includegraphics[width=5cm, height=4cm]{A4_0_0001.png}

When lambda is 0.01:

\includegraphics[width=5cm, height=4cm]{A4_0_01.png}

When lambda is 0.1:

\includegraphics[width=5cm, height=4cm]{A4_0_1.png}

From the graph, we can see that when lambda increase out model gets smoother. And the model tends to be simpler. And the weight tends to be smaller.

\section*{A.5}

\begin{minted}[mathescape,linenos,obeytabs=true,tabsize=2]{python}
def learningCurve(Xtrain, Ytrain, Xtest, Ytest, reg_lambda, degree):
"""
Compute learning curve

Arguments:
Xtrain -- Training X, n-by-1 matrix
Ytrain -- Training y, n-by-1 matrix
Xtest -- Testing X, m-by-1 matrix
Ytest -- Testing Y, m-by-1 matrix
regLambda -- regularization factor
degree -- polynomial degree

Returns:
errorTrain -- errorTrain[i] is the training accuracy using
model trained by Xtrain[0:(i+1)]
errorTest -- errorTrain[i] is the testing accuracy using
model trained by Xtrain[0:(i+1)]

Note:
errorTrain[0:1] and errorTest[0:1] won't actually matter, since we start displaying the learning curve at n = 2 (or higher)
"""
n = len(Xtrain)

errorTrain = np.zeros(n)
errorTest = np.zeros(n)

for i in range(3, n+1):
print("i = ", i, " Degree= ", degree, " lambda: ", reg_lambda)
model = PolynomialRegression(degree, reg_lambda)
model.fit(Xtrain[:i], Ytrain[:i])

trainPredicted = model.predict(Xtrain[:i])
singleErrorFromTrain = np.mean((trainPredicted- Ytrain[:i])**2)
errorTrain[i-1] = singleErrorFromTrain

test_predicted = model.predict(Xtest[:i])
singleErrorFromTest = np.mean((test_predicted - Ytest[:i])**2)
errorTest[i-1] = singleErrorFromTest

return errorTrain, errorTest
\end{minted}

\includegraphics[width=12cm, height=8cm]{A5_1.png}

\section*{A.6}

\subsection*{a.}
From the problem we know that:
\[ \sum_{i=0}^{n} || W^T x_i - y_i ||^2_2 + \lambda||W||^2_F = \sum_{j=0}^{k} \Big[ ||Xw_j - Ye_j||^2 + \lambda||w_j||^2\Big] \]
Let:
\[ A =  \sum_{j=0}^{k} \Big[ ||Xw_j - Ye_j||^2 + \lambda||w_j||^2\Big] \]
Take derivative:
\[   \frac{\partial A}{\partial w_j} = \sum_{j=0}^{k} 2X^T\big[ Xw_j - Ye_j \big] +2\lambda w_j \]
Set derivative to 0 to find minimum,
\[ 0 = \sum_{j=0}^{k} 2X^T\big[ Xw_j - Ye_j \big] +2\lambda w_j \]

\[ 0 = \sum_{j=0}^{k} X^TXw_j +\lambda w_j - X^TYe_j \]

\[ \sum_{j=0}^{k} (X^TX + \lambda I)w_j = \sum_{j=0}^{k} X^TYe_j\]

Here, $X^T \in \mathbb{R}^{n \times d}$, $Y^T \in \mathbb{R}^{n \times k}$, $w_j \in \mathbb{R}^{d \times 1}$, $x_i \in \mathbb{R}^{d \times 1}$, $y_i \in \mathbb{R}^{k \times 1}$, $\lambda \in \mathbb{R}$, $e_j \in \mathbb{R}^{k \times 1}$, let's write it in matrix form:
\[ (X^TX + \lambda I)\hat{W} = X^TY \]
\[ \hat{W} = (X^TX + \lambda I)^{-1}X^TY \]
\subsection*{b.}

\begin{minted}[mathescape,linenos,obeytabs=true,tabsize=2]{python}
import mnist
import numpy as np

mndata = mnist.MNIST("./python-mnist/data/")
X_train, labels_train = map(np.array, mndata.load_training())
X_test, labels_test = map(np.array, mndata.load_testing())
X_train = X_train/255.0
X_test = X_test/255.0

# Train function
def train(X,y,lam):
	n, d = X.shape
	y = np.eye(10)[y] # put y into a one hot encoding matrix
    reg_matrix = self.regLambda * np.identity(d )
    reg_matrix[0, 0] = 0
    self.theta = np.linalg.pinv((X_.T @ X_) + reg_matrix) @ (X_.T @ y)
	return W

def predict(W, X_new):
	return (X_new @ W_hat).argmax(axis=1)

# Compute weights
W_hat = train(X_train, labels_train, lam = 0.00001)
# Computed predicted training label
predicted_train = predict(W=W_hat, X_new=X_train)
# Compute predicted testing label
predicted = predict(W=W_hat, X_new=X_test)
# Compute error rate for both training and testing
train_error_rate = 1 -( sum(predicted_train == labels_train) / len(labels_train))
test_error_rate = 1 - (sum(predicted == labels_test) / len(labels_test))
print("Training Accuracy is: ", train_error_rate)
print("Testing Error Rate is: ", test_error_rate)
#Training Error is:  0.14806666666666668
#Testing Error Rate is:  0.14659999999999995
\end{minted}

\end{document}