\documentclass[PHIL101-Textbook.tex]{subfiles}
\begin{document}

\part{\tfl\ Truth Trees}\label{part:tfl.trees}

\chapter{Truth Trees}\label{ch:chaptertruthtrees}

\section{Truth Tables aren't Enough}

We've learned two ways of using truth tables to test for consistency, equivalence, validity, etc. Both of them have their uses and their drawbacks. A complete truth table always gives the right answer by following a mechanical process. But it can be long and tedious; and the complexity and length increases exponentially with the number of variables. A 2 variable table only has 4 rows, but a 20 variable table has over 1 million.

A partial truth table, on the other hand, requires a little bit of cunning to see which step should be taken next, and to spot contradictions. And it can be difficult to tell which steps were done in which order, unless you make many copies of your truth table.

So our first method takes too long for complex formulas, while our second method is too unstructured both when creating and reviewing. In this part, we will introduce a third method that is intrinsically more complex than either, but does not grow as fast as complete truth tables, and is more structured, rule-driven and checkable than partial truth tables. It will also work for the logic we introduce in the second half of this book, unlike truth tables.

The structures produced by this new method are called \define{truth trees}, because of the way the diagram of formulas can `branch' out to represent possibilities.

\section{Mutual Consistency}

The method of trees is most directly a way to test for the consistency of a set of \tfl\ formulas. Since all our other logical notions can be defined in terms of consistency (or lack of consistency), it indirectly gives a way to test for equivalence, validity, and so forth. For example, you may recall that one way to test if an argument is valid, is to  test if the premises and the negation of the conclusion are mutually consistent -- that is, if there is a counter-example, and thus the argument is invalid.

The basic idea of the tree method is that we'll write down the formulas we are testing for consistency, and then we'll break these formulas down into their component subformulas, like with did with our partial truth tables, in order to see whether it's possible to find a valuation where they are all true.

\paragraph{Our first Truth Tree}
We will begin by working through an example that illustrates the general method, then come back to give precise rules for the tree method. We want to know whether the following formulas are all mutually consistent in \tfl:

$$A\eand B,
\enot(C\eor D),
(\enot B \eor C) \eor E, \enot E$$

We begin by writing down the formulas we will test for consistency:

\begin{center}\begin{prooftree}
{close with=\ensuremath{\times}}
[A\eand B, just={Root}
[\enot(C\eor D), just={Root}
[(\enot B \eor C) \eor E, just={Root}
[\enot E, just={Root}
]]]]
\end{prooftree}\end{center}

The formulas we write down at the beginning of the tree are called the \define{root}. Trees are designed to show whether the root is consistent, and if so, provide a valuation. We will assume the formulas in the root are all true.
We then write down formulas which our previous formulas tell us must also be true, and repeat this process until we have found the information we need.

 Consider line (1), $A \eand B$. For this conjunction to be true, both conjuncts must be true. %So any valuation that makes the root formulas true must also make both $A$ and $B$ true.
The truth of each conjunct follows from what is already written down. We add them to the tree:

\begin{center}
\begin{prooftree}
{close with=\ensuremath{\times}}
[A\eand B, checked, name=p1, just={Root}
[\enot(C\eor D), just={Root}
[(\enot B \eor C) \eor E, just={Root}
[\enot E, just={Root}
	[A, just={\eand}:p1
	[B, just={\eand}:p1
	]]
]]]]
\end{prooftree}\end{center}

When we add lines (5) and (6), we note that they came from line (1), and that we considered conjunction (\eand) to produce them. This is useful for you when revising your work, and to help us to understand what you thought you were doing when we see some unexpected formulas in the truth tree. We also add a check mark on (1) once we've decomposed it, to avoid repeating our actions.  %This ensures that we only look at each line once.
%Think of the check mark as saying that we've extended the tree in a way that encodes the information from this line.

Now consider line (2) $\enot(C\eor D)$. This is a negated disjunction. A negated disjunction is true when a disjunction is false. Disjunctions are false iff \emph{both} disjuncts are false. We only write down formulas that are true, so if a formula is false, we write down its negation, which will be true. So we include new lines for \enot $C$ and \enot $D$, adding a check mark on line (2):

\begin{center}\begin{prooftree}
{close with=\ensuremath{\times}}
[A\eand B, checked, name=p1, just={Root}
[\enot(C\eor D), checked, name=p2, just={Root}
[(\enot B \eor C) \eor E, just={Root}
[\enot E, just={Root}
	[A, just={\eand}:p1
	[B, just={\eand}:p1
	[\enot C, just={\enot\eor}:p2
	[\enot D, just={\enot\eor}:p2
	]]]]
]]]]
\end{prooftree}\end{center}

Line (3) is a disjunction. Unlike the previous two cases, it doesn't tell us what \emph{must} be the case; it says that (at least) one of the disjuncts must be true. We represent this by \emph{branching} our tree :

\begin{center}\begin{prooftree}
{close with=\ensuremath{\times}}
[A\eand B, checked, name=p1, just={Root}
[\enot(C\eor D), checked, name=p2, just={Root}
[(\enot B \eor C) \eor E, checked, name=p3, just={Root}
[\enot E, just={Root}
	[A, just={\eand}:p1
	[B, just={\eand}:p1
	[\enot C, just={\enot\eor}:p2
	[\enot D, just={\enot\eor}:p2
			[\enot B \eor C \ \ , just={\eor}:p3
			]
			[E, just={\eor}:p3]
		]
		]
	]
	]
]
]
]
]
\end{prooftree}\end{center}

These two branches represent different potential valuations that we are considering. We always read a branch from the bottom up. So both branches contain all the root formulas. Another way of thinking about branches in a truth tree is that we have several lists, and we've been too lazy to write the formulas in common more than once.

So now we have to consider our branches in turn. We start by examining the right branch, just because it's simpler -- it just contains an atomic formula, $E$. Notice, however, that line (4) was \enot $E$. So this branch's valuation requires both $E$ and $\enot E$ to be true; that's impossible. If a branch contains any formula and its negation, we know that branch represents an impossible valuation. We'll call this branch \emph{closed}, and mark it with an `\ensuremath\times'.

The left branch on (9) is another disjunction $\enot B \eor C $. It too branches out into its two disjuncts:

\begin{center}\begin{prooftree}
{close with=\ensuremath{\times}}
[A\eand B, checked, name=p1, just={Root}
[\enot(C\eor D), checked, name=p2, just={Root}
[(\enot B \eor C) \eor E, checked, name=p3, just={Root}
[\enot E, just={Root}
	[A, just={\eand}:p1
	[B, just={\eand}:p1
	[\enot C, just={\enot\eor}:p2
	[\enot D, just={\enot\eor}:p2
			[\enot B \eor C \ \ , checked, name=p9, just={\eor}:p3
				[\enot B, close, just={\eor}:p9]
				[C, close, just={\eor}:p9]
			]
			[E, close, just={\eor}:p3]
	]]]]
]]]]
\end{prooftree}\end{center}

Both of these disjuncts also result in closed branches. The left branch at (10) is the negation of (6), and the right branch is the negation of (7). Now every branch in this tree is closed. This corresponds to the idea that every potential way to make the formulas consistent ended up requiring a contradiction, so are impossible. There is no possible valuation that makes (1)-(4) all true. In other words, the formulas are mutually inconsistent.

\paragraph{Our second Truth Tree}

Let's work through another consistency example. We've seen most of the elements of a truth tree, so we can make this one simpler. we want to test whether the following formulas are mutually consistent in \tfl:

$$ D \eor G, \enot D, \enot G \eor S $$

We begin by writing down the root:

\begin{center}\begin{prooftree}
{close with=\ensuremath{\times}}
[D \eor G, just={Root}
[\enot D, just={Root}
[\enot G \eor S, just={Root}
]]]
\end{prooftree}\end{center}

Line (1) is a disjunction, so it will produce a branch:

\begin{center}\begin{prooftree}
{close with=\ensuremath{\times}}
[D \eor G, checked, name=p1, just={Root}
[\enot D, just={Root}
[\enot G \eor S, just={Root}
	[D, just={\eor}:p1
	]
	[G, just={\eor}:p1
	]
]]]
\end{prooftree}\end{center}

Line (2) is already atomic, and can't be decomposed. But it does help us to close one of the branches:

\begin{center}\begin{prooftree}
{close with=\ensuremath{\times}}
[D \eor G, checked, name=p1, just={Root}
[\enot D, checked, name=p2, just={Root}
[\enot G \eor S, just={Root}
	[D, close, just={\eor}:p1	]
	[G, just={\eor}:p1	]
]]]
\end{prooftree}\end{center}

Line (3) is another disjunction, and will branch again. Note that if we hadn't closed one of the branches, we would have to add branches to both of them, leading to four open branches! It is important to try to close branches as soon as you can, to stop the tree from getting too complex.

\begin{center}\begin{prooftree}
{close with=\ensuremath{\times}}
[D \eor G, checked, name=p1, just={Root}
[\enot D, checked, name=p2, just={Root}
[\enot G \eor S, checked, name=p3, just={Root}
	[D, close, just={\eor}:p1	]
	[G, just={\eor}:p1
		[\enot G, close, just={\eor}:p3	]
		[S, open, just={\eor}:p3	]
	]
]]]
\end{prooftree}\end{center}

We've closed one of these branches, but we can see that the last branch is still open, and that all the formulas above it either have a check mark, or are atomic. There is nothing more we can do; our tree is complete. We mark the open branch with an $\uparrow$. This represents the valuation we were seeking.

To read off the valuation (or truth table row, if you prefer), start at the arrow and read all the atomic and negated atomic symbols from the tree. In this case, we get: $S, G, \enot D$. So when $G=1$, $S=1$ and $D=0$, all the formulas are true; they are mutually consistent.

\pagebreak
\practiceproblems

\noindent\solutions
\problempart \label{pr.tfl.ttreeconcepts}
How well do you understand the idea of a truth tree? Try to explain each of the following concepts:

\begin{earg}
\item What is the Root of a truth tree?
\item When should you tick (\checkmark) a formula, and why?
\item What does a branch of a truth tree represent?
\item When should you create new branches of your tree?
\item When should you close a branch, and why?
\item When is an open branch complete? Why is this important?
\item What does the collection of atomic formulas in an open branch mean?
\end{earg}

\chapter{Planting your Tree}\label{ch:tfl.tree.planting}
\section{Selecting the Right Root}

Truth trees are used to indicate whether the root set of formulas is consistent. To use a truth tree to answer a question, the first step is always to convert the question into a question about whether some formula, or set of formulas, is mutually consistent.

Consider how this would apply to validity. An argument in \tfl\ is invalid iff the set of its premises and the negation of its conclusion are mutually consistent. It is valid iff this set of formulas is mutually inconsistent. So we can test for the validity of a \tfl\ argument using a truth tree.
If at least one branch of the tree is open, then the set is consistent, and we've found a valuation that makes the premises true and the conclusion false. The argument is invalid, and the valuation is a counter-example. If the tree closes, the argument is valid.

Consistency and validity are the two main notions we are interested in. But we've already looked at several other \tfl\ properties, including tautologies (logical truths), contradictions (logical falsehoods), and equivalences. These can all be tested with truth trees too.

For example, suppose you want to test if a formula is a tautology. To test this with a truth tree, we'll check if the formula can be false -- that is whether it is consistent for its \emph{negation} to be true. So we will put the negation of the formula in the root. If this truth tree remains open, it shows us how to make the negation true, so the formula is \emph{not} a tautology. If it closes, the negation is inconsistent (can't be true), so the formula \emph{is} a tautology.

%But trees can be used to evaluate many other kinds of questions, aside from validity. Any question that can be converted into a question about consistency can be answered with a tree. For example, if you want to find out whether a set of formulas is consistent, put that set of formulas into the tree. A set of formulas is consistent if and only if there is an valuation where they are all true, so you can use a tree to find out.

You should be able to devise methods for testing for contradictions and equivalences from our complete and partial truth table methods, and the definitions of these notions. Give this a go, and then test your trees by trying some of the Exercises from previous chapters.

%A pair of formulas in \tfl\ are \emph{not} equivalent iff there is a valuation on which they have different truth values (i.e., there is  a valuation where one of the formulas and the negation of the other is consistent). The trick is to guess which formula you negate!

%To find out whether a formula is a contradiction, check and see whether it is possible to make it true, by putting it in the root of the tree. If the tree remains open, the formula is consistent, which means it's not a contradiction. If the tree closes, then it is a contradiction.

\section{Growing your Tree}
\label{sec.TFLinvalidtree}

If you have been wondering how to decompose each formula in the tree, we'll introduce the formal rules for \tfl\ truth trees in \S\ref{ch:TFLtreerules}. In the meantime, we are just reading our truth tables backwards, as we did for partial truth tables. We figure out what subformulas have to be true for the current formula to be true, and write those sub-formulas down. If either of two subformulas being true is enough to make the current formula true, then we have a choice. Instead of making that choice, we branch the tree and put one option on each branch. This allows us to delay making our choice until we have more information.

Let's now test an argument for validity.  Consider this argument form:

$$(D\eor A) \eand \enot N,  N \eor \enot A \ \therefore \ \enot N \eand A$$

We are looking to see if a counter-example is possible; that is, if the premises are consistent with the conclusion being false. So the root of our tree will have the premises and the negation of the conclusion. We'll be looking for a valuation that makes all these formulas true. It's also useful to note what each part of the root comes from. We'll label the premises `Premise' and the negated conclusion `Neg Conc'. This also serves as a reminder to negate the conclusion!

\begin{center}\begin{prooftree}
{close with=\ensuremath{\times}}
[(D\eor A) \eand \enot N, just={Premise}
[N \eor \enot A, just={Premise}
[\enot (\enot N \eand A), just={Neg Conc}
]]]
\end{prooftree}\end{center}

Line (1) is a conjunction; it decomposes into its conjuncts on (4) and (5). Line (2) is a disjunction; it branches into its disjuncts on line (6), and then the left branch closes as $N$ and $\enot N$ cannot both be true.

\begin{center}\begin{prooftree}
{close with=\ensuremath{\times}}
[(D\eor A) \eand \enot N, checked, name=p1, just={Premise}
[N \eor \enot A,  checked, name=p2, just={Premise}
[\enot (\enot N \eand A), just={Neg Conc}
	[D \eor A, just={\eand}:p1
	[\enot N, just={\eand}:p1
		[N, close, just={\eor}:p2]
		[\enot A, just={\eor}:p2]
	]]
]]]
\end{prooftree}\end{center}

Line (3) is a negated conjunction. A conjunction is true iff both conjuncts are true, so it is false if at least one conjunct is false. As we don't know which conjunct has to be false, the tree branches on line (7), with one negated conjunct on each branch: \enot\enot$N$ and \enot$A$. The first of these branches then closes because it is the negation of line (5). The final formula to decompose is line (4)'s disjunction; it branches on line (8), and one branch closes as it is the negation of line (7).

The remaining formulas are all atomic, so cannot be decomposed.

\begin{center}\begin{prooftree}
{close with=\ensuremath{\times}}
[(D\eor A) \eand \enot N, checked, name=p1, just={Premise}
[N \eor \enot A, checked, name=p2, just={Premise}
[\enot (\enot N \eand A), checked, name=p3, just={Neg Conc}
	[D \eor A, checked, name=p4, just={\eand}:p1
	[\enot N, just={\eand}:p1
		[N, close, just={\eor}:p2]
		[\enot A, just={\eor}:p2
			[\enot \enot N, close, just={\enot\eand}:p3]
			[\enot A, just={\enot\eand}:p3
				[D, open, just={\eor}:p4]
				[A, close, just={\eor}:p4]
			]
		]
	]]
]]]
\end{prooftree}\end{center}

The $\uparrow$ indicates that the open branch ending in $D$ is \emph{completed}. (We'll define `complete' in \S\ref{ch:TFLtreerules}.) This branch represents the valuation we were seeking. This valuation is a counter-example, and so this argument is \emph{not} valid in \tfl. We can read the counter-example off the tree by starting at the arrow, reading upwards, and listing the atomic symbols: \enot $A$, $D$, and \enot $N$.

Our counter-example is: $A=\vF, D=\vT, N=\vF$. A partial truth table can prove the mutual consistency of the formulas in the root:\\

\noindent \begin{tabular}{ccc | cccccc | cccc | ccccccc }
$A$ & $D$ & $N$ &
$(D$ & $\eor $ & $A) $ & $\eand $ & $\enot$ & $ N$ &
$N$ & $ \eor$ & $ \enot $ & $A$ &
$\enot$ & $ (\enot $ & $N $ & $\eand $ & $A)$\\
\hline
\vT & \vT & \vF &
\vT & \vT & \vF & \gT & \vT & \vF &
\vF & \gT & \vT & \vF &
\gT & \vT & \vF & \vF & \vF\\
\end{tabular} $ $ \\

All three formulas are true -- the counter-example is possible.\\

On line (7) we wrote $\enot\enot N$. Some students want to write $N$ instead. But that's not what our rules say.
We also wrote $\enot A$ on both lines (6) and (7). That's absolutely fine. There's more than one piece of information that led us to the same formula. Again, don't try to out-think the rules and leave out seemingly redundant steps. It will just cause you confusion and error.

\section{Completing your Tree}

A tree is complete when there are no more formulas to decompose on any of its branches. This occurs when each branch has either been closed, or all its complex formulas have already been decomposed. By a complex formula, we mean any formula which is not just an atomic symbol or negated atomic symbol.

\subsection{Closing a Branch}

We \emph{close} a branch when it contains a formula along with its negation:

\factoidbox{
A branch is \define{closed} iff it contains both some formula \meta{A} and its negation \enot\meta{A}. We mark closed branches with the `\ensuremath{\times}' symbol.

A branch is \define{open} iff it is not closed.

A tree is \define{closed} iff every branch in that tree is closed.

A tree is \define{open} iff it is not closed.
}

If a tree is closed, we have shown that its root is inconsistent.

%We've closed one of these branches, but we can see that the last branch is still open, and that all the formulas above it either have a check mark, or are atomic. There is nothing more we can do; our tree is complete. We mark the open branch with an $\uparrow$. This represents the valuation we were seeking.

\subsection{Completing an Open Branch}
\label{sec.TFL.tree.completion}
An open branch is only complete when we've decomposed all its complex formulas. We then write an `$\uparrow$' at the bottom of that branch.

\factoidbox{
An open branch is \define{complete} iff all its complex formulas have been decomposed, as indicated by a check mark.

A tree is \define{complete} iff all its branches are closed or complete.}

If there is at least one open branch in a completed tree, then each open branch represents a valuation that makes all the formulas in the root true.

\subsection{Reading the Tree Leaves}

To read off the valuation from an open branch on a completed tree, start at the `$\uparrow$' arrow and follow the tree upwards, writing down all the atomic and negated atomic symbols from the tree. An atomic symbol means that atom is true; a negated atomic symbol means that atom is false. If you have both an atomic symbol and its negation in your list, close the branch.

This valuation will correspond to a case where all the formulas in the root are true. The mutual consistency of these formulas will mean different things depending on whether you are testing for consistency, validity, logical truth, logical falsehood, or equivalence, as we've already discussed.

\pagebreak
\practiceproblems

\noindent\solutions
\problempart \label{pr.tfl.root}
List the formulas in the root of your tree when testing whether:
\begin{earg}
\item Are $(p \eif q), (p \eand r), (\enot r \lor \neg q)$ consistent?
\item Are $(p \eor \neg q), (q \eor \neg r), (r \eor \neg p)$ consistent?
\item Is $(\neg p \to p)$ a logical falsehood?
\item Is $(((p\to q)\to p)\eand\enot q)$ a logical falsehood?
\item Is $(((p\to q)\to p)\to p)$ a logical truth?
\item Is $((q \eor \enot q)\eif p)$ a logical truth?
\item Are $(p \eiff q), (\enot (p \eand q) \eiff (\enot p \eand \enot q))$ equivalent?
\item Are $((p \eor q) \eand (q \eor r) \eand (r \eor p)), ((p \eand q) \eor (q \eand r) \eor (r \eand p))$ equivalent?
\item Are $(p \eiff q), (p \eiff \enot q)$ contradictory?
\item Are $(p \eand q), (\enot p \eand \enot q)$ contradictory?
\item Is $(p\to r), (q\to r)\ \therefore\ (p \eif q) \to r)$ valid?
\item Is $(p\to r), (q\to r)\ \therefore\ (p \eor q) \to r)$ valid?
\end{earg}

\problempart \label{pr.tfl.treetest}
Complete the trees from Exercise \ref{pr.tfl.root}. For each closed tree, state what you have proved.\\
\medskip

\problempart \label{pr.tfl.treeprove}
For each open tree from Exercise \ref{pr.tfl.treetest}, complete the partial truth table, and state what you have proved.

\chapter{\tfl\ Tree Rules}
\label{ch:TFLtreerules}

The examples we've worked through should have given you an overview of the tree method for \tfl. But we haven't precisely defined how to decompose each type of main connective. We are now ready to give formal rules for trees. You should be able to recognize the following rules as a generalization of the steps of the proofs given above.

Our \define{decomposition} rules describe how the tree can be extended by decomposing formulas. The rules depend on the main connective of the formula. (If the main connective is a negation, then they also depend on the main connective inside that negation.)

\label{TFL.treerules.start}
\section{Decomposition Rules}

The decomposition rule for conjunction is that if you have a conjunction in a branch, you may make a linear extension of the branch that includes each conjunct, adding a check mark next to the conjunction. %Using our Greek symbols once again as variables to stand in for any formula of \tfl,
So, any time you have a conjunction $\meta{A}\eand\meta{B}$ on line ($i$) you may extend that branch of the tree by writing:

\begin{center}\begin{prooftree}
{close with=\ensuremath{\times},
 not line numbering}
%[\meta{A}\eand\meta{B}, checked, name=p1
	[\meta{A} , just={$i$ \eand}
	[\meta{B} , grouped, just={$i$ \eand}
	]]
%]
\end{prooftree}\end{center}

It is important to remember once again that \meta{A} and \meta{B} here can stand for \emph{any} formula of \tfl, including complex ones. The formulas must be copied exactly as written, including any negation signs. Also, write the line \# of the decomposed formula, and which decomposition rule you used, on the new line(s). This will help you when you check your work.
You do check your work, right?

%We won't have such detailed explanations for the remaining decomposition rules.

\pagebreak

\subsection{Conjunction}
\noindent Our conjunction decomposition rule is:

\factoidbox{
	\begin{center}\begin{prooftree}
	{close with=\ensuremath{\times},
	 not line numbering}
	[\meta{A}\eand\meta{B}, checked
		[\meta{A} , just={$i$, \eand}
		[\meta{B} , just={$i$, \eand}
		]]
	]
	\end{prooftree}\end{center}
}

\subsection{Negated conjunction}

A negated conjunction, branches into the negation of each conjunct. This is because there are two ways for a negated conjunct to be true -- either conjunct can be false:

\factoidbox{
	\begin{center}\begin{prooftree}
	{close with=\ensuremath{\times},
	 not line numbering}
	[\enot(\meta{A}\eand\meta{B}), checked
		[\enot\meta{A} , just={$i$, \enot\eand}]
		[\enot\meta{B} , just={$i$, \enot\eand}]
	]
	\end{prooftree}\end{center}
}

\subsection{Disjunction}
\label{subsec.DisjunctionTreeRule}
%\begin{groupitems}
Disjunctions branch into each disjunct:

\factoidbox{
	\begin{center}\begin{prooftree}
	{close with=\ensuremath{\times},
	 not line numbering}
	[\meta{A}\eor\meta{B}, checked
		[\meta{A} , just={$i$, \eor}]
		[\meta{B} , just={$i$, \eor}]
	]
	\end{prooftree}\end{center}
}

\subsection{Negated disjunction}

Since disjunctions are true any time either disjunct is true, they are only false if both disjuncts are false; that is, their negations are both true:

\factoidbox{
	\begin{center}\begin{prooftree}
	{close with=\ensuremath{\times},
	 not line numbering}
	[\enot(\meta{A}\eor\meta{B}), checked
		[\enot\meta{A} , just={$i$, \enot\eor}
		[\enot\meta{B} , just={$i$, \enot\eor}
		]
		]
	]
	\end{prooftree}\end{center}
}

\subsection{Conditional}

Conditionals are true when the antecedent is false \emph{or} the consequent is true. This means that conditionals in \tfl\ are treated similarly to disjunctions:

%\begin{center}
%\begin{tabular}{c c|c}
%\meta{A} & \meta{B} & $\meta{A}\eif\meta{B}$\\
%\hline
%\vT & \vT & \vT\\
%\vT & \vF & \vF\\
%\vF & \vT & \vT\\
%\vF & \vF & \vT
%\end{tabular}
%\end{center}
%
%We can represent this with a branching tree, similar to a disjunction.

\factoidbox{
	\begin{center}\begin{prooftree}
	{close with=\ensuremath{\times},
	 not line numbering}
	[\meta{A}\eif\meta{B}, checked
		[\enot\meta{A} , just={$i$, \eif}]
		[     \meta{B} , just={$i$, \eif}]
	]
	\end{prooftree}\end{center}
}

\subsection{Negated conditional}

The negation of a conditional is true when the conditional is false. That is, when its antecedent is true \emph{and} its consequent is false:

\factoidbox{
	\begin{center}\begin{prooftree}
	{close with=\ensuremath{\times},
	 not line numbering}
	[\enot(\meta{A}\eif\meta{B}), checked
		[     \meta{A} , just={$i$, \enot\eif}
		[\enot\meta{B} , just={$i$, \enot\eif}
		]]
	]
	\end{prooftree}\end{center}
}

\subsection{Double negation}

A doubly-negated formula is true iff the singly-negated formula is false iff the original formula is true. So we can simply remove a double negation. However, this has to be done as a separate step:

\factoidbox{
	\begin{center}\begin{prooftree}
	{close with=\ensuremath{\times},
	 not line numbering}
	[\enot\enot\meta{A}, checked
		[\meta{A} , just={$i$, \enot\enot}]
	]
	\end{prooftree}\end{center}
}

\logic{You may have noticed that the decomposition rules for the standard connectives and their negations are duals of each other -- if one branches, the other doesn't, if one negates \meta{A}, the other does not, and so forth. This means that if you know the decomposition rule for a connective, you also know the rule for its negation. Just do everything the opposite way!}

\subsection{Biconditional}

Biconditionals are really just a combination of two conditionals. That is, $\meta{A}\eiff\meta{B}$ iff $(\meta{A}\eif\meta{B}) \eand (\meta{B}\eif\meta{A})$. If we decompose this formula, we get:

\begin{center}\begin{prooftree}
{close with=\ensuremath{\times}}
[(\meta{A}\eif\meta{B}) \eand (\meta{B}\eif\meta{A}), checked , name=p1
	[\meta{A}\eif\meta{B} , checked , name=p2, just={\eand}:p1
	[\meta{B}\eif\meta{A} , checked , name=p3, just={\eand}:p1
		[\enot\meta{A} , just={\eif}:p2
			[\enot\meta{B}        , just={\eif}:p3]
			[     \meta{A} , close, just={\eif}:p3]
		]
		[\meta{B} , just={\eif}:p2
			[\enot\meta{B} , close, just={\eif}:p3]
			[     \meta{A}        , just={\eif}:p3]
		]
	]]
]
\end{prooftree}\end{center}

One open branch has \meta{A} and \meta{B}; the other \enot\meta{A} and\enot\meta{B}.  This might seem odd, but recall that a biconditional is true when its sub formulas have the same truth value: either both are true, or both are false. And that's exactly what this rule states:

\factoidbox{
	\begin{center}\begin{prooftree}
	{close with=\ensuremath{\times},
	 not line numbering}
	[\meta{A}\eiff\meta{B}, checked
		[\meta{A}, just={$i$, \eiff}
		[\meta{B}, just={$i$, \eiff}
		]]
		[\enot\meta{A}
		[\enot\meta{B}
		]]
	]
	\end{prooftree}\end{center}
}

\subsection{Negated biconditional}
%\begin{groupitems}
Negated biconditionals are true when the subformulas have different truth values. The tree rule looks remarkably similar to the standard biconditional:

\factoidbox{
	\begin{center}\begin{prooftree}
	{close with=\ensuremath{\times},
	 not line numbering}
	[\enot(\meta{A}\eiff\meta{B}), checked
		[     \meta{A}, just={$i$, \enot\eiff}
		[\enot\meta{B}, just={$i$, \enot\eiff}
		]]
		[\enot\meta{A}
		[     \meta{B}
		]]
	]
	\end{prooftree}\end{center}
}

You can derive this rule too, by creating a tree for the negation of a conjunction of conditionals:  $\enot((\meta{A}\eif\meta{B}) \eand (\meta{B}\eif\meta{A}))$.

%\section{Decomposition rule summary}
%These nine decomposition rules describe, in purely formal terms, how to deal with almost any \tfl\ formula that appears in a tree.

\section{Truth Tree Complexities}
%These decomposition rules work on almost every formula. The only formulas they don't apply to are atomic symbols and negated atomic symbols. Atoms and negated atoms cannot be decomposed for further information. Instead they are used to describe the valuation represented by an open branch.

%When creating a truth tree, you may apply any of these decomposition rules to any  formula (other than an atom or negated atom) in the branch. Mark decomposed formulas with a check mark.

\subsection{Incomplete Valuations}

Sometimes our open branch will not contain every atom in our set of formulas. That is, our valuation does not specify the truth value of some atomic symbols. When this occurs, the  truth value of the remaining atomic symbols does not affect the consistency of the formulas. However, a valuation must specify the truth value of every symbol, so you can arbitrarily pick some truth value for each, or indicate its truth value is unknown by a `?'.

For example, suppose an open branch for a set of formulas using the symbols $\{p,q,r\}$ contains only $p$ and $\neg q$. This valuation would be: $p$ is true, $q$ is false, $r$ is unknown; or in symbols: $p = 1, q=0, r=?$.

\subsection{Early Closing}
Note that we close a branch when it contains \emph{any} formula and its negation. We don't need an atomic formula and its negation. We saw this in an early example when we closed a branch containing \enot\enot$N$ and \enot$A$. Here is another example:

\begin{center}\begin{prooftree}
{close with=\ensuremath{\times}}
[(\enot S \eand T) \eif (\enot P \eiff (Q \eor R)), checked
[\enot S \eand T, name=p2
[\enot (\enot P \eiff (Q\eor R)),  name=p3
	[\enot(\enot S \eand T), close %close={:p2,!c}
	]
	[\enot P \eiff (Q \eor R), close %close={:p3,!c}
	]
]]]
\end{prooftree}\end{center}

In this tree, we only decomposed the first formula -- using the conditional rule -- and then the tree immediately closed. The left branch of (4) is the negation of (2), and the right branch is the negation of (3). Notice that we \emph{could} have ignored (or missed) this, and decomposed lines (2) and (3) using the conjunction or negated biconditional rules, respectively. However, this would have resulted in a more complicated tree that would also have eventually closed. Missing an early closing opportunity will never affect the final result of a truth tree, just create more work and a higher chance of error.
%Similarly, if we hadn't noticed that both branches close at line (4), we could have continued decomposing the tree, working on the left branch using the negated conjunction rule on (4), and then the right branch using the biconditionals rule. But this too would have involved needless complication for no benefit. (Eventually, the tree would have closed with simpler formulas too.)
You can occasionally save yourself a lot of effort by noticing when branches are ready to mark as closed. However, if you don't notice this early closing, no real harm will be done. %, and by thinking a bit strategically about which formulas to decompose first.

\subsection{Multiple Open Branches}
If you are decomposing a formula after the tree has already branched, you must add the new formulas under \emph{each} open branch that descends from that formula. Here is a tree illustrating how to do this:

\begin{center}\begin{prooftree}
{close with=\ensuremath{\times}}
[(D \eand \enot R) \eor Q, name={p1}, checked
[\enot Q \eor R, checked, name={p2}
	[D \eand \enot R, just={\eor}:p1, name={p3}, checked
		[\enot Q, just={\eor}:p2
			[D, just={\eand}:p3
				[\enot R, open, just={\eand}:p3]
			]
		]
		[R, name={R}
			[D, just={\eand}:p3
				[\enot R, close, just={\eand}:p3] %close={:R,!c}]
			]
		]
	]
	[Q
		[\enot Q, close, just={\eor}:p2 ] %, close={:!u,!c}
		[R, open, just={\eor}:p2]
	]
]]
\end{prooftree}\end{center}

This tree has two disjunctions in the root, so it will branch multiple times. We decomposed line (1) first, branching into the two disjuncts at line (3). When we decomposed (2), both branches descending from (2) were still open, so we added line (4) to both these branches. That's why our two branches split into four at line (4). The third branch contains $Q$ and $\enot Q$, and so closes. At line (5), the conjunction $D \eand \enot R$ at line (3) is decomposed. Only the left-most 2 of the 3 open branches descend from  this conjunction. So the conjuncts need to be added to both branches that descend from that conjunction, but not the right-most branch, which does not. %Decomposing a formula applies to every one of its open descendent branches, but not to branches that are off to the side.

\subsection {Multiple Valuations}

A tree can have more than one completed open branch. The tree above is an example of this. These completed open branches can have different valuations. This is because there can be more than one counter-example, or row that doesn't satisfy the property you are testing. It is fine to use any completed open branch. However, be careful.

First, don't combine your open branches. In the tree above, one valuation has $Q=1, R=1$, and the other $Q=0, R=0$. Any combination of these would fail to be a valuation where all the formulas in the root are true.

Second, indicate clearly which open branch your counter-example comes from. We suggest that you only use the symbol $\uparrow$ on the completed branch which you are reading your valuation from. Leave the other open branches blank.

\pagebreak

\section{Order of Decomposition}
You do \emph{not} need to decompose formulas in strict order from the top of the tree. You can decompose formulas in whichever order you find convenient. Some ways of decomposing trees are more efficient than others. The most basic principle is to decompose formulas that won't branch before those that will. When all the remaining complex formulas will branch, try to look a step ahead, and select a formula that will have a branch that closes immediately. This reduces the likelihood of having multiple open branches. Next, decompose simple formulas before complex ones, as you are less likely to muck this up. Finally, I suggest you decompose biconditionals last, as they have the most complex rules of all.

\begin{earg}
\item Non-branching rules before branching.
\item Close branches quickly.
\item Simple formulas before complex.
\item Biconditionals last.
\end{earg}

Suppose we want to test whether $\enot (C \eand A), D \eiff C, A \eor B, \enot B$ are mutually consistent. These formulas would then form the root of our tree. Here's the tree formed if we decompose the formulas in the order in which they're listed:

\begin{center}\begin{prooftree}
{close with=\ensuremath{\times}}
[\enot (C \eand A), checked, name=p1
[D \eiff C, checked, grouped, name=p2
[A \eor B, grouped, checked, name=p3
[\enot B, grouped, name=p4
	[\enot C, just={\enot\eand}:p1
		[D, just={\eiff}:p2
		[C, grouped, close]
		]
		[\enot D
		[\enot C
			[A, open,  just={\eor}:p3]
			[B, close, just={\eor}:p3]
		]]
	]
	[\enot A
		[D , just={\eiff}:p2
		[C , just={\eiff}:p2
			[A, close]
			[B, close]
		]]
		[\enot D
		[\enot C
			[A, close]
			[B, close]
		]]
	]
]]]]
\end{prooftree}\end{center}

This completed tree is open. We can read off the valuation from its one open branch. The atomic and negated atomic symbols on this open branch are: $A$, $\enot C$, $\enot D$, $\enot C$, $\enot B$. This tells us that a valuation where all the formulas are true is $A=1, B=0, C=0, D=0$.

This tree suffers from the problem of having multiple open branches during its construction. That's why, for example, decomposing line (3) at line (8) requires three different new branchings. So when there are more open branches, decomposing formulas requires more work on the tree, more repetition, and more chance of errors.

The tree above is a perfectly fine completed tree, and it gives a correct answer. However, it's possible to get there with less work, by choosing to decompose formulas that will close off branches right away. Here is another tree with the same root as in the previous example, but which decomposes formulas in a more efficient order:

\begin{center}\begin{prooftree}
{close with=\ensuremath{\times}}
[\enot (C \eand A), checked, name=p1
[D \eiff C,         checked, name=p2
[A \eor B,          checked, name=p3
[\enot B,                    name=p4
	[A, just={\eor:p3}
		[\enot C,      just={\enot\eand:p1}
			[D,        just={\eiff:p2}
			[C, close, just={\eiff:p2}
			]]
			[\enot D
			[\enot C, open
			]]
		]
		[\enot A, close]
	]
	[B, close]
]]]]
\end{prooftree}\end{center}

We can use a partial truth table to confirm our answer:\\

\noindent \begin{tabular}{cccc | cccc | ccc | ccc | cc}
$A$ & $B$ & $C$ & $D$ &
$\enot$ & $(C$ & $\eand$ & $A)$ &
$(D$& $\eiff$ & $C)$ &
$A$ & $\eor$  & $B$  &
$\enot$ & $B$\\
\hline
\vT & \vF & \vF & \vF &
\gT & \vF & \vF & \gT &
\vF & \gT & \vF &
\vT & \gT & \vF &
\gT & \vF\\
\end{tabular} $ $ \\

%All three formulas are true -- the counter-example is possible.\\

This tree gets us to the same result much more quickly, and generates the same valuation. Here's a description of the reasoning for deciding which formula to decompose at each step:

\begin{earg}
\item[5.] Line (4) contains a $\enot B$. Line (3) would generate a branch with $B$.
\item[6.] Line (5) contains an $A$. Line (1) would generate a branch with $\enot A$.
\item[7.] Line (6) contains a $C$. Line (2) would generate a branch with $\enot C$.
\end{earg}

You usually don't need to think more than one line ahead to select branches that create a much simpler tree.

%Trees require practice to understand which formulas to put in the root for which test, to reliably decompose formulas correctly, to interpret the result of a closed or open tree, to read of the valuation from an open branch, and to create efficient trees by selecting the order to decompose formulas. Practice each of these steps in turn, and don't be disheartened if you don't master all of them at your first attempt.

\pagebreak
\practiceproblems

\noindent\solutions
\problempart \label{pr.tfl.treeroot}
To evaluate each of the following claims with a tree, (a) what would you put in the root of the tree?, and (b) if the tree closes, does that show that the claim is true or false?
\begin{earg}
\item $P, P \eif Q, Q \eif \enot P$ is mutually consistent
%(a) $\{P, P \eif Q, Q \eif \enot P\}$, (b) true
\item $(P \eif Q) \eiff (Q \eif P)$ is a tautology.
%(a) $\enot((P \eif Q) \eiff (Q \eif P))$, (b) true
\item The following argument is valid:
 $P \eand Q$,  $\enot R \eif \enot Q$ $\ \therefore \ $ $P \eand R$

%(a) $\{P \eand Q, \enot R \eif \enot Q, \enot (P \eand R)\}$, (b) true
\item Every valuation making $P$, $P \eif Q$, and $\enot Q$ true also makes $A$ true.
%(a) $\{P, P \eif Q, \enot Q, \enot A\}$
\item There is no valuation that makes $A \eor B$, $B \eif C$, and $A \eiff C$ all true but C false.
%$\{A \eor B, B \eif C, A \eiff C, \enot C\}, (b) true
\item $A \eiff \enot A$ is a contradiction.
%(a) $A \eiff \enot A$, (b) false
\item There is at least one valuation that makes $P \eif Q$, $\enot P \eor \enot Q$, and $Q \eif P$ true.
%(a) $\{P \eif Q, \enot P \eor \enot Q, Q \eif P\}, (b) false.
 \end{earg}

\noindent\solutions
\problempart \label{pr.tfl.trees}
Evaluate each claim from Exercise \ref{pr.tfl.treeroot} by constructing a tree. If applicable, give the valuation that demonstrates the claim true or false, and use a partial truth table to confirm this.\\

\noindent\solutions
\problempart \label{pr.tfl.agtree}
Evaluate the argument
  $$A\eiff B, \enot B \eif (C \eor D), E \eif \enot C, (\enot D \eand F) \eor G, \enot A \eand E, \ \therefore \ H \eor G$$
by constructing a tree. If the tree is open, prove your counter-example is correct using a partial truth table.\\

\noindent\problempart The Sheffer stroke \meta{A} $|$ \meta{B} is a rarely-used connective of \tfl. It has the following characteristic truth table:

\begin{center}
\begin{tabular}{c|c|c}
\meta{A} & \meta{B} & \meta{A} $|$ \meta{B}\\
\hline
1 & 1 & 0\\
1 & 0 & 1\\
0 & 1 & 1\\
0 & 0 & 1
\end{tabular}
\end{center}

Create appropriate tree decomposition rules for the Sheffer stroke, and for the negated Sheffer stroke.\\

\noindent \problempart
Construct trees for any of the exercises from the complete and partial truth tables chapters. Check the valuations from any open branches are correct using partial truth tables.

\end{document}


%----------------------------------------------------------------------------------------
%	PACKAGES AND OTHER DOCUMENT CONFIGURATIONS
%----------------------------------------------------------------------------------------
\documentclass[paper=a4, fontsize=11pt]{scrartcl} % A4 paper and 11pt font size

\usepackage[T1]{fontenc} % Use 8-bit encoding that has 256 glyphs
\usepackage{fourier} % Use the Adobe Utopia font for the document - comment this line to return to the LaTeX default
\usepackage[english]{babel} % English language/hyphenation
\usepackage{amsmath,amsfonts,amsthm,amssymb} % Math packages
\usepackage{mathrsfs}
\usepackage{algorithm, algorithmic}
\renewcommand{\algorithmicrequire}{\textbf{Input:}} %Use Input in the format of Algorithm
\renewcommand{\algorithmicensure}{\textbf{Output:}} %UseOutput in the format of Algorithm
\usepackage{listings}
\lstset{language=Matlab}
\usepackage{enumerate}
\usepackage{graphicx}

\usepackage{lipsum} % Used for inserting dummy 'Lorem ipsum' text into the template
\usepackage{sectsty} % Allows customizing section commands
\allsectionsfont{\centering \normalfont\scshape} % Make all sections centered, the default font and small caps
\usepackage{fancyhdr} % Custom headers and footers
\pagestyle{fancyplain} % Makes all pages in the document conform to the custom headers and footers
\fancyhead{} % No page header - if you want one, create it in the same way as the footers below
\fancyfoot[L]{} % Empty left footer
\fancyfoot[C]{} % Empty center footer
\fancyfoot[R]{\thepage} % Page numbering for right footer
\renewcommand{\headrulewidth}{0pt} % Remove header underlines
\renewcommand{\footrulewidth}{0pt} % Remove footer underlines
\setlength{\headheight}{13.6pt} % Customize the height of the header

\numberwithin{equation}{section} % Number equations within sections (i.e. 1.1, 1.2, 2.1, 2.2 instead of 1, 2, 3, 4)
\numberwithin{figure}{section} % Number figures within sections (i.e. 1.1, 1.2, 2.1, 2.2 instead of 1, 2, 3, 4)
\numberwithin{table}{section} % Number tables within sections (i.e. 1.1, 1.2, 2.1, 2.2 instead of 1, 2, 3, 4)

\setlength\parindent{0pt} % Removes all indentation from paragraphs - comment this line for an assignment with lots of text

%----------------------------------------------------------------------------------------
%	TITLE SECTION
%----------------------------------------------------------------------------------------
\newcommand{\horrule}[1]{\rule{\linewidth}{#1}} % Create horizontal rule command with 1 argument of height

\title{
\normalfont \normalsize
\textsc{Shanghai Jiao Tong University, UM-SJTU JOINT INSTITUTE} \\ [25pt] % Your university, school and/or department name(s)
\horrule{0.5pt} \\[0.4cm] % Thin top horizontal rule
\huge Introduction to Numerical Analysis \\ HW9 \\ % The assignment title
\horrule{2pt} \\[0.5cm] % Thick bottom horizontal rule
}

\author{Yu Cang \\ 018370210001} % Your name

\date{\normalsize \today} % Today's date or a custom date

\begin{document}

\maketitle % Print the title

\section{Question 1}
	\begin{enumerate}[(a)]
		\item
			\begin{proof}
				As $\mathcal{A}$ is convex, the mean value theorem can be applied in terms of $y$ s.t.
				\begin{equation}
					\Phi(t, y_2) - \Phi(t, y_1) = (y_2 - y_1) \frac{\partial \Phi(t, y)}{\partial y}\Big|_{y=\xi}
				\end{equation}
				where $\xi \in (y_1, y_2)$.\\
				Since the there exists $c>0$ s.t. for all $(t, y)\in \mathcal{A}$
				\begin{equation}
					\Big| \frac{\partial \Phi(t, y)}{\partial y} \Big| \leq c
				\end{equation}
				Thus
				\begin{equation}
					\begin{aligned}
						& |\Phi(t, y_2) - \Phi(t, y_1)|\\
					  = & |y_2 - y_1| \Bigg| \frac{\partial \Phi(t, y)}{\partial y}\Big|_{y=\xi} \Bigg|\\
				   \leq & c |y_2 - y_1|
					\end{aligned}
				\end{equation}
				which implies that $\Phi(t, y)$ satisfies Lipschitz condition in $y$ on $\mathcal{A}$.
			\end{proof}

		\item
			\begin{proof}
				Let $P_1 = (t_1, y_1)$ and $P_2 = (t_2, y_2)$. Then any point $P'$ lies on the line segment joining $P_1$ and $P_2$ can be expressed as
				\begin{equation}
					\begin{aligned}
						P & = (1-\alpha)P_1 + \alpha P_2\\
						  & = ((1-\alpha)t_1 + \alpha t_2, (1-\alpha)y_1 + \alpha y_2)\\
						  & \triangleq (t', y')
					\end{aligned}
				\end{equation}
				where $\alpha \in [0, 1]$.\\
				It's clear that $t_1$ and $t_2$ lies between $t_0$ and $T$, and $t'$ lies between $t_1$ and $t_2$. Thus, $t'$ also lies between $t_0$ and $T$.\\
				Further, it's also clear that $-\infty < y' < +\infty$.\\
				Hence $P \in \mathcal{D}$, which implies that $\mathcal{D}$ is convex.
			\end{proof}

		\item
			\begin{proof}
				Let
				\begin{equation}
					\Phi(t, y) = \frac{4t^3y}{1+t^4}
				\end{equation}
				Then
				\begin{equation}
					\frac{\partial \Phi(t, y)}{\partial y} = \frac{4t^3}{1+t^4} =\frac{4 t}{t^2 + \frac{1}{t^2}} < \frac{4}{t^2 + \frac{1}{t^2}} < \frac{4}{2 \sqrt{t^2 \frac{1}{t^2}}} = 2
				\end{equation}
				as $t\in (0,1)$.\\
				which implies that $\Phi(t, y)$ satisfies a Lipschitz condition in $y$.\\
				Thus, the given IVP problem has a unique solution.
			\end{proof}

		\item
			Definitely not recommended.\\
			As $\Phi(t, y) = 1 + y^2$, then
			\begin{equation}
				\frac{\partial \Phi}{\partial y} = 2 y \triangleq \lambda y
			\end{equation}
			Here $\lambda > 0$, and the Euler's method is not stable as the error will be amplified at each iteration step. Finally the calculation will diverge.
	\end{enumerate}

\section{Question 2}
	\begin{enumerate}[(a)]
		\item
			As $\Phi(t, y) = arctan(y)$, then
			\begin{equation}
				\Bigg| \frac{\partial \Phi}{\partial y} \Bigg| = \frac{1}{|1+y^2|} < 1 \triangleq c
			\end{equation}
		\item
			As $\dot{y} = \Phi(t, y) = arctan(y)$, then
			\begin{equation}
				|\ddot{y}| = \Bigg| \frac{\dot{y}}{1+y^2} \Bigg| = \frac{|arctan(y)|}{1+y^2} < \frac{\pi}{2(1+y^2)} \leq \frac{\pi}{2}
			\end{equation}
		\item
			As the global error is bounded by
			\begin{equation}
				|e_k| \leq \frac{\tau^*}{hc}[e^{c(t_k - t_0)} - 1]
			\end{equation}
			where $\tau^* = \max\limits_{k}|\tau_k|$, $c$ is the Lipschitz constant and is taken as $1$ as has been illustrated above.\\
			Since
			\begin{equation}
				\begin{aligned}
					\frac{\tau^*}{h} & = \max\limits_{k}\Bigg|\frac{y(t_k) - y(t_{k-1}) - h\Phi(t_{k-1}, y(t_{k-1}))}{h}\Bigg|\\
									 & \leq \max\limits_{k}\Bigg| \frac{y(t_k) - y(t_{k-1})}{h}\Bigg| + |\Phi(t_{k-1}, y(t_{k-1}))|\\
									 & = 2 \max\limits_{k} |\Phi(t_{k-1}, y(t_{k-1}))| < \pi
				\end{aligned}
			\end{equation}
			Thus, $|e_k| < \pi[e^{(t_k - t_0)}-1]$.
	\end{enumerate}

\section{Question 3}
	\begin{enumerate}[(a)]
		\item
			With $y_0 = 4$, $y_k$ are calculated using Euler's method as follows
			\begin{equation}
				y_k = y_{k-1} + h \Phi(t_{k-1}, y_{k-1})
			\end{equation}
			The calculation is carried out with matlab, and the results are given as below
			\begin{figure}[!htbp]
				\centering
				\includegraphics[width=8cm]{../pic/Q3_a.eps}
			\end{figure}
		\item
			The initial value of $y_k^{(0)}$ is calculated using Euler's method.\\
			Then $y_k^{(i)}$ are calculated in $i-th$ iteration, and the back-ward Euler's method is adopted.
			\begin{equation}
				y_k^{(i)} = y_{k-1}^{(i)} + h \Phi(t_{k}, y_k^{(i-1)})
			\end{equation}
			The calculation is carried out with matlab, and the results are given as below
			\begin{figure}[!htbp]
				\centering
				\includegraphics[width=8cm]{../pic/Q3_b.eps}
			\end{figure}
		\item
			Iteration formula is given as
			\begin{equation}
				y_k = y_{k-1} + h \Phi(t_{k-1}, y_{k-1}) + \frac{h^2}{2}[\Phi_t(t_{k-1}, y_{k-1}) + \Phi_y(t_{k-1}, y_{k-1}) \Phi(t_{k-1}, y_{k-1})]
			\end{equation}
			The calculation is carried out with matlab, and the results are given as below
			\begin{figure}[!htbp]
				\centering
				\includegraphics[width=8cm]{../pic/Q3_c.eps}
			\end{figure}
		\item
			The calculation is carried out with the prediction-correction method. Iteration formulas are given as
			\begin{equation}
				y^* = y_{k-1} + h \Phi(t_{k-1}, y_{k-1})
			\end{equation}
			\begin{equation}
				y_k = y_{k-1} + \frac{h}{2} [\Phi(t_{k-1}, y_{k-1}) + \Phi(t_k, y^*)]
			\end{equation}
			The calculation is carried out with matlab, and the results are given as below
			\begin{figure}[!htbp]
				\centering
				\includegraphics[width=8cm]{../pic/Q3_d.eps}
			\end{figure}
		\item
			Here, an extra initial point($y_1$) is required, and it is calculated using the Euler's method as
			\begin{equation}
				y_1 = y_0 + h \Phi(t_0, y_0)
			\end{equation}
			Then, the iteration steps are taken as
			\begin{equation}
				y_k = y_{k-1} + \frac{h}{2}[3\Phi(t_{k-1}, y_{k-1}) - \Phi(t_{k-2}, y_{k-2})]
			\end{equation}
			The calculation is carried out with matlab, and the results are given as below
			\begin{figure}[!htbp]
				\centering
				\includegraphics[width=8cm]{../pic/Q3_e.eps}
			\end{figure}

	\end{enumerate}

\end{document}


\documentclass[12pt]{article}
\input{physics1}
\begin{document}

\section*{NYU Physics I---pulleys and blocks}

The instructor will draw two machines on the board. The first machine
consists of three blocks, of masses 7, 4, and $3\,\kg$, and two
pulleys (and three strings). The second machine consists of two
blocks, of masses 4 and $3\,\kg$, and one pulley (and two strings).

Choose a partner and work in pairs. Each member of each pair should
write down---on paper---their answers to each part of this worksheet.
You don't have to agree with your partner, but you must discuss and
understand one another.

\paragraph{\theproblem}\refstepcounter{problem}%
In the three-block machine, do you expect the $7\,\kg$ block to
accelerate upwards, or downwards, or not accelerate? You might have a
quick answer to this and want to move on, but talk it out. Consider
extreme scenarios (that is, consider changing the masses of the 3 and
$4\,\kg$ blocks). Are you \emph{sure} about your prediction?

\paragraph{\theproblem}\refstepcounter{problem}%
For the two-block machine, draw free-body diagrams for each of the two
blocks, and for the pulley. Make sure you can justify what you have
drawn. Make sure you are happy with your free-body diagram for the
pulley.

\paragraph{\theproblem}\refstepcounter{problem}%
We are going to assume that the strings in this problem are
inextensible. Why are we going to assume that? What does it do for us?

\paragraph{\theproblem}\refstepcounter{problem}%
We are going to assume that the strings and pulleys are massless. What
does that do for us? How does that help us?

\paragraph{\theproblem}\refstepcounter{problem}%
We are going to assume that the pulleys are frictionless. How does
\emph{that} help us?

\paragraph{\theproblem}\refstepcounter{problem}%
What is the relationship between the accelerations of the two blocks?
Set up a coordinate system and describe this relationship with an
equation.

\paragraph{\theproblem}\refstepcounter{problem}%
Put it all together and solve for the accelerations for the two
blocks, and the tensions in the two strings.

\paragraph{\theproblem}\refstepcounter{problem}%
What is the tension in the top string, and how does it compare to the
total mass times the acceleration due to gravity? Does this change
your thinking about the first question on this worksheet?

\end{document}


\documentclass{article}
\usepackage[utf8]{inputenc}
\usepackage[backend=biber]{biblatex}
\usepackage{amssymb}
\usepackage{amsmath}
\usepackage{dsfont}
\addbibresource{bib.bib}
\setlength{\parindent}{0em}
\bibliography{bib}
\setlength{\parskip}{6pt}
\usepackage[margin=1.0in]{geometry}
\usepackage{graphicx}
\usepackage{caption}
\usepackage{subcaption}
\usepackage{wrapfig}
\usepackage{url}

\title{Intro to deep learning with PyTorch}
\author{Miguel A. Saavedra-Ruiz}
\date{May 2020}
\linespread{1.0}

\nocite{*}

\begin{document}

\maketitle

\section*{Generative Adversarial Networks (GAN)}

Generative Adversarial Networks (GAN) are an architecture derived from fully connected layers and convolutional neural networks. This sort of networks have the capability to generate new samples similar to the data they were trained on. An example of a GAN would be a generate faces after being trained on a large data-set of real faces. The results of such example can be seen at Fig. \ref{fig:f1}

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.65\textwidth,height=0.65\textheight,keepaspectratio]{images/face_generator.png}
    \captionsetup{justification=centering}
    \caption{Generation of faces with GAN}
    \label{fig:f1}
\end{figure}

To understand the general idea of a GAN network it used the concept of two networks, a generator (G) and a discriminator (D). These two networks compete against each other. A simple way to understand a GAN network is use the analogy of the forger and the detective. Imagine a forger trying to forge paintings and make them real enough that the detective won't notice. Here it is possible to think about the forger as a generator and the detective as the discriminator Fig. \ref{fig:f2}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.7\textwidth,height=0.7\textheight,keepaspectratio]{images/gan.png}
    \captionsetup{justification=centering}
    \caption{GAN architecture}
    \label{fig:f2}
\end{figure}

The generator is represented with a blue box and the discriminator with the orange one.
The idea behind this architecture is that the generator makes fake data to pass to the discriminator. On the other hand, the discriminator also sees real data such as real images and it predicts if the data received is real or fake. The generator is trained to try to fool the discriminator by making its output data to look as close as possible to real data and the discriminator is trained to figure out which data is real and which data is fake. The final result is the generator learns to make data that is indistinguishable from real data to the discriminator Fig. \ref{fig:f2}.

Latent data is basically random noise that is feed into the generator as an input and this tries to produce an image as similar as possible to the real data to fool the discriminator. Overall, the latent sample is a random vector that the generator uses to construct its fake images. This is often called a latent vector and that vector space is called latent space. As the generator trains, it figures out how to map latent vectors to recognizable images that can fool the discriminator.

GANs are not only used in image data, they can also be used to generate audio data.

Training a GAN is a relatively simple process since it is essentially create two separate networks, one for the discriminator and other for the generator. Nevertheless, some of the drawbacks of GANs is the tuning of hyper-parameters and the training time involved.
Some other problems are when the discriminator is overpowering the generator. This would generrate that the discriminator eventually classifies all generated examples as fake.

Another problem with GANs is called mode collapse and is when the generator discovers some weakness in the generator. This will lead to the generator creating similar data examples that are passed as real by the discriminator.

The idea behind GANs is that there are two networks, a generator \(G\) and a discriminator \(D\), competing against each other. The generator makes "fake" data to pass to the discriminator. The discriminator also sees real training data and predicts if the data it's received is real or fake.

\begin{itemize}
    \item The generator is trained to fool the discriminator, it wants to output data that looks as close as possible to real, training data.
    \item The discriminator is a classifier that is trained to figure out which data is real and which is fake.
\end{itemize}

Once the generator is trained, if the idea is only generate images then the discriminator can be thrown away.

In the file \textit{MNIST\_GAN.ipynb} is a simple implementation of a GAN network where the discriminator is a linear. Similarly, the generator is almost the same configuration but with tanh activation function Fig. \ref{fig:f3}. It is important to mention that the discriminator is a classifier (MLP) so it reduces its dimension whereas the generator is more like a decoder that tries to create an output from the latent vector of minor dimension.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.6\textwidth,height=0.6\textheight,keepaspectratio]{images/simple_mnist.png}
    \captionsetup{justification=centering}
    \caption{GAN architecture for MNIST}
    \label{fig:f3}
\end{figure}

The discriminator loss can be denoted by Eq. \eqref{eq:1}. This loss is basically the sum of the losses for real and fake images. A real image as a label of 1 whereas a image generated by the generator has a label of 0.

\begin{equation}
Loss_D = Loss_{real} + Loss_{fake}
\label{eq:1}
\end{equation}

The generator loss is simply the loss of the Discriminator but flipped. Therefore, a generated image will have an output label of 1. This is done to fool the discriminator.

The training procedure of this network will involve alternating between training the discriminator and the generator. For the discriminator training the steps are:

\begin{itemize}
    \item Compute the discriminator loss on real, training images
    \item Generate fake images
    \item Compute the discriminator loss on fake, generated images
    \item Add up real and fake loss
    \item Perform backpropagation + an optimization step to update the discriminator's weights
\end{itemize}

Similarly, the training procedure for the generator is as follows:

\begin{itemize}
    \item Generate fake images
    \item Compute the discriminator loss on fake images, using flipped labels.
    \item Perform backpropagation + an optimization step to update the generator's weights
\end{itemize}

In \textit{DCGAN.ipynb} is an implementation of a Deep convolutional GAN. The main difference between this example and the last one is that now the discriminator and generator use convolutional neural networks. The discriminator architecture can be seen in Fig. \ref{fig:f4}.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.75\textwidth,height=0.75\textheight,keepaspectratio]{images/discriminator.png}
    \captionsetup{justification=centering}
    \caption{Discriminator as a CNN}
    \label{fig:f4}
\end{figure}

Similarly, the generator network looks as shown by Fig. \ref{fig:f5}. It is important to mention that the first fully connected layer is used to increase the amount of parameters of the latent vector and then the output of the fully connected layer is reshaped into a set of feature maps with an specific depth to then be passed to the first transposed convolution and so on.

\begin{figure}[ht]
    \centering
    \includegraphics[width=0.75\textwidth,height=0.75\textheight,keepaspectratio]{images/generator.png}
    \captionsetup{justification=centering}
    \caption{Generator as a CNN}
    \label{fig:f5}
\end{figure}

Finally, in \textit{CycleGAN.ipynb} is an implementation of a Cycle Gan to map images from one domain to another. Further details are in the notebook and the implementation was done by udacity.

\printbibliography

\end{document}


%%%%%%%%%%%
%% Home work template for Graduate School
%% Author : Thamme Gowda N.
%% Originally from  https://github.com/thammegowda/hw-tex-templ
%%%%%%%%%%%%%%

\documentclass[letterpaper,doc,notimes]{apa6}
%%\documentclass[tikz]{standalone}
\usepackage{tikz}
%Required by APA6 package
\usepackage[normalem]{ulem}
\usepackage[english]{babel}
\usepackage[utf8x]{inputenc}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{graphicx}
\usepackage{adjustbox}

%Oft-used, oft-abused
\usepackage{afterpage}
\usepackage{booktabs}
\usepackage{caption}
\usepackage{censor}
\usepackage{color}
\usepackage{csquotes}
\usepackage{enumitem}
\usepackage{float}
\usepackage{hyperref}
\usepackage{lmodern}
%\usepackage{media9}
\usepackage{multirow}
\usepackage{outlines}
\usepackage{pdfpages}
\usepackage{placeins}
\usepackage{soul}
\usepackage{tabularx}
\usepackage[colorinlistoftodos]{todonotes}
\usepackage{xcolor}
\usepackage{mathtools}
\graphicspath{ {images/}}

\usepackage{sectsty}
\sectionfont{\fontsize{16}{15}\selectfont}

\setenumerate[1]{label=\Roman*.}
\setenumerate[2]{label=\Alph*.}
\setenumerate[3]{label=\roman*.}
\setenumerate[4]{label=\alph*.}

% break page where ever is required
\allowdisplaybreaks

\title{ \textbf{ USC CSCI 567 HOMEWORK 2 SOLUTIONS} }
\shorttitle{USC CSCI567 FALL16 HW2}
\author{\textsc{ThammeGowda Narayanaswamy}}
\affiliation{ tnarayan@usc.edu \\ ID : 2074-6694-39 \\ Department of Computer Science \\ Viterbi School of Engineering \\ University of Southern California \\ Los Angeles, CA
	}

%\note{September 13, 2016}
\note{\today}
\note{ Collaborators : \\ Ishan Alok (ialok@usc.edu)}

\authornote{Produced for Fall 2016 session of CSCI 567, ``Machine Learning'', taught by Dr. Yan Liu at the University of Southern California}

\begin{document}

\maketitle
\newpage

\section{1. LOGISTIC REGRESSION}
\subsection{a. Negative Log Likelihood of Binary Logistic Regression}
The sigmoid-logistic function, defined by $\sigma(x) = \frac{1}{1 + e^{-x}}$ \\
Given that the dataset has $n$ examples, namely $\{(x_1,y_1), (x_2,y_2),... (x_n,y_n)\}$ \\
Where the class labels, $y_i = \{0, 1\}$  and $x_i$ $ \in \mathbb{R}^D$ \\
Let $w$ $ \in \mathbb{R}^D$ parameter which has to be learned by logistic regressor for optimal classification.\\
The probability estimation is defined by $P(Y=y_i|X=x_i) = \sigma(w^Tx_i) $. \\
The negative log likelihood is defined by:
\begin{align*}
		L(w) = & -log\bigg(\prod_{i=1}^{n} P(Y=y_i|X=x_i) \bigg) \\
		= & -\sum_{i=1}^{n}\bigg( log\big( P(Y=y_i|X=x_i) \big)\bigg) \\
		= & -\sum_{i=1}^{n}\bigg( log\big( P(Y=1|X=x_i)P(Y=0|X=x_i) \big)\bigg) \\
		= & -\sum_{i=1}^{n}\bigg( log\big[ \sigma(w^Tx_i)^{y_i} \times (1 - \sigma( w^Tx_i))^{1 - y_i} \big]\bigg) \\
		= & -\sum_{i=1}^{n}\bigg( y_i log\big[ \sigma(w^Tx_i) \big] + (1 - y_i ) log\big[ 1 - \sigma(w^Tx_i) \big]\bigg) \\
	%%	& w^Tx \text{ can also be expressed as sum of products, } \sum_{j}^{D} w_jx_j\\
\end{align*}

\section{1.b Find the update rule using gradient descent}
From the previous section, we have
\begin{align*}
	E(w) = & \mathcal{L}(w) = -\sum_{i=1}^{n}\bigg( y_i log\big[ \sigma(w^Tx_i) \big] + (1 - y_i ) log\big[ 1 - \sigma(w^Tx_i) \big]\bigg) \\
	& \text{we know that $w$ is a multivariate function of variables } \{w_0, w_1, w_2, ....w_d\} \\
	&  w^Tx \text{ can also be expressed as sum of products, } \sum_{j}^{d} w_jx_j\\
	= & -\sum_{i=1}^{n}\bigg( y_i log\big[ \sigma(\sum_{j}^{d} w_jx_j) \big] + (1 - y_i ) log\big[ 1 - \sigma(\sum_{j}^{d} w_jx_j) \big]\bigg) \\
	& \text{We find the gradient of $E(w)$ w.r.t variable $w_j$ for $j = 0, 1, ...d $} \\
 \dfrac{\partial E(w)}{\partial w_j} = & -\sum_{i=1}^{n} \bigg(
			  y_i \dfrac{\partial}{\partial w_j} log\big[ \sigma(\sum_{j}^{d} w_jx_j) \big]
			  + (1 - y_i ) \dfrac{\partial}{\partial w_j} log\big[ 1 - \sigma(\sum_{j}^{d} w_jx_j)
		\big]\bigg) \\
		\\
\end{align*}
Let us evaluate the derivatives individually
\begin{align*}
\dfrac{\partial}{\partial w_j} log\big[ \sigma(\sum_{j}^{d} w_jx_j) \big]  = &
		 \dfrac{\partial}{\partial w_j} log\big[  \frac{1}{1 + e^{-\sum_{j}^{d} w_jx_j} } \big]\\
	= &  (1 + e^{-\sum_{j}^{d} w_jx_j}) \times \frac{-1}{(1 + e^{-\sum_{j}^{d} w_jx_j})^2} \times e^{-\sum_{j}^{d} w_jx_j} \times (-x_j) \\
	= &  \frac{1}{1 + e^{-\sum_{j}^{d} w_jx_j}} \times e^{-\sum_{j}^{d} w_jx_j} \times (x_j) \\
	= & x_j \times [1 - \sigma(\sum_{j}^{d} w_jx_j)] \\
	= & x_j \times [1 - \sigma(w^Tx)] \\
	\\
\dfrac{\partial}{\partial w_j} log\big[ 1 - \sigma(\sum_{j}^{d} w_jx_j) = &
	\dfrac{\partial}{\partial w_j} log\big[ 1 - \frac{1}{1 + e^{-\sum_{j}^{d} w_jx_j} } \big] \\
	= &  \frac{1}{1 - \frac{1}{1 + e^{-\sum_{j}^{d} w_jx_j}}} \times \frac{-1 \times -1 }{(1 + e^{-\sum_{j}^{d} w_jx_j})^2} \times e^{-\sum_{j}^{d} w_jx_j} \times (-x_j) \\
	= & -x_j \times \sigma(w^Tx) \\
\end{align*}

Putting the above two values in the original equation, we have:
\begin{align*}
\dfrac{\partial E(w)}{\partial w_j} = & -\sum_{i=1}^{n} \bigg(
	 y_i \dfrac{\partial}{\partial w_j} log\big[ \sigma(\sum_{j}^{d} w_jx_j) \big]
	 + (1 - y_i ) \dfrac{\partial}{\partial w_j} log\big[ 1 - \sigma(\sum_{j}^{d} w_jx_j)
	 \big]\bigg) \\
	 = & -\sum_{i=1}^{n} \bigg(	x_j y_i (1 - \sigma(w^Tx)) - (1 - y_i ) x_j \sigma(w^Tx) \bigg) \\
\end{align*}

The update rule using gradient descent is given by,

\begin{equation*}
	w_j = w_j - \alpha \dfrac{\partial E(w)}{\partial w_j}
\end{equation*}
{Where $\alpha$ is the learning or step rate.

Provided proper learning rate $\alpha$, Gradient descent optimizer converges to global minimum for all convex functions. So, to test the convexity of the negative log likelihood function, let us compute second order derivatives.

\begin{align*}
\dfrac{\partial^2 E(w)}{\partial w_j^2} = & \dfrac{\partial}{\partial w_j} \dfrac{\partial E(w)}{\partial w_j} \\
	= & -\sum_{i=1}^{n} \dfrac{\partial}{\partial w_j} \bigg(	x_j y_i (1 - \sigma(w^Tx)) - (1 - y_i ) x_j \sigma(w^Tx) \bigg) \\
	= & -\sum_{i=1}^{n} \bigg( x_j y_i (- \dfrac{\partial}{\partial w_j} \sigma(w^Tx)) - (1 - y_i ) x_j \dfrac{\partial}{\partial w_j} \sigma(w^Tx) \bigg) \\
	= & -\sum_{i=1}^{n} \bigg( x_j y_i (-x_j \sigma(w^Tx) (1 - \sigma(w^Tx))) - (1 - y_i ) x_j x_j  \sigma(w^Tx) (1 - \sigma(w^Tx)) \bigg) \\
	= & -\sum_{i=1}^{n} \bigg( [ -y_i - (1 - y_i ) ] x_j^2 \sigma(w^Tx) (1 - \sigma(w^Tx)) \bigg)  \\
	= & \sum_{i=1}^{n} \bigg([ y_i + 1 - y_i ) ] x_j^2 \sigma(w^Tx) (1 - \sigma(w^Tx)) \bigg)  \\
	= & \sum_{i=1}^{n} \bigg( x_j^2 \sigma(w^Tx) (1 - \sigma(w^Tx)) \bigg) \\
\end{align*}

We know that, $\sigma(w^Tx)$ is the $(P(Y=1|X=x))$ which is  >= 0, also, $(1 - \sigma(w^Tx))$ is the $(P(Y=0|X=x))$ which is also >= 0. \\
Thus $\dfrac{\partial^2 E(w)}{\partial w_j^2} >= 0 $. \\
Therefore, negative log likelihood function is convex.  Hence Gradient descent finds the global minimum.

\subsection{c. Multi class logistic regression model's Negative log likelihood}

Given that we have `K` classes, namely k=\{1,2,3,4,...K\}.

We know that the posterior probability of each class sums up to 1.
\begin{align*}
	\sum_{k=1}^{K} P(Y = k | X = x_i) = 1
\end{align*}

To reduce the number of required parameters, we can use the above result to precisely compute the probability of a class if we know rest all other classes.
\begin{align*}
	P(Y = K | X = x_i) = 1 - \sum_{k=1}^{K-1} P(Y = k | X = x_i)
\end{align*}

Let us define an Indicator function
 $I(y_i = k)$ which hs value $1$ when $y_i = k$, and has $0$ if $y_i \ne k$

For a dataset of $N$ examples, the likelihood of this multi class classifier (using fewer parameters ) is defined by
\begin{align*}
	l(w) = \prod_{n=1}^{N} \prod_{k=1}^{K-1} P(Y = y_n | X = x_n ) ^ {I(y_n=k)} (1 - \sum_{k=1}^{K-1} P(Y = y_n | X = x_n ) ^ {I(y_n=k)})
\end{align*}
We can simplify this equation without adding any additional parameters but setting $W_K=0$ as given in the problem description.

\begin{align*}
	\ell (w) = & \prod_{n=1}^{N} \prod_{k=1}^{K} P(Y = y_n | X = x_n )^{I(y_n = k)} & &
\end{align*}
The negative log likelihood is given by
\begin{align*}
  \mathcal{L}(w) = & - log \bigg [ \prod_{n=1}^{N} \prod_{k=1}^{K} P(Y = y_n | X = x_n )^{I(y_n = k)} \bigg]  & & \\
	   = & - \sum_{n=1}^{N} \sum_{k=1}^{K} log P(Y = y_n | X = x_n ) ^{I(y_n = k)}  & & \\
	   = & - \sum_{n=1}^{N} \sum_{k=1}^{K} log \bigg[ \frac{e^{w_k^Tx_n}}{1 + \sum_{t=1}^{K} e^{w_t^Tx_n}}  \bigg]^{I(y_n = k)}  & & \\
	   = & - \sum_{n=1}^{N} \sum_{k=1}^{K} I(y_n = k) \bigg[ w_k^Tx_n - log [1 +  \sum_{t=1}^{K} e^{w_t^Tx_n} ] \bigg]  & & \\
\end{align*}

\subsection{d. Gradient descent update rule}
Let us consider the final equation of previous section.
The $\mathcal{L}$ is a multivariate function with variables $w_i$ for i = 1, 2,3, ... K . \\

\begin{align*}
	\mathcal{L}(w_1, w_2, ...w_K ) = & - \sum_{n=1}^{N} \sum_{k=1}^{K} I(y_n = k) \bigg[ w_k^Tx_n - log [1 +  \sum_{t=1}^{K} e^{w_t^Tx_n} ] \bigg]
\end{align*}
The first order derivative of $\mathcal{L}$ w.r.t $w_i$ is:

\begin{align*}
\frac{\partial \mathcal{L}}{\partial w_i} = & - \frac{\partial }{\partial w_i } \sum_{n=1}^{N} \sum_{k=1}^{K} I(y_n = k) \bigg[ w_k^Tx_n - log [1 +  \sum_{t=1}^{K} e^{w_t^Tx_n} ] \bigg ] \\
= & - \sum_{n=1}^{N} \sum_{k=1}^{K} I(y_n = k) \bigg[ x_{ni}  - \frac{1}{1 + \sum_{t=1}^{K} e^{w_t^Tx_n}} \times \sum_{t=1}^{K} e^{w_t^Tx_n} \times x_{ni} \bigg] \\
& \text{$x_n$ is a vector of all atttributes of $n^{th}$ example}
\\ & \text{where as $x_{ni}$ is a scalar from $i^{th}$ attribute of $n^{th}$ example} \\
\\
= & - \sum_{n=1}^{N} \sum_{k=1}^{K} x_{ni} I(y_n = k) \bigg[ 1  - \frac{\sum_{t=1}^{K} e^{w_t^Tx_n}}{1 + \sum_{t=1}^{K} e^{w_t^Tx_n}}  \bigg] \\
\frac{\partial \mathcal{L}}{\partial w_i} = & - \sum_{n=1}^{N} \sum_{k=1}^{K} x_{ni} I(y_n = k) \bigg[ \frac{1}{1 + \sum_{t=1}^{K} e^{w_t^Tx_n}}  \bigg] \\
\end{align*}
The update rule for gradient descent optimizer with step rate $\alpha$ is:
\begin{align*}
	w_i = & w_i - \alpha \frac{\partial \mathcal{L}}{\partial w_i} \\
	& \text{for all } i = \{1, 2, 3,...K\}
\end{align*}

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
\section{2. LINEAR/GAUSSIAN DISCRIMINANT ANALYSIS}

\subsection{a. Log likelihood and MLE estimates}
Given:

The dataset $\mathcal{D}$ has $N$ examples, $\{(x_1,y_1), (x_2, y_2), ...(x_N, y_N)\}$

There are two classes $y_n = \{1, 2\} $

$P(y_n = 1) = p_1$ and $P(y_n = 2) = p_2$

$P(x_n | y_n=1) = \frac{1}{\sqrt{2\pi } \cdot \sigma_1} \exp(-\frac{(x_n-\mu_1)^2}{2\sigma_1^2})$

$P(x_n | y_n=2) = \frac{1}{\sqrt{2\pi } \cdot \sigma_2} \exp(-\frac{(x_n-\mu_2)^2}{2\sigma_2^2})$ \\

Let us define Indicator functions $I(y_n=1)$ and $I(y_n=2)$

\paragraph{Log Likelihood}
The log likelihood is defined by
\begin{align*}
\mathcal{L} = & ln \bigg[ \prod_{n=1}^{N} P(x_n, y_n) \bigg] \\
	= &  \ln \bigg[ \prod_{n=1}^{N} P(x_n, y_1)^{I(y_n=1)} \times P(x_n, y_1)^{I(y_n=2)} \bigg] \\
	= & \ln \bigg[ \prod_{n=1}^{N} P(x_n, y_1)^{I(y_n=1)} \times P(x_n, y_1)^{I(y_n=2)} \bigg] \\
	= &  \sum_{n=1}^{N}\bigg[ ln P(x_n, y_1)^{I(y_n=1)} + ln P(x_n, y_1)^{I(y_n=2)} \bigg] \\
	= &  \sum_{n=1}^{N}\bigg[ I(y_n=1) \ln \big[ p_1 \times \frac{1}{\sqrt{2\pi } \cdot \sigma_1} \exp(-\frac{(x_n-\mu_1)^2}{2\sigma_1^2}) \big]
			 + I(y_n=2) \ln \big[ p_2 \times \frac{1}{\sqrt{2\pi } \cdot \sigma_2} \exp(-\frac{(x_n-\mu_2)^2}{2\sigma_2^2}) \big] \bigg]  \\
	= & \sum_{n=1}^{N}\bigg[
		I(y_n=1)\big[ \ln(p_1) -\frac{1}{2}\ln(2\pi) - \ln(\sigma_1) - \frac{1}{2\sigma_1^2} (x_n-\mu_1)^2 \big] \\
		 & \indent  +
		I(y_n=2)\big[ \ln(p_2) -\frac{1}{2}\ln(2\pi) - \ln(\sigma_2) - \frac{1}{2\sigma_2^2} (x_n-\mu_2)^2 \big] \bigg] \\
\end{align*}

The above likelihood expression is a multivariate function with parameters $(p1, p2, \mu_1, \mu_2, \sigma_1, \sigma_2 )$.
From a fundamental law of probability, we know that
\begin{align*}
	p_1 + p_2 = & 1\\
	 p_2 = & 1 - p_1 \\
	 p_1 = & 1 - p_2
\end{align*}
Thus,
\begin{multline}
	\mathcal{L} (p1, \mu_1, \mu_2, \sigma_1, \sigma_2 )	=  \sum_{n=1}^{N}\bigg[
	I(y_n=1)\big[ \ln(p_1) -\frac{1}{2}\ln(2\pi) - \ln(\sigma_1) - \frac{1}{2\sigma_1^2} (x_n-\mu_1)^2 \big] \\
		\indent \indent \indent +
	I(y_n=2)\big[ \ln(1 - p_1) -\frac{1}{2}\ln(2\pi) - \ln(\sigma_2) - \frac{1}{2\sigma_2^2} (x_n-\mu_2)^2 \big] \bigg] \\
\end{multline}
Let us find an estimate of these parameters using maximum likelihood estimator method.\\\
To find an estimate of $p_1$, we solve $\frac{\partial \mathcal{L} }{\partial p_1} = 0$
\begin{align*}
\frac{\partial \mathcal{L} }{\partial p_1} =& 0 \\
\sum_{n=1}^{N}\big[I(y_n=1)\frac{1}{p_1}  + I(y_n=2)\frac{1}{1-p_1} \times(-1) \big]  =& 0 \\
\frac{1}{p_1} \sum_{n=1}^{N} I(y_n=1)  - \frac{1}{1-p_1} \sum_{n=1}^{N} I(y_n=2) =& 0 \\
(1-p_1) \sum_{n=1}^{N} I(y_n=1)  - p_1 \sum_{n=1}^{N} I(y_n=2) =& 0 \\
p_1 =&  \frac{\sum_{n=1}^{N} I(y_n=1) }{\sum_{n=1}^{N} I(y_n=1) + \sum_{n=1}^{N} I(y_n=2)}\\
\implies p_1^* =&  \frac{\sum_{n=1}^{N} I(y_n=1) }{N} \\
\text{Similarly,} \\
1 - p_1 =& 1 - \frac{\sum_{n=1}^{N} I(y_n=1) }{N} = \frac{N - \sum_{n=1}^{N} I(y_n=1) }{N} \\
\implies p_2^* =&  \frac{\sum_{n=1}^{N} I(y_n=2) }{N} \\
\end{align*}

To find an estimate of $\mu_1$ when $\mathcal{L}$ is maximum, let us solve for $\frac{\partial \mathcal{L} }{\partial \mu_1} = 0$
\begin{align*}
\frac{\partial \mathcal{L} }{\partial \mu_1} =& 0 \\
	\sum_{n=1}^{N} I(y_n=1) \times \frac{-1}{2\sigma_1} \times 2 \times (x_n - \mu_1) \times -1 =& 0 \\
	\sum_{n=1}^{N} I(y_n=1) \times (x_n - \mu_1) =& 0 \\
		\sum_{n=1}^{N} I(y_n=1) x_n =&  \sum_{n=1}^{N} I(y_n=1) \mu_1 \\
				\sum_{n=1}^{N} I(y_n=1) x_n =& \mu_1 \sum_{n=1}^{N} I(y_n=1) \\
				\frac{ \sum_{n=1}^{N} I(y_n=1) x_n } {\sum_{n=1}^{N} I(y_n=1)}=& \mu_1  \\
				\implies \mu_1^* = & \frac{ \sum_{n=1}^{N} I(y_n=1) x_n } {\sum_{n=1}^{N} I(y_n=1)}  \\
		\text{We already know that, }  p_1 \times N = \sum_{n=1}^{N} I(y_n=1) & \\
	 \implies \mu_1^* = & \frac{ \sum_{n=1}^{N} I(y_n=1) x_n } {p_1 N} \\
	 \\
 \text{$\mu_2$ when $\mathcal{L}$ is maximum can be estimated by solving} & \\ \frac{\partial \mathcal{L} }{\partial \mu_2} = &0 \\
	 \implies \mu_2^* = & \frac{ \sum_{n=1}^{N} I(y_n=2) x_n } {\sum_{n=1}^{N} I(y_n=2)}  \\
	 	  = & \frac{ \sum_{n=1}^{N} I(y_n=2) x_n } {p_2 N} \\
\end{align*}

To find an estimate of $\sigma_1$ when $\mathcal{L}$ is maximum, let us solve
\begin{align*}
	\frac{\partial \mathcal{L} }{\partial \sigma_1} = &0  \\
	\sum_{n=1}^{N} I(y_n=1) \big[ -\frac{1}{\sigma_1} -\frac{(x_n - \mu_1)^2}{2} \times \frac{-2}{\sigma^3} \big] =& 0 \\
	\sum_{n=1}^{N} I(y_n=1) \big[ -\sigma_1^2 + (x_n - \mu_1)^2 \big] =& 0 \\
	\sigma_1^2 \sum_{n=1}^{N} I(y_n=1) =& \sum_{n=1}^{N} I(y_n=1) (x_n - \mu_1)^2 \\
    \sigma_1^2 =& \frac{ \sum_{n=1}^{N} I(y_n=1) (x_n - \mu_1)^2 }{\sum_{n=1}^{N} I(y_n=1)}\\
  \implies \sigma_1^* =& \sqrt{ \frac{ \sum_{n=1}^{N} I(y_n=1) (x_n - \mu_1)^2 }{\sum_{n=1}^{N} I(y_n=1)} } = \sqrt{ \frac{ \sum_{n=1}^{N} I(y_n=1) (x_n - \mu_1)^2 }{p_1 N} } \\
\end{align*}

Similarly, to find an estimate of $\sigma_2$ when $\mathcal{L}$ is maximum, let us solve

\begin{align*}
\frac{\partial \mathcal{L} }{\partial \sigma_2} = &0  \\
\implies \sigma_2^* =& \sqrt{ \frac{ \sum_{n=1}^{N} I(y_n=2) (x_n - \mu_2)^2 }{\sum_{n=1}^{N} I(y_n=2)} } = \sqrt{ \frac{ \sum_{n=1}^{N} I(y_n=2) (x_n - \mu_2)^2 }{p_2 N} } \\
\end{align*}

\subsection{b. Reduction to Logistic Sigmoid function form}
Given :\\
\indent  There are two classes, $y_n = \{c_1, c_2\}$ \\
\indent  $P(x | y= c_1) = \mathcal{N}(\mu_1, \sum)$  \\
\indent  $P(x | y=c_2) = \mathcal{N}(\mu_2, \sum) $ \\

\noindent We know \\
\indent for $D$ variables, \\
\indent \indent $\mathcal{N}(\mu, \sum) = (2\pi)^{-D/2} \sum^{-1/2} \exp[-\frac{1}{2} (x-\mu)^T \space \sum^{-1} \space (x-\mu)]$

\noindent Goal: \\
\indent  To express $P(y=c_1|x) $ in logistic sigmoid form

\noindent Solution:
\begin{align*}
\text{Using the Bayes rule} & \\
	P(y =& c_1|x) = \frac{P(x | y=c1) P(y=c1)}{P(x)} \\
\text{Using the law of otal probability} \\
	P(y =& c_1|x) = \frac{P(x | y=c1) P(y=c1)}{P(x | y=c1) P(y=c1) + P(x | y=c2) P(y=c2)} \\
	P(y =& c_1|x) = \frac{1}{1 + \frac{ P(x | y=c2) P(y=c2)}{P(x | y=c1) P(y=c1)} } \\
\text{This is of form} \\
	P(y =& c_1|x) = \frac{1}{1 + f(x)} \\
\text{where } f(x) = & \frac{P(y=c2)}{P(y=c1)} \times \frac{P(x|y=c2) }{P(x|y=c1)} \\
	 = & \frac{P(y=c2)}{P(y=c1)} \times \frac{\mathcal{N}(\mu_2, \sum) }{\mathcal{N}(\mu_1, \sum)}
\end{align*}
Now, substituting the values in $f(x)$
\begin{align*}
f(x) = & \frac{P(y=c2)}{P(y=c1)} \times \frac{(2\pi)^{-D/2} \sum^{-1/2} \exp[-\frac{1}{2} (x-\mu_2)^T \space \sum^{-1} \space (x-\mu_2)] }  {(2\pi)^{-D/2} \sum^{-1/2} \exp[-\frac{1}{2} (x-\mu_1)^T \space \sum^{-1} \space (x-\mu_1)]}\\
  = & \frac{P(y=c2)}{P(y=c1)} \times \exp \bigg[\frac{1}{2}  \big[(x-\mu_1)^T \space \sum^{-1} \space (x-\mu_1) - (x-\mu_2)^T \space \sum^{-1} \space (x-\mu_2)  \big] \bigg] \\
  \\
&  \text{For the sake of similification, we can use the form, } x = \exp(\ln(x))\\
\exp(\ln(f(x))) = &\exp(\ln\bigg( \frac{P(y=c2)}{P(y=c1)} \times \exp \bigg[\frac{1}{2}  \big[(x-\mu_1)^T \space \sum^{-1} \space (x-\mu_1) - (x-\mu_2)^T \space \sum^{-1} \space (x-\mu_2)  \big] \bigg] \bigg)) \\
\\
f(x) = &\exp(\ln(\frac{P(y=c2)}{P(y=c1)}) + \frac{1}{2} \big[(x-\mu_1)^T \space \sum^{-1} \space (x-\mu_1) - (x-\mu_2)^T \space \sum^{-1} \space (x-\mu_2)  \big] ) \\
f(x) = &\exp( - \big(- \ln(\frac{P(y=c2)}{P(y=c1)}) + \frac{1}{2} \big[ (x-\mu_2)^T \space \sum^{-1} \space (x-\mu_2) - (x-\mu_1)^T \space \sum^{-1} \space (x-\mu_1)\big] \big)) \\
\end{align*}
This equation is of  form $\exp(-(\theta_0 + {\theta}^Tx))$
Where, $\theta_0$ is a scalar computed by - $\ln(\frac{P(y=c2)}{P(y=c1)}) $ \\
 and $\theta^Tx$ is another scalar computed by $\frac{1}{2} \big[(x-\mu_2)^T \space \sum^{-1} \space (x-\mu_2) - (x-\mu_1)^T \space \sum^{-1} \space (x-\mu_1)  \big]$, but $\theta$ and $x$ are vectors of $D$ dimensions.

 Substituting the value of $f(x)$ in the original equation above, we get:
\begin{align*}
	P(y =& c_1|x) = \frac{1}{1 + f(x)}
		= \frac{1}{1 + exp(-(\theta_0 + \theta^Tx))}
 \end{align*}
 The above expression is in logistic sigmoid form.
 However, we can further simplify the above equation by packing $\theta_0$ scalar parameter inside $\theta$ vector by inserting $1$ into $x$ vector at the corresponding dimension. \\
 \begin{equation}
	\implies 	P(y = c_1|x) = \frac{1}{1 + exp(-\theta^Tx)}
 \end{equation}
 where $\theta$ and $x$ are vectors of $D+1$ dimensions

 \section{4. Data Analysis}

 \subsection{a. Histograms of Attributes and the Target variable}

Here are the histograms of the dataset.\\
\includegraphics[scale=0.45]{Histograms}

Here are the correlations among the attributes:\\
\includegraphics[scale=1]{correlation-heatmap}

\subsection{Program output}
\begin{verbatim}
$ python CSCI567_hw2_fall16.py

The following parameters will be used for gradient descent optimizer:
 Learning rate= 0.001
 Convergence Tolerance= 0.005
 Max Iterations= 50000
 ##### 3.1 DATA ANALYSIS / EXPLORATION
 Feature Names:
 0
 0      CRIM
 1        ZN
 2     INDUS
 3      CHAS
 4       NOX
 5        RM
 6       AGE
 7       DIS
 8       RAD
 9       TAX
 10  PTRATIO
 11        B
 12    LSTAT
 Dataset shape: (506, 13) (506, 1)
 Test shape: (73, 13) (73, 1)
 Train shape: (433, 13) (433, 1)
 Correlations between the attributes:
 0         1         2         3         4         5         6   \
 0   1.002315  0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
 1  -0.197276  1.002315  0.000000  0.000000  0.000000  0.000000  0.000000
 2   0.396280 -0.537783  1.002315  0.000000  0.000000  0.000000  0.000000
 3  -0.054158 -0.038522  0.050507  1.002315  0.000000  0.000000  0.000000
 4   0.410524 -0.525903  0.769347  0.065319  1.002315  0.000000  0.000000
 5  -0.206593  0.314229 -0.387829  0.066559 -0.308899  1.002315  0.000000
 6   0.351814 -0.586247  0.654674  0.081180  0.734113 -0.239191  1.002315
 7  -0.370380  0.672366 -0.713743 -0.097458 -0.771612  0.209400 -0.750088
 8   0.607248 -0.317635  0.597033 -0.021546  0.611365 -0.211677  0.470596
 9   0.567033 -0.318496  0.722513 -0.051311  0.668685 -0.288305  0.520563
 10  0.279365 -0.383817  0.376030 -0.145421  0.183605 -0.356928  0.264867
 11 -0.358765  0.180031 -0.360012  0.054895 -0.386504  0.134887 -0.284053
 12  0.456168 -0.422381  0.591894 -0.057371  0.591094 -0.597977  0.608152

 7         8         9         10        11        12
 0   0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
 1   0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
 2   0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
 3   0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
 4   0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
 5   0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
 6   0.000000  0.000000  0.000000  0.000000  0.000000  0.000000
 7   1.002315  0.000000  0.000000  0.000000  0.000000  0.000000
 8  -0.498835  1.002315  0.000000  0.000000  0.000000  0.000000
 9  -0.535925  0.913672  1.002315  0.000000  0.000000  0.000000
 10 -0.236001  0.463870  0.457994  1.002315  0.000000  0.000000
 11  0.296206 -0.452862 -0.449102 -0.177750  1.002315  0.000000
 12 -0.498400  0.497685  0.545339  0.374000 -0.391823  1.002315

 Magnitude of Correlation with the target
 ('CRIM', 0.38859443435207547)
 ('ZN', 0.36382754420139768)
 ('INDUS', 0.48418563338201681)
 ('CHAS', 0.20407144132697344)
 ('NOX', 0.4258130776462164)
 ('RM', 0.69252269454509585)
 ('AGE', 0.39108230278590012)
 ('DIS', 0.25300487309124547)
 ('RAD', 0.38638415658601943)
 ('TAX', 0.46993468487617812)
 ('PTRATIO', 0.5064403651254078)
 ('B', 0.34422912357951185)
 ('LSTAT', 0.74168271373323336)
 ##### Generating Histograms

 ###### 3.2 a LINEAR REGRESSION ALGORITHM
 Using analytical optimizer
 W= [ 22.46351039  -0.9673241    1.04542152  -0.17351707   0.92139354
 -1.6205974    2.72573311  -0.26723668  -3.11240878   2.48363548
 -1.91078756  -1.88799963   0.82940209  -3.67532974]
 Train MSE:: 20.950144508 	Test MSE:: 28.4179164975
 Using gradient_desc optimizer
 W= [ 22.46351039  -0.93347984   0.97861438  -0.35174534   0.94966261
 -1.54772785   2.76534798  -0.30369741  -3.0952704    2.03099152
 -1.41039412  -1.86141796   0.82666393  -3.65780415]
 Train MSE:: 20.9818495194 	Test MSE:: 28.7478497502

 ##### 3.2 b RIDGE REGRESSION########

 Lambda		Train MSE		Test MSE
 0.01		21.0581242724		28.8462622276
 0.1		25.8139532086		33.5094712654
 1.0		155.792730262		165.411777892

 ###### 3.2 c RIDGE REGRESSION : 10-FOLD CROSS VALIDATION########

 Lambda		Train MSE		Test MSE
 0.0001		20.9523058443		28.7708292066
 0.001		20.9895200484		28.7758027829
 0.01		21.2170522694		28.8607963801
 0.1		25.7050072249		33.5041252869
 1		156.108135228		165.411512266
 10		475.657189134		500.398146416

 ###### 3.3 a FEATURE SELECTION: TOP 4 CORRELATIONS WITH TARGET
 Top4 attrs:: ['LSTAT', 'RM', 'PTRATIO', 'INDUS']
 Train MSE:: 26.4066042155  Test MSE:: 31.4962025449

 ###### 3.3 b FEATURE SELECTION: TOP 4 CORRELATIONS WITH RESIDUAL ERROR
 Choosing:  LSTAT
 Train MSE:: 37.5682291124  Test MSE:: 43.9181708494
 Choosing:  RM
 Train MSE:: 29.5454831972  Test MSE:: 36.2967947962
 Choosing:  B
 Train MSE:: 28.6939844449  Test MSE:: 34.3784509663
 Choosing:  CHAS
 Train MSE:: 26.9461754396  Test MSE:: 38.5301578153

 ###### 3.3 c FEATURE SELECTION: TOP 4 USING BRUTEFORCE
 Performing bruteforce feature selection. Hang tight for results or press CTRL+C to skip
 Best Combination::  (5, 10, 11, 12) , Least Test MSE:: 30.1004061101 , Train MSE:: 25.7444167198

 ###### 3.4 FEATURE EXPANSION
 Train Shape: (433, 104)
 Test Shape: (73, 104)
 Train MSE:: 5.05978429711  Test MSE:: 14.5553049758
\end{verbatim}

\end{document}


\section{Structure from Motion}
\subsection{Factorization}
Factorise observations in structure of the scene and motion/calibration of the camera
Use all points in all images at the same time

\subsubsection{perspective factorization}
$$\lambda m = P M $$

\textit{something}

\subsection{structure from motion}

seriously no plan


\subsection{Pascal Array Functions} % (fold)
\label{sec:pascal_array_functions}

Pascal includes a number of functions to help you work with arrays. These allow you to determine the length of the array, and its range of indexes. These functions are typically used in conjuncture with the for loop.

\begin{table}[h]
  \centering
  \begin{tabular}{|c|p{9cm}|}
    \hline
    \multicolumn{2}{|c|}{\textbf{Function Prototypes}} \\
    \hline
    \multicolumn{2}{|c|}{} \\
    \multicolumn{2}{|c|}{\texttt{function Length( arg: array of \ldots ): Integer}} \\
    \multicolumn{2}{|c|}{} \\
    \hline
    \multicolumn{2}{|c|}{\textbf{Returns}} \\
    \hline
    \texttt{Integer} & The number of elements in the \texttt{arg} array. \\
    \hline
    \textbf{Parameter} & \textbf{Description} \\
    \hline
    \texttt{ arg } & The array you want to get the length of.\\
    \hline
    \multicolumn{2}{c}{} \\
    \hline
    \multicolumn{2}{|c|}{} \\
    \multicolumn{2}{|c|}{\texttt{function Low( arg: array of \ldots ): Integer}} \\
    \multicolumn{2}{|c|}{} \\
    \hline
    \multicolumn{2}{|c|}{\textbf{Returns}} \\
    \hline
    \texttt{Integer} & The lowest index in the \texttt{arg} array. \\
    \hline
    \textbf{Parameter} & \textbf{Description} \\
    \hline
    \texttt{ arg } & The array you want to get the lowest index of.\\
    \hline
    \multicolumn{2}{c}{} \\
    \hline
    \multicolumn{2}{|c|}{} \\
    \multicolumn{2}{|c|}{\texttt{function High( arg: array of \ldots ): Integer}} \\
    \multicolumn{2}{|c|}{} \\
    \hline
    \multicolumn{2}{|c|}{\textbf{Returns}} \\
    \hline
    \texttt{Integer} & The last index in the \texttt{arg} array. \\
    \hline
    \textbf{Parameter} & \textbf{Description} \\
    \hline
    \texttt{ arg } & The array you want to get the last index of.\\
    \hline
  \end{tabular}
  \caption{Parameters that must be passed to \texttt{Length}, \texttt{High}, and \texttt{Low}}
  \label{tbl:pas-array-fns}
\end{table}

\passection{\pascode{plst:array_fn_test}{}{code/pascal/array/TestArrayFunctions.pas}}

% section pascal_array_functions (end)