Normal equation for linear regression I am going through the derivation of normal equation for multivariate linear regression. The equation is given by :

$\theta = (X^{T}X)^{-1}X^{T}Y$

The cost function is given by:

$J(\theta) = \frac{1}{2m}(X\theta-Y)^{T}(X\theta-Y)$

Simplifying, 

$J(\theta) = \frac{1}{2m}(\theta^{T}X^{T}X\theta - 2(X\theta)^{T}Y + Y^{T}Y)$

Differentiating w.r.t $\theta$ and equating to zero

$\frac{dJ(\theta)}{d\theta} = \frac{d}{d\theta}(\theta^{T}X^{T}X\theta)-\frac{d}{d\theta}(2(X\theta)^{T}Y) = 0$

I want to specifically understand the differentiation of the left term:

$\frac{d}{d\theta}(\theta^{T}X^{T}X\theta) = X^{T}X\frac{d}{d\theta}(\theta^{T}\theta) $

$\frac{d}{d\theta}(\theta^{T}\theta) = [\frac{d}{d\theta_1}(\theta_1^{2}+\theta_2^{2}+...\theta_n^{2}),\frac{d}{d\theta_2}(\theta_1^{2}+\theta_2^{2}+...\theta_n^{2}) ,...., \frac{d}{d\theta_n}(\theta_1^{2}+\theta_2^{2}+...\theta_n^{2})]$

$\frac{d}{d\theta}(\theta^{T}\theta) = [2\theta_1,2\theta_2,...,2\theta_n] $

$\frac{d}{d\theta}(\theta^{T}\theta) = 2\theta^{T} $

But the final equation is obtained by using $\frac{d}{d\theta}(\theta^{T}\theta) = 2\theta$

How is $\frac{d}{d\theta}(\theta^{T}\theta) = 2\theta$ and not $2\theta^{T}$
It is basically a matter of convention, which becomes a bit more clear if you write the whole thing in terms of elements, rather than vectors. Consider

$$ \theta^T \theta = \sum_{n=1}^N \theta_n \theta_n = \sum_{n=1}^N \theta_n^2 $$

What is usually meant if you write $\frac{df}{d\theta}$ is that you take the gradient of a scalar $f$, i.e. you get a vector $\frac{df}{d\theta}$ where each element $i$ is the derivative with respect to the respective coordinate $\theta_i$:

$$ \left( \frac{df}{d\theta} \right)_i = \frac{df}{d\theta_i} $$

Let's apply this to $\theta^T\theta$:

$$ \left( \frac{d}{d\theta} \theta^T\theta \right)_i = \frac{d}{d\theta_i} \sum_{n=1}^N \theta_n^2 = 2 \sum_{n=1}^N \theta_n \frac{d\theta_n}{d\theta_i}  $$

and since $\frac{d\theta_n}{d\theta_i} = \delta_{i,n}$:

$$ \left( \frac{d}{d\theta} \theta^T\theta \right)_i = 2 \theta_i $$

If you interpret this as columns or rows is pretty much up to you. Commonly we would take $\theta_i$ as the elements of a column vector $\theta$ and for practicality we usually want the gradient to live in the same vector space as our coordinates, so $\left( \frac{df}{d\theta} \right)_i$ would also be a column vector. Hence

$$ \frac{d}{d\theta} \theta^T\theta = 2 \theta $$