What is the purpose of creating symbolic variables in Theano?
Question: In Theano, variables are written as 'symbols':

    
    
    x = T.matrix("x")
    y = T.vector("y")
    

From reading the documentation, it is implied that the reason we create these
symbols may be due to the fact that these variables are compiled into C code.
But I'm not sure if this is the case, much less the only reason for using
symbolic variables.

What is the purpose of creating symbolic variables in Theano? What can they do
that a, out of the box assignment in Python can't do?


Answer: The Theano web site opens with

> Theano is a Python library that allows you to define, optimize, and evaluate
> mathematical expressions involving multi-dimensional arrays efficiently.

which seems like quite a good summary of what it does but perhaps not why it
does it.

One of the main features of Theano is its symbolic differentiation feature.
That is, given a symbolic mathematical expression, Theano can automatically
differentiate the expression with respect to some variable within the
expression, i.e. it can automatically determine the gradient of the expression
along some dimension(s) of interest.

For example, if `y=x**2` (where `**` is the power operator) then the gradient
of `y` with respect to `x` is `dy/dx = 2*x`. Theano can do this automatically:

    
    
    import theano
    import theano.tensor
    x = theano.tensor.scalar('x')
    y = x ** 2
    theano.printing.pp(y)
    dy_dx = theano.grad(y, x)
    theano.printing.pp(dy_dx)
    

If you run this code you'll see the output

    
    
    (x ** TensorConstant{2})
    ((fill((x ** TensorConstant{2}), TensorConstant{1.0}) * TensorConstant{2}) * (x ** (TensorConstant{2} - TensorConstant{1})))
    

The `fill` operation just creates a tensor of the correct shape (a scalar in
this case) filled with ones and `TensorConstant{a}` is just the number `a` so
this can be simplified to

    
    
    (x ** 2) == x ** 2
    ((1 * 2) * (x ** (2 - 1))) == (2 * (x ** 1)) == 2 * x
    

As expected.

Clearly this is not particularly helpful for such a simple mathematical
expression, but now imagine you're constructing an arbitrarily large and
complex mathematical expression for which the gradient is not so immediately
obvious, as is often the case in neural network research.

But there's more. Theano can do the above for expressions that involve
operations over not just scalars but also vectors, matrices, or any other
tensor.

Besides symbolic differentiation, Theano's symbolic approach offers other
significant benefits:

  * Theano can compile symbolic mathematical expressions into executable code. It uses various techniques that can yield executable code that can run faster than plain Python code.
  * Theano programs can often be switched between running on CPU and GPU with no code changes whatsoever.
  * When running on CPU, Theano makes full use of numpy BLAS and OpenMP facilities, when available, to parallelize the execution over multiple CPU cores.
  * When running on GPU, Theano makes full use of of the many cores on modern GPUs to parallelize the most costly operations (most notably matrix multiplication).
  * Theano's compiler is an optimizing compiler in that it can change the expression (also known as the computation graph) using various transforms that maintain the semantics of the expression (it still computes the same results given the same inputs) but achieving various performance and, crucially, [numerical stability](http://mathworld.wolfram.com/NumericalStability.html) gains.

Plain Python does not offer any of the above and using numpy alone only offers
some of the parallelization features (via BLAS and OpenMP).

More on this topic [in the Theano
documentation](http://deeplearning.net/software/theano/introduction.html).

P.S. this is an expansion of @eickenberg brief comment to the question.

