Can TensorFlow minimize "symbolically"
If by "symbolic" you mean finding an analytical solution, that is, an equation for each weight, then the answer is no. The example you chose results in a system linear equations, which can be solved analytically. However once you introduce non linearities (by using activation functions with more than one layer), most non trivial cases will have no analytical solution and will need to be solved numerically. This is not a problem specific to tensorflow, it is  a mathematical issue, it will not be possible on any language, current or future. Unless there is some revolution in math first.