Mechanism for deleting old memories Do we have any mechanism for deleting neuronal connections? 

That is, if I learn a better version for doing something, I assume the brain will create a new neuronal connection, and we will use this connection. However, what happens to the old, non-used version? Does it fades away after a while? or is it enough to have a better, stronger connection to have the older version ignored, even if present?
Yes.

One particular mechanism for reducing synaptic strength is called long-term depression (note: this terminology is not related to the mental health term depression). Much like the opposite process that strengthens synapses, certain firing patterns tend to weaken synapses.

Additionally, synaptic scaling is a homeostatic mechanism by which more excited cells become less excitable over time, including by reducing their response to inputs globally (this is also a particular category of long-term depression). If you keep re-strengthening certain synapses, and then scaling all of them down, the result over multiple iterations is that those synapses that don't get re-strengthened get weaker.

The synaptic homeostasis hypothesis of sleep suggests that this process occurs during sleep and is a key reason why sleep is important (there are many other theories about sleep, however).