How can I choose the address bitcoins are sent from? So let's say I have multiple bitcoin addresses (some of them already received bitcoins), created a new bitcoin address, received bitcoins on it and would like to use the very same address to send them elsewhere - how can I do that?

I'm using Bitcoin Core (bitcoin-qt) as of right now.
As long as you're running a relatively recent version of Bitcoin Core you can use Coin Control. This feature let's you pick which unspent outputs will be used in the transaction.

To enable Coin Control, go to Settings->Options->Wallet->'Enable coin control features'.

Now in the Send window, you can click 'Inputs...' to select which unspent outputs you want to use. Easy!