ATMega328P not running sketch in standalone (works in Uno R3 Board though)
I found the solution by accident.

While I used my multimeter to measure the voltages between GND and the Pins (for the last fact), I noticed the program running at weird speeds. I. e. the LED started blinking.

What did I measure?

Actually I was just measuring one of the crystal pins (Pin 9 or 10). The sketch went running for a few seconds and very different speeds and then stopped.

This small change in voltage gave the ATMega the clock signal it needs to step through the program.

Maybe I need less capacitance of the two capacitors as mentioned in the comments. I will update the answer with more information later on.

The perfect solution

As pointed out in the comments, I really chose the wrong capacitors. It worked without, but we want the stability so use a bright light and a magnifying glass if you are unsure. Really, do it.