Why do VFR-flights need a clearance in class D airspace?
Because class D is controlled airspace.
While you are correct that VFR-IFR separation is not provided in class D airspace, the major advantage compared to class E airspace is that, because VFR is subject to a clearance, ATC knows exactly where all VFR aircraft are, enabling ATC to provide very accurate traffic information and/or collision avoidance.

So why for example is a VFR pilot given the clearance (or is that an instruction) to maintain 4000 ft in class D airspace, if he is resposible for the separation, and not the controller?

ATC can certainly limit you to a specific altitude, but as a VFR flight ATC can never lock you on a specific heading (since you must also be able to turn to stay clear of clouds). All rules regarding right-of-way in the air are based on turning - not climbing or descending - so being instructed to maintain a specific altitude should not limit your abilities to follow the normal rules of right-of-way.
Although not technically required, ATC will often provide some sort of separation between VFR and IFR in class D, at least in cases where there is an obvious risk of collision. But the normal separation minima do not apply, and ultimatively it is pilot responsibility to practise see-and-avoid (which is made significantly easier compared to class E or below because ATC can provide accurate traffic information).
If we go one step &quot;up&quot; the ladder, to class C - where VFR-IFR separation is provided - a major difference for VFR is the requirement for a mode A/C transponder. A transponder is normally not required in class D for VFR traffic, which is another reason ATC is not necessarily able to provide IFR-VFR separation in class D.
There are different use cases for the different airspace classes, and different countries and ANSP's will make use of them as they see fit in their local environment. The existance of the different airspace classes makes it possible to graduate the service provided to suit most situations:

Class G: No separation
Class F: No separation, but advisory service
Class E: IFR-IFR separation, no radio/transponder for VFR
Class D: IFR-IFR separation, radio+transponder required for VFR
Class C: IFR-IFR and IFR-VFR separation
Class B: IFR-IFR, IFR-VFR and VFR-VFR separation
Class A: IFR-IFR separation, VFR not allowed