Self-Collision not work well
The  cloth self-collision is affected by the scale of the faces making up the cloth. This means that when you have a fairly coarse level of subdivision the self-collision Distance will result in a cloth that is quite separated whereas if you subdivide the mesh more finely the cloth is permitted to get closer before colliding. Since you only have a single face-width between your folds the self collision is forcing the sections apart in your first example, resulting in the cloth being blown out into the unnatural shape.

The solution is therefore to simply add more subdivisions. This will result in  the scaling of the self-collision to be more appropriate and will also produce a more natural behaviour of the cloth - but this will also significantly affect the simulation time.

To demonstrate this working, here's my subdivided example mesh :



I first ran the simulation with lower subdivision and a Self-Collision Distance of 0.55. This resulted in overlapping faces as in your second example. Using a high Distance also caused the cloth to 'blow out' as in your first example.

However, the higher subdivisions allow the Distance to be left at the default value of 0.75 without the cloth being 'blown out' while also preventing it from self-intersecting. In addition, the cloth was allowed to collide quite close together so there is not a significant gap between the layers. This produced the following result :



This was produced with the following cloth settings :



If you still have intersection issues (particularly from fast moving collisions) you may need to increase the Cloth Quality Steps and/or the Collision and Self-Collision Quality settings (this will also slow down the simulation).