Cloth simulation makes object "cripple"
A large part of the problem here is with collisions and the boundaries of collisions. This is resulting in some of the meshes being effectively intersecting at the start of the simulation - or becoming trapped below the surface part way through the simulation - which is then dragging the surface into the mesh in an unpredictable manner.
The first thing to change is the collision on the button meshes - these are hooked onto the front of the shirt and have their own collision settings. This is having two effects - one is that it is interferring with the shirt itself (since it is pinned to the surface and is also colliding with the surface it is causing a conflict in the forces that result). Since the buttons are small and should not significantly affect the simulation it is easiest to simply remove the Collision from all of the buttons.
Secondly, I noticed that the 'neck' section of the tie is getting dragged under the surface of the shirt at the start of the simulation. This is a sign that the meshes are starting the simulation effectively already intersecting. While they don't appear to be intersecting, the collision settings (in the Collision physics modifier and also the Cloth settings themselves) include 'bounds' that can extend beyond the mesh surface. When diagnosing such issues it is a good start point to change those bounds (the Inner and Outer settings in the Soft Body and Cloth settings of the Collision settings and the 'Distance' in the Cloth Collision settings). Note that as soon as you get one cloth vertex dragged below the surface of a collider, it can result in more of the cloth being pulled through the mesh.
Once the above changes had been applied I re-ran the simulation with a much better result. Any further discrepancies should be able to be resolved by ensuring the meshes start slightly further apart in the problem areas, by reviewing your geometry to avoid large variations in edge lengths (areas of cloth where the vertices are close together will behave differently to those where they are further apart) and I also noticed that the tie itself is made up of two layers of faces - it would be better to model as a single surface and then add a Solidify modifer to give it thickness.