Cause of Frayed Edges
There're some faces with twisted vertices the scaled model reveals them:



To fix it remove the faces and recreate them by selecting the edges followed by F.



You might also want to consider to remove the loop cut in the upper part.

Related:


What is non-manifold geometry?
How to create a smooth surface onto a rectangle frame?