Deepening the front fork dropout
Yes- structural integrity and your own well being. 4mm is a lot of metal to remove in the scheme of things. If I were you I would be looking for a long reach brake caliper instead of modifying the dropout by removing material.