Why effects flow during camera movement in After Effects 2020?
I think it's because the Roughen Edges effect is a pixel-based effect applied to the entire display of your comp.  If you pre-comp the effect first, it won't display the animating edges.
Or you could try applying this expression to the Roughen Edges offset (transform) property, as explained here:
parent.position * 400 / effect(“Roughen Edges”)(“Scale”)