invoke_props_dialog: callback during change?
The props_dialog shows Properties, so you can define set/get functions for them like for any other Property to control changes: https://docs.blender.org/api/2.79/bpy.props.html#get-set-example

Example for your case:

class SomeOperator(Operator):
    bl_idname = "test.some_operator"
    bl_label = "Example"

    def set_start(self, value):
        if value &gt; self["stop_frame"]:
            print("start frame has to be smaller or equal to stop frame")
            self["start_frame"] = self["stop_frame"]
        else:
            print("setting start frame")
            self["start_frame"] = value

    def get_start(self):
        return self["start_frame"]

    start_frame = bpy.props.IntProperty(set=set_start, get=get_start)
    stop_frame = bpy.props.IntProperty()


Notes:


You have to define both a setter and a getter (you can't define only a setter, the property will return always 0 if you set no getter).
In the setter/getter you should call the operator's properties with self["property_name"], not self.property_name. This way you can get/set the value without calling your own setter/getter functions (which would lead to infinite recursion).