How to reopen folder select dialog (probbably by clearing string property)
I'd suggest just recall the operator, eg  the operator file export template operator_file_export.py  Used a directory property for a folder path, to keep filepath as a file path (usual usage).

class ExportSomeData(Operator, ExportHelper):
    """This appears in the tooltip of the operator and in the generated docs"""
    bl_idname = "export_test.some_data"  # important since its how bpy.ops.import_test.some_data is constructed
    bl_label = "Export Some Data"

    # ExportHelper mixin class uses this
    filename_ext = ".txt"

    filter_glob = StringProperty(
            default="*.txt",
            options={'HIDDEN'},
            maxlen=255,  # Max internal buffer length, longer would be clamped.
            )

    directory = StringProperty(subtype='DIR_PATH')

    def execute(self, context):
        if os.path.isfile(os.path.join(self.directory, "test.txt")):
            return {'FINISHED'}
        else:
            return bpy.ops.export_test.some_data('INVOKE_DEFAULT')