How to pass a property from popup to execute
invoke_props_dialog() opens a "blocking" popup, it won't call execute() with the updated values until user confirm with OK button.

Use invoke_props_popup() instead, if you want to immediately re-run the operation:

import bpy

class CurveOutline(bpy.types.Operator):
    """Curve Outliner"""
    bl_idname = "object._curve_outline"
    bl_label = "Create Outline"
    bl_options = {'REGISTER', 'UNDO'}

    outline = bpy.props.FloatProperty(name="Outline", default=0.1, min=-10, max=10)

    @classmethod
    def poll(cls, context):
        return (context.object is not None and
                context.object.type == 'CURVE')

    def execute(self, context):
        createOutline(context.object, self.outline)
        return {'FINISHED'}

    def invoke(self, context, event):
        return context.window_manager.invoke_props_popup(self, event)

def register():
    bpy.utils.register_class(CurveOutline)


def unregister():
    bpy.utils.unregister_class(CurveOutline)

if __name__ == "__main__":
    register()


Added a check for context.object is not None to ensure the operator isn't run with no active object and replace selected_objects[0] by object, as it's not the same. If you intended to work on multiple selected objects, get all selected objects and remove the active one from list to separate them.