set special mouse button to toggle between selection modes
To cycle through the possible selection options, you will need an operator that looks at the current selection to choose which selection option should be enabled.

That can be done in a simple operator, which you can then assign to your mouse button.

class SelectCycle(bpy.types.Operator):
    bl_idname = 'mesh.select_cycle'
    bl_label = 'Cycles through selection modes.'

    def execute(self, context):
        ts = context.tool_settings
        if ts.mesh_select_mode[0]:
            ts.mesh_select_mode = [False,True,False]
        elif ts.mesh_select_mode[1]:
            ts.mesh_select_mode = [False,False,True]
        else:
            ts.mesh_select_mode = [True,False,False]
        return {'FINISHED'}


You can find a copy of this as an addon here, so you can enable it in your startup.blend.

In your input settings, you just need to enter mesh.select_cycle for the idname and choose the mouse button.