How can I align a model to three custom points with a Python script?
Create the matrix world

The rotation part of an objects world matrix is an orthogonal matrix where the columns represent the local axes. 

To demonstrate, in test code below,  I have added a plane and removed one vertex to visualize the  three points A, B and C ensuring the angle between  BA and CA is 90 degrees.  


Cube (orange wireframe) aligned to 3 vertex, 2 edge "plane" Vertex 0 for B, 1 for A and 2 for C

The vectors BA and CA, normalized,,  are used for the x and y axis directions.

The cross product of x and y is the z axis.  (The negated value would be the "flipped" option)

Finally build the matrix world from the 3 axis vectors, make it 4x4 and the translation A.

Test script. Move and rotate the "plane" then run script to align "Cube" such that its origin is at A, and the x and y axis align.

import bpy
from mathutils import Matrix
context = bpy.context

scene = context.scene
tri = scene.objects.get("Plane")
# global coords of 3 verts in plane.
b, a, c = [tri.matrix_world @ v.co for v in tri.data.vertices]

x = (b - a).normalized()
y = (c - a).normalized()
z = x.cross(y)

M = Matrix([x, y, z]).transposed().to_4x4()
M.translation = a

# Cube is the test object

ob = scene.objects.get("Cube")
ob.matrix_world = M


Note I have used blender 2.8. Replace matrix - vector multiplication @ with * for prior versions.