How to select concave quads
This may be a naive answer, I'm no coder, but is there anything wrong with letting BMLoop.is_convex do this for you?
(Starting from Edit Mode):
import bpy
import bmesh

obj = bpy.context.edit_object
me = obj.data

bm = bmesh.from_edit_mesh(me)
bm.faces.active = None

for face in bm.faces:
    face.select_set(False)
    for loop in face.loops:
        if not loop.is_convex:
            face.select_set(True)
            break

bmesh.update_edit_mesh(me, False)