How do you calculate cos(theta) for diffuse in flat shading?
With flat shading, the normal is the same across the whole triangle (or quad or whatever). There are some assumptions that were made here and two approaches. The latter approach will probably be best for most basic problems (although more work). We don't have the winding order, so we need to calculate the normal based on the vertices.

Average normal approach

The assumption is that each vertex contributes the same amount of weight towards the triangle normal.

We first calculate the average of each:
$$normal = normalize(0,0,1) + normalize(1,1,0) + normalize(-1,-1,1) \approx \frac{(0,0,1)+(0.7,0.7,0)+(-0.58,-0.58,0.58)}{3} \approx (0.04,0.04,0.53)$$
$$normalize(0.04,0.04,0.53) \approx (0.076,0.076,0.994)$$

For Lambert shading:
$$cos(\theta) = normalize(0,3,4) \cdot (0.76,0.76, 0.994) \approx 0.84 $$

Geometric approach (recommended for this problem)

Another assumption Nathan Reed suggested was that each vertex normal holds equal weight, which would make sense in more common use cases. For this, we will want to find the face normals in both winding directions to compare this value to:

Winding order 1

$$normal1 = normalize((B-A) \times (O-A)) \approx (0.29,0.43,-0.85)$$

Winding order 2

$$normal2 = normalize((O-A) \times (B-A)) \approx (-0.29,-0.43,0.85)$$

Now, we can take the dot product between these two vectors and the one we computed above. Whichever one is greater or equal to 0:
$$geometric\ normal1 = normal \cdot normal1 \approx -0.79$$
$$geometric\ normal2 = normal \cdot normal2 \approx 0.79$$

So then,
$$normal = geometric\ normal2$$

For Lambert shading:
$$cos(\theta) = normalize(0,3,4) \cdot (-0.29,-0.43,0.85) \approx 0.42 $$