What type of AI would you recommend for this complex problem?
You are trying to solve a variant of The Halting Problem, which is the problem of detecting whether a computer program is going to stop, or run forever.

The Halting Problem is incomputable, which means it is not possible to write a computer program that solves it. It is easy to see that your problem is also incomputable. If you could predict whether a program would generate an error, then for any program X that someone wants to solve the halting problem for, we could write a new program:


run(X)
Error("X finished running").


and use your algorithm to determine whether X would finish running or not. Since the halting problem is known to be incomputable, this means your problem must be incomputable too.

That's not to say all is lost though. Formal verification is a field that uses some AI techniques (mostly reasoning-based, but I think there's some machine learning now too) to try to solve this problem for some programs. It can't work for every program though.