how to identify patterns in program execution flows?
There's probably no existing turnkey solution that will do what you want, out of the box.  I suspect this would be a research-level task, so you might want to start by searching the research literature to see if anyone else has tried to do something like this.  (Google Scholar is useful.)

If you some advice on how to pursue this yourself, my advice would be:


Try to narrow the scope.  Rather than trying to detect all bugs, pick a single class of bugs (e.g., use-after-free would be a fine choice) and see if you can have any success detecting those, as a first step.  If you're successful then you could broaden the scope.
Think now about how you'll obtain enough data to work with.  I suspect one of the biggest challenges will be how to obtain a large training set with known labels, that have many instances of in-the-wild use-after-free bugs, as well as many other traces with no instances of those type of bugs, labelled with whether there was a bug or not and which function invocations were responsible.
Take a look at association rule learning, sequential pattern mining, grammar induction and automata learning (e.g., my links here; e.g., you might try to learn a small automaton that detects use-after-free bugs), and recurrent neural networks (especially LSTM).
Also, you might need to do some feature engineering.  For instance, when you have pointers or handles (like 0x100), you might want to canonicalize them, as the only useful information is whether two calls used the same pointer or a different pointer -- the actual value of the pointer is irrelevant.  Since you have domain knowledge of that fact, by transforming the data before you feed it to the data mining / machine learning framework, you can help it learn a lot faster and more effectively.