What topologies support recognition of action sequences?
This is an old area of AI called "Plan Recognition", which has about 3.5 million results in Google Scholar.

A lot of the modern work is done with classical search techniques coupled with expert domain knowledge, or related reasoning concepts like Hierarchical Task Networks.

I'm not aware of or able to find recent research using deep neural networks for this problem, but I think there are some data-drive approaches to the related work in video-game player modeling.