good algorithm for outliers detection
I'd go out on a limb to say this is a difficult problem without incorporating other data. Models just try to explain future action with past information. Traders act differently in different scenarios. 

Without data on their analysis or financial data on the stocks they are analyzing, a model on the previous transaction history of traders would not be effective. Unless, if the "traders" you describe are more like etf fund managers that operate on a predefined set of rules.