Anomaly detection for transaction data
There are a few factors to consider in anomaly detection.  A simple method would be to plot a boxplot of the data and calculate outliers this way (boxplot description).

If the time series is seasonal, you could take an approach similar to Twitter's anomaly detection algorithm (Twitter Anomaly Detection).  This uses an outlier detection method (ESD) applied to the seasonal decomposition of the time series.

You could use clustering and SVMs as well.

There is also change-point detection which I understand less but can be very effective.

Really, your answer depends on how complex your data is.  Sometimes, a simple heuristic algorithm works the best.

Here are some similar questions:

Anomaly detection in Time Series Data - Help Required

How to classify and cluster this time series data