I am using the Association Rules algorithm using this:http://spark.apache.org/docs/latest/mllib-frequent-pattern-mining.htmlI'm have 83945 transactions in my dataset. And I want to filter some products that only appears X times in my dataset. Basically, I want to set my cut-off. My question is: How can I define my cut-off, this is how to define the minimum number of occurences that my products need to have?Many thanks!
Cut-Off using Frequent Pattern Mining - Spark Mllib
data mining;apache spark;association rules
Like @SeanOwen pointed out its called support. spark.mllibs FP-growth implementation takes it as a hyper-parameter under minSupport.It is the minimum support for an itemset to be identified as frequent, e.g : if an item appears 4 out of 5 transactions, it has a support of 4/5=0.8.Usage:import org.apache.spark.mllib.fpm.FPGrowthval transactions: RDD[Array[String]] = ???val fpg = new FPGrowth().setMinSupport(0.2)val model = fpg.run(transactions)I hope this helps. 