title: Spark ALS implicit exception
body: <p>We use ALS on Azure Spark to build our recommender system.</p>
  
  <p>For the computation power reason that we cannot output different recommendation list for each individual user. So we divide users into clusters and use ALS to output recommendation list for each individual cluster centroid.</p>
  
  <p>Before clustering on users, we user standardscaler and normalizer on Spark to preprocess data in order to have better clustering result. However, this will cause the following exception when using ALS.trainImplicit</p>
  
  <blockquote>
    <p>15/11/16 15:43:11 INFO TaskSetManager: Lost task 30.0 in stage 15.0 (TID 197) on executor localhost: java.lang.AssertionError (assertion failed: lapack.dppsv returned 4.) [duplicate 9]
    Traceback (most recent call last):
      File "/home/rogeesjir_huasqngfda/woradofkapkspace/jigsusLaudfadfecher/scripts/RecommendationBackend/AzureSpark/src/collaborativeFiltering/spark_als.py", line 92, in 
        main()
      File "/home/rogeesjir_huasqngfda/rogeesjir_huasqngfda/jigsusLaudfadfecher/scripts/RecommendationBackend/AzureSpark/src/collaborativeFiltering/spark_als.py", line 39, in main
        model = ALS.trainImplicit(ratings, rank, numIter, alpha=0.01)
      File "/home/jigsusLaudfadfecher/spark-1.3.1-bin-hadoop2.6/python/pyspark/mllib/recommendation.py", line 147, in trainImplicit
    15/11/16 15:43:11 INFO TaskSetManager: Lost task 25.0 in stage 15.0 (TID 192) on executor localhost: java.lang.AssertionError (assertion failed: lapack.dppsv returned 4.) [duplicate 10]
        iterations, lambda_, blocks, alpha, nonnegative, seed)
      File "/home/jigsusLaudfadfecher/spark-1.3.1-bin-hadoop2.6/python/pyspark/mllib/common.py", line 120, in callMLlibFunc
        return callJavaFunc(sc, api, *args)
      File "/home/jigsusLaudfadfecher/spark-1.3.1-bin-hadoop2.6/python/pyspark/mllib/common.py", line 113, in callJavaFunc
        return _java2py(sc, func(*args))
      File "/home/jigsusLaudfadfecher/spark-1.3.1-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip/py4j/java_gateway.py", line 538, in <strong>call</strong>
      File "/home/jigsusLaudfadfecher/spark-1.3.1-bin-hadoop2.6/python/lib/py4j-0.8.2.1-src.zip/py4j/protocol.py", line 300, in get_return_value
    py4j.protocol.Py4JJavaError15/11/16 15:43:11 INFO TaskSetManager: Lost task 16.0 in stage 15.0 (TID 183) on executor localhost: java.lang.AssertionError (assertion failed: lapack.dppsv returned 4.) [duplicate 11]</p>
    
    <p>: An error occurred while calling o39.trainImplicitALSModel.
    : org.apache.spark.SparkException: Job aborted due to stage failure: Task 8 in stage 15.0 failed 1 times, most recent failure: Lost task 8.0 in stage 15.0 (TID 175, localhost): java.lang.AssertionError: assertion failed: lapack.dppsv returned 4.
        at scala.Predef$.assert(Predef.scala:179)
        at org.apache.spark.ml.recommendation.ALS$CholeskySolver.solve(ALS.scala:355)
        at org.apache.spark.ml.recommendation.ALS$$anonfun$org$apache$spark$ml$recommendation$ALS$$computeFactors$1.apply(ALS.scala:1131)
        at org.apache.spark.ml.recommendation.ALS$$anonfun$org$apache$spark$ml$recommendation$ALS$$computeFactors$1.apply(ALS.scala:1092)
        at org.apache.spark.rdd.PairRDDFunctions$$anonfun$mapValues$1$$anonfun$apply$15.apply(PairRDDFunctions.scala:674)
        at org.apache.spark.rdd.PairRDDFunctions$$anonfun$mapValues$1$$anonfun$apply$15.apply(PairRDDFunctions.scala:674)
        at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
        at org.apache.spark.storage.MemoryStore.unrollSafely(MemoryStore.scala:249)
        at org.apache.spark.CacheManager.putInBlockManager(CacheManager.scala:172)
        at org.apache.spark.CacheManager.getOrCompute(CacheManager.scala:79)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:242)
        at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
        at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)
        at org.apache.spark.rdd.RDD.iterator(RDD.scala:244)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
        at org.apache.spark.scheduler.Task.run(Task.scala:64)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)</p>
  </blockquote>
  
  <p>When we remove the "regularization" component(i.e, not to do standardscaler and normalizer), everything works fine.
  By the way, ALS.train() call for explicit rating also works fine even we regularize data before ALS model training.</p>
  
  <p>Has anyone ever encounter this kind of problem?
  We are still new so please help! Thanks. </p>