title: Parallelized machine learning (recommendation) algorithm with hadoop
body: <p>I am working on a recommendation algorithm : <strong>Matrix Factorization using a stochastic gradient as optimizer.</strong></p>
  
  <p>I would like to parallelized my algorithm. I found this article <a href="http://www.research.rutgers.edu/~lihong/pub/Zinkevich11Parallelized.pdf" rel="nofollow noreferrer">Parallelized Stochastic Descent Gradient</a>. 
  They give an algorithm (p3):</p>
  
  <pre><code>Algorithm 3 SimuParallelSGD(Examples {c1, . . . cm}, Learning Rate η, Machines k)
  
  Deﬁne T = ⌊m/k⌋
  Randomly partition the examples, giving T examples to each machine.
  for all i ∈ {1, . . . k} parallel do
    Randomly shufﬂe the data on machine i.
    Initialize w(i,0) = 0.
    for all t ∈ {1, . . . T }: do
      Get the tth example on the ith machine (this machine), c(i,t)
      w(i,t) ← w(i,t−1) − η∂(w)  ci*(wi,t−1)
    end for
  end for
  Aggregate from all computers v = 1/k SUM(W(i,t) and return v.
  </code></pre>
  
  <p>And they commented with : </p>
  
  <blockquote>
    <p>"The algorithm requires no communication
    between machines until the end. This is perfectly suited to MapReduce settings."</p>
  </blockquote>
  
  <p>I can't see how they implement it. Did they distribute the data on each machine "<em>manually</em>"? or using <em>HDFS</em>?</p>
  
  <p>As HDFS take care of distribution of the data; how can I restrict my program to work only with the data, on his current data-node?</p>
  
  <p>Is there a suited way to implement it with a <strong>spark + hadoop cluster</strong>? </p>
  
  <p>Any leads is welcome</p>
  
  <hr>
  
  <p>I forgot to mention something. I saw the example <strong>regression logistic</strong> with <strong>SGD</strong> on spark.</p>
  
  <p>But the dataset is just a </p>
  
  <pre><code>      UserID | ItemID | ratings
  </code></pre>
  
  <p>and in my case to reinforce the the algorithm we add some Users informations and Items information. So I have two more files (UserID age sex..) and (ItemID color size..) that I need on each data-node in order to complete the computation.</p>
  
  <p>I was thinking maybe to put the number of replicate of HDFS equal to the number of machines and in this way all the data will be on each node. But the thing I don't know is when I will do on spark that: </p>
  
  <pre><code> val textFile = sc.textFile("README.md")
  </code></pre>
  
  <ol>
  <li>What will happen?  </li>
  <li>Will rdd be created on each datanode's memory?</li>
  <li>Does all the task could be process only in local by each datanode?</li>
  </ol>