title: Embarassingly parallelizable computation with CUDA, how to start?
body: <p>I need to accelerate many computations I am now doing with PyLab. I thought of using CUDA. The overall unit of computation (A) consists in doing several (thousands) entirely independent smaller computations (B). Each of them involves, at their initial stage, doing 40-41 independent, even smaller, computations (C). So parallel programming should really help. With PyLab the overall (A) takes 20 minutes and (B) takes some tenth of a second.</p>
  
  <p>As a beginner in this realm, my question is what level I should parallelize the computation at, whether at (C) or at (B). </p>
  
  <p>I should clarify that the (C) stage consists in taking a bunch of data (thousands of floats) which is shared between all the (C) processes, and doing various tasks, among which one of the most time consuming is linear regression, which is, too, parallelizable! The output of each procedure (C) is a single float. Each computation (B) consists basically in doing many times procedure (C) and doing a linear regression on the data that comes out. Its output, again, is a single float. </p>
  
  <p>I'm not familiar with CUDA programming so I am basically asking what would be the wisest strategy to start with.</p>