Speedup GPU vs CPU for matrix operations
Question: I am wondering how much GPU computing would help me speed up my simulations.

The critical part of my code is matrix multiplication. Basically the code
looks like the following python code with matrices of order 1000 and long for
loops.

    
    
    import numpy as np
    Msize = 1000
    simulationLength = 50
    
    a = np.random.rand(Msize,Msize)
    b = np.random.rand(Msize,Msize)
    
    for j in range(simulationLength):
        result = np.dot(a,b)
    

Note: My matrices are dense, mostly random and for loops are compiled with
cython.

My naive guess would be that I have two factors:

  * More parallel threads (Currently of order 1 thread, GPUs of order 100 threads?) --> Speedup of order 100? [[Source](http://gamedev.stackexchange.com/a/17255) is quite outdated, from 2011]
  * Lower processor frequency (Currently 3Ghz, GPUs typically 2 Ghz) --> Neglect

I expect that this viewpoint is to naive, so what am I missing?


Answer: Generally speaking GPUs are much faster than CPU at highly parallel simple
tasks (that is what they are made for) like multiplying big matrices but there
are some problems coming with GPU computation:

  * transfering data between normal RAM and graphics RAM takes time
  * loading/starting GPU programs takes some time

so while multiplication itself may be 100 (or more) times faster, you might
experience an actually much smaller speedup or even a slowdown

There are more issues with GPUs being "stupid" in comparison to CPUs like
massive slowdowns on branching code, having to handle caching by hand and
others which can make writing fast programs for GPUs quite challenging.

