from theano import function, config, shared, sandbox
import theano.tensor as T
import numpy
import time

vlen = 10 * 30 * 768  # 10 x #cores x # threads per core
iters = 1000

rng = numpy.random.RandomState(22)
x = shared(numpy.asarray(rng.rand(vlen), config.floatX))
f = function([], T.exp(x))
print(f.maker.fgraph.toposort())
t0 = time.time()
for i in range(iters):
    r = f()
t1 = time.time()
print("Looping %d times took %f seconds" % (iters, t1 - t0))
print("Result is %s" % (r,))
if numpy.any([isinstance(x.op, T.Elemwise) for x in f.maker.fgraph.toposort()]):
    print('Used the cpu')
else:
    print('Used the gpu')

"""
Explanation: To understand what CUDA is and why it is important, I read 
this article. It is a subtle plug for their services, but it aligns well with all that I have so far heard about GPUs in data science and analytics. I even learned who made it and what the acronym means: "Nvidia introduced ... CUDA (Compute Unified Device Architecture." 
CUDA  is apparently hard to get to work on Windows, and we're going to see if that is true or not. This is a link to the  tutorial/instruction set I'm going to try out. 
Make sure your computer has a compatible CUDA graphics card
Head here to check it out for your own card. I the card name in my device manager, a GeForce GTX 960M, and it is on the CUDA list. Then I confirmed I had the card my system thinks it has by running the Nvidia hardware scanner. Check! Onward!
Download CUDA
I grabbed the download from here, and chose these options:
    - version 8.0 
    - for windows 10
    - x86_64 architecture
    - local installer
I opted for local in case I have to run the whole thing over again after some problem comes up. It's 1.2 GB, so the download is going to take a while. 
Download and Install Visual Studio 2013 Community version
I tried to grab it from here. That failed because it wanted me to sign up for some account, and the process was painful to start. So I searched for another way to avoid having to sign up for anything, and I found this site. Downloaded the 2013 vs-community.exe, and it starts up saying it is VS 2013, hooray!
In the options box during the installer, chose to install 
    - "Microsoft Foundation Classes for C++"
as the only optional feature, and the download is still 9GB. I'll be back in an even longer while. 
Install CUDA, then check it
VS 2013 appears to have installed just fine, so now to try CUDA. I ran the extractor and installer, no problems. 
I tried to find the file, vs2013.sln, the instructions say to run, but no luck in the spot it suggested it would be. The CUDA folder isn't there. I tried searching the C drive, but no luck either. I tried re-extracting into a different folder, and as an administrator, and that turns out that was the problem. 
The first time I ran it wasn't as an administrator, but it just crashed without error, which is always handy. Now running it as an admin, it ran fine. I chose the custom install when it asked, and I selected everything under CUDA and de-selecting everything else because I already had newer versions installed for my graphics card. It then installed, for real, and it reported no errors. 
I ran 
bash
C:\ProgramData\NVIDIA Corporation\CUDA Samples\v7.0\1_Utilities\deviceQuery\deviceQuery_vs2013.sln
in Visual studio, and VS built the run time and it reported PASS at the bottom of this report. 
```bash
 CUDA Device Query (Runtime API) version (CUDART static linking)
Detected 1 CUDA Capable device(s)
Device 0: "GeForce GTX 960M"
  CUDA Driver Version / Runtime Version          8.0 / 8.0
  CUDA Capability Major/Minor version number:    5.0
  Total amount of global memory:                 2048 MBytes (2147483648 bytes)
  ( 5) Multiprocessors, (128) CUDA Cores/MP:     640 CUDA Cores
  GPU Max Clock rate:                            1176 MHz (1.18 GHz)
  Memory Clock rate:                             2505 Mhz
  Memory Bus Width:                              128-bit
  L2 Cache Size:                                 2097152 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(65536), 2D=(65536, 65536), 3D=(4096, 4096, 4096)
  Maximum Layered 1D Texture Size, (num) layers  1D=(16384), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(16384, 16384), 2048 layers
  Total amount of constant memory:               65536 bytes
  Total amount of shared memory per block:       49152 bytes
  Total number of registers available per block: 65536
  Warp size:                                     32
  Maximum number of threads per multiprocessor:  2048
  Maximum number of threads per block:           1024
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)
  Maximum memory pitch:                          2147483647 bytes
  Texture alignment:                             512 bytes
  Concurrent copy and kernel execution:          Yes with 1 copy engine(s)
  Run time limit on kernels:                     Yes
  Integrated GPU sharing Host Memory:            No
  Support host page-locked memory mapping:       Yes
  Alignment requirement for Surfaces:            Yes
  Device has ECC support:                        Disabled
  CUDA Device Driver Mode (TCC or WDDM):         WDDM (Windows Display Driver Model)
  Device supports Unified Addressing (UVA):      Yes
  Device PCI Domain ID / Bus ID / location ID:   0 / 1 / 0
  Compute Mode:
     < Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 8.0, CUDA Runtime Version = 8.0, NumDevs = 1, Device0 = GeForce GTX 960M
Result = PASS
```
Setup System Variables
As per instruction, I added 
bash
THEANO_FLAGS = "floatX=float32,device=gpu,nvcc.fastmath=True"
then added 
bash
VS2013path = C:\Program Files (x86)\Microsoft Visual Studio 12.0\VC\bin
and finally tacked on
bash 
%VS2013path%
to the system path. 
Final Check
I tried importing Theano on the command line in the machine learning environment I created previously, and got the success report out the other side!
```bash



import theano
Using gpu device 0: GeForce GTX 960M (CNMeM is disabled, cuDNN not available)
```
That seems too easy, but I'll take it. Now to start playing with it in PyMC3, and see if I notice a difference. 



Making Theano work the the GPU in jupyter
In the theano documentation, I found this script test to see if I am using the GPU. I goig to ru it from this notebook, and I think I know what will happen.
End of explanation
"""


import theano.sandbox.cuda
theano.sandbox.cuda.use("gpu0")

"""
Explanation: Yup, it looks like it is not using the GPU :( Running the same thing from the command line results in success, though.
bash
Using gpu device 0: GeForce GTX 960M (CNMeM is disabled, cuDNN not available)
[GpuElemwise{exp,no_inplace}(&lt;CudaNdarrayType(float32, vector)&gt;), HostFromGpu(GpuElemwise{exp,no_inplace}.0)]
Looping 1000 times took 0.875091 seconds
Result is [ 1.23178029  1.61879349  1.52278066 ...,  2.20771813  2.29967761
  1.62323296]
Used the gpu
Argh!
I found this stack overflow question, and I think I have the same problem. Lets see if I get the same error message.
End of explanation
"""


import sys
sys.path

"""
Explanation: I do, and the solution says it has something to do with my path not being set properly. Lets see what my path is for this notebook instance.
End of explanation
"""


cuda_paths = ['C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v8.0\\bin',
                'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v8.0\\libnvvp',
                'C:\\Program Files (x86)\\NVIDIA Corporation\\PhysX\\Common',
                'C:\\Program Files (x86)\\Microsoft Visual Studio 12.0\\VC\\bin',
             ]

for path in cuda_paths:
    sys.path.append(path)

import theano.sandbox.cuda
theano.sandbox.cuda.use("gpu0")

"""
Explanation: It doesn't include any of the CUDA or VS stuff, so I'm betting this is the problem. I'll add the path items related to CUDA or VS I have in my path to this path, and try again.
End of explanation
"""


sys.path

"""
Explanation: Still not working!
End of explanation
"""


%%cmd
nvcc -V
path

"""
Explanation: Doesn't work. Lets see if I can find this nvcc compiler on the system. It is on the system, and also on the system path because this command worked fine on the command line.
```bash

nvcc -V
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Sat_Sep__3_19:05:48_CDT_2016
Cuda compilation tools, release 8.0, V8.0.44
```
If I try running the cmd from a notebook cell, I get an error.
End of explanation
"""


%%cmd
nvcc -V

import theano.sandbox.cuda
theano.sandbox.cuda.use("gpu0")

from theano import function, config, shared, sandbox
import theano.tensor as T
import numpy
import time

vlen = 10 * 30 * 768  # 10 x #cores x # threads per core
iters = 1000

rng = numpy.random.RandomState(22)
x = shared(numpy.asarray(rng.rand(vlen), config.floatX))
f = function([], T.exp(x))
print(f.maker.fgraph.toposort())
t0 = time.time()
for i in range(iters):
    r = f()
t1 = time.time()
print("Looping %d times took %f seconds" % (iters, t1 - t0))
print("Result is %s" % (r,))
if numpy.any([isinstance(x.op, T.Elemwise) for x in f.maker.fgraph.toposort()]):
    print('Used the cpu')
else:
    print('Used the gpu')

"""
Explanation: But I threw in a call to show the system path as well, to see what it says. I note that the system path is not updated with the CUDA variables, but it does have everything else I have on my path normally, unlike the python system path above. I think I understand now. 
I started this jupyter session before I installed CUDA and VS, so it was started before I or the installers made any changes to the system path. The system path variable the jupyter notebook server grabbed and is holding in memory is the old, out of date one. If I'm right, this problem with CUDA should fix itself  when I restart the notebook server.
Restarted Jupyter Notebook server
Lets try the commands and tests again.
End of explanation
"""


