1. A computer-executable method for performing a general matrix-vector multiplication (GEMV) operation on a graphics processing unit (GPU), comprising: generating a set of kernels that includes at least one of a variable-N GEMV kernel and a constant-N GEMV kernel, wherein a respective kernel of the set of kernels is a function compiled for execution on the GPU; maintaining a set of rules, wherein a respective rule of the set of rules includes a mapping between a set of dimensional parameters, which includes a number of rows of a matrix and a number of rows of a vector, and a corresponding kernel from the set of kernels, wherein the mapping indicates that the corresponding kernel can perform the GEMV operation faster than any other kernel in the set of kernels for the set of dimensional parameters in the mapping; receiving data representing at least a matrix and a vector; determining a first set of dimensional parameters associated with the matrix and the vector; loading a kernel from the set of kernels on the GPU for performing the GEMV operation on the GPU, which involves looking up the first set of dimensional parameters in the set of rules; and executing the loaded kernel on the GPU using the matrix and the vector as parameters, thereby performing the GEMV operation on the matrix and the vector.