What are GPUs bad at?
GPUs are bad at doing one thing at a time.  A modern high-end GPU may have several thousand cores, but these are organized into SIMD blocks of 16 or 32.  If you want to compute 2+2, you might have 32 cores each compute an addition operation, and then discard 31 of the results.

GPUs are bad at doing individual things fast.  GPUs only recently topped the one-gigahertz mark, something that CPUs did more than twenty years ago.  If your task involves doing many things to one piece of data, rather than one thing to many pieces of data, a CPU is far better.

GPUs are bad at dealing with data non-locality.  The hardware is optimized for working on contiguous blocks of data.  If your task involves picking up individual pieces of data scattered around your data set, the GPU's incredible memory bandwidth is mostly wasted.