What are some well-known problems where neural networks don't do very well?
Here's a snippet from an article by Gary Marcus 


  In particular, they showed that standard deep learning nets often fall
  apart when confronted with common stimuli rotated in three dimensional
  space into unusual positions, like the top right corner of this
  figure, in which a schoolbus is mistaken for a snowplow:
  
  
  
  .
  .
  .
  
  Mistaking an overturned schoolbus is
  not just a mistake, it’s a revealing mistake: it that shows not only
  that deep learning systems can get confused, but they are challenged
  in making a fundamental distinction known to all philosophers: the
  distinction between features that are merely contingent associations
  (snow is often present when there are snowplows, but not necessary)
  and features that are inherent properties of the category itself
  (snowplows ought other things being equal have plows, unless eg they
  have been dismantled). We’d already seen similar examples with
  contrived stimuli, like Anish Athalye’s carefully designed, 3-d
  printed foam covered dimensional baseball that was mistaken for an
  espresso
  
  
  
  Alcorn’s results — some from real photos from the natural
  world — should have pushed worry about this sort of anomaly to the top
  of the stack.


Please note that the opinions of the author are his alone and I do not necessarily share all of them with him.

Edit: Some more fun stuff

1) DeepMind's neural network that could play Breakout and Starcraft
saw a dramatic dip in performance when the paddle was moved up by a few pixels.

See: General Game Playing With Schema Networks

While in the latter, it performed well with one race of the character but not on a different map and with different characters.

Source

2) 


  AlphaZero searches just 80,000 positions per second in chess and
  40,000 in shogi, compared to 70 million for Stockfish and 35 million
  for elmo.


What the team at Deepmind did was to build a very good search algorithm. A search algorithm that includes the capability to remember facets of previous searches to apply better results to new searches. This is very clever; it undoubtedly has immense value in many areas, but it cannot be considered general intelligence.

See: AlphaZero: How Intuition Demolished Logic (Medium)