Is it legal to take advantage of my opponent's priority mistakes?
Your opponent attempting to cast Lightning Bolt when they don't have priority is a legal shortcut as outlined in the comprehensive rules:


  716.1. When playing a game, players typically make use of mutually understood shortcuts rather than explicitly identifying each game
  choice (either taking an action or passing priority) a player makes.


and also:


  716.1a The rules for taking shortcuts are largely unformalized. As long as each player in the game understands the intent of each other
  player, any shortcut system they use is acceptable.


Clearly your opponent's intent was to cast Lightning Bolt assuming you just planned to pass priority. Therefore, after your opponent has tried to cast Lightning Bolt, you have the option of either:


Telling your opponent that he doesn't have priority and forcing him to undo the casting of Lightning Bolt
Allow the shortcut to happen at which point you can still cast Giant Growth as you should now have priority


Yes, it is legal for you to cast Giant Growth in response to your opponent ignoring your priority.

Edit

Some people claim that only the player with priority can propose a shortcut. While the rules do state this, they also state that any shortcut method that is understood and unambiguous is acceptable. Since players in every magic game I have ever seen (including MTG pro tour and similar high level play) ignore their opponent's priority when moving from their upkeep to their draw phase it seems to me that this behavior should be acceptable during other phases as well. Players may also go into the declare attackers phase by just tapping some of their creatures effectively skipping their opponent's priority. This is fairly common and took me no more than about one minute to find this example and this example. Skipping your opponent's priority in this manner is exactly the same as skipping it after they declared attackers so you can cast Lightning Bolt.

What is not okay is saying you are going to do something and then not actually doing it after your opponent decides to not respond to it. (I'm not really sure how this situation came up as it isn't in the original question, but some answers have addressed it)