Why does the Boeing 787 engine nacelle exhaust have such an unusual shape?
Well, it does look pretty, doesn't it?

Boeing wanted to reduce the noise generated because of jet blast of the engines. Many airports around the world are implementing new noise regulations.

As mentioned by Boeing:


  To combat the sound of jet-blast from the rear of the engine, Boeing,
  General Electric, and NASA developed serrated edges called chevrons
  for the back of the nacelle and the engine exhaust nozzle. The
  chevrons reduce jet blast noise by controlling the way the air mixes
  after passing through and around the engine.
  
  The acoustic liners and chevrons are such effective noise suppressors
  that several hundred pounds of sound insulation may be eliminated from
  the fuselage.


As NASA was involved, they state (my paraphrasing):


  Chevrons are the sawtooth pattern seen on the trailing edges of some
  jet engine nozzles. As hot air from the engine core mixes with cooler
  air blowing through the engine fan, the shaped edges serve to smooth
  the mixing, which reduces turbulence that creates noise.
  
  ... The new Boeing 787 is among the most modern jets relying on
  chevrons to reduce engine noise levels. It sports chevrons on the
  nacelles, or fan housings.