Choosing between internal oscillator and external crystal- ATXMega32E5
The main advantage of external quartz crystals are their frequency stability over a wide range of temperatures. RC oscillators, on the other hand, tend to vary considerably over temperature.

A crystal, for example, may advertise a frequency stability of 50ppm. That's just 0.005% drift over the entire range of temperatures specified. Taking a quick look at the ATXMega32E5's datasheet shows the internal RC is only good for 0.5% and that's just at 25C. Go hotter or colder and your oscillator frequency will drift even more. To make matters worse, manufacturers make no promise which way the oscillator frequency will drift. It may be different from chip to chip. In the worst case scenario, the oscillator in one device will drift faster and the other will drift slower, causing possibly sporadic and difficult to trace communication corruptions.

Under "nice" conditions, you should have no problem successfully using UART communication with the internal RC oscillator. But if the devices will experience large temperature swings, it's best to switch to something with better accuracy, like a crystal.