Learning about security of UART and RS232
RS232 is an electrical and connector specification for how the voltages are interpreted as 1 or 0, and how they are connected, nothing more. 

A UART essentially does the parallel to serial conversion and adds (and strips) some usually fairly minimal amount of framing around the (usually) bytes it sends, its interface to the processor is usually a memory mapped register and an interrupt or polling, DMA as such would be uncommon but possible, uart serial is usually just not that quick. There is nothing that requires either that RS232 be async serial from a UART (But it usually is) or that a UART ultimately drive RS232.

Hyperterminal is a program that can be configured to talk to a serial port, but you cannot assume that is the only attack vector.

The place you will find software vulnerabilities is in the higher levels of whatever protocol you are carrying over the RS232 link, for example you might be carrying TCP/IP encapsulated as SLIP over that link, which is a huge attack surface even if the high level application is something simple. The UART and the dozen or so lines of code that read bytes and stuff them into the rx ring buffer is seldom the problem, it is all the complex stuff that happens afterwards.
Best case, I can convince some of the higher level stuff to crash, possibly the developer has left an assert on with the dump set to the serial port... Game on!

For hardware vulnerabilities, I would be thinking side channel attacks, either looking at noise on the signal lines (Can be used to estimate processor power consumption, which with poorly done crypto can leak key information). 
Possibly I might be able to abuse an input protection diode to force the power supply for the processor out of tolerance, maybe I learn something useful.... 
Maybe I can apply some RF to the serial port and get a crash that way, maybe I apply RF, sense the leakage on the supply wires and find that it is phase modulated in a way that leaks information, lots of things to try.
Been there, done that. 

You may find a talk given at 29C3 by Timo entitled "Milking the digital cash cow" interesting (It is on youtube as well as in the CCCs archives) as it demonstrates a side channel attack in the real world.