Best way to diagnose embedded computer in field problems?
Recently, I deployed a few units of a new embedded product for a friendly trial. I was trying to discover how well the IP stack performed out in the real world, so using the on-board ethernet for debug wasn't an option.

I bought several serial dataloggers from Sparkfun (many similar devices are available) and attached one to each unit over TTL serial. When I got the units back, I retrieved the logs from the SD cards.

In the case of a system crash, dumping registers out to the serial port will let you find the offending code via the map file later.