How do you determine if a new microcontroller is defective?
It is rare to have that kind of failure.  You might expect to see a little more noise on a pin, or have that pin completely non-functional.  But to have it "somewhat work, but not in a useful way" is rare.  I would suspect there are design issues that are causing the problems, and have something to do with a difference between the 164A and 164P.  Since jitter is high, I would look at power related things.  Are all the power/gnd pins connected?  Are are the I/O pins either driven or pulled high or low?  Etc.

But there still remains the possibility that the parts are bad.  It's rare, but not unheard of.  The only real way to tell is to get some more parts, from a different supplier, and try them.  If they work, then you need to investigate further and see if you killed them in handling/soldering or if they really did come from Digikey bad.