Is it a rules violation for a pilot to intentionally frighten a passenger to the point of hysterics?
There's no regulation that I can see that says "don't frighten your passengers". And there probably shouldn't be: almost anything could frighten a nervous person who isn't familiar with aircraft, even if the maneuver (or whatever) is completely normal.

Having said that, if the FAA thinks you've done something stupid then they do have a 'catch-all' regulation that they could argue you violated.

14 CFR 91.13


  No person may operate an aircraft in a careless or reckless manner so
  as to endanger the life or property of another


I haven't watched the video but I think it's at least plausible that the FAA could say that there was a real risk that the panicked passenger would take the controls and try to fly the aircraft themselves. In a dive, pulling up suddenly could easily stall the aircraft and stall recovery is not intuitive or obvious to a non-pilot. In the worst case, pulling up suddenly from a dive can actually tear the wings off. So it would be very easy for the passenger to damage or destroy the aircraft and potentially fly into people or property on the ground. 

Finally, apart from the aviation regulations, it might be possible for the passenger to sue the pilot for causing stress and emotional damage. You could always ask on law.SE about that.