Played with some lead solder without washing my hands for awhile, should I be concerned?
Lead solder is not all that bad- it's cumulative exposure that can cause problems.

Here is the MSDS (Material Safety Data Sheet) for a typical tin-lead solder. It says, in part, 


  Keep away from foodstuffs, beverages and feed.
  Immediately remove all soiled and contaminated clothing.
  Wash hands before breaks and at the end of work.


Note also that, as @MattYoung says, the "no-clean" flux is quite nasty- LD50 on rats is 4mg/kg. From this occupational health and safety site: 


  What does LD50 mean?
  
  LD stands for "Lethal Dose". LD50 is the amount of a material, given
  all at once, which causes the death of 50% (one half) of a group of
  test animals. The LD50 is one way to measure the short-term poisoning
  potential (acute toxicity) of a material.
  
  Toxicologists can use many kinds of animals but most often testing is
  done with rats and mice. It is usually expressed as the amount of
  chemical administered (e.g., milligrams) per 100 grams (for smaller
  animals) or per kilogram (for bigger test subjects) of the body weight
  of the test animal. The LD50 can be found for any route of entry or
  administration but dermal (applied to the skin) and oral (given by
  mouth) administration methods are the most common.


The reasons for only washing hands before breaks and after work is so that you don't ingest lead from handling food etc. or other contact with mucous membranes. 

Solder paste is considerably easier to ingest, so more care is called for. 

Also, if you are pregnant or could become pregnant, much more care should be taken.