Do dirty brake pads cause them to wear faster?
Yes, dirt is a realistic cause for increased brake pad wear

According to Chain Reaction Cycles


  Organic pads – also referred to as ‘resin’ pads – are made from a high-density ceramic and have a softer compound than metallic sintered pads. Organic pads generally provide better stopping power and heat dissipation, which means that they are slower to heat up and so are advisable for use in braking systems with a low boiling point. However organic pads can wear out more quickly in wet and gritty conditions. 
  
  Sintered pads: Also referred to as ‘metallic’ pads, these use a compound with added metal content – usually copper shavings. This makes them noiser, and they heat up faster, but in wet conditions they last far, far longer than organic pads. Ultimately your choice may depend on the prevalent conditions you ride in, with sintered pads advisable for the typical wet/mixed weather encountered in the UK (unless you have no problem changing your pads often).http://hub.chainreactioncycles.com/buying-guides/components/brake-pad-buying-guide/


It depends on many variables but cleaning the pads after each ride might help. 


The dirt on the pads at the end of a ride won't be there at the beginning of the next ride. 


If the next ride has less dirt then not having the previous ride's dirt might help a lot. 
If the next ride gets dirty quickly then the benefit is reduced.



Sintered/Metallic Pads  

There is an on-going debate over the wisdom of switching from resin to metallic
According to Shimano:


  The SM-RT30/53/51 rotor should be used together with resin pads. If it is used with metal pads, the pads will wear out very rapidly. https://si.shimano.com/pdfs/si/SI-8JW0A-001-ENG.pdf


Determining what works best for you would require some experimentation.