Egg of a dove in my flowerpot
According to the American Dove Association:


  How many eggs do doves lay?
  Most doves lay two eggs; some species only lay one. Wild doves are notorious for infertile eggs.
  
  Why do foreign doves abandon their eggs or young?
  Abandoning eggs may be due to biting pests (lice, mites), noises that frighten brooding birds, or overcrowding and fighting. Frightened doves will abandon young, or if the nest is disturbed in any way, parents may refuse to care for the young.


There is a chance the egg was not fertilized, and there is also a chance the bird could have had pests (whether or not that affects the egg, I do not know).

Based on this info, if it was me, I'd steer clear.