Disabling "access to contacts" dialog box with terminal
Mail will ask you only one time, and if you deny it should stop asking.

You can use the

defaults write com.apple.mail SuppressAddressHistory -bool true


However if you need to deny the access already given following are the instructions for that.

According to OS X Man pages you can reset privacy for apps to use your address book. 

I did not try it since I do not want to revoke my settings, but it is safe to say it works on 10.9 according to Apple.

DESCRIPTION

The tccutil command manages the privacy database, which stores decisions the user has made about whether apps may access personal data.

One command is current supported:

reset    Reset all decisions for the specified service, causing apps to prompt again the next time they
              access the service.

EXAMPLES
     To reset all decisions about whether apps may access the address book:

   tccutil reset AddressBook