How to find operation type in validating form?
A new node will not have a node ID.

So you can check the form values or the original form for that to determine whether it is a new node or not.

For example:

if (empty($form_state['values']['nid'])) {
  // This is a new node.
}
else {
  // This is an existing node.
}