How to configure programmatically a unique constraint on several fields of a custom entity?
As suggested by elstevenson, I used the validation API.

Here the steps.


Create a repository in your module repo. 
/src/Plugin/Validation/Constraint
Create a constraint (name = Unique)

namespace Drupal\module_name\Plugin\Validation\Constraint;
use Symfony\Component\Validator\Constraint;

/**
* @Constraint(
*   id = "Unique",
*   label = @Translation("Unique ", context = "Validation"),
*   type = "entity:entity_name"`
* )
*/


class UniqueConstraint extends Constraint {
   public $message= 'Unique entity!';
   }

Create a constraintValidator, and we populate the validate method with an entity query to make sure that there is no record having the same value of field1 and field 2:

 namespace Drupal\module_name\Plugin\Validation\Constraint;
 use Symfony\Component\Validator\Constraint;
 use Symfony\Component\Validator\ConstraintValidator;

class UniqueConstraintValidator extends ConstraintValidator {

/**
 * {@inheritdoc}
 */
public function validate($entity, Constraint $constraint) {
 $field1 = $entity-&gt;field1-&gt;value;
 $field2 = $entity-&gt;field2-&gt;value;
 $entity_type_id = $entity-&gt;getEntityTypeId();
 $value_taken = (bool) \Drupal::entityQuery($entity_type_id)
  -&gt;condition('field1',  $field1)
  -&gt;condition('field2',  $field2)
  -&gt;range(0, 1)
  -&gt;count()
  -&gt;execute();
if ($value_taken)
{
$this-&gt;context-&gt;addViolation($constraint-&gt;message);

}


}
  }
Then configure our constraint in our entity (annotations part):


/**

* Defines the entity_name entity.

*

    * @ingroup entity_name
    *
    * @ContentEntityType(
    *   id = "entity_name",
    *   label = @Translation("entity_name"),
    *   handlers = {
    *     ...
    *     "form" = {
    *     ..
    *     },
    *  ....
    *  
    *   constraints = {
    *     "Unique" = {}
    *   }
    * )
    */



Then you should uninstall your module and then reinstall it . (if your entity is populated with data, you have to delete all the date).
In your code (for example in your form) , you have to ensure the validation :

$violations = $entity-&gt;validate();
if ($violations-&gt;count() &gt; 0) 
{
 $violation = $violations[0];

 drupal_set_message($violation-&gt;getMessage(),'warning');
}



Hope it will help.