Constraint based rules engine
Constraints will basically allow you to enforce data integrity - limiting the values that may be inserted into a column and there by enforcing business rules. The database engine (SQL Server) will automatically enforce integrity of a database based on the constraints defined 
e.g. NOT NULL, CHECK constraints, UNIQUE constraints,  PRIMARY KEY constraints, FOREIGN KEY Constraints, etc.

For Rule Engine, 


  A rules engine is a schedule based data validity application that typically runs as a meta-layer on top of an OLTP application. It fires a set of queries (defined as rules) which determine whether the underlying data elements comply with a specific rule’s definition. The compliance to the rule is then recorded indicating a pass or fail.


I will just point you to some Microsoft articles and other useful links (as I am not a pro in this area).


Building a Rule Engine with SQL Server
Choosing a Rules Engine Design
Creating a Rules Engine