How to put this particular constraint on this link table in MySQL?
The solution is to add the group category column in the StoreGroups table and then add a unique constraint on (storeId, groupCategory). The table definitions would be like this:

CREATE TABLE Groups
  ( groupID INT NOT NULL PRIMARY KEY
  , name VARCHAR(255) NOT NULL
  , groupCategory NOT NULL VARCHAR(30)
  , exprireDate DATE NULL
  , UNIQUE (groupCategory, groupID)       -- this is needed for the FK below
  ) ;

CREATE TABLE StoreGroups
  ( storeID VARCHAR(16) NOT NULL 
  , groupID INT NOT NULL PRIMARY KEY
  , groupCategory NOT NULL VARCHAR(30)
  , PRIMARY KEY (storeId, groupID)
  , FOREIGN KEY (groupCategory, groupID)
      REFERENCES Groups (groupCategory, groupID)
  , UNIQUE (storeID, groupCategory)              -- your rule enforced
  ) ;


The UNIQUE constraint enforces that there cannot be 2 StoreGroups with the same storeID and the same groupCategory.