it's an implicit right covered by the 1st amendment, backed up by supreme court interpretation. you cannot have the right to assemble/practice religion/speak freely if you do not also necessarily have freedom of association
United States