Gregory died at hospital in Washington DC after being admitted with an undisclosed illness.
The family shared the news of their "enormous sadness" in a post on social media on Saturday.
From the 1960s Gregory mocked racism in his stand-up routines. He was among the first US black comedian to perform regularly in front of white audiences.
"It is with enormous sadness that the Gregory family confirms that their father, comedic legend and civil rights activist Mr Dick Gregory departed this earth tonight in Washington, DC," his son, Christian, wrote on Instagram.
He added that the family "appreciates the outpouring of support and love" but requested that their privacy be respected "during this very difficult time".
US civil rights activist Jesse Jackson paid tribute to Gregory on Twitter:
End of Twitter post  by @RevJJackson
Gregory was a no-holds-barred comedian, delivering lines in his performances that were likely to raise eyebrows.
"A Southern liberal?" he once said, "that's a guy that'll lynch you from a low tree".
"Segregation is not all bad," Gregory said on another occasion. "Have you ever heard of a collision where the people in the back of the bus got hurt?"
Gregory was also involved in a number of civil rights activities in the 1960s. He campaigned against the Vietnam War and his protests often involved hunger strikes.
US comedian and civil rights activist Dick Gregory has died at the age of 84, his family has announced.