A woman died and 19 people were injured when a car rammed a crowd of people opposing a far-right rally there.
The FBI has opened a civil rights investigation into the incident.
Earlier, street brawls erupted between white nationalists and counter-protesters.
Attorney General Jeff Sessions said "the violence and deaths in Charlottesville strike at the heart of American law and justice.
"When such actions arise from racial bigotry and hatred, they betray our core values and cannot be tolerated,"
Twenty-year-old James Fields from Ohio, the alleged driver of the car, is in detention on suspicion of second-degree murder,
In addition to those injured in the car incident, the Charlottesville Police Department said 15 were wounded in other violence related to the far-right march.
Late in the afternoon, a Virginia State Police helicopter crashed in woodland south-west of the city, killing two people. There has been no sign that this was linked to the violence, however.
The "Unite the Right" march was called to protest against plans to remove a statue of General Robert E Lee, who had fought for the pro-slavery Confederacy during the US Civil War.
Mr McAuliffe told a press conference: "I have a message for all the white supremacists, and the Nazis who came into Charlottesville today. Our message is plain and simple: Go home. You are not wanted in this great commonwealth. Shame on you. You pretend that you're patriots, but you are anything but a patriot.
"You came here today to hurt people. And you did hurt people. But my message is clear: We are stronger than you."
The governor, who is a Democrat, said he had spoken to President Trump, and twice urged him to begin a movement to bring people together.
He thanked the police and law enforcement officials, who he said had prevented "a much worse day", and praised the emergency services who helped the wounded.
The violence in Charlottesville - a liberal college town - is a stark demonstration of the growing political divide in the US, which has intensified since President Trump's election last year.
Right-wing blogger Jason Kessler had called for a "pro-white" rally, and white nationalists promoted the gathering widely.
Oren Segal, director of the Anti-Defamation League's Center on Extremism, said several "white power" groups were present - including neo-Nazis and factions of the Ku Klux Klan.
The New York Times reports that some were chanting "You will not replace us," and "Jew will not replace us."
Anti-racism organisations such as Black Lives Matter held protests at the scene.
On Friday, the white nationalists held lit torches and chanted "White lives matter" as they marched through the University of Virginia in the city.
President Donald Trump condemned "in the strongest possible terms this egregious display of hatred, bigotry, and violence on many sides".
"The hate and the division must stop right now," he told reporters, speaking in New Jersey, where he is on a working holiday. "We have to come together as Americans with love for our nation."
Democrats and Republicans alike took issue with his choice of words, noting that he failed to refer to the central role of white nationalists.
Republican Senator Cory Gardner tweeted: "Mr. President - we must call evil by its name. These were white supremacists and this was domestic terrorism."
Senator Orrin Hatch of Utah, also a Republican, felt similarly.
End of Twitter post  by @senorrinhatch
"What we've seen today in Charlottesville needs to be condemned and called what it is: hatred, evil, racism & homegrown extremism," tweeted Democrat John Kerry, who served as US Secretary of State under Barack Obama.
One picture tweeted by author J K Rowling showed a man on the street carrying a swastika flag.
Speaking in Charlottesville, the city's Mayor Michael Signer said the city would fight back. "We've overcome McCarthyism, we've overcome segregation - and we're gonna overcome this."
He said those who brought burning torches to Virginia's lawns "belong in the trash heap of history".
The governor of Virginia, Terry McAuliffe, has said that his only message for the white supremacists who brought chaos to the city of Charlottesville is "Go home".