Human Rights Watch (HRW) said at least 72 civilians had been killed in the strikes and the bloc needed to bear responsibility where appropriate.
But Nato's spokeswoman said the campaign was conducted "with unprecedented care and precision".
She said Nato "did everything possible to minimise risks to civilians".
"But in a complex military campaign, that risk can never be zero," spokeswoman Oana Lungescu acknowledged.
She added that the alliance "looked into each credible allegation" of harm to civilians and "confirmed that the specific targets struck by Nato were legitimate military targets".
Aircraft from the US, the UK and France conducted most of the 9,658 strike sorties last year, targeting forces loyal to Libyan leader Col Muammar Gaddafi.
The point of the Nato air campaign in Libya last year was to protect civilians, so how many innocent people died is still a sensitive issue, BBC diplomatic correspondent Bridget Kendall reports.
In its report published on Monday
, the US-based HRW said it had examined in detail evidence of claims of civilian deaths from eight separate Nato strikes.
In one instance, it said a first bomb killed 14 people and a second, moments later, killed 18 more who had rushed to help victims.
"We're calling for prompt, credible and thorough investigations," HRW's Fred Abrahams, the main author of the report, told BBC News.
Mr Abrahams says Nato refuses to acknowledge the deaths and has offered no compensation to families.
"Until now, Nato has taken a position of denial," he said.
"I think it will lead to unnecessary civilian deaths in the future if Nato refuses to look at what went wrong and make corrections."
In March, another human rights organisation,  
          Amnesty International, said it had documented 55 cases
 of named civilians, including 16 children and 14 women, killed in air strikes.
It said Nato had failed to investigate these cases thoroughly.
Source: Nato
Adel Ebsat, a relative of three victims who died from an air strike near the eastern town of Zlitan, told the BBC he had repeatedly asked the Libyan government to launch a case against Nato and for an investigation to be carried out.
But there appears to be a reluctance by Libyan officials to hold Nato accountable for any civilian deaths that may have been caused by air strikes, the BBC's Rana Jawad in Tripoli reports.
Most Libyans feel that the alliance helped the country succeed in its uprising against Col Gaddafi, which toppled and killed him last year, our correspondent says.
The overwhelming belief, she adds, is that countless deaths were avoided because of the West's intervention.
Nato has stressed that it is ready to co-operate with the new Libyan authorities in assessing what further action is appropriate.
However, the alliance says it cannot take responsibility because it has had no presence on the ground to confirm the deaths.
Nato has hit back at a report urging the alliance to investigate fully the deaths of civilians in air strikes in Libya last year.