Sea Watch said men from the speedboat boarded the dinghy and beat migrants with sticks, causing a "mass panic".
At least four of the 150 people on board the dinghy fell into the water and drowned, Sea Watch said.
A spokesman for Libya's naval forces in Tripoli said he had not heard about the incident.
The motive for the attack was not clear, but Ruben Neugebauer, a spokesman for Sea Watch, said the attackers were attempting to steal the dinghy's engine.
Mr Neugebauer said that 25 people remained missing.
Sea Watch, a German organisation, said in a statement: "The violent intervention of the Libyan Coast Guard caused a situation of mass panic on board the rubber boat in distress.
"One tube of the rubber boat collapsed, causing the majority of the 150 people to slip into the water."
About 120 people were rescued and brought aboard the Sea Watch 2 ship. Four were unconscious and receiving treatment.
Italy's coast guard confirmed that Sea Watch had been involved in some of the nine rescue operations taking place on Friday.
Both the Libyan and Italian coast guards said they had received no information about the attack yet.
Italy has taken in more than 146,000 boat migrants so far this year and is on track to see more arrivals than last year, when 153,000 arrived.
In August, the Doctors Without Borders humanitarian group, which operates rescue ships off the coast of Libya, said it was attacked by armed men on a Libyan navy boat.
The Libyan navy said it thought the vessel was involved in people smuggling.
At least four migrants have died after a speedboat labelled "Libyan Coast Guard" attacked a dinghy, according to a migrant rescue organisation.