The study, published on the British Medical Journal website, showed that the failure rate could be cut in half.
Researchers analysed data from patients who were taking bisphosphonates, which are used to prevent the loss of bone material.
However, scientists say further studies are still needed.
When joints become damaged or worn out, commonly due to osteoarthritis, it can be painful and limits movement.
An implant can dramatically improve a patient's mobility. More than 50,000 hip and 70,000 knee, replacements take place in the UK each year.
While they can last for decades, some fail within years. This can happen when the bone around the implant is dissolved by the body meaning the replacement joint becomes loose.
The theory was that bisphosphonates, which are used by patients with osteoporosis to prevent bone being broken down, would prevent the loosening.
The researchers looked at data from General Practice Research Database for joint replacements and compared what happened to 1,912 patients taking bisphosphonates with 41,995 patients who did not.
After five years, 1.96% of implants failed without the drug, compared with 0.93% in those taking medication.
Prof Nigel Arden, a specialist in rheumatic diseases at the Universities of Oxford and Southampton, said the first implant would cost the NHS about Â£7,000, but replacements would cost Â£34,000.
He told the BBC: "It has the potential to have a huge impact."
He said two of the risk factors, age and obesity, meant the number of cases was "increasing dramatically".
Prof Arden is not arguing that people should be given the drug yet. He has applied for funding for a further trial to test the effectiveness of bisphosphonates and which patients would benefit most.
The chief medical officer for England, Prof Dame Sally Davies welcomed the findings.
"With such a high incidence of knee and hip replacement surgery, the possibility that the life of joint implants could be lengthened and reduce the number of complex revision surgeries means that these results have the potential to make significant improvements to the lives of many NHS patients," she said.
Using a bone-strengthening drug could make joint replacements last longer, according to an analysis of GPs' records.