Mr Jepson, who died in August, began his affiliation with the club in the 1960s and was named president in 2004.
The service took place at Headingley Stadium earlier.
Rhinos chief executive Gary Hetherington said: "Harry had a wonderful life and touched the lives of so many others."
Mr Jepson's love of the sport dated back to the 1920s when he first watched Hunslet at their Parkside Stadium.
Leeds players past and present were amongst those to pay tribute to Mr Jepson during the service.
Among those who spoke during the ceremony were Jamie Jones-Buchanan and Tony Currie, while former Leeds Rhinos and England captain Kevin Sinfield read Rudyard Kipling's poem 'If'.
The service was brought to a close by Lizzie Jones singing the Challenge Cup anthem 'Abide With Me' as the hearse carrying Mr Jepson's coffin left the ground for a family service.
Mr Hetherington said: "He knew people who had been involved in the formation of the game in 1895, and his death coincides with the 121st anniversary of the game he loved so much.
"He was a Rugby League legend, despite never playing the game and his passing is so sad."
A book of remembrance has been opened for fans in the club shop.
A memorial service has taken place to remember the life of Leeds Rhinos president and life-long rugby league fan Harry Jepson OBE, who died aged 96.