 When Comptroller Nathan Lewis Miller was appointed to the New York Supreme Court, Kelsey was appointed New York State Comptroller to fill the vacancy, and was elected at the New York state election, 1904, to succeed himself.On May 2, 1906, Kelsey was appointed by Governor Frank W. Higgins to a three-year term as Superintendent of Insurance, and resigned the comptrollership.Early in 1907, Governor Charles Evans Hughes asked Kelsey to resign, but he refused.