Fred Stobaugh, whose wife Lorraine died in April, has no previous musical experience and wrote the song on a whim for a competition.
He submitted his handwritten lyrics by post and, although the contest was online-only, the organisers were so moved they put the words to music.
Oh Sweet Lorraine is number seven on US iTunes and has 1.9m YouTube views.
Billboard magazine said the song had sold 6,000 copies so far, placing it at number 49 in its rock digital songs sales chart.
The track is also in the iTunes charts for Austria, Switzerland, Germany, Sweden, Norway and Luxembourg.
"Oh sweet Lorraine," the chorus begins, "I wish we could do all the good times over again."
Stobaugh does not perform the song himself - on his original submission, he wrote: "I don't sing, I would scare people, haha!"
Vocals are provided instead by Jacob Colgan, whose Green Shoe Studio organised the original contest.
A documentary about the recording of the song, uploaded to YouTube in July, explains how the recording came about.
"What made Fred's entry so special to us was, one, it's an online contest. People are supposed to upload their videos," Colgan says in the clip.
"But, instead of a video, we received a very large manila envelope. Well, I was excited to see what it was. Lo and behold, it was a letter from a 96-year-old man who said, 'I've written a song for my wife'.
"He sounds like a sweet guy," he continues, "but, as I'm reading through the letter, I begin to realise that his wife had just passed away a month earlier."
Stobaugh says the song, his first, came to him out of the blue.
"After she passed away, I was just sitting in the front room one evening by myselfâ€¦it just [came] to me. I kept humming it."
The finished recording is a simple, country-tinged lullaby. The documentary shows Stobaugh overcome with emotion as he hears it for the first time, covering his face with his hand as he tells Colgan: "It's wonderful, it's wonderful".
"She was just the prettiest girl I ever saw," he says of meeting Lorraine in 1938. "Real timid-like. I just fell in love with her right there."
"I really, really miss her."
A 96-year-old who wrote a song for his late wife has made the US iTunes top 10, alongside Katy Perry and Lady Gaga.