Damon, the star of the Bourne franchise, was originally going to direct and star in the film, which is nominated for five Golden Globe awards.
But scheduling meant he had to pull out of both roles, remaining as a producer.
Affleck believes that "there aren't many parts like this".
And that's even for male actors at the height of their career.
"It's so exhilarating and fun to get a part like this," he explains.
"You get to do what you thought you'd do when you first started being an actor. The reality is, you end up doing so much stuff you thought you'd never have to do, and would never want to do again."
Affleck plays Boston janitor Lee, who, having suffered painful tragedy in his own life, has to return to his home town of Manchester by the Sea to take care of his teenage nephew, following the death of his brother.
The film was written and directed by Kenneth Lonergan, who was nominated for an Oscar for his writing on Martin Scorsese's Gangs of New York.
"The truth is there's so much media out there, so much TV, and there's a lot of material written for comedies and dramas, but there are very few things that have been brewing for years in the way Kenny writes things," the actor claims.
"It's the antithesis of what our culture has come to be, I mean our Western pop culture of churning it out and gobbling it up. There are also a lot of great actors out there, and sometimes those scripts go to other people. I got lucky."
Affleck, who was nominated for an Oscar in 2008 for his role in The Assassination of Jesse James by the Coward Robert Ford, grew up in the Massachusetts area where the film is set, along with his brother Ben and neighbour, Matt Damon.
Affleck says he was "aware of the project for some time" and knew Damon was working on it.
"When they asked me to do the part I said, 'Sure, that might mean we never get it made, but I'm honoured that you asked.' It was very clear to me that it was the kind of movie I would love even if no-one else did."
However, not only did the film receive glowing reviews after its debut at the Sundance Film Festival last year, but Affleck is the favourite to receive the Globe for best actor in a drama.
The film is also nominated for best screenplay, best director, best motion picture drama, and his co-star Michelle Williams is up for best supporting actress.
She plays Affleck's ex-wife, and confesses that she "burst into tears" when she got the role.
"I'd wanted to work with Kenny for so long," she says. "Casey and I had actually read for a play with him years ago. I was pregnant with my daughter at the time. Just knowing that it was finally going to happen, that we were all going to work together - I got a little tearful, yes.
"It felt like a momentous occasion when you want something and it comes true, even when you have to wait a long time.
"Casey and Kenny are good men and more than anything I am really happy to see their toil and efforts come good at this end because they deserve it. Casey looked like he carried the weight of the world on his shoulders when we were making it."
The drama is a study in grief, and how the three main characters, played by Williams, Affleck and Lucas Hedges as Lee's nephew Patrick, deal with their losses.
Affleck agrees that "there were bright spots in the experience" and "a lot of light in the film", but admits it was a "demanding role".
"Talking about acting sounds so precious and pretentious, it's almost unbearable, but there was a lot required emotionally - showing up there, and being able to be very upset and sad and tortured, yet contain it all, and keep it tight."
Kenneth Lonergan has received critical praise for not providing a so-called "Hollywood answer" to suffering, calling such stories "dishonest fantasies".
"Nobody needs me to tell anyone that real life can be difficult enough without watching something that tells you that everything will be OK, and in time you will understand about the circle of life and all this palaver," he says. "But to see my own experience reflected back at me helps me and makes me feel less alone. The sentimental approach which is so common is a cheat."
However, Michelle Williams believes that Manchester by the Sea does offer "a glimmer of hope" in its portrayal of bereavement.
"I think ultimately one of the things the movie is about is endurance. After hard times, you have to find ways to cleave to life and to people, even when you feel there isn't any hope. There's always a glimmer, I think that's what the movie offers, a glimmer of hope."
Manchester by the Sea is released in the UK on 13 January. The Golden Globes take place on 8 January.
Follow us on Facebook, on Twitter @BBCNewsEnts, or on Instagram at bbcnewsents. If you have a story suggestion email entertainment.news@bbc.co.uk.
Golden Globe nominated actor Casey Affleck says that he "got lucky" to get his part in drama Manchester by the Sea - after replacing his friend Matt Damon at the last minute.