The singer - who is the first woman to win the award twice - also won best pop vocal album and best music video for her track Bad Blood.
Rapper Kendrick Lamar won the most prizes on the night with five awards.
British singer Ed Sheeran also won his first two Grammys for song of the year and best pop solo performance for his number one hit Thinking Out Loud.
See the list of key winners
A surprised Sheeran beat Lamar, Swift, country group Little Big Town and Wiz Khalifa to the song of the year accolade.
"We wrote [the song] on a couch in my house. Thanks to my parents who've flown to the Grammys every year for the last four years and every time I lose I say 'maybe next year'," he said as he collected the award.
"If you'd told my 11-year-old self I would receive an award from Stevie Wonder I'd be chuffed."
Swift was equally surprised at winning the top honour. Her album, which was released in 2014, came out too late to be eligible for last year's awards so was finally recognised at this year's ceremony.
She gave a powerful acceptance speech saying: "As the first woman to win album of the year at the Grammys twice, I want to say to young women: There are going to be people along the way who will try to undercut your success or take credit for your accomplishments or your fame.
"But if you focus on your work and don't let those people sidetrack you, someday when you get where you're going, you'll look around and you'll know it was you and the people that love you that put you there. And that will be the greatest feeling in the world."
Lamar's wins included best rap album for To Pimp A Butterfly; best rap song and rap performance for Alright; best rap/sung collaboration and best music video, which he shared with Swift for Bad Blood.
Alabama Shakes were triple winners. They collected best alternative music album while their song Don't Wanna Fight won best rock song and rock performance.
The Weeknd also picked up two Grammys for best urban contemporary album and best R&B performance.
Mark Ronson and Bruno Mars won two awards for their hit Uptown Funk, including record of the year, while Meghan Trainor was named best new artist.
British rock band Muse also won best rock album for Drones, and Tony Bennett collected his 18th Grammy for best traditional pop vocal album.
Justin Bieber also won his first Grammy for his collaboration with Skrillex and Diplo, winning best dance recording for Where Are U Now.
Outside the main categories, Amy - Asif Kapadia's documentary on the life of Amy Winehouse - won the Grammy for best music film.
Swift opened the ceremony with a rendition of her song Out of the Woods and was one of many who performed during the show, along with Lamar, Adele, Ellie Goulding, James Bay and The Weeknd.
Adele's performance was slightly hampered, however, by audio problems which appeared to throw the singer.
She later tweeted: "The piano mics fell on to the piano strings, that's what the guitar sound was. It made it sound out of tune."
Rihanna had also been due to perform, but pulled out at the last minute on doctor's orders due to bronchitis.
There were also a number of acts who gave tribute performances - Lady Gaga sang a medley of David Bowie songs, Jackson Browne and The Eagles performed in honour of Glenn Frey and Stevie Wonder remembered Earth Wind and Fire's Maurice White.
Hollywood Vampires - a supergroup made up of stars including Alice Cooper, Johnny Depp and Joe Perry - also performed in tribute of Motorhead's Lemmy.
Taylor Swift's 1989 has won the coveted album of the year award at this year's Grammys in Los Angeles.