It was the first time a single album had been a top-seller for two years in a row since Nielsen began tracking album sales in 1991, the organization said.
But U.S. album sales overall fell 4 percent in 2012 to 315.96 million albums, after 2011 saw a rare 3 percent bump in sales.
Adele's "21" sold 4.41 million units in the United States in 2012 to top Swift's "Red," which sold 3.11 million copies. In 2011, "21" sold 5.82 million units.
"It's a sort of a once-in-a-lifetime album," Keith Caulfield, associate director of charts at Billboard, told Reuters of "21." "Only a few of these albums come along in history."
The heartbreak record, with hits like "Rolling in the Deep" and "Someone Like You," earned Adele six Grammy Awards in early 2012, boosting the profile of the 24-year-old singer and songwriter, who records on indie label XL Recordings.
The album sold at a furious pace, reaching the 10 million albums-sold plateau in the span of two years, Caulfield noted. The last album to achieve that feat was boy band 'N Sync's "No Strings Attached," which was released in 2000.
"It's really the right combination of artistry and hit singles," Caulfield said of "21's" success.
"She really crossed over from pop to Latin to adult contemporary to dance," he added. "Young and old consumers bought it, and because of its mixture of fans, she was able to sell it as well as she did."
Adele's success came despite the drop in 2012 U.S. album sales.
"Last year (2011) was a fluke," Caulfield said. "A year gain in album sales is a mega achievement. ... It's the way the market works now, people buy songs and not albums."
Indeed, digital song sales rose 5 percent in 2012 to a record high 1.336 billion downloads.
The year's best-selling albums in the United States had a particularly British flavor as Swift was the lone American in the top five. Swift records for the independently owned Nashville-based Big Machine, distributed by Universal Music Group.
British boy band One Direction's "Up All Night," released in 2011 on Sony Music Entertainment's SYCO/Columbia label, placed third with 1.62 million units sold, while their 2012 follow-up, "Take Me Home," took the fifth spot with 1.34 million units sold.
Britain's folk revivalists Mumford & Sons, on indie record label Glassnote, placed fourth with their album "Babel" selling 1.46 million units.
(Reporting by Eric Kelsey; Editing by Jill Serjeant and Peter Cooney)