Calling it a "very difficult decision," Spears said in a statement: "I had an incredible time doing the show and I love the other judges and I am so proud of my teens but it's time for me to get back in the studio.
"Watching them all do their thing up on that stage every week made me miss performing so much! I can't wait to get back out there and do what I love most," the "Circus" singer said.
Spears' most recent album, "Femme Fatale," was released in March 2011.
Celebrity website TMZ.com reported on Friday that Spears was in talks about a long-term residency gig in Las Vegas. The gambling city is already host to stars like Celine Dion and Shania Twain, who perform for about two years in one of the many hotel and casino venues.
Spears, 31, was recruited to the Fox singing contest "The X Factor" with a reported $15 million salary after a 14-year singing career that made her one of the biggest pop stars of the 2000s.
"X Factor" creator Simon Cowell had banked on Spears' huge fan base and a strong curiosity factor to give his show a second chance with audiences after a disappointing first season in 2011 that ended with the firing of two judges as well as host Steve Jones.
But audiences in fact slumped and the TV show lost about three million regular viewers from its first season to about 9.7 million per episode in 2012. Many fans and TV critics found Spears bland and boring.
The exit of Spears leaves Cowell searching for two new judges to lift the "The X Factor" past its NBC rival, "The Voice," in the ratings when it returns in September.
Judge and record producer L.A. Reid announced in December finale that he would be returning full time to his job as the head of Epic Records. Spears took "The X Factor" gig with singer Demi Lovato, 20, in May 2012 to fill the judges' seats left by Paula Abdul and Nicole Scherzinger, who were both fired by Cowell.
(Reporting By Jill Serjeant; Editing by David Brunnstrom)