The 2010 European Ryder Cup captain was one shot off the lead after an opening-round 69 during Local Final Qualifying at Gullane in Scotland before a round of 76, featuring four bogeys and a double bogey, saw him miss out by four strokes.
"I got to five under and threw it away - never mind," the 50-year-old Scot said in a news release. "It's not my schedule. I just played badly this afternoon. I didn't get going at all."
Montgomerie, who won a record eight order of merit titles on the European Tour between 1993 and 2005, has never won a major despite several close calls with five runner-up spots among a total of 10 top-10 finishes.
There were three qualifying spots up for grabs at Gullane with amateurs Ben Stow and Matthew Fitzpatrick joining Sweden's Oscar Floren in securing their places at the July 18-21 major at Muirfield.
At the qualifying event at The Musselburgh, Steven Tiley dominated the field to win by three shots to book his berth along with Scot Lloyd Saltman and Tyrrell Hatton.
At Dunbar, amateur Grant Forrest shot 67 and 65 for an eight-under-par aggregate and was joined in The Open field by Indian Shiv Kapur and Australian John Wade, who finished a stroke behind.
In the event at North Berwick, amateur Jimmy Mullen, Gareth Wright and George Murray took the coveted qualifying places.
(Reporting by Sonia Oxley; editing by Ken Ferris; Editing by)