Lance Armstrong, meanwhile, can hold onto his bronze medal from the 2000 Sydney Games for a little while longer.
The International Olympic Committee executive board disqualified four athletes whose Athens doping samples were retested earlier this year and came back positive for steroids, including shot put gold medalist Yuriy Bilonog of Ukraine.
Also stripped were hammer throw silver medalist Ivan Tskikhan of Belarus and two bronze medalists women's shot putter Svetlana Krivelyova of Russia and discus thrower Irina Yatchenko of Belarus.
The case of a fifth bronze medalist, weightlifter Oleg Perepechenov of Russia, remains pending.
"Athletes who cheat by using doping substances must understand that just because they get away with it one day, there is a very good chance that they will be caught in the future," WADA President John Fahey said in a statement. "The retesting and subsequent decisions of the IOC are proof of that."
The IOC said it will ask the International Association of Athletics Federations to get the four medals back and readjust the results and rankings from the Athens Games.
Until then, no decision will be taken on reallocating the medals. Adam Nelson of the United States finished second in the shot put in Athens behind Bilonog and would stand to move up to gold.
The IOC, meanwhile, held off stripping Armstrong of the bronze he won 12 years ago in the cycling road time trial in Sydney, citing procedural reasons for the delay.
IOC leaders want the medal back following the damning U.S. Anti-Doping Agency report that painted Armstrong as a systematic drug cheat and led to him being stripped of his seven Tour de France titles from 1999-2005.
But the IOC said it must wait for cycling's governing body UCI to formally notify Armstrong of the loss of all his results since August 1998. The IOC wants to avoid any legal problems in connection with the eight-year statute of limitations in the Olympic rules.
"The IOC today will not move," IOC President Jacques Rogge said at a news conference following a two-day executive board meeting in Lausanne. "We need to have the situation whereby the UCI notifies officially Mr. Armstrong of the fact that he will be disqualified, declared ineligible and that he should hand over his medal.
"This is a legal obligation not for the IOC but for the International Cycling Union. When he will be notified, Mr. Armstrong will have 21 days to launch an appeal if he wishes. It is only after this period of 21 days that the IOC can legally take action."
The intention of the IOC to wipe Armstrong from the Olympic record books remains clear.
"Absolutely," IOC vice president Craig Reedie of Britain told The Associated Press. "If the UCI have the ability to remove all these titles, we should have the ability to remove a bronze medal. Once they go through their procedures, then we'll go through ours."
Armstrong finished behind winner and U.S. Postal Service teammate Vyacheslav Ekimov of Russia and Jan Ullrich of Germany in Sydney. The IOC has no plans to reallocate Armstrong's bronze medal, just as the UCI decided not to declare any winners for the Tour titles once held by the American.
In August, the IOC stripped Tyler Hamilton, a former Armstrong teammate, of his time-trial gold medal from the 2004 Athens Olympics after he admitted to doping.
The Athens Games were already considered the dirtiest on record, producing 26 doping cases and catching six medalists including two gold winners at the time. The retroactive tests bring the number of Athens cases to 31, including 11 medal winners and three gold medalists.
Since Athens, the IOC has been storing doping samples from each Olympics for eight years to allow for retesting when new detection methods become available. Last year, under prodding from the World Anti-Doping Agency, the IOC reanalyzed about 100 samples from Athens.
The IOC said Bilonog and Krivelyova tested positive for oxandrolone, while the samples from Tsikhan and Yatchenko were positive for methandienone.
Bilonog's disqualification means that both shot put winners in Athens have now been disqualified for doping: Women's champion Irina Korzhanenko of Russia was stripped of gold at the games after testing positive for stanozolol.
In what had been intended as a symbolic and uplifting return to the birthplace of the Olympics, the IAAF held the 2004 shot put competition in Ancient Olympia. The event has now been tarnished by the two drug scandals.
In the men's event, Nelson and Bilonog finished with the same best throw, but the Ukrainian was declared the winner because his second-best attempt was longer. It was the first time an Olympic field event was decided by a second-best mark.