Rohan Dennis of Australia took the overall lead from David Veilleux of Canada by finishing second, 47 seconds behind Martin.
Froome wound up third, 2:45 ahead of Contador. A two-time Tour de France winner, Contador was also overtaken by Australian rider Richie Porte, who started two minutes behind him.
''It was not a good day for me,'' Contador told France 3 television. ''I didn't have much energy in the legs, but there is still time before my form reaches the top. I hope to be better and better each stage. For the Tour de France, there's still time. I knew that Chris Froome was superior in time trialing.''
Froome, the runner-up behind Bradley Wiggins at last year's Tour, moved to second place in the overall standings, five seconds behind Dennis.
''Today it was a good exercise and a good test for the form,'' Froome said. ''I had good legs. I don't know what is Alberto's strategy. Maybe he keeps some energy for the mountains.''
Martin completed the 20-mile time trial from Villars-les-Dombes to Parc des Oiseaux in just under 37 minutes. His goal was to win the stage, but he wasn't sure if that was possible after stomach problems the last couple of days.
''I'm happy to give Chris Froome some time because we'll be the main contenders for the time trials at the Tour de France too,'' he said.
Thursday's fifth stage is an 86-mile trek from Gresy-sur-Aix to Valmorel with four climbs. The weeklong race ends Sunday.