Downer was first elected as a Democrat to the state House in the first ever nonpartisan blanket primary in the fall of 1975. He represented District 52 from 1976 until January 2004. In his penultimate term, 1996–2000, he was Speaker during the administration of Republican Governor Murphy J. "Mike" Foster Jr. Foster removed Downer as Speaker and replaced him with another Democrat, Charles W. DeWitt Jr. of Rapides Parish. Former legislator Ron Gomez of Lafayette said that Downer "did a terrific job before being unceremoniously dumped by Governor Foster after the fall 1999 elections." In 1987, Downer, as a Democrat, was named "Legislator of the Month" by the Louisiana Municipal Association. He was named "Outstanding Young Man of the Year" in 1977 by the Jaycees. He won the American Jurisprudence Award in 1971. During much of his legislative career through December 2003, Downer was affiliated with the law firm of Waitz & Downer in Houma. In 2000, Downer endorsed the Republican presidential ticket of George W. Bush and Richard B. Cheney. A year later, he formally switched parties, at Bush's invitation, and announced his candidacy for governor in 2003.
2003 Louisiana Gubernatorial race
Downer formally switched parties in 2001, and announced his candidacy for governor as a Republican in 2003. Three Republican gubernatorial candidates dropped out of the race early and endorsed Downer: former Governor David C. Treen of St. Tammany Parish, veteran state Senator Ken Hollis of Jefferson Parish, and Louisiana Public Service Commissioner Jack A. "Jay" Blossman Jr. of Mandeville in St. Tammany Parish. However, the majority of Republican voters supported Bobby L. Jindal, then of Baton Rouge. Downer secured the endorsement of thirty-three members of the state legislature from both parties and both chambers, including Republican Representatives Carl Crane of Baton Rouge and Ernie Alexander of Lafayette as well as the Democrat Francis C. Thompson of Delhi in Richland Parish in northeast Louisiana. Yet, Downer received only 84,718 votes in the primary, compared to 444,389 ballots for Jindal. In turn, Jindal was defeated by the leading Democrat Kathleen Babineaux Blanco of Lafayette in the general election. Soon after her election, Blanco appointed Downer as Secretary of the Louisiana Department of Veterans Affairs, and later as her legislative director in the Office of the Governor.
2010 U.S. Congressional race
Downer was a candidate for, being vacated by Democrat Charlie Melancon in a bid for the United States Senate. In January 2010, Downer told his hometown paper, The Courier, that he was "99.9 percent" sure that he would run. He lost a party runoff on October 2, 2010, to newcomer Jeff Landry, who had the support of the Tea Party movement.