His comments came to light this week in a secretly recorded video of the Republican nominee speaking at a fundraiser in May. He made the comments while explaining how his campaign would not try to win over staunch supporters of President Barack Obama.

After media outlets began talking about the comments, Romney hurriedly held a news briefing Monday night. He said that his words "were not elegantly stated," but he refused to back away from them.

"I'm sure I can state it more clearly and in a more effective way than I did in a setting like that, and so I'm sure I'll point that out as time goes on," he said.

The claim that nearly half of Americans pay no taxes is based on a 2011 finding from the Tax Policy Center, a joint tax research arm of the centrist Urban Institute and center-left Brookings Institution. The center reported in 2011 that 53.6 percent of an estimated 164 million U.S. households paid some federal income taxes, while the rest -- actually 46.4 percent -- paid none.

More than 76 million households paid no income taxes last year, according to the Tax Policy Center. But about 60 percent still paid federal payroll taxes that support Medicare and Social Security, said Roberton Williams, a senior fellow at the tax center. Many also paid federal excise taxes, along with state and local sales, property and income taxes.

"So it's not that they're not paying any taxes," Williams said. "It's just that they're not paying federal income taxes."

The reason, he said, is that more than 38 million, or roughly half of the 76 million households that paid no taxes last year, simply didn't earn enough to have a tax liability under the current tax code.

But it isn't just low-income households that pay no federal income taxes, Williams said. He estimated that 1,400 millionaires didn't pay federal income taxes in 2009, with many likely taking advantage of foreign tax credits or charitable donations to lower their tax liability.

"These are things that people do to lower their tax rates," he said.

Apart from the more than 38 million households that paid no taxes because they didn't earn enough, the other half of the 76 million without federal tax liabilities are mostly low- to moderate-income families that benefited from special tax deductions targeting specific populations.

Nearly two-thirds of these households -- 28.3 million -- are made up of seniors and low-income families with children.

In fact, nearly 17 million elderly households paid no federal taxes last year because tax laws allow them to exclude municipal bond interest and some Social Security benefits from their taxable income, Williams said.

Another 11.6 million low-income families with children paid no income taxes because they benefited from the child earned income and child care tax credits, which erased their tax liability. The Earned Income Tax Credit is the nation's largest anti-poverty program and helps pull the incomes of more than 5 million families over the federal poverty line each year.

The tax credit is just one example of Congress using the tax code to provide financial support for families. The tax cuts enacted under former President George W. Bush, which targeted billions in tax relief for wealthy Americans, also canceled out the tax liability for 7.8 million low- and moderate-income families as well, according to a 2004 study by The Tax Foundation, a nonpartisan tax research group.

In 2010, Romney himself saved $2 million in income taxes because his capital gains income was taxed at a lower rate, Williams said.