He became acting head of the agency in November, when IRS Commissioner Douglas Shulman's five-year term expired. Shulman had been appointed by President George W. Bush, a fact highlighted by several Democrats at Friday's hearing.
Camp said Miller's departure wouldn't be enough.
"The reality is this is not a personnel problem. This is a problem of the IRS being too large, too powerful, too intrusive and too abusive of honest, hardworking taxpayers," Camp said.
Two other committees have hearings scheduled for next week, and the Justice Department has launched its own criminal investigation. Miller is also scheduled to testify on Tuesday before the Senate Finance Committee. He will be joined by Shulman and George.
Underscoring the seriousness of his testimony Friday, Miller was sworn in as a witness, an unusual step for the Ways and Means panel.
He told committee members that before the episode became public, he had no contact with the Treasury Department, the White House or Obama's re-election campaign about targeting conservative groups.
"Absolutely not," Miller said.
He surprised committee members when he said "it is absolutely not illegal" for IRS agents to single out conservative groups for additional scrutiny.
"Please don't get me wrong," he added. "It should not happen."
George, the inspector general, backed up Miller's assertion when he said the yearlong investigation did not uncover illegal activity.
"It is not illegal, but it was inappropriate," George said of targeting conservative groups.
George's report concluded that an IRS office in Cincinnati, which screened applications for the tax exemptions, improperly singled out tea party and other conservative groups for tougher treatment. The report says the practice began in March 2010 and lasted in various forms until May 2012.
Agents did not flag similar progressive or liberal labels, though some liberal groups did receive additional scrutiny because their applications were singled out for other reasons, the report said.
The IG's report blamed ineffective management in Washington for letting the inappropriate singling out occur for so long.
Miller said he was notified that conservative groups had been singled out for additional scrutiny on May 3, 2012. After he was briefed, members of Congress repeatedly asked the IRS about complaints from tea party groups that said they were being harassed by the IRS.
But Miller was not forthcoming back then about groups being targeted in at least two letters to members of Congress and in testimony before a Ways and Means subcommittee.
"Didn't this committee have the right to know that groups were treated differently?" asked Rep. Dave Reichert, R-Wash.
"I answered all the questions I was asked," Miller replied.
"So your answer is a non-answer, once again," Reichert retorted.
As the hearing drew to a close, Camp, the committee chairman, declared, "I promise the American people this investigation has just begun."