WASHINGTON -- The Secret Service is moving quickly to quell a prostitution scandal that has given President Barack Obama's critics political ammunition, forcing three agents out of government less than a week after the embarrassing incident came to light. Lawmakers welcomed the move but said more needed to be done.

"It's certainly not over," said Rep. Peter King of New York, chairman of the House Homeland Security Committee, which oversees the Secret Service.

The Secret Service did not identify the agents being forced out or eight more it said remain on administrative leave. In a statement, it said one supervisor was allowed to retire and another will be fired for cause. A third employee, who was not a supervisor, has resigned.

The agents were implicated in the prostitution scandal in Colombia that also involved about 10 military service members and as many as 20 women. All the Secret Service employees who were involved had their security clearances revoked.

The embarrassing scandal erupted last week after 11 Secret Service agents were sent home from Cartagena, on Colombia's Caribbean coast, after a night of partying that reportedly ended with at least some of them bringing prostitutes back to their hotel. The special agents and uniformed officers were in Colombia in advance of Obama's arrival for the Summit of the Americas.

In Washington and Colombia, separate U.S. government investigations are already under way. King said he has assigned four congressional investigators to the probe. The House Committee on Oversight and Government Reform, led by Rep. Darrell Issa, R-Calif., sought details of the Secret Service investigation, including the disciplinary histories of the agents involved. Secret Service investigators are in Colombia interviewing witnesses.

In a letter to Secret Service Director Mark Sullivan, Issa and Rep. Elijah Cummings of Maryland, the committee's ranking Democrat, said the agents "brought foreign nationals in contact with sensitive security information." A potential security breach has been among the concerns raised by members of Congress.

King said Sullivan took employment action against "the three people he believes the case was clearest against." The lawmaker said the agency was "reasonably confident" that drug use was not an issue with the three agents who have been forced out. But he said Secret Service investigators would continue to look into whether drugs played a role in the incident as it continues talking to the other eight agents involved.

Hotel workers told Secret Service investigators they found no drugs or drug paraphernalia in the rooms where the agents stayed, according to a person familiar with the investigation. The person was not authorized to discuss the probe publicly and spoke only on condition of anonymity.

Sen. Chuck Grassley, the ranking Republican on the Senate Judiciary Committee, said news of the three agents leaving the Secret Service was a positive development.

"I've always said that if heads don't roll, the culture in a federal agency will never change," the Iowa lawmaker said in a statement.

The episode took a sharp political turn Wednesday when presumptive Republican presidential nominee Mitt Romney said he would fire the agents involved.

Romney told radio host Laura Ingraham that he'd "clean house" at the Secret Service.

"The right thing to do is to remove people who have violated the public trust and have put their play time and their personal interests ahead of the interests of the nation," Romney said.

While Romney suggested to Ingraham that a leadership problem led to the scandal, he told a Columbus, Ohio, radio station earlier that he has confidence in Sullivan, the head of the agency.

At least 10 military personnel who were staying at the same hotel are also being investigated for misconduct. The troops are suspected of violating curfews set by their commanders.

Two U.S. military officials have said they include five Army Green Berets. One of the officials said the group also includes two Navy Explosive Ordinance Disposal technicians, two Marine dog handlers and an Air Force airman. The officials spoke on condition of anonymity because the investigation is still under way.

The Secret Service's Office of Professional Responsibility, which handles that agency's internal affairs, is investigating, and the Homeland Security Department's inspector general also has been notified.

Sullivan, who this week has briefed lawmakers behind closed doors, said he has referred to the case to an independent government investigator.