The US Secret Service said Wednesday that three employees will leave their jobs over the sex scandal in a hotel in Colombia which tarnished the elite presidential protection agency's image.

Two leading congressmen meanwhile revealed in a letter to Secret Service chief Mark Sullivan obtained by ABC News that agents accused of consorting with prostitutes may have been careless with "sensitive security information."

Representatives Darrell Issa and Elijah Cummings also warned that the party hijinks in the Caribbean resort of Cartagena before President Barack Obama's arrival last weekend brought the entire culture of the agency into question.

One "supervisory" employee will retire over the allegations that agents consorted with prostitutes, another has been told he will be sacked and a third "non-supervisory" employee has resigned, said Paul Morrissey, of the service's Office and Government and Public Affairs.

Morrissey said that a probe into the alleged scandal was still at an early stage, and that eight other agents remained under investigation, in a process which included the use of polygraph lie detector technology.

A total of 11 agents and at least 10 military personnel are being investigated over the incident, which reportedly came to light when a prostitute got into a dispute over payment with one of the agents.

The departure of the three agents came amid a growing clamor for heads to roll in the Secret Service following the alleged incident, which overshadowed Obama's important visit to Colombia for the Summit of the Americas.

Republican presidential candidate Mitt Romney, who is a Secret Service protectee, said Wednesday he would "clean house" at the agency.

"The right thing to do is to remove people who have violated the public trust and have put their playtime and their personal interests ahead of the interests of the nation," Romney said.

The White House said that Obama would be "angry" if the allegations were true and retains confidence in Sullivan, but is waiting for the outcome of the investigation before commenting further.

Issa and Cummings told Sullivan that America's capacity to protect the president depended on the "character and judgment" of Secret Service agents.

"The actions of at least 11 agents and officers in Colombia last week showed an alarming lack of both. Your task is to restore the world's confidence in the US Secret Service," the lawmakers wrote.

The accused agents made "a range of bad decisions, from drinking too much, to engaging with prostitutes, to bringing foreign nationals into contact with sensitive security information," the letter said.

The agents had also opened themselves to the threat of blackmail and other forms of potential compromise, the lawmakers wrote.

They asked Sullivan to provide details of any disciplinary action against US Secret Service agents overseas since 2007 and to ascertain whether the women involved were over 18 years of age.

If they are not, the agents could have broken a US law which makes it illegal for an American to have illicit sexual contact with anyone under that age in a foreign country.

The New York Times on Wednesday carried an interview with one of the prostitutes, who said that the affair came to light after an agent offered her $30 for sex after earlier agreeing to pay $750.

The 24-year-old woman, who was not identified, said the early morning row led to the matter being taken up by hotel and police in Colombia. She said her group of women had approached by the agents in a disco the night before.

The image of the US Secret Service, known for sharp suited agents with sunglasses and earpieces ready to take a bullet for the president, is being dragged through the mud in the scandal.

The agency, which protects presidents, their families, foreign leaders in the United States and presidential candidates, is enduring a painful trial by media and questions about the agency's entire culture.

The top Republican on the Senate Judiciary Committee, Chuck Grassley, said that if the lurid stories were true, it could do lasting damage to the Service's reputation.

"It's going to bring a black mark to the whole Secret Service because, for 150 years or so, I think they've been a pretty respected organization," Grassley said on MSNBC. (AFP)