US President Joe Biden has said that "people will pay" after images emerged showing border agents corralling migrants on horseback back to a river.
The pictures, shared widely online, show the riders using their reins against the migrants and pushing them back towards the Rio Grande in Texas.
US officials have vowed to investigate the incident, which occurred near a migrant camp in the town of Del Rio.
The Del Rio camp grew to over 15,000 people before being cleared on Friday.
"I take responsibility," Mr Biden told reporters at the White House on Friday. "It was horrible to see, to see people treated like they did, with horses nearly running them over. People being strapped. It's outrageous."
The images of the horseback charge on mostly black migrants have been condemned by lawmakers and sparked comparisons to America's slavery era.
In his first public remarks on the situation in Del Rio, Texas - where thousands of mostly Haitian migrants massed this week - Mr Biden added: "I promise you those people will pay. They will be investigated. There will be consequences."
Amid growing uproar on Thursday, the Department of Homeland Security (DHS) on Thursday announced that it was suspending the uses of horses in the Del Rio area.
DHS Secretary Alejandro Mayorkas has said the photographs "horrified" him and promised the incident would be investigated at the "highest levels".
Speaking at the White House on Friday, Mr Mayorkas said the expulsions were conducted out of a concern for public health amid the ongoing coronavirus pandemic.
The Biden administration has extended the Trump-era pandemic policy that allows the US to swiftly expel undocumented immigrants.
"It is not an immigration policy that we would embrace," Mr Mayorkas said.
Some Republicans have criticised the promise to pause the use of horses, saying the administration should act as quickly to fix bigger immigration issues.
Jenn Budd, a immigrant rights activist and former Border Patrol agent, said that while agents are not trained to use their reins on migrants, "that type of behaviour is very common."
On Friday, Vice-President Kamala Harris was also grilled on the situation in Haiti during a television appearance on The View talk show, in which she said she "fully supports" the ongoing investigation. "I was outraged by it," she said.
The Biden administration has come under increasing criticism for his handling of the situation in Del Rio, and the subsequent deportation of thousands of migrants back to Haiti.
On Thursday, US special envoy for Haiti Daniel Foote resigned over what he called the "inhumane, counterproductive" deportations.
Pamela White, a former US ambassador to Haiti, told the BBC that she "does not understand" how the US government didn't foresee the influx of migrants at the border and set up "a system to handle it".
"This deportation is a nightmare," she said. "We do not have a good immigration policy in the United States of America."
Ms White added that she agrees with Mr Foote's assessment of the situation and the administration's handling of it.
"In order for us to make any progress in Haiti, we have to have security. Food security, housing security and physical security," she said.