The initial phase of Iraq's vast offensive to retake the city of Fallujah from IS was supported by several Shia militia, which raised fears of reprisals against the area's Sunni Muslim population.
UN rights chief Zeid Ra'ad Al Hussein said there was strong evidence that one group, Ketaeb Hizballah, perpetrated atrocities after telling civilians that they were there to help.
"This appears to be the worst - but far from the first - such incident involving unofficial militias fighting alongside government forces," Zeid said in a statement.
Ketaeb Hizballah fighters approached the village of Saqlawiyah near Fallujah - which lies only 50 kilometres (30 miles) west of Baghdad - on June 1, Zeid's office said in a statement, citing witness testimony.
Some 8,000 civilians spotted the fighters as they were leaving Saqlawiyah amid the assault on IS.
The militia members "hailed them with loudspeakers, saying the villagers had nothing to fear from them," according to the rights office.
"Witnesses said that hidden behind the Iraqi flags they saw the flags of a militia called Ketaeb Hizballah," the UN statement added.
According to witnesses, those who asked for water were dragged outside and shot, strangled, or severely beaten.
Women and children were sent to a displaced persons camp while men and teenage boys were taken to a series of locations.
According to witnesses, those who asked for water "were dragged outside and shot, strangled, or severely beaten," the UN said.
The abducted males were separated on June 5, with 605 men and boys taken to the displaced persons camp.
The whereabouts of a second group, with an estimated 900 people, is "unknown," according to Zeid.
The rights chief said locals made a list of 643 missing men and boys and "49 others believed to have been summarily executed or tortured to death while in the initial custody of Ketaeb Hizballah."
Locals said 200 additional abductees have not been accounted for.
Women in the displaced persons camp at Amriyat al-Fallujah told AFP last month that their sons, husbands and nephews were missing.
Zeid's spokesman Rupert Colville said Iraq's government had launched an investigation but had no details on its progress.
"People who escape from (IS) should be treated with sympathy and respect, not tortured and killed simply on the basis of their gender and where they had the misfortune to be living when (IS) arrived," Zeid said in the statement.