The US Court of Appeals in Chicago on Tuesday upheld a lower court ruling last year allowing the men, Donald Vance and Nathan Ertel, to pursue claims that Rumsfeld and unnamed others should be found personally liable for their treatment - despite efforts by the former Bush and current Obama administration to get the case dismissed.
The two men worked for a private security company in Iraq in 2006 and said they became concerned the firm was engaging in illegal bribery or other corruption activities. They notified US authorities and began co-operating with them.
In early 2006, they were taken into custody by US military forces and eventually taken to Camp Cropper near Baghdad's airport. Vance and Ertel claimed they were subjected to harsh interrogations and physical and emotional abuse.
Months later they said they were unceremoniously dropped at the airport and never charged with a crime.
They sued, seeking unspecified damages and saying their constitutional rights had been violated and US officials knew they were innocent.
"Having judges second guess the decisions made by the armed forces halfway around the world is no way to wage a war"
David Rivkin, lawyer
The appeals court ruled that while it may have been unusual for Rumsfeld to be personally responsible for the treatment of detainees, the two men had sufficiently argued that the decisions were made at the highest levels of government.
"We agree with the district court that the plaintiffs have alleged sufficient facts to show that Secretary Rumsfeld personally established the relevant policies that caused the alleged violations of their constitutional rights during detention," the court ruled in a split decision.
The three-judge panel voted 2-1 to affirm the lower court ruling. Judge Daniel Manion dissented, saying Congress has yet to decide whether courts should have a role in deciding whether such claims against the US military can be pursued.
A lawyer representing Rumsfeld said the appeals court decision was a blow to the US military.
"Having judges second guess the decisions made by the armed forces halfway around the world is no way to wage a war," attorney David Rivkin said in a statement on Monday.
"It saps the effectiveness of the military, puts American soldiers at risk, and shackles federal officials who have a constitutional duty to protect America."
A spokesman for the US Justice Department, which has been representing the former defense secretary, had no immediate comment. The Justice Department could appeal to the full appeals court or to the US Supreme Court.
There have been other lawsuits against Rumsfeld and the US government over allegations of abuse and torture overseas, but most involved foreigners, not US citizens, so federal courts have typically dismissed those cases.
A district judge in Washington last week allowed a similar case to proceed involving an American translator who worked in Iraq with the US military and who said he was later detained and subjected to harsh interrogation techniques and abuse.