Christopher Gunness, a spokesman for the United Nations Relief and Works Agency (UNRWA) which looks after the refugees said it was deeply concerned for their safety.
If the militants don't leave within the two-day time-frame, the army said it will forcefully begin evacuating the area.
"There are reports that large numbers of people have been displaced from Yarmouk Camp to the neighbouring area of Yalda", said Gunness.
Their capture would leave the opposition holding only two strongholds, located in the northwest and southwest along Syria's global borders.
But the single volley of air strikes, hitting three targets far from any frontline, had no effect on the wider war which has killed 500,000 people and made more than half of Syrians homeless.More news: New York Mets vs. Atlanta Braves, 4/21/2018 Prediction & Odds
More news: TV Presenter Dale Winton dies, 62
More news: Giants' GM expresses interest in Dez Bryant after releasing Brandon Marshall
The inspectors have delayed their visit to Douma after their security team were shot at during a reconnaissance trip on Tuesday, the Organisation for the Prohibition of Chemical Weapons said. Syria and its ally Russian Federation deny that. The Western countries say the Syrian government, which now controls the town, is keeping the inspectors out and may be tampering with evidence, both accusations Damascus and Moscow deny.
A government media tour on Monday of Douma, the biggest town in the former rebel enclave of eastern Ghouta just outside Damascus, revealed severe destruction and the plight of residents who had survived years of siege.
In the nearby enclave of Eastern Qalamoun, which consists of several towns and an area of hills and has also been covered by an informal ceasefire, rebels said they were also negotiating a withdrawal deal with Russian Federation. All the rebel groups controlling areas of eastern Ghouta eventually agreed surrender deals that involved withdrawal to opposition-held areas of northwestern Syria.
Dumayr is at the Qalamoun mountains, near the southern Ghouta region that came under complete government management every week soon after the alleged chemical weapon strike and also having a offensive.
The daily further said, Syrian forces have for days been launching air strikes on the area controlled by Daesh (IS) in the south of Damascus, primarily around the Palestinian Yarmouk refugee camp and its surrounding areas.