According to county deputy head Hu Jinliang, in 1997 the population of the 2.25-square-kilometer town was 45,000, yet the density was about 16 times of that of Beijing.
In response, the local government has moved many public institutions and enterprises out of town, with the purpose of transforming it from a community to a tourist site.
Local residents have also been leaving the area due to inconvenient transportation and substandard living conditions.
Local resident Gao Guihong said: "It is too humid in the house. There is no big supermarket, school or hospital in the town, and the air is also bad since we still use coal for heating."
Gao, 45, said that in an effort to save enough money to finance a move out of the county, has found work as a waiter in a hostel and is also employed part-time as a masseuse.
Ji Taiping, director of the county's urban-rural planning bureau, said that since the number of local residents emotionally attached to the traditional houses decreased and the number of tourists and newcomers grew the ancient town had begun to slowly lose its vigor and vitality.
The current situation also threatens the protection of traditional structures as this influx of outsiders usually lack any interest in protecting or restoring the houses, Ji said.
The local government should regulate the development of tourism and improve infrastructure and living conditions for local residents, he added.
The county has 3,797 traditional dwellings and other historical sites within its 6.4-km-long walls. But only about 400 are well-preserved and the majority are dilapidated due to a lack of adequate protection, said county chief Cao Zhisheng.
The protection of the buildings in this ancient town should develop alongside improvements for the local residents, Zhang Peng said.