2020 census

As of the 2020 United States census, there were 4,185 people, 1,682 households, and 1,066 families residing in the town.

2010 census

As of the census of 2010,^[10] there were 3,675 people, 1,583 households, and 985 families living in the town. The population density was 680.5 inhabitants per square mile (262.7/km²). There were 1,908 housing units with a density of 353.3 per square mile (136.3/km²). The racial makeup of the town was 85.2% White, 9.3% African American, 1.3% Asian, 0.6% Native American, 2.0% from other races, and 1.6% from two or more races. Hispanic or Latino of any race were 4.8% of the population.

There were 1,583 households, out of which 23.2% had children under the age of 18 living with them, 46.5% were married couples living together, 12.1% had a female householder with no husband present, and 37.8% were non-families. 31.7% of all households were made up of individuals, and 14.5% had someone living alone who was 65 years of age or older. The average household size was 2.24 and the average family size was 2.78.

The town population was distributed with 22.8% under the age of 20, 22.5% from 20 to 40, 32.2% from 40 to 65, and 22.4% aged 65 and over. The median age was 43.9 years. For every 100 females, there were 85.8 males. For every 100 females age 18 and over, there were 79.1 males.

The median income for a household in the town was $41,616, and the median income for a family was $52,284. Males had a median income of $36,591 versus $30,000 for females. The per capita income for the town was $23,397. About 4.7% of families and 10.2% of the population were below the poverty line, including 3.4% of those under age 18 and 7.7% of those age 65 or over.