Poverty is a key challenge to socioeconomic development globally. However, the degree to which distance from a market contributes to poverty remains unclear. To provide insights into this relationship, we quantified the relationships between distance from markets and the per capita income of rural and urban people in China based on data from 29 provinces and 2651 counties. Our results illustrate the existence of a “geographical curse”; that is, a large separation between producers and consumers can exacerbate poverty for less-affluent rural residents, who pay a larger proportion of their income to send their products to market and to purchase goods from those markets. Programs to alleviate poverty should therefore consider seeking solutions associated with reducing the impact of that distance, such as subsidizing the transport of goods, improving the transportation infrastructure, supporting innovative business practices, and balancing the locations of producers and their markets.