As a reflection of social norms towards childbearing, the ideal number of children is crucial to project the long-term fertility trend. Comparing China and other selected countries, our results show that the average ideal family size is much lower among Chinese women aged between 18 and 49. We also observe high one-child and low 3-or-more-children intentions in China. Capitalizing on the 2018 China Family Panel Study, we examine the effects of macro-level contextual factors on the ideal number of children in China. Results show that the higher economic development level, higher income inequality level, lower total fertility rate, and higher educational competition level will lead to smaller ideal family size. The mediation analysis shows that contextual fertility level and educational pressure are more important in shaping Chinese people’s ideal family size. Our study implies that fertility attitudes should be understood by taking the historical, cultural, and social context into account.