This study aims to find the optimal vegetation indices (VIs) to remotely estimate plant nitrogen concentration (PNC) in winter oilseed rape across different growth stages. Since remote sensing cannot “sense” N in live leaves, remote estimation of PNC should be based on understanding the relationships between PNC and chlorophyll (Chl), carotenoid concentration (Car), Car/Chl, dry mass (DM), and leaf area index (LAI). The experiments with eight nitrogen fertilization treatments were conducted in 2014 to 2015 and 2015 to 2016, and measurements were acquired at six-leaf, eight-leaf, and ten-leaf stages. We found that at each stage, Chl, Car, DM, and LAI were all strongly related to PNC. However, across different growth stages, semipartial correlation and linear regression analysis showed that Chl and Car had consistently significant relationships with PNC, whereas LAI and DM were either weakly or barely correlated with PNC. Therefore, the most suitable VIs should be sensitive to the change in Chl and Car while insensitive to the change in DM. We found that anthocyanin reflectance index and the simple ratio of the red band to blue band fit the requirements. The validation with the 2015 to 2016 dataset showed that the selected VIs could provide accurate estimates of PNC in winter oilseed rape.