As the visual perception window of the drone system, the lens provides great help for obtaining visual information, detection, and recognition. However, traditional lenses carried on drones cannot have characteristics of a large field of view (FoV), small size, and low weight at the same time. To meet the above requirements, we propose a panoramic annular lens (PAL) system with 4K high resolution, a large FoV of (30 deg to 100 deg)   ×  360 deg, an angular resolution of 12.2 mrad of aerial perspective, and great imaging performance. We equip a drone system with our designed PAL to collect panoramic image data at an altitude of 100 m from the track and field and obtain the first drone-perspective panoramic scene segmentation dataset Aerial-PASS, with annotated labels of track and field. We design an efficient deep architecture for aerial scene segmentation. Trained on Aerial-PASS, the yielded model accurately segments aerial images. Compared with the ERF-PAPNet and SwiftNet semantic segmentation networks, the network we adopted has higher recognition accuracy with the mean IoU greater than 86.30%, which provides an important reference for the drone system to monitor and identify specific targets in real-time in a large FoV.