Rapid damage assessment after disasters is crucial for humanitarian relief and emergency response. The abrupt and unpredictable nature of disasters causes variations in the time, location, and sensors used for image collection, resulting in significant data disparities in satellite imagery. This poses a significant challenge for assessment tasks. To enable a rapid response, training models from scratch using a sufficient amount of on-site data is impractical due to time constraints. Thus, in practical applications, a model with robust adaptability and generalization is essential for autonomously adjusting to data variations. However, the majority of current models are trained and tested on a singular dataset, neglecting the aforementioned issues. To address these challenges, this study created datasets from the Turkey earthquake with large and dense building features and introduced datasets from the Louisiana hurricane with small and sparse building features. These datasets exhibit significant style differences and cover a broader range of building characteristics, providing a comprehensive evaluation of the model’s adaptability. In the building damage assessment task, the model is trained on public datasets and validated using the newly introduced scenario data. Compared with existing assessment models, the U-net model demonstrates the highest adaptability performance in objectively evaluating damage levels.