Abstract: To address the issues of sample imbalance, poor model generalization, and insufficient accuracy in identifying small cracks in building facade defect detection, this paper proposes an intelligent recognition method based on the You Only Look Once version 11 for segmentation (YOLOv11-seg) network. A balanced defect dataset containing cracks, spalling and background is constructed by integrating open-source data with images captured by unmanned aerial vehicles. Two training strategies, transfer learning and layer freezing, are compared to determine their adaptation patterns for different defects while balancing accuracy and training efficiency. To overcome the low recognition accuracy and long training time for small cracks, KAN_C3K2, Dysample and Segment_RSCD modules are introduced to build the enhanced YOLOv11-KUR network, which improves crack segmentation accuracy without sacrificing training efficiency. Experimental results show that transfer learning significantly improves the efficiency of small-sample training, achieving a mean Average Precision at an intersection-over-union threshold of 0.5 (mAP@0.5) of 86.8% for spalling. Freezing only the Backbone layer reduces the training time for spalling by 88% while maintaining a mAP@0.5 of 86.6%. The improved network attains a crack segmentation mAP@0.5 of 91.5%, which is 5.8% higher than that of the baseline model. This method provides an effective technical solution for intelligent detection of building facade defects.