Detecting small objects in complex remote sensing environments presents significant challenges, including insufficient extraction of local spatial information, rigid feature fusion, and limited global feature representation. In addition, improving model performance requires a delicate balance between improving accuracy and managing computational complexity. To address these challenges, we propose the SMA-YOLO algorithm. First, we introduce the Non-Semantic Sparse Attention (NSSA) mechanism in the backbone network, which efficiently extracts non-semantic features related to the task, thus improving the model's sensitivity to small objects. In the model's throat, we design a Bidirectional Multi-Branch Auxiliary Feature Pyramid Network (BIMA-FPN), which integrates high-level semantic information with low-level spatial details, improving small object detection while expanding multi-scale receptive fields. Finally, we incorporate a Channel-Space Feature Fusion Adaptive Head (CSFA-Head), which fully handles multi-scale features and adaptively handles consistency problems of different scales, further improving the robustness of the model in complex scenarios. Experimental results on the VisDrone2019 dataset show that SMA-YOLO achieves a 13% improvement in mAP compared to the baseline model, demonstrating exceptional adaptability in small object detection tasks for remote sensing imagery. These results provide valuable insights and new approaches to further advance research in this area.