Enhancing Malware Detection via RGB Assembly Visualization and Hybrid Deep Learning Models

Malicious software presents significant challenges in cybersecurity, leveraging rapidly evolving technologies to bypass traditional defense mechanisms. This research introduces a novel image-based malware classification framework that uses hybrid-model Convolutional Neural Networks to process RGB images generated from assembly code. We present MalevisAsm, an enriched dataset that merges MaleVis malware samples with benign files, and propose a hybrid deep learning model that combines EfficientNetB0 and DenseNet121 for robust feature extraction. The approach transforms Portable Executable files into assembly code, maps opcode transitions into three-channel images, and uses a fine-tuned CNN to classify malware families. Additionally, we implemented Uniform Manifold Approximation and Projection a contemporary nonlinear dimensionality reduction technique, to enhance the identification of previously unseen malware samples via binary classification. Our experiments achieve a top-tier accuracy of 98.45%, surpassing existing benchmarks on the MaleVis dataset. This research contributes to the field by integrating static binary analysis with advanced computer vision techniques, offering a scalable and effective solution for malware detection.