Harnessing self-supervised learning to boost malicious traffic detection with enhanced attention

SUN Jianwen; ZHANG Bin; LI Hongyu; CHANG Heyu

doi:10.11959/j.issn.2096-109x.2025025

您当前的位置：

首页 >

文章列表页 >

Harnessing self-supervised learning to boost malicious traffic detection with enhanced attention

Papers | 更新时间：2025-05-14

- Harnessing self-supervised learning to boost malicious traffic detection with enhanced attention
- Chinese Journal of Network and Information Security Vol. 11, Issue 2, Pages: 136-151(2025)
- 作者机构：
  
  信息工程大学，河南郑州 450001
- 作者简介：
- 基金信息：
  
  Innovation Fund for Graduate Students of the Cryptography Engineering Institute, Information Engineering University(2019f113)
- DOI：10.11959/j.issn.2096-109x.2025025
  CLC： TP393
- Received：23 September 2024，
  
  Revised：2025-02-28，
  
  Published：25 April 2025
- 稿件说明：
移动端阅览
孙剑文, 张斌, 李红宇, 等. 自监督学习驱动的注意力增强恶意流量检测方法[J]. 网络与信息安全学报, 2025, 11(2): 136-151.

SUN J W, ZHANG B, LI Hongyu, et al. Harnessing self-supervised learning to boost malicious traffic detection with enhanced attention[J]. Chinese Journal of Network and Information Security, 2025, 11(2): 159-174.
孙剑文, 张斌, 李红宇, 等. 自监督学习驱动的注意力增强恶意流量检测方法[J]. 网络与信息安全学报, 2025, 11(2): 136-151. DOI： 10.11959/j.issn.2096-109x.2025025.

SUN J W, ZHANG B, LI Hongyu, et al. Harnessing self-supervised learning to boost malicious traffic detection with enhanced attention[J]. Chinese Journal of Network and Information Security, 2025, 11(2): 159-174. DOI： 10.11959/j.issn.2096-109x.2025025.

摘要

现有的基于深度学习的恶意流量检测方法普遍面临3个主要问题：标记样本稀缺、恶意行为流量特征表示不够充分、行为关联模式整合效率低导致检测误报率高。针对以上问题，提出一种端到端的注意力增强恶意流量检测（malicious traffic detection with enhanced attention

MTAttention）方法。该方法首先对网络行为流量的异构头部特征和有效载荷实施统一编码，通过结构化的多包序列流量表示，形成数据包级别的标准化流和会话表示；其次，基于MAE（masked auto-encoder）模型采用自监督的掩码预训练策略，利用视觉Transformer提取丰富的流量特征表示。通过选择性地关注输入数据包序列中不同部分的空间和变量间的依赖关系，学习通用流量表示，将编码器模型的权重参数用于下游任务初始化，以加速模型训练的收敛过程；最后，引入基于通道注意力的数据包序列特征融合策略，利用多维注意力机制和有标签数据对模型权重进行微调以适应流量识别和分类任务，在分类决策前增强高权重特征的整合能力，进一步提高模型检测精确度。实验采用CIC-IDS2017数据集，结果表明，在恶意流量识别和细粒度分类场景下，MTAttention方法的精确度平均可达98.7%，推理速度高于每秒1 590个样本。与基于MAE范式的改进方法Flow-MAE相比，MTAttention在保持高精度的同时，仅需1.56%的参数量和63.89%的内存开销，推理速度平均提升约一倍，且模型大小仅为5.17 MB。

Abstract

The existing deep learning-based malicious traffic detection methods generally suffered from three main problems: labeled sample scarcity

inadequate representation of malicious behavior traffic features

and a high false positive rate due to ineffective integration of behavioral association patterns during detection. To address these issues

an end-to-end malicious traffic detection method named MTAttention was proposed. The heterogeneous header features and payloads of network behavior traffic were unified and encoded

resulting in a standardized flow and session representation at the packet level through structured multi-packet sequential traffic representation. Based on the MAE model

a self-supervised mask pre-training strategy was adopted

and a visual Transformer was utilized to extract rich traffic feature representations. By selectively focusing on the spatial and intervariable dependencies among different parts of the input packet sequence

a general traffic representation was learned

and the weight parameters of the encoder model were used to initialize subsequent tasks

which accelerated the model training process. A packet sequence feature fusion strategy based on channel attention was introduced

and a multi-dimensional attention mechanism

along with labeled data

were used to fine-tune the model’s weights to adapt to traffic identification and classification tasks. Before making classification decisions

the integration ability of high-weight features was enhanced to further improve the model's detection precision. The CIC-IDS2017 dataset was employed for the experiment

and the results showed that in the multi-classification scenario for malicious traffic identification

the precision of the MTAttention method could reach an average of 98.7%

and the inference efficiency exceeded 1590 samples per second. Compared with Flow-MAE

an improvement over the MAE paradigm

MTAttention maintained high precision while requiring only 1.56% of the parameter count and 63.89% of the memory overhead of Flow-MAE. Additionally

the average inference speed enhancement was approximately 100%

and the model size was only 5.17 MB.

关键词

Keywords

references

侯剑 , 鲁辉 , 刘方爱 , 等 . 加密恶意流量检测及对抗综述 [J ] . 软件学报 , 2024 , 35 ( 1 ): 333 - 355 .

HOU J , LU H , LIU A F , et al . Detection and countermeasure of encrypted malicious traffic: A survey [J ] . Journal of Software , 2024 , 35 ( 1 ): 333 - 355 .

XU C , CHEN S , SU J , et al . A survey on regular expression matching for deep packet inspection: Applications, algorithms, and hardware platforms [J ] . IEEE Communications Surveys & Tutorials , 2016 , 18 ( 4 ): 2991 - 3029 .

XU G , GUO B , SU C , et al . Am I eclipsed? A smart detector of eclipse attacks for Ethereum [J ] . Computers & Security , 2020 , 88 : 101604 .

SHEN M , ZHANG J , ZHU L , et al . Accurate decentralized application identification via encrypted traffic analysis using graph neural networks [J ] . IEEE Transactions on Information Forensics and Security , 2021 , 16 : 2367 - 2380 .

WANG T , CAI X , NITHYANAND R , et al . Effective attacks and provable defenses for website fingerprinting [C ] // 23rd USENIX Security Symposium (USENIX Security 14). 2014 : 143 - 157 .

ASHIKU L , DAGLI C . Network intrusion detection system using deep learning [J ] . Procedia Computer Science , 2021 , 185 : 239 - 247 .

QU J , MA X , LI J , et al . An input-agnostic hierarchical deep learning framework for traffic fingerprinting [C ] // 32nd USENIX Security Symposium . 2023 : 589 - 606 .

李丽娟 , 李曼 , 毕红军 , 等 . 基于混合深度学习的多类型低速率DDoS攻击检测方法 [J ] . 网络与信息安全学报 , 2022 , 8 ( 1 ): 73 - 85 .

LI L J , LI M , BI H J , et al . Multi-type low-rate DDoS attack detection method based on hybrid deep learning [J ] . Journal of Cyber Security and Information , 2022 , 8 ( 1 ): 73 - 85 .

PRASAD M , TRIPATHI S , DAHAL K . An efficient feature selection based Bayesian and Rough set approach for intrusion detection [J ] . Applied Soft Computing , 2020 , 87 : 105980 .

YIN Y , JANG-JACCARD J , SABRINA F , et al . Improving multilayer-perceptron (MLP)-based network anomaly detection with birch clustering on CICIDS-2017 Dataset [C ] // 2023 26th International Conference on Computer Supported Cooperative Work in Design (CSCWD) . IEEE , 2023 : 423 - 431 .

ZHANG H , HUANG L , WU C Q , et al . An effective convolutional neural network based on smote and gaussian mixture model for intrusion detection in imbalanced dataset [J ] . Computer Networks , 2020 , 177 ( 18 ): 107315 .

HOJJATI H , HO T K K , ARMANFARD N . Self-supervised anomaly detection in computer vision and beyond: A survey and outlook [J ] . Neural Networks , 2024 : 106106 .

YUANPENG T , BOSHEN Z , LIANG L , et al . Self-supervised feature adaptation for 3D industrial anomaly detection [J ] . Computer Vision - ECCV 2024 , PT II, 2025 , 15060 : 75 - 91 .

XIE R , WANG Y , CAO J , et al . Rosetta: Enabling robust tls encrypted traffic classification in diverse network environments with tcp-aware traffic augmentation [C ] // Proceedings of the ACM Turing Award Celebration Conference-China 2023 . 2023 : 131 - 132 .

WANG W , JIAN S , TAN Y , et al . Robust unsupervised network intrusion detection with self-supervised masked context reconstruction [J ] . Computers & Security , 2023 , 128 : 103131 .

HAN Y , HUI L , JIANG H , et al . Generative subgraph contrast for self-supervised graph representation learning [C ] // European Conference on Computer Vision . 2022 : 91 - 107 .

LIN X , XIONG G , GOU G , et al . Et-bert: A contextualized datagram representation with pre-training transformers for encrypted traffic classification [C ] // Proceedings of the ACM Web Conference 2022 . 2022 : 633 - 642 .

HE K , CHEN X , XIE S , et al . Masked autoencoders are scalable vision learners [C ] // Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 2022 : 16000 - 16009 .

XU K , ZHANG X , WANG Y , et al . Self-supervised learning malware traffic classification based on masked auto encoder [J ] . IEEE Internet of Things Journal , 2024 , 11 ( 10 ): 17330 - 17340 .

HANG Z , LU Y , WANG Y , et al . Flow-MAE: leveraging masked autoencoder for accurate, efficient and robust malicious traffic classification [C ] // Proceedings of the 26th International Symposium on Research in Attacks, Intrusions and Defenses . 2023 : 297 - 314 .

XU C , SHEN J , DU X . A method of few-shot network intrusion detection based on meta-learning framework [J ] . IEEE Transactions on Information Forensics and Security , 2020 , 15 : 3540 - 3552 .

SHAPIRA T , SHAVITT Y . FlowPic: A generic representation for encrypted traffic classification and applications identification [J ] . IEEE Transactions on Network and Service Management , 2021 , 18 ( 2 ): 1218 - 1232 .

SHARAFALDIN I , LASHKARI A H , GHORBANI A A . Toward generating a new intrusion detection dataset and intrusion traffic characterization [C ] // International Conference on Information Systems Security & Privacy . 2018 : 108 - 116 .

DOSOVITSKIY A , BEYER L , KOLESNIKOV A , et al . An image is worth 16x16 words: transformers for image recognition at scale [C ] // International Conference on Learning Representations . 2021 .

VASWANI A , SHAZEER N , PARMAR N , et al . Attention is all you need [J ] . Advances in Neural Information Processing Systems , 2017 , 30 : 5998 - 6008 .

HU J , SHEN L , SUN G , et al . Squeeze-and-excitation networks [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2019 , 42 ( 8 ): 2011 - 2023 .

WANG Q , WU B , ZHU P , et al . ECA-Net: efficient channel attention for deep convolutional neural networks [C ] // 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE , 2020 : 11531 - 11539 .

邓明阳 , 郭应时 , 刘通 . 基于分层重组的不平衡数据采样方法研究 [J ] . 重庆理工大学学报(自然科学) , 2021 , 35 ( 8 ): 122 - 128 .

DENG M Y , GUO Y S , LIU T , et al . Research on imbalanced data sampling method based on stratification and recombination [J ] . Journal of Chongqing University of Technology(Natural Science) , 2021 , 35 ( 8 ): 122 - 128 .

ENGELEN G , RIMMER V , JOOSEN W . Troubleshooting an intrusion detection dataset: the CICIDS2017 case study [C ] // 2021 IEEE Security and Privacy Workshops (SPW) . IEEE , 2021 : 7 - 12 .

HE H , YANG Z , CHEN X . PERT: payload encoding representation from transformer for encrypted traffic classification [C ] // ITU Kaleidoscope Academic Conference , 2021 , 19 ( 4 ): 90 - 97 .

HUANG J , LING C X . Using AUC and accuracy in evaluating learning algorithms [J ] . IEEE Transactions on Knowledge and Data Engineering , 2005 , 17 ( 3 ): 299 - 310 .

SALEH S M , SAYEM I M , MADHAVJI N , et al . Advancing software security and reliability in cloud platforms through AI-based anomaly detection [C ] // Proceedings of the 2024 on Cloud Computing Security Workshop . 2024 : 43 - 52 .

ROSAY A , RIOU K , CARLIER F , et al . Multi-layer perceptron for network intrusion detection: From a study on two recent data sets to deployment on automotive processor [J ] . Annals of Telecommunications , 2022 , 77 ( 5 ): 371 - 394 .

ZHOU G , GUO X , LIU Z , et al . TrafficFormer: An efficient pre-trained model for traffic data [C ] // 2025 IEEE Symposium on Security and Privacy (SP) . IEEE Computer Society , 2024 : 102 - 102 .

MCINNES L , HEALY J , MELVILLE J . Umap: Uniform manifold approximation and projection for dimension reduction [J ] . arXiv preprint arXiv: 1802.03426 , 2018 .

Views

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Metric-based learning approach to botnet detection with small samples

Related Author

Honggang LIN

Junjing ZHU

Lin CHEN

Related Institution

School of Cyberspace Security, Chengdu University of Information Technology

Sichuan Key Laboratory of Advanced Cryptography and System Security

Anhui Key Laboratory of Cyberspace Security Situational Awareness and Assessment

⁰