Survey on anomaly detection technology based on logs

Yingjun ZHANG; Ushangqi LI; Mu YANG; Haixia ZHANG; Kezhen HUANG

doi:10.11959/j.issn.2096-109x.2020072

您当前的位置：

首页 >

文章列表页 >

Survey on anomaly detection technology based on logs

Special Column:Network Application and Protection Technology | 更新时间：2024-06-03

- Survey on anomaly detection technology based on logs
- Chinese Journal of Network and Information Security Vol. 6, Issue 6, Pages: 1-12(2020)
- 作者机构：
  
  1. 中国科学院软件研究所可信计算与信息保障实验室，北京 100190
  2. 北京市公安局网络安全保卫总队，北京100029
- 作者简介：
- 基金信息：
- DOI：10.11959/j.issn.2096-109x.2020072
  CLC： TP393
- Online First：2020-12，
  
  Published：15 December 2020
- 稿件说明：
移动端阅览
Yingjun ZHANG, Ushangqi LI, Mu YANG, et al. Survey on anomaly detection technology based on logs[J]. Chinese Journal of Network and Information Security, 2020, 6(6): 1-12.
DOI：

Yingjun ZHANG, Ushangqi LI, Mu YANG, et al. Survey on anomaly detection technology based on logs[J]. Chinese Journal of Network and Information Security, 2020, 6(6): 1-12. DOI： 10.11959/j.issn.2096-109x.2020072.

摘要

日志信息是信息系统快速发展中产生的重要信息资源，通过日志的分析，可以进行异常检测、故障诊断和性能诊断等。研究基于日志的异常检测技术，首先对主要使用的基于日志的异常检测框架进行介绍，然后对日志解析、日志异常检测等关键技术进行详细介绍。最后对当前技术进行总结，并对未来研究方向给出建议。

Abstract

Log information has become an important information resource in the rapid development of information systems.Through the analysis of logs

abnormal detection

fault diagnosis and performance diagnosis can be performed.The log-based anomaly detection technology was focused on.Firstly

the currently used log-based anomaly detection framework was introduced

and then the key link technologies such as log analysis and log anomaly detection were focused on.Finally

the current technology was summarized and suggestions for future research directions were given.

关键词

Keywords

references

廖湘科 , 李姗姗 . 大规模软件系统日志研究综述 [J ] . 软件学报 , 2016 , 27 ( 8 ): 1934 - 1947 .

LIAO X K , LI S S . Survey on log research of large scale software system [J ] . Journal of Software , 2016 , 27 ( 8 ): 1934 - 1947 .

OLINER A J , GANAPATHI A , XU W . Advances and challenges in log analysis [J ] . Communications of the ACM , 2012 , 55 ( 2 ): 55 - 61 .

RAPIDS . cyBERT:neural network,that’s the tech; to free your staff from,bad regex [EB ] .

MI H , WANG H , ZHOU Y , et al . Toward finegrained,unsupervised,scalable performance diagnosis for production cloud computing systems [J ] . IEEE Trans Parallel Distrib Syst , 2013 , 24 ( 6 ): 1245 - 1255 .

崔元 , 张琢 . 基于大规模网络日志的模板提取研究 [J ] . 计算机科学 , 2017 , 44 ( 11A ): 448 - 452 .

CUI Y , ZHANG Z . Research on Template Extraction Based on Large-scale Network Log [J ] . Computer Science , 2017 , 44 ( 11A ): 448 - 452 .

HE P , ZHU J , HE S , et al . An evaluation study on log parsing and its use in log mining [C ] // Proc of the 46th Annual IEEE/IFIP International Conference on Dependable Systems and Networks . 2016 : 654 - 661 .

HE SL , ZHU J M , HE P J , et al . Experience report:system log analysis for anomaly detection [C ] // 27th International Symposium on Software Reliability Engineering . 2016 : 207 - 218 .

FU Q , LOU J , et al . Contextual analysis of program logs for understanding system behaviors [C ] // MSR ’13 . 2013 : 397 - 400 .

PECCHIA A , COTRONEO D , KALBARCZYK Z , et al . Improving log-based field failure data analysis of multi-node computing systems [C ] // Dependable Systems and Networks . 2011 : 97 - 108 .

LU J , LI F , LI L , et al . CloudRaid:hunting concurrency bugs in the cloud via log-mining [C ] // Foundations of Software Engineering , 2018 : 3 - 14 .

AIT EL HADJ M , KHOUMSI A , BENKAOUZ Y , et al . Efficient security policy management using suspicious rules through accesslog analysis [J ] . Lecture Notes in Computer Science , 2019 , 11704 : 250 - 266 .

STUDIAWAN H , FERDOUS S , PAYNE C . A survey on forensic investigation of operating system logs [J ] . Digital Investigation , 2019 , 29 : 1 - 20 .

CHOW M , MEISNER D , FLINN J , et al . The mystery machine:end-to-end performance analysis of large-scale internet services [C ] // 11th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’14) . 2014 : 217 - 231 .

KARTHIK N , CHARLES K , JENNIFER N . Structured comparative analysis of systems logs to diagnose performance problems [C ] // Networked Systems Design and Implementation . 2012 : 26 - 26 .

CHANDOLA V , BANERJEE A , KUMAR V . anomaly detection:a survey [J ] . ACM Computing Surveys , 2009 , 41 ( 3 ): 30602 - 30611 .

KULKARNI J , JOSHI S , BAPAT S , et al . Analysis of system logs for pattern detection and anomaly prediction [C ] // Proceeding of International Conference on Computational Science and Applications . 2020 : 427 - 436 .

ARIEL R , RANDY K . Chukwa:a system for reliable large-scale log collection [C ] // Usenix Large Installation Systems Administration Conference . 2010 : 1 - 15 .

ZHU J M , HE S L , LIU J Y , et al . Tools and benchmarks for automated log parsing [C ] // International Conference on Software Engineering . 2019 : 121 - 130 .

Splunk [EB ] .

Logentries [EB ] .

Logz.io [EB ] .

DU M , LI F F . Spell:streaming parsing of system event logs [C ] // ICDM 2016 . 2016 : 859 - 864 .

XU W , HUANG L , FOX A , et al . Detecting large-scale system problems by mining console logs [C ] // Symposium on Operating Systems Principles , 2009 : 117 - 132 .

NAGAPPAN M , WU K , MLADEN A . Efficiently extracting operational profiles from execution logs using suffix arrays [C ] // ISSRE . 2009 : 41 - 50 .

BAO L , LI Q , LU P Y , et al . Execution anomaly detection in large-scale systems through console log analysis [J ] . Journal of Systems and Software , 2018 , 143 : 172 - 186 .

LONVICK C , . The BSD syslog protocol [EB ] .

VAARANDI R , . Mining event logs with slct and loghound [C ] // Proceedings of the 2008 IEEE/IFIP Network Operations and Management Symposium . 2008 : 1071 - 1074 .

RISTO V , PIHELGAS M . LogCluster — a data clustering and pattern mining algorithm for event logs [C ] // Conference on Network and Service Management (CNSM) . 2015 : 1 - 7 .

MAKANJU A , ZINCIR-HEYWOOD N , MILIOS E E . A lightweight algorithm for message type extraction in system application logs [J ] . IEEE Transactions on Knowledge and Data Engineering , 2012 , 24 ( 11 ): 1921 - 1936 .

TATSUAKI K , KEISUKE I , TATSUYA Mori , et al . Spatio-temporal factorization of log data for understanding network events [C ] // IEEE INFOCOM 2014 . 2014 : 610 - 618 .

FU Q , LOU J G , WANG Y , et al . Execution anomaly detection in distributed systems through unstructured log analysis [C ] // (ICDM’09)Proc of International Conference on Data Mining . 2009 : 149 - 158 .

TANG L , LI T , PERNG C S . LogSig:generating system events from raw textual logs [C ] // CIKM’11:Proc.of ACM International Conference on Information and Knowledge Management . 2011 . 785 - 794 .

HE P J , ZHU J M , HE S L , et al . Towards automated log parsing for large-scale log data analysis [J ] . IEEE Transactions on Dependable and Secure Computing , 2018 , 15 ( 6 ): 931 - 944 .

STUDIAWAN H , SOHEL F , PAYNE C . Automatic event log abstraction to support forensic investigation [C ] // ACSW 2020 . 2020 : 1 - 9 .

STUDIAWAN H , PAYNE C , SOHEL F . Automatic graph-based clustering for security logs [C ] // Advanced Information Networking and Applications(AINA) . 2019 : 914 - 926 .

DAI H , LI H , CHEN C S , et al . Logram:efficient log parsing using n-gram dictionaries [R ] . 2020 .

NICOLAS A , YOHAN P , SOPHIE C , et al . Improving performances of log mining for anomaly prediction through NLP-based log parsing [C ] // Modeling Analysis And Simulation on Computer and Telecommunication Systems . 2018 : 237 - 243 .

Li G F , ZHU P J , CAO N , et al . Improving the system log analysis with language model and semi-supervised classifier [J ] . Multimedia Tools and Applications , 2019 , 78 ( 15 ): 21521 - 21535 .

PI A D , CHEN W , ZELLER W , et al . It can understand the logs,literally [C ] // International Parallel and Distributed Processing Symposium . 2019 : 446 - 451 .

LIU W Y , LIU X , DI X Q , et al . FastlogSim:a quick log pattern parser scheme based on text similarity [C ] // Knowledge Science Engineering and Management . 2020 : 211 - 219 .

DU M , LI F F . Spell:online streaming parsing of large unstructured system logs [J ] . IEEE Transactions on Knowledge and Data Engineering , 2019 , 31 ( 11 ): 2213 - 2227 .

MESSAOUDI S , PANICHELLA A , BIANCULLI D , et al . A search-based approach for accurate identification of log message formats [C ] // ICPC . 2018 : 167 - 177 .

HE P J , ZHU J M , ZHENG Z B , et al . Drain:an online log parsing approach with fixed depth tree [C ] // ICWS . 2017 : 33 - 40 .

BAO L F , BUSANY N , LO D , et al . Statistical log differencing [C ] // Automated Software Engineering . 2019 : 851 - 862 .

SIDDHARTHA S , SUPRATIM D , SRIKANT R , et al . Learning latent events from network message logs [J ] . IEEE ACM Transactions on Networking , 2019 , 27 ( 4 ): 1728 - 1741 .

XIE X S , WANG Z , XIAO X H , et al . A confidence-guided evaluation for log parsers inner quality [J ] . Mobile Networks and Applications , 2020 : 1 - 12 .

ZHANG D X , ZHENG Y , WEN Y , et al . Role-based log analysis applying deep learning for insider threat detection [C ] // SecArch'18 . 2018 : 18 - 20 .

EL-MASRIA D , PETRILLOB F , YANN-GAËL G , et al . A systematic literature review on automated log abstraction techniques [J ] . Information ＆ Software Technology , 2020 , 22 : 1 - 18 .

OPREA A , LI Z , YEN T F , et al . Detection of early-stage enterprise infection by mining large-scale log data [C ] // Proceedings of the 45th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN) . 2015 : 45 - 56 .

MARCELLO C , DOMENICO C , ANTONIO P . Event logs for the analysis of software failures:a rule-based approach [J ] . IEEE Transactions on Software Engineering , 2013 , 39 ( 6 ): 806 - 821 .

LOU J G , FU Q , YANG S G , et al . Mining program workflow from interleaved traces [C ] // Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . 2010 : 613 - 622 .

BEZERRA F , WAINER J . Algorithms for anomaly detection of traces in logs of process aware information systems [J ] . Information Systems , 2013 , 38 ( 1 ): 33 - 44 .

JIA T , CHEN G , YANG L , et al . An approach for anomaly diagnosis based on hybrid graph model with logs for distributed services [C ] // Proceedings of the IEEE International Conference on Web Services (ICWS) . 2017 : 25 - 32 .

LI T , MA J F , PEI Q Q , et al . AClog:attack chain construction based on log correlation [C ] // Global Communications Conference . 2019 : 1 - 6 .

MENG W B , LIU Y , ZHANG S L , et al . Device-agnostic log anomaly classification with partial labels [C ] // International Workshop on Quality of Service . 2018 : 1 - 6 .

LIU Z L , QIN T , GUAN X H , et al . An integrated method for anomaly detection from massive system logs [J ] . IEEE Access , 2018 : 30602 - 30611 .

XU W , HUANG L , ATREJA S , et al . Online system problem detection by mining patterns of console logs [C ] // ICDM’09 . 2009 : 588 - 597 .

NANDI A , MANDAL A , ATREJA S , et al . Anomaly detection using program control flow graph mining from execution logs [C ] // KDD 2016 . 2016 : 215 - 224 .

LIN Q W , ZHANG H Y , LOU J G , et al . Log clustering based problem identification for online service systems [C ] // ICSE 2016 . 2016 : 1 - 10 .

LIU F C , WEN Y , ZHANG D X , et al . Log2vec:a heterogeneous graph embedding based approach for detecting cyber threats within enterprise [C ] // CCS’19 . 2019 : 1777 - 1794 .

RISTO V , BERNHARDS B , MARKUS K . An unsupervised framework for detecting anomalous messages from syslog log files [C ] // Network Operations and Management Symposium . 2018 : 1 - 6 .

LOU J G , FU Q , YANG S Q , et al . Mining invariants from console logs for system problem detection [C ] // USENIX Annual Technical Conference . 2010 : 1 - 14 .

YUAN Y , ANU H , SHI W C , et al . Learning-based anomaly cause tracing with synthetic analysis of logs from multiple cloud service components [C ] // Computer Software and Applications Conference . 2019 : 66 - 71 .

BIBLOP D , MOHIUDDIN S , MUHAMMADALI G , et al . LogLens:a real-time log analysis system [C ] // International Conference on Distributed Computing Systems . 2018 : 1052 - 1062 .

DUNIA R , QIN J S . Multi-dimensional fault diagnosis using a subspace approach [C ] // ACC . 1997 : 1 - 5 .

PAPINENI K , . Why inverse document frequency? [C ] // NAACL ’01 . 2001 : 1 - 8 .

ASTEKIN M , OZCAN S , SOZER H . Incremental analysis of large-scale system logs for anomaly detection [C ] // International Conference on Big Data . 2019 : 2119 - 2127 .

ASTEKIN M , ZENGIN H ， SÖZER H . Evaluation of distributed machine learning algorithms for anomaly detection from large-scale system logs:a case study [C ] // 2018 IEEE International Conference on Big Data (Big Data) . 2018 : 2071 - 2077 .

BROWN A , TUOR A , HUTCHINSON B , et al . Recurrent neural network attention mechanisms for interpretable system log anomaly detection [C ] // MLCS 2018 . 2018 : 1 - 8 .

BERTERO C , ROY M , SAUVANAUD C , et al . Experience report:log mining using natural language processing and application to anomaly detection [C ] // International Symposium on Software Reliability Engineering(2017) . 2017 : 351 - 360 .

AMEY W , TANISHQ G , ROHIT V , et al . Hybrid CAE-VAE for unsupervised anomaly detection in log file systems [C ] // International Conference on Computing Communication and Networking Technologies . 2019 : 1 - 7 .

YOON-HO C , PENG L , SHANG Z T , et al . Using deep learning to solve computer security challenges:a survey [J ] . arXiv:Cryptography and Security . 2019 .

STEVEN Y , MELODY M , TENG-SHENG M . CausalConvLSTM:semi-supervised log anomaly detection through sequence modeling [C ] // International Conference on Machine Learning and Applications . 2019 : 1334 - 1341 .

DU M , LI F F , VIVEK S . DeepLog:anomaly detection and diagnosis from system logs through deep learning [C ] // CCS . 2017 : 1285 - 1298 .

XIA B , YIN J J , XU J , et al . LogGAN:a sequence-based generative adversarial network for anomaly detection based on system logs [C ] // SciSec 2019:Science of Cyber Security,Switzerland . 2019 : 61 - 76 .

ZHANG XU , XU Y , ZHANG H Y , et al . Robust log-based anomaly detection on unstable log data [C ] // ESEC/FSE’19 . 2019 : 807 - 817 .

WANG X , WANG D , ZHANG Y , et al . Unsupervised learning for log data analysis based on behavior and attribute features [C ] // International Conference on Artificial Intelligence . 2020 : 510 - 518 .

梅御东 , 陈旭 , 孙毓忠 , 等 . 一种基于日志信息和CNN-text的软件系统异常检测方法 [J ] . 计算机学报 , 2020 , 43 ( 2 ): 366 - 380 .

MEI Y D , CHEN X , SUN Y Z , et al . A method for software system anmaly detection based on log information and CNN-Text [J ] . Chinese Journal of Computers , 2020 , 43 ( 2 ): 366 - 380 .

LU S Y , WEI X , LI Y D , et al . Detecting anomaly in big data system logs using convolutional neural network [C ] // Dependable Autonomic and Secure Computing . 2018 : 151 - 158 .

Views

3144

下载量

CSCD

Alert me when the article has been cited

提交

Tools

Publicity Resources

Anomaly detection in DNS: an empirical study

Research progress on detection of software vulnerabilities based on code similarity

method for machine learning backdoor attacks based on defense method based on diffusion models

Personal data privacy protection method based onvertical partitioning

Malicious user detection in Momo application based on spatio-temporal behavior and static attribute analysis

Related Author

ZUO Peng

CHEN Yong

KONG Kaichuan

HE Ming

HOU Ronghao

LI Qinghao

LIU Chunling

TANG Yonghe

Related Institution

China Internet Network Information Center

School of Cyber Science and Engineering, Jinan University

Information Engineering University

School of Medical Information, Wannan Medical College

School of Physics and Electronic Information, Anhui Normal University

⁰