面向金融领域大语言模型的提示注入攻击防御机制研究及应用

牟大恩; 卫志华; 孙铭隆; 宋娜; 倪琳

doi:10.11959/j.issn.2096-109x.2024071

您当前的位置：

首页 >

文章列表页 >

面向金融领域大语言模型的提示注入攻击防御机制研究及应用

学术论文 | 更新时间：2024-12-20

- 面向金融领域大语言模型的提示注入攻击防御机制研究及应用
- Research and application of defense mechanism for prompt injection attack of large language model in financial industry
- 网络与信息安全学报 2024年10卷第5期页码：119-133
- 作者机构：
  
  1.同济大学电子与信息工程学院，上海 200092
  2.上海证券有限责任公司，上海 200002
- 作者简介：
  
  [ "牟大恩（1985- ），男，湖北襄阳人，同济大学博士生，主要研究方向为机器学习、深度学习、数据挖掘及推荐算法。" ]
  [ "卫志华（1979- ），女，山西晋中人，博士，同济大学教授、博士生导师，主要研究方向为机器学习、图像内容分析、自然语言处理。" ]
  [ "孙铭隆（1995- ），男，辽宁沈阳人，上海证券有限责任公司研究员，主要研究方向为机器学习、深度学习、自然语言处理、统计学及推荐算法。" ]
  [ "宋娜（1994- ），女，江苏连云港人，上海证券有限责任公司研究员，主要研究方向为机器学习、深度学习、自然语言处理、数据挖掘及推荐算法。" ]
  [ "倪琳（1990- ），女，江苏南通人，上海证券有限责任公司研究员，主要研究方向为统计学、机器学习、深度学习、数据挖掘及推荐算法。" ]
- 基金信息：
  
  国家自然科学基金(62376199)
- DOI：10.11959/j.issn.2096-109x.2024071
  中图分类号： TP391
- 收稿：2024-01-23，
  
  修回：2024-03-11，
  
  纸质出版：2024-10-25
- 稿件说明：
移动端阅览
牟大恩, 卫志华, 孙铭隆, 等. 面向金融领域大语言模型的提示注入攻击防御机制研究及应用[J]. 网络与信息安全学报, 2024, 10(5): 119-133.

Format: MOU D E, WEI Z H, SUN M L, et al. Research and application of defense mechanism for prompt injection of large language models in financial industry[J]. Chinese Journal of Network and Information Security, 2024, 10(5): 119-133.
牟大恩, 卫志华, 孙铭隆, 等. 面向金融领域大语言模型的提示注入攻击防御机制研究及应用[J]. 网络与信息安全学报, 2024, 10(5): 119-133. DOI： 10.11959/j.issn.2096-109x.2024071.

Format: MOU D E, WEI Z H, SUN M L, et al. Research and application of defense mechanism for prompt injection of large language models in financial industry[J]. Chinese Journal of Network and Information Security, 2024, 10(5): 119-133. DOI： 10.11959/j.issn.2096-109x.2024071.

摘要

大语言模型在金融领域具有广阔的应用前景，无论资产管理端还是财富管理端，大语言模型都将会发挥重要的作用。随着ChatGPT和GPT-4等大型语言模型的快速发展和广泛应用，大语言模型的安全问题也愈发受到人们的关注。金融行业是一个严监管和强监管行业，对安全性有着更高的要求。围绕金融行业大语言模型提示注入攻击问题及安全防御机制进行深入研究，构建了包含8类输入侧提示注入攻击形式和5类大语言模型输出侧安全场景的金融领域大模型风险分类体系，通过人机对抗方式构建了金融领域的大模型提示注入攻击数据集FIN-CSAPrompts，提出了一套端到端的针对大语言模型提示注入攻击的安全防御框架，选取主流的中文大模型进行了实验验证。结果表明，在金融领域应用该提示词防御框架，中文大模型的防御性能显著提升，有效减少了不当内容的生成，并增强了模型对提示注入攻击的抵抗力。为进一步研究中文大模型在金融领域的安全性提供了数据集、衡量指标及解决方案的参考与借鉴。

Abstract

The large language models had a broad application prospect in the financial sector

and they were expected to play an important role in both asset management and wealth management. With the rapid development and wide application of large language models such as ChatGPT and GPT-4

attention to the security of large language models increased. The financial industry

characterized by strict regulations and supervision

demanded heightened security measures. Consequently

a comprehensive study on prompt injection attacks and a security defense framework was delved into in large language models within the financial sector. A risk taxonomy encompassing eight forms of input prompt injection attacks and five categories of safety scenarios on the output side was developed

and a financial domain large model prompt injection attack dataset

FIN-CSAPrompts

was collected using a human-machine adversarial approach. An end-to-end security defense framework against prompt injection attacks in large language models was proposed and tested

and comparative evaluations were performed using prevalent open-source large language models. The research indicated that in the financial industry

the application of the proposed security defense framework significantly enhanced the defensive capabilities of Chinese large language models

effectively reducing the generation of inappropriate content and improving their resilience against prompt injection attacks. This research provided a reference and foundation for further research on the security issues of Chinese large language models in the financial domain

offering datasets

evaluation metrics

and solutions for consideration and adaptation.

关键词

Keywords

references

百度安全 . 2023大模型安全解决方案白皮书 [R ] . 2023 .

Baidu Security . White paper on large model security solution in 2023 [R ] . 2023 .

CHEN C Y , SHU K . Can LLM-generated misinformation be detected? [C ] // The Twelfth International Conference on Learning Representations . 2023 .

BHARDWAJ R , PORIA S . Red-teaming large language models using chain of utterances for safety-alignment [J ] . arXiv eprints , arXiv: 2308. 09662 . 2023 .

XU G H , LIU J Y , YAN M , et al . CValues: measuring the values of Chinese large language models from safety to responsibility [J ] . arXiv eprints , 2023 , arXiv: 2307.09705 .

SUN H , ZHANG Z X , DENG J W , et al . Safety assessment of Chinese large language models [J ] . arXiv eprints , 2023 , arXiv: 2304.10436 .

WANG Y X , LI H N , HAN X D , et al . Do-not-answer: a dataset for evaluating safeguards in LLMs [J ] . arXiv eprints , 2023 , arXiv: 2308.13387 .

ZHANG H F , GUO Z M , ZHU H S , et al . On the safety of open-sourced large language models: does alignment really prevent them from being misused [J ] . arXiv eprints , arXiv: 2310. 01581 . 2023 .

BIANCHI F , SUZGUN M , ATTANASIO G , et al . Safety-tuned LLaMAs: lessons from improving the safety of large language models that follow instructions [C ] // The Twelfth International Conference on Learning Representations . 2023 .

O’NEILL C , MILLER J , CIUCA I , et al . Adversarial fine-tuning of language models: an iterative optimisation approach for the generation and detection of problematic content [J ] . arXiv eprints , arXiv: 2308. 13768 . 2023 .

CAO B C , CAO Y P , LIN L , et al . Defending against alignment-breaking attacks via robustly aligned LLM [EB ] . arXiv preprint , arXiv: 2309. 14348 . 2023 .

DEVLIN J , CHANG M W , LEE K , et al . BERT: pre-training of deep bidirectional transformers for language understanding [C ] // Proceedings of NAACL-HLT . 2019 : 1 - 2 .

LAN Z Z , CHEN M , GOODMAN S , et al . Albert: a lite bert for self-supervised learning of language representations [EB ] . arXiv preprint , arXiv: 1909 .11942. 2019 .

LIU Y H , OTT M , GOYAL N , et al . RoBERTa: a robustly optimized BERT pretraining approach [EB ] . arXiv preprint , arXiv: 1907 .11692. 2019 .

JOHNSON J , DOUZE M , JÉGOU H . Billion-scale similarity search with GPUs [J ] . IEEE Transactions on Big Data , 2021 , 7 ( 3 ): 535 - 547 .

MALKOV Y A , YASHUNIN D A . Efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs [J ] . IEEE Transactions on Pattern Analysis and Machine Intelligence , 2020 , 42 ( 4 ): 824 - 836 .

浏览量

下载量

CSCD

文章被引用时，请邮件提醒。

提交

工具集

关联资源

暂无数据