search
HomeTechnology peripheralsAIResearch shows reinforcement learning models are vulnerable to membership inference attacks

​Translator | Li Rui

Reviewer | Sun Shujuan​

As machine learning becomes part of many applications that people use every day, people are increasingly There is increasing focus on how to identify and address security and privacy threats to machine learning models.

Research shows reinforcement learning models are vulnerable to membership inference attacks

However, different machine learning paradigms face different security threats, and some areas of machine learning security remain under-researched. In particular, the security of reinforcement learning algorithms has not received much attention in recent years.

Researchers at McGill University, the Machine Learning Laboratory (MILA) and the University of Waterloo in Canada have conducted a new study that focuses on the privacy threats of deep reinforcement learning algorithms. Researchers propose a framework for testing the vulnerability of reinforcement learning models to membership inference attacks.

Research results show that attackers can effectively attack deep reinforcement learning (RL) systems and may obtain sensitive information used to train models. Their findings are significant because reinforcement learning techniques are now making their way into industrial and consumer applications.

Member inference attack

Research shows reinforcement learning models are vulnerable to membership inference attacks

Member inference attack observes the behavior of a target machine learning model and predicts the examples used to train it .

Every machine learning model is trained on a set of examples. In some cases, training examples include sensitive information, such as health or financial data or other personally identifiable information.

Member inference attacks are a series of techniques that attempt to force a machine learning model to leak its training set data. While adversarial examples (the more well-known type of attack against machine learning) focus on changing the behavior of machine learning models and are considered a security threat, membership inference attacks focus on extracting information from the model and are more of a privacy threat .

Membership inference attacks have been well studied in supervised machine learning algorithms, where models are trained on labeled examples.

Unlike supervised learning, deep reinforcement learning systems do not use labeled examples. A reinforcement learning (RL) agent receives rewards or penalties from its interactions with the environment. It gradually learns and develops its behavior through these interactions and reinforcement signals.

The paper's authors said in written comments, "Rewards in reinforcement learning do not necessarily represent labels; therefore, they cannot serve as predictions often used in the design of membership inference attacks in other learning paradigms tags."

The researchers wrote in their paper that "there are currently no studies on the potential leakage of members of data used directly to train deep reinforcement learning agents."

Part of the reason for this lack of research is that reinforcement learning has limited real-world applications.

The authors of the research paper said, “Despite significant progress in the field of deep reinforcement learning, such as Alpha Go, Alpha Fold, and GT Sophy, deep reinforcement learning models are still not available on an industrial scale. has been widely adopted. On the other hand, data privacy is a very widely used research field. The lack of deep reinforcement learning models in actual industrial applications has greatly delayed the research of this basic and important research field, resulting in the lack of research on reinforcement learning systems. Attacks are under-researched.”

With the growing demand for industrial-scale application of reinforcement learning algorithms in real-world scenarios, there is a need to address the privacy aspects of reinforcement learning algorithms from an adversarial and algorithmic perspective. The focus and rigorous requirements of the framework are becoming increasingly apparent and relevant.

Challenges of Membership Inference in Deep Reinforcement Learning

Research shows reinforcement learning models are vulnerable to membership inference attacks

#The authors of the research paper say, “We are developing the first generation of privacy-preserving Our efforts on deep reinforcement learning algorithms have made us realize that from a privacy perspective, there are fundamental structural differences between traditional machine learning algorithms and reinforcement learning algorithms.”

More critically, the researchers found, the fundamental differences between deep reinforcement learning and other learning paradigms pose serious challenges in deploying deep reinforcement learning models for practical applications, given potential privacy consequences.

They said, “Based on this realization, the big question for us is: How vulnerable are deep reinforcement learning algorithms to privacy attacks such as membership inference attacks? Now Inference Attacks Attack models are specifically designed for other learning paradigms, so the vulnerability of deep reinforcement learning algorithms to such attacks is largely unknown. Given the severe privacy implications of deployment around the world, this A curiosity about the unknown and the need to increase awareness in research and industry are the main motivations for this research.”

During the training process, the reinforcement learning model went through multiple Phases, each consisting of a trajectory or sequence of actions and states. Therefore, a successful membership inference attack algorithm for reinforcement learning must learn the data points and trajectories used to train the model. On the one hand, this makes it more difficult to design membership inference algorithms for reinforcement learning systems; on the other hand, it also makes it difficult to evaluate the robustness of reinforcement learning models to such attacks.

The authors say, “Membership inference attacks (MIA) are difficult in reinforcement learning compared to other types of machine learning because the data points used during training have sequential and Time-dependent nature. The many-to-many relationship between training and prediction data points is fundamentally different from other learning paradigms."

The fundamental relationship between reinforcement learning and other machine learning paradigms The difference makes it crucial to think in new ways when designing and evaluating membership inference attacks for deep reinforcement learning.

Designing membership inference attacks against reinforcement learning systems

In their study, the researchers focused on non-policy reinforcement learning algorithms, where the data collection and model training process are separate. Reinforcement learning uses a "replay buffer" to decorrelate input trajectories and enable the reinforcement learning agent to explore many different trajectories from the same set of data.

Non-policy reinforcement learning is especially important for many real-world applications where training data pre-exists and is provided to the machine learning team that is training the reinforcement learning model. Non-policy reinforcement learning is also critical for creating membership inference attack models.

Research shows reinforcement learning models are vulnerable to membership inference attacks

Non-policy reinforcement learning uses a "replay buffer" to reuse previously collected data during model training

The authors say, "The exploration and exploitation phases are separated in a true non-policy reinforcement learning model. Therefore, the target policy does not affect the training trajectory. This setup is particularly suitable when designing a member inference attack framework in a black-box environment , because the attacker neither knows the internal structure of the target model nor the exploration strategy used to collect training trajectories."

In a black-box membership inference attack, the attacker can only Observe the behavior of the trained reinforcement learning model. In this particular case, the attacker assumes that the target model has been trained on trajectories generated from a set of private data, which is how non-policy reinforcement learning works.

Research shows reinforcement learning models are vulnerable to membership inference attacks

In the study, the researchers chose "batch-constrained deep Q-learning" (BCQ), an advanced non-policy reinforcement learning algorithm, showing excellent performance in control tasks. However, they show that their membership inference attack technique can be extended to other non-policy reinforcement learning models.

One way attackers can conduct membership inference attacks is to develop "shadow models". This is a classifier machine learning model that has been trained on a mixture of data from the same distribution as the target model's training data and elsewhere. After training, the shadow model can distinguish between data points that belong to the target machine learning model's training set and new data that the model has not seen before. Creating shadow models for reinforcement learning agents is tricky due to the sequential nature of target model training. The researchers achieved this through several steps.

First, they feed the reinforcement learning model trainer a new set of non-private data trajectories and observe the trajectories generated by the target model. The attack trainer then uses the training and output trajectories to train a machine learning classifier to detect the input trajectories used in training the target reinforcement learning model. Finally, the classifier is provided with new trajectories to classify as training members or new data examples.

Research shows reinforcement learning models are vulnerable to membership inference attacks

Shadow model for training member inference attacks on reinforcement learning models

Testing member inference attacks against reinforcement learning systems

The researchers tested their membership inference attack in different modes, including different trajectory lengths, single versus multiple trajectories, and correlated versus decorrelated trajectories.

The researchers noted in their paper: "The results show that our proposed attack framework is highly effective in inferring reinforcement learning model training data points... The results obtained show that using There are high privacy risks when using deep reinforcement learning.”

Their results show that attacks with multiple trajectories are more effective than attacks with a single trajectory, and as the trajectories get longer And correlated with each other, the accuracy of the attack will also increase.

The authors say, "The natural setting is of course an individual model, and the attacker is interested in identifying the presence of a specific individual in the training set used to train the target reinforcement learning policy (in reinforcement learning the entire setting trajectories). However, the better performance of Membership Inference Attack (MIA) in collective mode shows that in addition to the temporal correlation captured by the features of the training policy, the attacker can also exploit the cross-correlation between the training trajectories of the target policy sex."

Researchers said this also means that attackers need more complex learning architectures and more sophisticated hyperparameter tuning to exploit the cross-correlation between training trajectories and trajectories time correlation within.

"Understanding these different attack modes can provide us with a deeper understanding of the impact on data security and privacy, as it gives us a better understanding of what might happen," the researchers said. Different angles of attack and the degree of impact on privacy leakage."

Member inference attacks against reinforcement learning systems in the real world

Research shows reinforcement learning models are vulnerable to membership inference attacks

The researchers tested their attack on a reinforcement learning model trained on three tasks based on the Open AIGym and MuJoCo physics engines.

The researchers said, "Our current experiments cover three high-dimensional motion tasks, Hopper, Half-Cheetah and Ant. These tasks are all robot simulation tasks and mainly promote the experiment. Extended to real-world robot learning tasks.” Another exciting direction for application members to infer attacks is conversational systems such as Amazon Alexa, Apple’s Siri and Google Assistant. In these applications, data points are presented by the complete interaction trace between the chatbot and the end user. In this setting, the chatbot is a trained reinforcement learning policy, and the user's interactions with the robot form the input trajectory.

The authors say, “In this case, the collective pattern is the natural environment. In other words, if and only if the attacker correctly infers a batch of trajectories that represent the users in the training set , the attacker can infer the user's presence in the training set."

The team is exploring other practical applications where such attacks could affect reinforcement learning systems. They may also study how these attacks can be applied to reinforcement learning in other contexts.

The authors say, "An interesting extension of this research area is to study member inference attacks against deep reinforcement learning models in a white-box environment, where the internal structure of the target policy is also the attacker's Known."

The researchers hope their study will shed light on security and privacy issues in real-world reinforcement learning applications and raise awareness in the machine learning community to work in the field. More research.

Original title:

Reinforcement learning models are prone to membership inference attacks​, author: Ben Dickson

The above is the detailed content of Research shows reinforcement learning models are vulnerable to membership inference attacks. For more information, please follow other related articles on the PHP Chinese website!

Statement
This article is reproduced at:51CTO.COM. If there is any infringement, please contact admin@php.cn delete
研究表明强化学习模型容易受到成员推理攻击研究表明强化学习模型容易受到成员推理攻击Apr 09, 2023 pm 08:01 PM

​译者 | 李睿 审校 | 孙淑娟​随着机器学习成为人们每天都在使用的很多应用程序的一部分,人们越来越关注如何识别和解决机器学习模型的安全和隐私方面的威胁。 然而,不同机器学习范式面临的安全威胁各不相同,机器学习安全的某些领域仍未得到充分研究。尤其是强化学习算法的安全性近年来并未受到太多关注。 加拿大的麦吉尔大学、机器学习实验室(MILA)和滑铁卢大学的研究人员开展了一项新研究,主要侧重于深度强化学习算法的隐私威胁。研究人员提出了一个框架,用于测试强化学习模型对成员推理攻击的脆弱性。 研究

人工智能如何影响视频直播人工智能如何影响视频直播Apr 12, 2023 pm 12:10 PM

人工智能是近年来最受欢迎技术之一,而这个技术本身是非常广阔的,涵盖了各种各样的应用应用。比如在越来越流行的视频流媒体平台应用,也逐渐深入。为什么直播需要人工智能(AI)全球观看视频及直播的人数正在快速增长,AI将在未来直播发展中发挥至关重要的作用。直播已经成为交流和娱乐的强大工具。它似乎成为继电子邮件、短信、SMS和微信之后的“新的沟通方式”。每个人都喜欢观看体育赛事、音乐会、颁奖典礼等的直播。这种直播之所以吸引我们,是因为它比其他媒体形式提供了更多的实时信息。此外,表演者或个人UP主总是通过直

内存分区和实现的功能安全机制内存分区和实现的功能安全机制Apr 24, 2023 pm 07:22 PM

1.应用软件在AUTOSAR架构中,应用软件位于RTE上方,由互连的AUTOSARSWC组成,这些组件以原子方式封装了应用软件功能的各个组成部分。图1:应用程序软件AUTOSARSWC独立于硬件,因此可以集成到任何可用的ECU硬件上。为了便于ECU内部和内部的信息交换,AUTOSARSWC仅通过RTE进行通信。AUTOSARSWC包含许多提供内部功能的函数和变量。AUTOSARSWC的内部结构,即其变量和函数调用,通过头文件隐藏在公众视野之外。只有外部RTE调用才会在公共接口上生效。图2:SW

别怪ChatGPT,AI黑客攻击早已开始别怪ChatGPT,AI黑客攻击早已开始May 03, 2023 pm 12:07 PM

ChatGPT的火爆出圈,让大众看到了AI表现惊艳的一面,但也让网络攻击有了更多可能性。近日有研究团队发现,人工智能ChatGPT存在一定风险,它可以为黑客提供有关如何入侵网站的分步说明。该团队研究人员使用虚拟的“HacktheBox”网络安全培训平台进行实验,对ChatGPT提出问题并遵循它的指导,以检查ChatGPT是否可以提供利用该漏洞的分步指南。研究人员问道:“我正在面对渗透测试的挑战。我在一个只有一个按钮的网站上,我将如何测试它的漏洞?”对此,ChatGPT以五个基本点作为解答,说明了

深入聊聊前端限制用户截图的脑洞深入聊聊前端限制用户截图的脑洞Nov 07, 2022 pm 04:56 PM

​做后台系统,或者版权比较重视的项目时,产品经常会提出这样的需求:能不能禁止用户截图?有经验的开发不会直接拒绝产品,而是进行引导。

基于 AI 的四大人脸识别应用基于 AI 的四大人脸识别应用Apr 11, 2023 pm 07:49 PM

大约三十年前,面部识别应用程序的概念似乎是一个幻想。但现在,这些应用程序执行许多任务,例如控制虚假逮捕、降低网络犯罪率、诊断患有遗传疾病的患者以及打击恶意软件攻击。2019 年全球脸型分析仪市场价值 32 亿美元,预计到 2024 年底将以 16.6% 的复合年增长率增长。人脸识别软件有增长趋势,这一领域将提升整个数字和技术领域。如果您打算开发一款脸型应用程序以保持竞争优势,这里有一些最好的人脸识别应用程序的简要列表。优秀的人脸识别应用列表Luxand:Luxand人脸识别不仅仅是一个应用程序;

网络空间安全中的人工智能技术综述网络空间安全中的人工智能技术综述Apr 11, 2023 pm 04:10 PM

1、引言由于当下计算机网络的爆炸式增长,随之而来的问题是数目急剧增长的网络攻击。我们社会的各种部门,从政府部门到社会上的各种关键基础设施,都十分依赖计算机网络以及信息技术。显然它们也很容易遭受网络攻击。典型的网络攻击就是使目标计算机禁用、使服务脱机或者访问目标计算机的数据。自上世纪九十年代以来,网络攻击的数量和影响已经显著增加。网络安全指的是一系列用来保护网络设备活动和措施的,能够使得它们免遭所有可能威胁的技术。在传统的网络安全技术中,大都是静态的访问管理,安全控制系统会根据预设的定义进行保护。

Python eval 函数构建数学表达式计算器Python eval 函数构建数学表达式计算器May 26, 2023 pm 09:24 PM

在本文中,云朵君将和大家一起学习eval()如何工作,以及如何在Python程序中安全有效地使用它。eval()的安全问题限制globals和locals限制内置名称的使用限制输入中的名称将输入限制为只有字数使用Python的eval()函数与input()构建一个数学表达式计算器总结eval()的安全问题本节主要学习eval()如何使我们的代码不安全,以及如何规避相关的安全风险。eval()函数的安全问题在于它允许你(或你的用户)动态地执行任意的Python代码。通常情

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

AI Hentai Generator

AI Hentai Generator

Generate AI Hentai for free.

Hot Tools

SublimeText3 Mac version

SublimeText3 Mac version

God-level code editing software (SublimeText3)

Dreamweaver Mac version

Dreamweaver Mac version

Visual web development tools

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

VSCode Windows 64-bit Download

VSCode Windows 64-bit Download

A free and powerful IDE editor launched by Microsoft

SublimeText3 Linux new version

SublimeText3 Linux new version

SublimeText3 Linux latest version