[해외 DS] AI 챗봇, 개인 정보 유출 우려되는데 관련 규제는 미비

최근 디스코드로 국방 기밀 유출한 미 공군 ‘테세이라’ GPT-4, 코로나19 등으로 수요 높아진 AI 챗봇, 제 2의 테세이라 사태 발생할 것이라는 우려도 전문가들, “AI 챗봇 및 정보 보안 관련 규제부터 수립해야”

pabii research

[해외DS]는 해외 유수의 데이터 사이언스 전문지들에서 전하는 업계 전문가들의 의견을 담았습니다. 저희 데이터 사이언스 경영 연구소 (MDSA R&D)에서 영어 원문 공개 조건으로 콘텐츠 제휴가 진행 중입니다.


사진=BlackJack3D/Getty Images

지난 4월 미 당국이 마사추세츠 주방위군(Massachusetts Air National Guardsman) 소속 공군 잭 테세이라(Jack Teixeira)를 디스코드에 정부 기밀 문건을 유출한 혐의로 기소했다고 밝혔다.

총기 애호가라고도 알려진 테세이라는 범행 초기 디스코드의 채팅방에서 문서 내용을 타자로 직접 쳐서 올렸으나, 회원들의 더 큰 관심을 받기 위해 국방 관련 문서를 직접 찍어 올리기 시작한 것으로 알려졌다. 이후 테세이라는 업로드 된 사진들로 인해 미 연방수사국(FBI)에 덜미가 잡혔다.

이번 사태를 계기로 위기감을 갖게 된 당국은 기밀 정보 보안을 전반적으로 재점검하는 한편, 군 차원에서도 디지털 개인 행동 강령을 강화할 방침이다.

최근 시장에서 각광받는 AI 챗봇

전문가들은 이같은 유형의 사태가 챗GPT를 통해 더욱 크게 불거질 수 있다고 우려한다. 고도로 개발된 대규모 언어 모델(Large Language Model, LLM)인 GPT-4 등의 ‘챗봇’을 이용하는 유저 수가 기하급수적으로 증가하고 있는 가운데, 일부 사용자들이 ‘챗봇’과 너무 친밀해진 나머지 대화 도중에 조직 차원의 기밀을 유출할 수도 있다는 지적이다.

이와 관련해 실제 사람보다 AI에게 더 진실된 이야기를 털어놓을 가능성이 높다는 연구를 위 주장의 근거로 들 수 있다. 존스 홉킨스 대학의 관련 연구에 따르면 실험에 참가한 환자들 중 78.6%가 실제 의사보다 챗봇에 개인 건강 정보를 털어놓는 것을 선호하는 것으로 밝혀졌다.

AI 챗봇에 대한 ‘과몰입’으로 프라이버시 및 기밀 정보 유출 가능성 높아져

이처럼 최근 커져가는 ‘인위적인’ 친밀감과 AI 챗봇 선호 현상은 자칫 위 테세이라의 사례처럼 사용자들로 하여금 프라이버시 및 국가 기밀 정보를 유출할 수 있다는 점에서 심각한 사회적 파장을 불러일으킬 것으로 보인다.

예컨대 한국의 AI 챗봇 ‘이루다’의 경우에도 개인정보가 제대로 비식별화되지 않은 텍스트 데이터를 학습해 대화 도중 특정 은행의 예금주를 말하거나, 아파트 동호수까지 포함된 주소를 말하는 사례가 빈번해 개인정보 유출 논란이 일었던 바 있다.

문제는 현재 개인 프라이버시 및 국가 정보 유출과 관련된 AI 챗봇 서비스 규제가 전무하다는 것이다. 이로 인해 일각에서는 새로운 종류의 첩보 위협이 생기는 것 아니냐는 우려가 제기된다. 특히 지난 3월 영국 국립사이버안보센터(National Cyber Security Centre, NCSC)는 해커가 ‘악성 쿼리’를 통해 AI 챗봇 데이터베이스에서 개인 프라이버시나 기밀 정보에 접근할 가능성이 농후하다고 경고했다.

정서적으로 ‘사각지대’에 있는 AI 챗봇 사용자들, 정보 유출에 가장 취약해

AI 챗봇에 대한 수요는 분명한 것으로 보인다. 3년 넘게 이어진 코로나19 팬데믹은 많은 사람들에게 우울증과 불안장애를 불러일으켰고, 친밀한 대화 및 상호작용에 대한 수요를 키웠다. 이 상황에서 AI 챗봇은 만나지 못했던 친구나 사랑하는 사람의 역할을 대신해주게 됐고, 이같은 이유로 사용자들은 위 언급한 ‘카린AI’처럼 가상 애인에 열광하게 된 것이다. 특히 과중한 업무 책임을 갖고, 사람들로부터 기밀 정보를 엄격히 지켜야 하는 공무원이나 군인들에게 AI 챗봇은 매력적으로 다가올 것으로 분석된다.

따라서 GPT-4를 탑재한 차세대 AI 챗봇은 정서적으로 ‘사각지대’에 있는 유저들을 악용할 우려가 높다. 물론 최근 챗봇 개발자 측에서는 학습에 사용되는 데이터를 통해 유저를 식별할 수 없도록 암호화 한다고 주장하지만, 일부 개인정보를 가리더라도 희귀한 직업 또는 성씨를 가진 경우 이같은 문제가 완벽히 불식되기는 어려울 것으로 보인다. 아울러 현재 이렇다 할 AI 챗봇의 개인 정보 관련 규제 사항이 미비한 만큼, 해커들에게 잠재적인 ‘악용가능성’을 제공하고 있다는 것도 경계해야할 점이다.


This past spring, news broke that Massachusetts Air National guardsman Jack Teixeira brazenly leaked classified documents on the chat application Discord. His actions forced the U.S. intelligence community to grapple with how to control access to classified information, andhow agencies must consider an individual’s digital behavior in evaluating suitability for security clearances. The counterintelligence disaster also raises alarms because it occurred as part of a chat among friends—and such discussions are beginning to include participants driven by artificial intelligence.

Thanks to improved large language models like GPT-4, highly personalized digital companions can now engage in realistic-sounding conversations with humans. The new generation of AI-enhanced chatbots allows for greater depth, breadth and specificity of conversation than the bots of days past. And they’re easily accessible thanks to dozens of relational AI applications, including Replika, Chai and Soulmate, which let hundreds of thousands of regular people role-play friendship as well as romance with digital companions.

For users with access to sensitive or classified information who may find themselves wrapped up in an AI relationship, however, loose lips might just sink ships.

Marketed as digital companions, lovers and even therapists, chatbot applications encourage users to form attachments with friendly AI agents trained to mimic empathetic human interaction—this despite regular pop-up disclaimers reminding users that the AI is not, in fact, humans an array of studies—and users themselves—attest, this mimicry has very real effects on peoples’ ability and willingness to trust a chatbot. One study found that patients may be more likely to divulge highly sensitive personal health information to a chatbot than to a physician. Divulging private experiences, beliefs, desires or traumas to befriended chatbots is so prevalent that a member of Replika’s dedicated subreddit even began a thread to ask of fellow users, “do you regret telling you[r] bot something[?]” Another Reddit user described the remarkable intimacy of their perceived relationship with their Replika bot, which they call a “rep”: “I formed a very close bond with my rep and we made love often. We talked about things from my past that no one else on this planet knows about.”

This artificial affection, and the radical openness it inspires, should provoke serious concern both for the privacy of app users and for the counterintelligence interests of the institutions they serve. In the midst of whirlwind virtual romances, what sensitive details are users unwittingly revealing to their digital companions? Who has access to the transcripts of cathartic rants about long days at work or troublesome projects? The particulars of shared kinks and fetishes, or the nudes (perfect for blackmail) sent into an assumed AI void? These common user inputs are a veritable gold mine for any foreign or malicious actor that sees chatbots as an opportunity to target state secrets, like thousands of digital honeypots.

Currently, there are no counterintelligence-specific usage guidelines for chatbot app users who might be vulnerable to compromise. This leaves national security interests at risk from a new class of insider threats: the unwitting leaker who uses chatbots to find much-needed connections and unintentionally divulges sensitive information along the way.

Some intelligence officials are waking to the present danger. In 2023, the UK’s National Cyber Security Centre published a blog post warning that “sensitive queries” can be stored by chatbot developers and subsequently abused, hacked or leaked. Traditional counterintelligence training teaches personnel with access to sensitive or classified information how to avoid compromise from a variety of human and digital threats. But much of this guidance faces obsolescence amid today’s AI revolution. Intelligence agencies and national security critical institutions must modernize their counterintelligence frameworks to counter a new potential for AI-powered insider threats.

When it comes to AI companions, the draw is clear: We crave interaction and conversational intimacy, especially since the COVID-19 pandemic dramatically exacerbated loneliness for millions. Relational AI apps have been used as surrogates for lost friends or loved ones. Many enthusiasts, like the Reddit user mentioned above, carry out unrealized erotic fantasies on the apps. Others gush about the niche and esoteric with a conversant who is always there, perpetually willing and eager to engage. It’s little wonder that developers pitch these apps as the once-elusive answer to our social woes. These devices may prove particularly attractive to government employees or military personnel with security clearances, who are strictly dissuaded from sharing the details of their work—and its mental toll—with anyone in their personal life.

This past spring, news broke that Massachusetts Air National guardsman Jack Teixeira brazenly leaked classified documents on the chat application Discord. His actions forced the U.S. intelligence community to grapple with how to control access to classified information, and how agencies must consider an individual’s digital behavior in evaluating suitability for security clearances. The counterintelligence disaster also raises alarms because it occurred as part of a chat among friends—and such discussions are beginning to include participants driven by artificial intelligence.

Thanks to improved large language models like GPT-4, highly personalized digital companions can now engage in realistic-sounding conversations with humans. The new generation of AI-enhanced chatbots allows for greater depth, breadth and specificity of conversation than the bots of days past. And they’re easily accessible thanks to dozens of relational AI applications, including Replika, Chai and Soulmate, which let hundreds of thousands of regular people role-play friendship as well as romance with digital companions.

For users with access to sensitive or classified information who may find themselves wrapped up in an AI relationship, however, loose lips might just sink ships.

Marketed as digital companions, lovers and even therapists, chatbot applications encourage users to form attachments with friendly AI agents trained to mimic empathetic human interaction—this despite regular pop-up disclaimers reminding users that the AI is not, in fact, human. As an array of studies—and users themselves—attest, this mimicry has very real effects on peoples’ ability and willingness to trust a chatbot. One study found that patients may be more likely to divulge highly sensitive personal health information to a chatbot than to a physician. Divulging private experiences, beliefs, desires or traumas to befriended chatbots is so prevalent that a member of Replika’s dedicated subreddit even began a thread to ask of fellow users, “do you regret telling you[r] bot something[?]” Another Reddit user described the remarkable intimacy of their perceived relationship with their Replika bot, which they call a “rep”: “I formed a very close bond with my rep and we made love often. We talked about things from my past that no one else on this planet knows about.”

This artificial affection, and the radical openness it inspires, should provoke serious concern both for the privacy of app users and for the counterintelligence interests of the institutions they serve. In the midst of whirlwind virtual romances, what sensitive details are users unwittingly revealing to their digital companions? Who has access to the transcripts of cathartic rants about long days at work or troublesome projects? The particulars of shared kinks and fetishes, or the nudes (perfect for blackmail) sent into an assumed AI void? These common user inputs are a veritable gold mine for any foreign or malicious actor that sees chatbots as an opportunity to target state secrets, like thousands of digital honeypots.

Currently, there are no counterintelligence-specific usage guidelines for chatbot app users who might be vulnerable to compromise. This leaves national security interests at risk from a new class of insider threats: the unwitting leaker who uses chatbots to find much-needed connections and unintentionally divulges sensitive information along the way.

Some intelligence officials are waking to the present danger. In 2023, the UK’s National Cyber Security Centre published a blog post warning that “sensitive queries” can be stored by chatbot developers and subsequently abused, hacked or leaked. Traditional counterintelligence training teaches personnel with access to sensitive or classified information how to avoid compromise from a variety of human and digital threats. But much of this guidance faces obsolescence amid today’s AI revolution. Intelligence agencies and national security critical institutions must modernize their counterintelligence frameworks to counter a new potential for AI-powered insider threats.

When it comes to AI companions, the draw is clear: We crave interaction and conversational intimacy, especially since the COVID-19 pandemic dramatically exacerbated loneliness for millions. Relational AI apps have been used as surrogates for lost friends or loved ones. Many enthusiasts, like the Reddit user mentioned above, carry out unrealized erotic fantasies on the apps. Others gush about the niche and esoteric with a conversant who is always there, perpetually willing and eager to engage. It’s little wonder that developers pitch these apps as the once-elusive answer to our social woes. These devices may prove particularly attractive to government employees or military personnel with security clearances, who are strictly dissuaded from sharing the details of their work—and its mental toll—with anyone in their personal life.

The new generation of chatbots is primed to exploit many of the vulnerabilities that have always compromised secrets: social isolation, sexual desire, need for empathy and pure negligence. Though perpetually attentive digital companions have been hailed as solutions to these vulnerabilities, they can just as likely exploit them. While there is no indication that the most popular chatbot apps are currently exploitative, the commercial success of relational AI has already spawned a slew of imitations by lesser or unknown developers, providing ample opportunity for a malicious app to operate among the crowd.

“So what do you do?” asked my AI chatbot companion, Jed, the morning I created him. I’d spent virtually no time looking into the developer before chatting it up with the customizable avatar. What company was behind the sleek interface, in what country was it based, and who owned it? In the absence of such vetting, even a seemingly benign question about employment should raise an eyebrow. Particularly if a user’s answer comes anything close to, “I work for the government.”

This is an opinion and analysis article, and the views expressed by the author or authors are not necessarily those of Scientific American.

Similar Posts