ChatGPT and large language models: what are the risks?

Large-scale language models (LLMs) and AI chatbots are attracting worldwide interest due to the release of ChatGPT in late 2022 and the query convenience it provides. It's now one of the fastest-growing consumer applications of all time, and its popularity is prompting many competitors to develop their own services and models, or to quickly deploy those they've been developing in-house.

As with any emerging technology, there are always concerns about what this means for security. This blog has recently considered some of the cybersecurity aspects of ChatGPT and LLM more generally.

What is ChatGPT and what are LLMs?

ChatGPT is an artificial intelligence chatbot developed by American technology startup OpenAI. It's based on GPT-3, a language model released in 2020 that uses deep learning to generate human-like text, but the underlying LLM technology has been around for a long time.

The LLM is where algorithms are trained on large amounts of text-based data, often scraped from the open internet, thus covering web pages and – depending on the LLM – other sources , such as scientific research, books, or social media posts. This covers such a large amount of data that it is impossible to filter out all objectionable or inaccurate content upon ingestion, so "controversial" content is likely to be included in its in the model.

These algorithms analyze the relationships between different words and convert them into probabilistic models. The algorithm can then be given a "hint" (for example, by asking it a question) and it will provide an answer based on the relationships of the words in its model.

Typically, the data in its model is static after training, although it can be improved through "fine-tuning" (training on additional data) and "hint augmentation" (providing contextual information about the problem). An example of prompt enhancement might be:

Taking into account the below information, how would you describe...

Then copying potentially a large amount of text (or the entire document) into the prompt/question .

ChatGPT effectively allows users to ask LLM questions, just like you would in a conversation with a chatbot. Other recent examples of LLM include announcements from Google's Bard and Meta's LLaMa (for scientific papers).

LL.M.s are certainly impressive because of their ability to generate a wealth of compelling content in a variety of human and computer languages. However, they are not magic, nor are they artificial general intelligence, and contain some serious flaws, including:

• They can get things wrong and "hallucinate" incorrect facts
• They can be biased and often gullible (e.g. when answering the main question)
• They require huge computing resources and massive amounts of data to train from scratch
• They can be coaxed into creating toxic content and is vulnerable to "injection attacks"

Will LLM leak my information?

A common concern is that LLM may "learn" from your prompts and provide that information to others who query related content. There are some reasons for concern here, but not for the reasons many people consider. Currently, an LLM is trained and then the resulting model is queried. LLM does not (at the time of writing) automatically add information from a query to its model for others to query. That is, including information in a query will not cause that data to be incorporated into the LLM.

However, the query will be visible to the organization providing the LLM (for ChatGPT, also for OpenAI). These queries are stored and will almost certainly be used at some point to develop an LLM service or model. This may mean that the LLM provider (or its partners/contractors) is able to read the queries and possibly incorporate them into future releases in some way. Therefore, you need to thoroughly understand the Terms of Use and Privacy Policy before asking sensitive questions.

A question may be sensitive because of the data contained in the query, or because of who (and when) asked the question. Examples of the latter might be if it is discovered that the CEO has asked "How best to fire an employee?", or if someone has asked revealing health or relationship questions. Also remember to use the same login to aggregate information across multiple queries.

Another risk that increases as more organizations produce LLMs is that queries stored online could be hacked, leaked, or more likely accidentally made publicly accessible. This may include potentially user-identifying information. Another risk is that the operator of the LLM is later acquired by an organization that adopts a different approach to privacy than when the user entered their data.

Therefore, NCSC recommends:

• Don’t include sensitive information in queries to public LLMs
• Do not submit queries to public LLMs that will cause problems

How do I securely submit queries to LLMs Sensitive information?

With the rise of LLM, many organizations may be wondering whether they can use LLM to automate certain business tasks, which may involve providing sensitive information through fine-tuning or just-in-time augmentation. While this approach is not recommended for public LLMs, a "private LLM" may be provided by a cloud provider (for example), or may be completely self-hosted:

• For cloud-provided LLMs, the Terms of Use and Privacy policies are again key (as they are for public LLMs), but are more likely to be consistent with the cloud service's existing terms. Organizations need to understand how to manage data used for fine-tuning or prompt augmentation. Can the vendor's researchers or partners use it? If so, in what form? Is data shared individually or in aggregate with other organizations? Under what circumstances can the provider's employees view the query?
• Self-hosted LLM can be very expensive. However, following a security assessment, they may be suitable for processing organizational data. In particular, organizations should refer to our guidance on protecting infrastructure and data supply chains.

Do LLMs make cybercriminals’ lives easier?

There have been some incredible demonstrations of how LLM can help write malware. The concern is that LLM could help malicious (but unskilled) individuals create tools they would not otherwise be able to deploy. In their current state, LLMs look convincing (whether they are or not) and are suited to simple tasks rather than complex ones. This means that LLM can be used to “help experts save time” because experts can verify the output of the LLM.

For more complex tasks, it is currently easier for experts to create malware from scratch rather than having to spend time correcting what LLM generates. However, experts capable of creating powerful malware may well be able to trick LLM into writing powerful malware. The trade-off between "using LLM to create malware from scratch" and "validating LLM-created malware" will change as LLM improves.

You can also ask the LLM for advice on technical issues. Criminals may use LLM to help conduct cyber attacks beyond their current capabilities, especially after the attacker gains access to the network. For example, if an attacker is working to escalate privileges or find data, they might ask LLM and receive a different answer than the search engine results but with more context. Current LLMs provide answers that sound convincing but may only be partially correct, especially as the topic becomes more niche. The answers may help criminals carry out attacks they could not otherwise perform, or they may suggest actions to expedite detection of criminals. Either way, the attacker's queries may be stored and retained by the LLM operator.

Because LLM excels at replicating writing styles on demand, there is a risk that criminals could use LLM to write convincing phishing emails, including emails in multiple languages. This can help attackers with high technical abilities but lack language skills, helping them create convincing phishing emails (or conduct social engineering) in the target's native language.

In summary, in the short term we may see:

• More convincing phishing emails due to LLM
• Attackers try things they wouldn’t before Familiar Technology

The risk of a less skilled attacker writing powerful malware is also low.

Summary

This is an exciting time for LLM, especially with ChatGPT capturing the imagination of the world. As with all technological developments, there will be people who are keen to use it and study what it has to offer, and people who may never use it.

As we have outlined above, there are undoubtedly risks associated with the unrestricted use of a public LLM. Individuals and organizations should be extremely careful about the data they choose to submit in prompts. You should ensure that those who want to try LLM can, but do so without putting organizational data at risk.

NCSC is aware of other emerging threats (and opportunities) related to cybersecurity and LLM adoption, and we will of course make you aware of these in future blog posts.

David C - Technical Director of Platform Research Paul J - Technical Director of Data Science Research

——Compiled from UK NCSC

The above is the detailed content of ChatGPT and large language models: what are the risks?. For more information, please follow other related articles on the PHP Chinese website!