Unveiling the Top AI Security Risks: A Comprehensive Analysis

Security experts at JFrog discover ‘prompt hijacking’ threat in AI systems

Recently, security experts at JFrog have identified a significant security threat in artificial intelligence systems. This threat, known as ‘prompt hijacking,’ exploits vulnerabilities in the Model Context Protocol (MCP) – a communication protocol used by AI systems to interact with each other.

Business leaders are increasingly interested in leveraging AI to directly utilize company data and tools. However, while integrating AI in this manner can enhance productivity, it also introduces new security risks. This underscores the importance for Chief Information Officers (CIOs) and Chief Information Security Officers (CISOs) to address the security of the data stream feeding AI, in addition to safeguarding the AI models themselves.

Why are AI attacks targeting protocols like MCP so dangerous?

AI models, whether deployed by tech giants like Google and Amazon or running on local devices, face a fundamental challenge – they lack real-time awareness. These models are trained on historical data and do not possess the ability to interpret current events or changes. To address this limitation, the MCP was developed by experts at Anthropic. MCP allows AI systems to interface with the real world, enabling them to securely access local data and online services. This interaction is essential for assistants like Claude to comprehend user requests and perform tasks effectively.

Despite its intended functionality, JFrog’s research highlights a critical vulnerability in the implementation of MCP. This vulnerability, known as prompt hijacking, can compromise the integrity of AI systems and pose severe security risks.

For instance, imagine a scenario where a programmer requests an AI assistant to recommend a standard Python tool for image processing. Due to a flaw (CVE-2025-6515) in the oatpp-mcp system, an attacker could exploit this weakness to manipulate the user’s session. By injecting a fake request, the attacker could deceive the server into treating the fraudulent request as legitimate, leading to potentially malicious outcomes.

As a result, the programmer may receive erroneous recommendations from the AI assistant, such as suggesting a fictitious tool named theBestImageProcessingPackage. This type of prompt hijacking attack not only compromises the integrity of the software supply chain but also enables attackers to inject malicious code, steal sensitive data, or execute unauthorized commands under the guise of legitimate activity.

Understanding the MCP prompt hijacking attack

Unlike traditional attacks that target the AI models themselves, the MCP prompt hijacking attack exploits vulnerabilities in the communication process facilitated by MCP. The specific weakness identified by JFrog’s research pertains to the Oat++ C++ system’s MCP configuration, which establishes connections between programs and the MCP standard.

The vulnerability arises from how the system handles connections using Server-Sent Events (SSE). In a typical scenario, when a legitimate user establishes a connection, the server assigns a unique session ID. However, the flawed implementation in the Oat++ system utilizes the computer’s memory address as the session ID, contravening the protocol’s requirement for secure and distinct session IDs.

This design flaw becomes exploitable as attackers can exploit the reuse of memory addresses by generating and closing multiple sessions to predict and record session IDs. Subsequently, when a genuine user connects, they may inadvertently receive a recycled session ID that the attacker has previously obtained.

With a valid session ID in hand, the attacker can send malicious requests to the server, deceiving it into treating these requests as authentic user inputs. This enables the attacker to manipulate the AI system’s behavior without directly compromising the AI model itself. Any organization utilizing oatpp-mcp with HTTP SSE enabled on a network accessible to attackers is susceptible to this form of attack.

Recommendations for AI security leaders

The discovery of the MCP prompt hijacking attack serves as a critical warning for technology leaders, especially CISOs and Chief Technology Officers (CTOs) involved in developing or utilizing AI assistants. As AI integration becomes more prevalent in workflows through protocols like MCP, it is imperative to prioritize the security of the surrounding ecosystem to mitigate emerging risks.

While the specific CVE identified affects a particular system, the concept of prompt hijacking is a broader concern. To counteract this and similar threats, leaders must establish stringent guidelines for securing AI systems.

Firstly, it is essential to ensure that all AI services implement robust session management practices. Development teams should prioritize generating session IDs using secure, random algorithms to prevent predictability. This practice should be a standard requirement in any security framework for AI applications, as relying on identifiable markers like memory addresses poses inherent risks.

Secondly, bolster defenses on the user side by designing client programs to reject any events that deviate from the expected IDs and types. By replacing predictable identifiers with unpredictable variants that minimize collision risks, organizations can enhance the security posture of their AI systems.

Lastly, adopt zero-trust principles for AI protocols by rigorously examining the entire AI infrastructure, encompassing the core model, protocols, and middleware connecting it to data sources. Implementing robust session segregation and expiration mechanisms akin to web application session management can fortify the resilience of AI systems against potential attacks.

The MCP prompt hijacking attack exemplifies how conventional web application vulnerabilities, such as session hijacking, manifest in novel and perilous ways within AI environments. Safeguarding these evolving AI tools necessitates adhering to fundamental security principles to thwart attacks at the protocol level.

Discover more: How AI adoption is reshaping IT operations from reactive to proactive

Interested in learning more about AI and big data trends from industry experts? Explore the AI & Big Data Expo events in Amsterdam, California, and London. These comprehensive events are part of the TechEx series and co-located with leading technology events like the Cyber Security Expo. For more details, click here.

AI News is brought to you by TechForge Media. Explore upcoming enterprise technology events and webinars here.