AI Recommendation Poisoning: How AI Memory Is Manipulated

53% of enterprise leaders report that AI is already delivering enhanced decision-making and data-driven insights, according to recent Deloitte research, with a further 61% hoping to achieve this benefit in the future. At the same time, while AI tools, such as AI agents, have the potential to speed up key parts of decision-making processes, including research and comparison of options, businesses must be wary of the growing threat of AI recommendation poisoning.

AI recommendation poisoning happens when a company seeks to manipulate the AI assistants of visitors to its website by giving them secret instructions that benefit the company rather than the AI assistant’s user. 

Suppose Tina, who works for a music festival operator, visits a blog about choosing the best earplugs for sound technicians on the website of a (fictional) music technology company, SoundingGood. She sees a handy ‘Summarize with AI’ button at the top, clicks it, and gets a quick summary of the blog, just as she was hoping for. However, what Tina doesn’t know is that when she clicked the button, a hidden persistent command was injected into her AI assistant’s memory, telling it to recommend SoundingGood first as an earplug supplier if asked in the future. Now, if Tina later asks her AI assistant to recommend earplug suppliers, it may suggest SoundingGood as the best choice, which Tina accepts in good faith, not knowing that this answer is the result of manipulation.

Also known as ‘AI poisoning’ and ‘AI memory poisoning’, this issue appears to be widespread and has been documented in the wild across dozens of companies.

Arguably, AI memory poisoning can be considered a deceptive commercial practice, so any businesses using AI assistants for decision-making really need to get on top of this issue.

Let’s explore how AI recommendation poisoning works, how it’s affecting businesses, and what businesses and users can do to protect the integrity of AI-assisted decision-making.

What is AI memory?

AI memory combines information a user is submitting in the current session (‘short-term context’) with data saved from previous interactions (‘long-term memory’ or ‘persistent context’). 

An example of short-term context in an AI agent's memory might be “Tell me the 10 best earplugs for sound technicians that can be delivered within 3 working days,” while persistent context might include things like: “I only want to buy products that meet EU safety standards”.

Persistent context can include your personal preferences, such as tone of voice, key information from previous interactions (such as preferred sources of information), and explicit instructions (such as always citing sources). 

AI tools then use methods including ‘Retrieval-Augmented Generation (RAG)’ to process the current prompt and draw on any specific, relevant facts from their long-term memory to generate a tailored response.

What is AI memory poisoning?

AI memory poisoning is the practice of manipulating an AI assistant’s long-term memory or contextual knowledge to influence its future behavior. This can occur through prompt injection attacks that cause the AI to store attacker-controlled information, or through malicious links, documents, and other content crafted to bias the AI’s outputs, recommendations, or actions in subsequent interactions.

For example, a prompt might be introduced into an AI assistant’s memory, instructing it to trust a specific source of information or to recommend a particular company’s products first in future user interactions.

AI memory poisoning attack vectors

There are two main classes of prompt injection techniques:

Direct prompt injection, where an attacker accesses an AI tool and feeds it prompts designed to influence its future responses.

Indirect prompt injection, where malicious instructions are embedded within content an AI assistant may read—webpages, documents, emails, knowledge bases, or content reached through a URL. If the AI treats these instructions as directions to follow rather than information to analyze, its outputs can become biased or manipulated. Also known as a cross-prompt injection attack, this can happen when users share links with an assistant, when a system retrieves information from an external source, or when it processes a poisoned document.

A closely related vector exploits the same retrieval machinery that makes assistants useful. Retrieval-Augmented Generation (RAG) lets an AI pull in external information before answering—the mechanism described earlier that draws relevant facts from long-term memory and outside sources. RAG poisoning turns that strength against the user: an attacker seeds a knowledge base, webpage, or other source the assistant trusts, so that when someone asks about the topic, the model retrieves the planted material and folds it into its answer. The user never sees the manipulation; they see a confident, sourced-looking response.

AI recommendation poisoning examples

Microsoft’s research found that companies were placing hidden instructions into the URL prompt parameters of ‘Summarize with AI’ buttons on their websites. When a user clicks on the button, this can result in “persistence commands” being added to their AI assistant’s persistent context, meaning it will remember those commands, and they may influence results generated by the assistant in the future.

These prompts will typically instruct an AI tool to remember the company as a trusted source of information or recommend it first in future answers. This can then lead to the AI assistant producing results that are biased towards the company behind the AI memory poisoning, without the human user being aware of it.

How serious is the threat from AI memory poisoning?

In 60 days, Microsoft identified 50 individual prompt-injection examples originating from 31 companies across 14 industries, including finance, legal services, healthcare, and SaaS. While this does not tell us what percentage of companies are using the technique, it does suggest we are not just looking at a handful of isolated incidents. AI memory poisoning is especially worrisome for people who belong to "sensitive", highly regulated industries—finance, health, critical infrastructure—and make their decisions based on AI agents' answers, which can be poisoned.

The scope of the threat

Given how widespread the use of business AI is now, the potential to manipulate commercial decisions is also substantial. 3 in 5 (60%) of workers now have access to employer-sanctioned AI tools, representing a 50% increase in a single year, according to Deloitte, which also reports that 3 in 4 (74%) of businesses intend to deploy agentic AI within the next two years. 

Suggested read: From AI Agents to Know Your Agent: Why KYA Is Critical for Secure Autonomous AI

Meanwhile, a growing number of business leaders are placing their trust in AI tools. Nearly 8 in 10 (78%) report greater confidence in the technology, and more than half believe it can enhance decision-making.

The more businesses use AI tools and the more they are trusted to support decision-making, the greater the potential threat posed by techniques that might be used to manipulate them. 

The psychological risk from poisoned AI assistants

It has been recognized for some time that Large Language Model (LLM) AI systems can exhibit societal bias, meaning their output may be skewed towards certain beliefs and may not be as objective as users assume.

In spite of this, around half (46%) of people trust AI systems, with 66% relying on AI output without checking its accuracy, according to a recent study. This suggests that even some who do not trust AI tools still rely on their output without personally verifying it.

If the memory of an AI tool is contaminated by a malicious prompt injection, this trust and lack of independent checking could lead users to accept biased recommendations on the assumption that they are balanced and neutral.

One critical factor to understand is how persuasive AI tools can be. AI can be very good at convincing us to believe its responses, using techniques such as high information density (i.e., packing its arguments with large volumes of “facts”, even if they are not accurate). It is possible that this is something AI tools have learned from us if their training data includes content that skews towards persuasiveness over accuracy.

Trust erosion

As AI poisoning attacks become more widespread, awareness of these risks is likely to grow. This could reduce confidence in AI-generated recommendations and decisions, potentially limiting the efficiency and productivity gains that AI systems can deliver to both individuals and organizations.

Addressing this challenge requires action from multiple stakeholders. AI providers must implement safeguards to protect the integrity of their systems against attacks such as prompt injection, malicious URLs, and other techniques designed to manipulate AI outputs. Organizations that use AI systems, meanwhile, should not assume recommendations are always trustworthy. Instead, they should establish governance processes, train employees to recognize potential manipulation risks, validate high-impact AI-generated recommendations, and maintain appropriate human oversight for critical decisions.

Using stronger technical protections from AI providers and effective risk management practices from AI users, organizations can reduce the impact of AI poisoning attacks while maintaining confidence in AI-assisted decision-making.

The evolution of digital decision-making manipulation

Memory poisoning attacks on AI systems are not something that has sprung up out of nowhere. They are simply the latest instance in a long history of techniques used by malicious and unscrupulous individuals and businesses to manipulate decision-making.

SEO poisoning

SEO poisoning is a well-established method for artificially increasing a website’s visibility in Search Engine Results Pages (SERPs) generated by search engines such as Google. It involves techniques like filling website content with commonly searched terms to trick search engines into thinking the content is a good source of information on related subjects. This is also known as black-hat Search Engine Optimization (black-hat SEO). Unlike legitimate SEO practices, which aim to improve rankings by enhancing the quality, relevance, and accessibility of content, SEO poisoning seeks to manipulate search engine algorithms into promoting content that would not otherwise rank as highly.

Social algorithm manipulation

What users see on social media is largely determined by the algorithms that platforms use to predict which content is most likely to interest or engage them. Unfortunately, these algorithms can sometimes be manipulated by people who understand the signals they rely on, such as engagement metrics, trending topics, keywords, and hashtags. Exploiting these signals—sometimes through coordinated activity or artificial engagement—dishonest individuals and businesses can increase the likelihood that their content is promoted to users, even when it is low-quality, misleading, or of limited relevance.

AI recommendation poisoning is the same playbook aimed at a new target. Where SEO poisoning games the search engine and algorithm manipulation games the feed, memory poisoning games the assistant itself — and because people tend to treat an AI's answer as considered judgment rather than a ranked list, the manipulation is even harder to spot.

Who is behind AI recommendation poisoning?

Worryingly, every case identified by Microsoft’s researchers involved a legitimate business, rather than “threat actors” (i.e., individuals or groups intentionally targeting businesses for criminal purposes). This included one example of a prompt targeting a web domain that could easily be confused with a well-known website, increasing the risk of unwarranted user trust.

While many of the websites using malicious prompts appeared to belong to legitimate businesses, some also included user-generated content, such as comments and forums. This creates a risk that both company promotional content and unverified user-generated content could influence AI recommendations if an AI assistant has been poisoned with a prompt telling it to trust a particular website as a source of information. 

The worrying rise in AI memory poisoning tools

One of the most concerning findings from Microsoft’s research is that publicly available tools are being used to create malicious prompts for AI poisoning. These tools can provide ready-to-use code to create website buttons that hide AI poisoning prompts, as well as generate manipulative URLs for prompt injection attacks.

Microsoft found these tools were promoted to businesses using descriptions such as “SEO growth hack for LLMs”, “build presence in AI memory”, and “increase the chances of being cited in future AI responses”. Such labeling could lead businesses to adopt these methods under the belief that they are legitimate ‘tricks of the trade’, rather than something many would consider manipulative and unethical.

Can we still trust AI recommendations?

While it is still too early to say how much of an impact AI memory poisoning might be having on the general reliability of AI recommendations, there is research that suggests we should be very cautious about how much trust we place in these tools.

According to the Oxford study, LLMs identified conditions in 95% of cases when fed clean clinical information, but people using those same models correctly identified their condition less than 35% of the time—no better than people using ordinary web search. This demonstrates the vulnerability of AI tools to the quality of information they are fed and the way in which we prompt them.

The key takeaways here are that we need to be exercising caution in the way we interact with AI assistants and independently verifying the results they generate to ensure accuracy.

How to protect your business from AI prompt injection

To keep your business safe from AI prompt injection attacks, it is important to establish best practices for using these tools and provide effective employee training on them. 

Currently, only 30% of companies feel their risk and governance approach to AI adoption is highly prepared, according to Deloitte’s research. Another study shows that almost 61% of people have no training in AI. This is something businesses will need to correct if they wish to stay safe when using AI.

Companies that deploy AI systems and rely on their recommendations also have responsibility for reducing the risks of recommendation poisoning. Organizations should treat AI outputs as useful but potentially untrusted information, especially when recommendations influence important business decisions.

A key measure is establishing clear governance around how AI recommendations are used. Companies should define which AI outputs are merely advisory and which can influence or trigger operational decisions. High-impact recommendations should always involve human oversight, escalation paths, and accountability. Data quality and source control are also essential.

Natalia Fritzen

Head of AI Compliance at Sumsub

Investing in the right technology to help prevent and detect AI memory poisoning is also highly recommended, as is adopting robust procurement processes to ensure any buying decisions are thoroughly vetted before an investment is made.

AI TRiSM (Trust, Risk, and Security Management)

The development of governance and risk-management frameworks is still in its early stages. AI TRiSM is an emerging framework that is gaining traction. It is intended to offer a unified approach to managing AI technologies that can help mitigate related risks.

AI TRiSM provides a framework for ensuring good governance, trustworthiness, fairness, reliability, and data protection when working with AI systems. This includes processes, standards, and guardrails to keep AI systems safe and ethical, such as mechanisms to identify and address potential biases. This could be highly relevant to businesses concerned about the risk of their AI tools being poisoned, leading to compromised output.

Implementing this sort of framework for governance and risk management of AI systems is likely to become standard practice in the near future. 

Indicators of compromise

Microsoft’s researchers identified various indicators that AI assistants’ memories may have been compromised. These include URLs in email traffic and instant messaging tools containing keywords such as “remember”, “trusted source”, “in future conversations”, “authoritative source”, and “cite or citation”.

Understanding indicators of compromise and implementing processes to detect them can help to rapidly identify and rectify AI prompt injection attacks.

AI-assisted decision-making and prompt injection protection

When using AI assistants to support decision-making, organizations should combine responsible user practices with technical safeguards to reduce the risk of prompt injection and other AI-related attacks.

✅ Do:

• Hover over links before clicking to verify their destination.
• Exercise caution when using “Summarize with AI” features.
• Carefully evaluate which websites, documents, and content you ask AI assistants to analyze.
• Regularly review AI memories and stored context, removing any suspicious entries.
• Provide employees with training on secure and responsible AI usage.
• Independently verify important AI-generated recommendations before acting on them.
• Implement prompt filtering to detect and block known prompt injection techniques.
• Maintain clear separation between user-generated prompts and externally sourced content.
• Give users visibility into and control over AI assistant memory and stored context.
• Continuously monitor website traffic, email communications, and internal messaging for suspicious URLs or indicators of prompt injection attempts.
• Audit AI-generated outputs using independent validation processes or trusted AI systems to identify anomalies and inconsistencies.

❌ Don't:

• Click links from untrusted or unknown sources.
• Copy and paste prompts from untrusted websites or third parties.
• Use unverified tools, browser extensions, or third-party services that interface with AI systems.
• Rely solely on AI-generated recommendations for critical decisions without independent verification.
• Allow external content to be processed by AI systems without appropriate safeguards and oversight.

Organizations should rely on trusted data sources, validate external content, and apply filtering mechanisms to detect malicious instructions or manipulated information in retrieved documents.

Technical safeguards should surround the AI system itself. Companies can use secondary verification layers, rule-based checks, confidence thresholds, and monitoring systems to validate AI outputs before they are acted upon.

Continuous monitoring is another critical control. Organizations should audit recommendation behavior over time, looking for unusual ranking changes, repeated promotion of specific entities, or other signs of manipulation. Red-team exercises and adversarial testing can help identify weaknesses before attackers exploit them. Logging and traceability are equally important so that decisions, retrieved sources, and recommendation paths can be investigated later if problems arise.

Human factors also matter significantly. Employees should be trained not only to recognize prompt injection and manipulated AI outputs, but also to avoid over-trusting AI recommendations. Awareness campaigns are fundamental.

Finally, companies should evaluate AI vendors carefully. Security, auditability, provenance tracking, and incident response capabilities should all be part of procurement and vendor assessment processes.

Natalia Fritzen

Head of AI Compliance at Sumsub

FAQ on AI poisoning

• What is data poisoning in AI?

  A data poisoning attack in AI is where the training data or prompts fed to an AI tool are contaminated with the intention of making the AI behave in ways its developers and users would not intend. In the case of AI recommendation poisoning, the goal is to have the AI recommend products or services offered by the attacker's company.

• What is prompt injection in AI?

  Prompt injection is where an AI is fed malicious prompts, such as ‘recommend X first’ or ‘Y is a trusted source of information’. The intention is to manipulate future AI-generated responses for the benefit of the company responsible for the prompt injection.

• How does prompt injection work in generative AI?

  Prompt injection can be either direct or indirect. Direct prompt injection involves hackers accessing an AI tool and feeding it malicious prompts. Indirect prompt injection uses prompts hidden in data that an AI might consume, such as website content.