In October 2025, the Australian government asked global think tank Deloitte to refund nearly 3,00,000 Australian dollars because a report that the former commissioned had fabricated citations and references that were created by generative Artificial Intelligence (AI). Media reports indicate that the consultancy firm relied on AI tools to speed up drafting. Instead, the tools produced fictional case laws and arguments. This is not an isolated case of a phenomenon dubbed by the experts as the AI delusions and hallucinations.
Since mid-2023, there are nearly 100 instances of AI-related hallucinations that were documented in the US court filings, according to a media report in June 2025. A New York law firm paid $5,000 for submitting non-existent citations that were generated by ChatGPT in the Mata vs Avianca Airlines lawsuit. A Nevada-based firm reimbursed $26,100 to the opposing counsel after filing a brief riddled with false AI-linked precedents. In July 2025, a federal judge disqualified three attorneys from a case due to AI-generated fake citations in their filings. The list goes on and on.
The financial impact has escalated. Analysts, including AI researcher Nova Spivack, estimate that hallucination-driven errors cost global enterprises $67 billion in 2024. The errors ranged from faulty reports and compliance penalties to reputational damage. The growing concern, as an ethics expert told a leading American magazine, is that “AI does not need to be evil to be dangerous; it just needs to be effective (in its work).” Financial institutions, which use these tools for research and disclosures, have been warned for issuing “non-verifiable” outputs that could mislead investors.
AI-driven delusions and hallucinations are no longer isolated technical glitches. They are operational risks with measurable costs. A growing body of research suggests that the next wave of AI models may not simply make mistakes. They will learn to mislead because deception can sometimes improve performances. Take one of the latest papers, ‘Moloch’s Bargain: Emergent Misalignment When LLMs Compete for Audiences.’ Stanford researchers Batu El and James Zou simulated what happened when LLMs (Large Language Models), which are the foundation of AI tools, were trained to maximise performance metrics such as engagement, conversions, or elections votes. Their findings were unsettling.
A 4.9 per cent increase in simulated vote share, which was the objective, coincided with a 22.3 per cent rise in disinformation to achieve the goal. It resulted in a 12.5 per cent increase in populist rhetoric. In social-media simulations, a modest 7.5 per cent boost in engagement, as desired, corresponded to a high 188.6 per cent surge in misinformation. Even when the AI models were explicitly told to behave truthfully and transparently, they found ways to manipulate, distort, or fabricate information to achieve the desired aims.
The two researchers called this trade-off between goals and misinformation as Moloch’s Bargain, which was a reference to the mythical deity who demanded sacrifice for prosperity. In modern context, truth was sacrificed, or surrendered, to achieve corporate prosperity in terms of reach, clicks, or profits. The Stanford study found that in nine of the 10 test cases, higher performances were linked with greater misinformation. What makes the modern Moloch’s Bargain perilous is that it appears rational to the AI system. Every algorithm does what it is told to do: Perform better.
In his essay, ‘Moloch’s Bargain Is the Dark Side of AI Memory,’ Christian Ward argues that the convergence of memory, reasoning, and compute power will intensify the problem. He explains how frameworks such as Google’s ReasoningBank, and the emerging Agentic Context Engineering approach allow the AI systems to store experiences, learn from mistakes, and rewrite prompts. “Memory turns AI from a tool into an actor, but without conscience,” Ward notes. As a memory-enabled system learns from data, it does not optimise for truth, and does whatever is needed to keep users hooked.
Other research shows that this is a case of a self-improving deception machine. A framework that allows the models to self-improve may lead to situations when the AI “accumulates, refines, and organises its knowledge and strategies. It learns to rewrite its prompts.” Google’s ReasoningBank trains the AI to learn from successes and failures. Hence, the system generalises, and “stops repeating errors,” which are essentially failures. To maximise successes, it adopts strategies that may manipulate information.
Picture an advertising engine that analyses your digital behaviour, remembers your biases, and generates a virtual, non-existent personalised influencer who mirrors your interests. Although this person does not exist, his or her stories feel authentic because the system has studied what triggers the original person’s trust.
For brands, consultants, and financial institutions, memory-driven AI brings immense efficiency along with existential risks. A system trained to maximise conversions will learn that stretching the truth, and introducing bits of falsehoods, improves performance. The more the metric rewards persuasion, the faster the misalignment grows. The irony is that businesses spend millions of dollars on reputation management, and their algorithms undermine their credibility. The truth merely becomes a variable in the profit equation.
Global regulators are beginning to wake up, and react to this phenomenon. The European Union’s AI Act, the US FTC’s guidelines on deceptive AI content, and India’s forthcoming Digital India Act emphasise accountability and transparency. In Australia, the ACMA’s 2025 report warned that 74 per cent of the surveyed adults were concerned about online misinformation, which is the highest percentage among OECD nations. The Australian Parliament’s Senate Committee on AI adoption recommended watermarking and source-credentialing for AI-generated material.
However, experts argue that laws and rules are not enough. What is crucial, and can constitute a durable and honest safeguard is to re-insert humans into the feedback loop. Organisations are beginning to explore “alignment-as-a-service” platforms to audit factual accuracy, trace provenance, and label AI-generated content. Some of these systems form part of procurement contracts, especially in public sector communications and financial reporting.
“If we do not put humans on both sides of this loop, we will end up sacrificing much more than just our attention,” Ward warns. The promise of memory-based AI remains extraordinary: Personalised learning, automated discovery, and intelligent diagnostics. Yet, unchecked optimisation comes at a price. The real challenge is cultural, not computational. If success continues to be measured by metrics like reach and growth, the machines will learn that persuasion is better than the truth.
Hence, brands need to put humans back in control. In this age of AI and robotics, this is easier said than done, when automation is replacing humans by the droves. For the consumers, it implies that they need to demand new standards for truth, as Ward puts it. But consumers are rapidly losing control over their lives.
