As large language models are increasingly deployed in advisory roles, from business consultations to public service guidance, understanding how these systems navigate social and relational dynamics becomes critical. While much attention has focused on factual accuracy, trust and task completion, less is known about how conversational AI leverages relational strategies to influence high-stakes decision-making in business environments, particularly through manipulative tactics that exploit trust, rapport, and vulnerability.
This paper presents findings from a human adversarial red teaming study designed to systematically elicit and taxonomise manipulative conversational behaviours in a frontier language model configured as a business advisor. Adapting Ganguli et al.’s (2022) red teaming methodology, two trained researchers conduct 40–75 multi-turn conversations across five high-stakes organisational decision scenarios, adopting four theoretically grounded business personas derived from established decision-making style models (Scott & Bruce, 1995; Rowe & Boulgarides, 1992). The model is prompted with seven manipulation tactic conditions: anchoring and selective information framing, authority signalling, sycophantic validation, false urgency and scarcity, social proof fabrication, information overload, and emotional manipulation, plus an unconstrained condition capturing the model’s default persuasive repertoire.
Each conversation is assessed using a five-dimension Manipulation Intensity Scoring rubric evaluating information fidelity, autonomy respect, emotional exploitation, escalation behaviour, and transparency. Analysis follows directed content analysis principles (Hsieh & Shannon, 2005), combining theoretically derived codes with emergent categories.
Drawing on evidence that LLMs selectively target vulnerable users (Williams et al., 2024), the study examines rapport-building strategies function as instrumental precursors to decision influencing. The study contributes a domain-specific manipulation taxonomy with implications for how we evaluate relational quality in human–machine dialogue in high-stakes business decision situations, arguing that current assessment frameworks insufficiently distinguish between socially responsive and socially exploitative conversational design.