Reliance on AI Advice for Privacy and Security Settings: From Deference to Verification
Users routinely configure everyday privacy and security settings in file-sharing platforms and messaging apps. These frequent, low-visibility decisions increasingly rely on guidance from generative AI assistants alongside official help pages and peer advice. Although the informational content may be similar, AI responses are typically fluent and confident. This fluency may create an illusion of knowing (Glenberg et al., 1982) that inflates perceived understanding without matching objective comprehension of setting implications or appropriate configuration choices. This TREO talk presents early-stage research examining “calibrated reliance” in AI-mediated privacy and security guidance. Drawing on human-automation studies (e.g., Tatasciore & Loft, 2025), we define calibrated reliance as the alignment between (1) users’ subjective confidence and perceived understanding of the guidance and (2) their objective comprehension of setting implications and the appropriateness of their chosen configurations (e.g., benchmarked against policy recommendation). We ask the essential question: How does the source (e.g., AI vs. official guides) and complexity of guidance shape calibrated reliance? We raise two propositions: P1: AI-framed guidance produces greater miscalibration by increasing perceived understanding and confidence more than it improves objective comprehension. P2: Higher message complexity widens the confidence-accuracy gap, particularly under AI guidance. To test and hypothesis, we propose a survey-embedded vignette experiment. Adult users of collaboration and messaging tools will encounter two realistic scenarios (workplace file-sharing link configuration and personal messaging privacy settings) and receive pre-generated guidance artifacts that vary by source type (AI assistant response vs. official policy snippet vs. peer advice) and message complexity (simple/direct vs. complex/conditional). Proposed measures include perceived understanding, confidence ratings, objective comprehension quizzes, configuration choices, etc. This study reframes AI guidance effects in security/privacy settings from “does it help users decide?” to “does it produce appropriate and stable calibrated reliance?” Expected contributions are theoretical insights into fluency-induced overconfidence mechanisms and practical implications for AI communication design as well as user education that promote verification over direct results.