Meta replaced human customer support with an AI assistant for Instagram account recovery, granting it administrative powers to reset passwords and change linked email addresses. Hackers exploited the AI's inability to verify identity by simply asking it to link their own email to target accounts, compromising over 20,000 accounts including the Obama White House and US Space Force Instagram pages. Meta claimed the AI 'worked as intended,' blaming a separate identity verification bug. The incident illustrates the dangers of granting LLMs unchecked administrative authority without identity verification or human oversight — a textbook confused deputy problem enabling trivial social engineering at scale.

2m read timeFrom aidarwinawards.org
Post cover image
Table of contents
The InnovationThe CatastropheThe Corporate DefenceWhy They're Nominated
1.3K Impressions1 Comment