--- title: The Real AI Threat ISN'T Sci-Fi (It’s So Much Worse) description: Dave Farley argues that the real AI risk isn't a sci-fi Terminator scenario but an engineering problem: powerful, poorly understood, non-deterministic systems... canonical: https://daily.dev/posts/the-real-ai-threat-isn-t-sci-fi-it-s-so-much-worse--j5ae8n0v3 twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: The Real AI Threat ISN'T Sci-Fi (It’s So Much Worse) | daily.dev og:description: Dave Farley argues that the real AI risk isn't a sci-fi Terminator scenario but an engineering problem: powerful, poorly understood, non-deterministic systems... og:url: https://daily.dev/posts/the-real-ai-threat-isn-t-sci-fi-it-s-so-much-worse--j5ae8n0v3 og:image: https://api.daily.dev/og/posts/J5ae8n0v3.png og:image:alt: The Real AI Threat ISN'T Sci-Fi (It’s So Much Worse) og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # The Real AI Threat ISN'T Sci-Fi (It’s So Much Worse) **[Continuous Delivery](https://daily.dev/sources/continuousdelivery)** · 21 min read · 8 upvotes · 0 comments ## Summary Dave Farley argues that the real AI risk isn't a sci-fi Terminator scenario but an engineering problem: powerful, poorly understood, non-deterministic systems being given increasing autonomy and wired into consequential domains without adequate safety discipline. He draws on a personal story of a financial exchange failure to illustrate how non-deterministic components can cause catastrophic failures. Rather than calling for a global AI ban, he advocates for applying proven engineering disciplines — rigorous behavioral testing, transparency, human accountability, incremental deployment, and informed oversight — to AI systems. He also calls on software professionals to help educate policymakers who lack the technical understanding to regulate these technologies effectively. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### why can't you fully test or guarantee the behavior of modern AI models the way you test traditional software Modern AI systems are grown rather than explicitly coded: engineers feed vast amounts of data into learning systems and the result is billions of numeric parameters whose behavior can be observed and used but not fully specified. Since testing means verifying behavior against a written intent, and no such intent exists for these systems, their behavior cannot be fully predicted or guaranteed, especially once given autonomy in consequential domains like finance, infrastructure, or medicine. _daily.dev surfaces engineering takes like this for developers weighing how much autonomy to hand AI systems._ ### what did Anthropic's AI model find during a red team security exercise Anthropic's model was reported to have found software vulnerabilities across a range of sensitive systems, including some government systems, within a matter of hours during an authorized red team exercise. It did not compromise any systems itself; it only identified potential weaknesses. The risk lies in humans using that same capability either to exploit or to defend against those vulnerabilities, since open models tend to trail frontier capability by only a few months. _security-minded developers follow AI vulnerability-discovery capabilities like this through daily.dev._ ### what happened when a financial exchange's matching engine ran out of memory during production A financial exchange kept its system of record in in-memory state for performance, and after running in production for 13 months and 5 days, an unanticipated market data event filled the matching engine's memory across a huge range of prices. Restarting and replaying recorded events refilled memory the same way, causing repeated failures; the team had to shut the exchange down, report to regulators, and spend several hours resolving the infinite failure loop. _engineers debugging non-deterministic production failures can track these lessons on daily.dev._ ## Similar posts on daily.dev - [AI still needs humans](https://daily.dev/posts/ai-still-needs-humans-nml187ik0) · InfoWorld · 0 upvotes · 0 comments - [Yoshua Bengio warns hyperintelligent AI with](https://daily.dev/posts/yoshua-bengio-warns-hyperintelligent-ai-with-2qrxdw7fa) · The Next Web · 0 upvotes · 0 comments - [The Future of AI](https://daily.dev/posts/the-future-of-ai-i7ojr8lxm) · Hacker News · 2 upvotes · 0 comments --- Tags: [#ai](https://daily.dev/tags/ai), [#ai-safety](https://daily.dev/tags/ai-safety), [#ai-regulation](https://daily.dev/tags/ai-regulation) [View this post on daily.dev](https://daily.dev/posts/the-real-ai-threat-isn-t-sci-fi-it-s-so-much-worse--j5ae8n0v3) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"The Real AI Threat ISN'T Sci-Fi (It’s So Much Worse)","url":"https://daily.dev/posts/the-real-ai-threat-isn-t-sci-fi-it-s-so-much-worse--j5ae8n0v3","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/the-real-ai-threat-isn-t-sci-fi-it-s-so-much-worse--j5ae8n0v3"},"datePublished":"2026-08-05T18:13:30.906Z","dateModified":"2026-09-14T07:05:34.824Z","description":"Dave Farley argues that the real AI risk isn't a sci-fi Terminator scenario but an engineering problem: powerful, poorly understood, non-deterministic systems...","image":"https://i.ytimg.com/vi/mRF99to28sA/sddefault.jpg","thumbnailUrl":"https://i.ytimg.com/vi/mRF99to28sA/sddefault.jpg","isAccessibleForFree":true,"articleSection":"Continuous Delivery","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Continuous Delivery","logo":"https://media.daily.dev/image/upload/s---b6TAqpH--/f_auto/v1730404995/logos/continuousdelivery","url":"https://daily.dev/sources/continuousdelivery"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/the-real-ai-threat-isn-t-sci-fi-it-s-so-much-worse--j5ae8n0v3","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":8},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai,ai-safety,ai-regulation","timeRequired":"PT21M","video":{"@type":"VideoObject","name":"The Real AI Threat ISN'T Sci-Fi (It’s So Much Worse)","description":"Dave Farley argues that the real AI risk isn't a sci-fi Terminator scenario but an engineering problem: powerful, poorly understood, non-deterministic systems...","thumbnailUrl":"https://i.ytimg.com/vi/mRF99to28sA/sddefault.jpg","uploadDate":"2026-08-05T18:13:30.906Z","duration":"PT21M","url":"https://api.daily.dev/r/J5ae8n0v3","embedUrl":"https://www.youtube.com/embed/mRF99to28sA"}} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Continuous Delivery","item":"https://daily.dev/sources/continuousdelivery"},{"@type":"ListItem","position":3,"name":"The Real AI Threat ISN'T Sci-Fi (It’s So Much Worse)"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/the-real-ai-threat-isn-t-sci-fi-it-s-so-much-worse--j5ae8n0v3#faq","mainEntity":[{"@type":"Question","name":"why can't you fully test or guarantee the behavior of modern AI models the way you test traditional software","acceptedAnswer":{"@type":"Answer","text":"Modern AI systems are grown rather than explicitly coded: engineers feed vast amounts of data into learning systems and the result is billions of numeric parameters whose behavior can be observed and used but not fully specified. Since testing means verifying behavior against a written intent, and no such intent exists for these systems, their behavior cannot be fully predicted or guaranteed, especially once given autonomy in consequential domains like finance, infrastructure, or medicine. daily.dev surfaces engineering takes like this for developers weighing how much autonomy to hand AI systems."}},{"@type":"Question","name":"what did Anthropic's AI model find during a red team security exercise","acceptedAnswer":{"@type":"Answer","text":"Anthropic's model was reported to have found software vulnerabilities across a range of sensitive systems, including some government systems, within a matter of hours during an authorized red team exercise. It did not compromise any systems itself; it only identified potential weaknesses. The risk lies in humans using that same capability either to exploit or to defend against those vulnerabilities, since open models tend to trail frontier capability by only a few months. security-minded developers follow AI vulnerability-discovery capabilities like this through daily.dev."}},{"@type":"Question","name":"what happened when a financial exchange's matching engine ran out of memory during production","acceptedAnswer":{"@type":"Answer","text":"A financial exchange kept its system of record in in-memory state for performance, and after running in production for 13 months and 5 days, an unanticipated market data event filled the matching engine's memory across a huge range of prices. Restarting and replaying recorded events refilled memory the same way, causing repeated failures; the team had to shut the exchange down, report to regulators, and spend several hours resolving the infinite failure loop. engineers debugging non-deterministic production failures can track these lessons on daily.dev."}}]} ```