Safety, Alignment & Governance

Building with AI means owning its failure modes. This section is deliberately neutral and tight, enough to reason about risk without ideology.

What alignment / RLHF targets. Alignment is the effort to make a model's behavior track human intent and values: helpful, honest, and harmless, roughly. RLHF and its relatives (RLAIF, DPO, see How LLMs Actually Work) are the primary technical levers. They shape the model to prefer responses people judge as good and to refuse ones they judge as harmful. But alignment is imperfect and shallow in important ways. A model can be helpful and honest in ordinary use yet still be steered off the rails by adversarial input. That gap is why the next paragraph matters so much.

Jailbreaks & prompt injection: the critical one for builders. Two related but distinct threats:

• A jailbreak is a user crafting input that coaxes the model past its safety training (role-play framings, obfuscation, "ignore previous instructions").
• Prompt injection is more insidious and is the security problem for agents and tool use. When your app feeds the model untrusted content (a web page, an email, a document, a tool's output), that content can contain instructions the model follows as if they came from you. An attacker who controls a web page your agent browses can tell it to exfiltrate data, misuse a connected tool, or take destructive actions. The OWASP Top 10 for LLM Applications ranks prompt injection as the number-one risk. The core danger is that the model doesn't reliably distinguish trusted instructions from untrusted data in its context. It's all just tokens. The more capable your agent (tools, memory, autonomy, access to real systems), the higher the stakes. Defenses in depth: constrain tool permissions and scopes, keep a human in the loop for consequential actions, sandbox execution, sanitize and clearly delimit untrusted content, and never assume the model will "know better." Treat any model handling untrusted input as a partially-compromised component and design blast radius accordingly.

Red-teaming & evals basics. Red-teaming is adversarially probing a system for failures (harmful outputs, jailbreak susceptibility, injection vulnerabilities) before attackers or users find them. Evals are systematic, repeatable tests of behavior: correctness, safety, refusal rates, regressions between versions. The discipline mirrors software testing. You can't manage what you don't measure, and vibes don't survive contact with production. Build a small eval set early, automate it, and gate releases on it. This is covered practically in the agents and production parts of the handbook.

The regulatory & governance landscape as of mid-2026 🔄. This moves fast, so verify the current status before relying on any specific date. The short version: if you ship AI, three separate things can constrain you, a law in the EU, a patchwork of rules in the US, and your own company's internal policies.

• EU AI Act (Europe's main AI law). It sorts AI systems by how risky they are and puts the heaviest duties on the riskiest uses, and it is rolling out in phases. The worst uses (like government social scoring) were banned in February 2025. Rules for providers of general-purpose models began in August 2025. From August 2, 2026, you have to tell users when they are talking to an AI and label AI-generated images, audio, and video (summary). The heaviest "high-risk" requirements were pushed back by a mid-2026 simplification package, to December 2027 and August 2028. What it means for you: if you have EU users, plan for the transparency and labeling duties now. The heavier compliance paperwork for high-risk uses comes later, but it is coming.
• United States (no single federal law). The US picture is unsettled. In December 2025 the White House issued an executive order pushing a light-touch, pro-industry approach and moving to challenge stricter state laws (analysis). But individual states still have their own AI laws on the books (California's SB 53, Colorado's SB 205), and many state attorneys general are pushing back. What it means for you: there is no one national rulebook. Requirements vary by state and are likely to keep shifting through 2026. 🔄
• Enterprise governance (your own company's rules). Separate from any law, most companies set their own internal AI policies: which tools are allowed, how models and vendors get reviewed, how customer data and personal information are handled, when a human has to sign off, and what gets logged. Frameworks like the NIST AI Risk Management Framework are common starting points. What it means for you: inside a company, this internal governance is usually the real gate that blocks or clears a launch, so plan for it early instead of treating it as an afterthought.

The throughline of this section is that capability and risk scale together. The same autonomy that makes an agent useful makes it dangerous when fed hostile input or pointed at real systems. Design for that from the start.