--- title: Checkpoint/Restore Systems: Evolution, Techniques, and... description: A comprehensive survey of checkpoint/restore (C/R) technology covering its evolution from early OS-level process snapshotting and HPC fault tolerance through... canonical: https://daily.dev/posts/checkpoint-restore-systems-evolution-techniques-and-applications-in-ai-agents-9ulta2ppr twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Checkpoint/Restore Systems: Evolution, Techniques, and Applications in AI Agents | daily.dev og:description: A comprehensive survey of checkpoint/restore (C/R) technology covering its evolution from early OS-level process snapshotting and HPC fault tolerance through... og:url: https://daily.dev/posts/checkpoint-restore-systems-evolution-techniques-and-applications-in-ai-agents-9ulta2ppr og:image: https://api.daily.dev/og/posts/9UlTA2ppr.png og:image:alt: Checkpoint/Restore Systems: Evolution, Techniques, and Applications in AI Agents og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Checkpoint/Restore Systems: Evolution, Techniques, and Applications in AI Agents **[eunomia eBPF Blog](https://daily.dev/sources/eunomia)** · 87 min read · 0 upvotes · 0 comments ## Summary A comprehensive survey of checkpoint/restore (C/R) technology covering its evolution from early OS-level process snapshotting and HPC fault tolerance through VM/container live migration, and into emerging AI applications. The piece examines C/R at five layers (OS, container, VM, application, library), compares stateless vs. stateful restoration strategies for AI systems, and details technical mechanisms including memory snapshotting, GPU state capture (CRIUgpu, CUDA Checkpoint APIs), and distributed consistency. New AI use cases covered include LangGraph agent state persistence, Cursor IDE codebase checkpointing, ROS robotics fault recovery via DMTCP, and multi-agent coordinated snapshots. A comparative table summarizes tools like CRIU, BLCR, DMTCP, and vMotion across transparency, portability, and capability dimensions. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### How does CRIU checkpoint a running container and what state does it preserve? CRIU freezes all processes in the container using a freezer cgroup or ptrace, dumps their memory pages, CPU registers, open file descriptors, and OS resources (including TCP socket state via a kernel socket-repair API) to disk. The container runtime (e.g., Podman) additionally saves namespace configurations, cgroup state, and network stack metadata. On restore, namespaces are re-established first, then CRIU restores processes inside them, resuming open connections and in-memory caches exactly as they were. _Teams migrating stateful services between hosts track CRIU compatibility and container runtime support on daily.dev._ ### What is CRIUgpu and how does it checkpoint GPU workloads like deep learning training jobs? CRIUgpu extends CRIU to support GPU-accelerated workloads by leveraging low-level driver APIs from NVIDIA and AMD that expose CUDA context, GPU memory contents, and kernel state. It coordinates GPU state extraction with CPU process checkpointing, allowing a running deep learning training job to be transparently snapshotted — including all GPU tensors — and later restored without any modifications to the training code. This enables GPU schedulers to preempt jobs and resume them later with zero lost progress. _ML infrastructure engineers choosing between framework-level and system-level GPU checkpointing follow developments like CRIUgpu on daily.dev._ ### What is the difference between stateless and stateful checkpoint restore for AI agents? Stateful restore recreates the exact process memory image so an agent resumes mid-execution with no divergence — ideal for real-time or tightly-coupled multi-agent scenarios but requires identical hardware and OS environment. Stateless restore relaunches a fresh process and reloads serialized state (conversation history, model weights, agent memory) from files or a database — simpler and portable across versions and machines, but may miss transient in-memory variables and requires recomputation time. Hybrid approaches use stateful snapshots for fast local failover and stateless saves for cross-version durability. _Developers designing fault-tolerant AI agent architectures weigh these trade-offs alongside emerging tooling news on daily.dev._ ## Similar posts on daily.dev - [AI agents need continuity, not just context](https://daily.dev/posts/ai-agents-need-continuity-not-just-context-ytamjkqsn) · Pulumi · 3 upvotes · 1 comments - [Why Agent Crashes Are Nothing Like Database Crashes](https://daily.dev/posts/why-agent-crashes-are-nothing-like-database-crashes-ksqsnije3) · Daily Dose of Data Science \| Avi Chawla \| Substack · 2 upvotes · 0 comments - [Announcing the Checkpoint/Restore Working Group](https://daily.dev/posts/announcing-the-checkpoint-restore-working-group-edpg79z7b) · K8S Contributors · 3 upvotes · 0 comments --- Tags: [#ai-agents](https://daily.dev/tags/ai-agents), [#gpu](https://daily.dev/tags/gpu), [#distributed-systems](https://daily.dev/tags/distributed-systems) [View this post on daily.dev](https://daily.dev/posts/checkpoint-restore-systems-evolution-techniques-and-applications-in-ai-agents-9ulta2ppr) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Checkpoint/Restore Systems: Evolution, Techniques, and Applications in AI Agents","url":"https://daily.dev/posts/checkpoint-restore-systems-evolution-techniques-and-applications-in-ai-agents-9ulta2ppr","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/checkpoint-restore-systems-evolution-techniques-and-applications-in-ai-agents-9ulta2ppr"},"datePublished":"2026-08-11T08:54:13.334Z","dateModified":"2026-08-11T09:17:48.385Z","description":"A comprehensive survey of checkpoint/restore (C/R) technology covering its evolution from early OS-level process snapshotting and HPC fault tolerance through...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/b59e04eefe0ea85df0b7e50151b0b12e?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/b59e04eefe0ea85df0b7e50151b0b12e?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"eunomia eBPF Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"eunomia eBPF Blog","logo":"https://media.daily.dev/image/upload/s--z-nTl7TK--/f_auto,q_auto/v1786438304/logos/eunomia?_a=BAMAMicg0","url":"https://daily.dev/sources/eunomia"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/checkpoint-restore-systems-evolution-techniques-and-applications-in-ai-agents-9ulta2ppr","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-agents,gpu,distributed-systems","timeRequired":"PT87M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"eunomia eBPF Blog","item":"https://daily.dev/sources/eunomia"},{"@type":"ListItem","position":3,"name":"Checkpoint/Restore Systems: Evolution, Techniques, and Applications in AI Agents"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/checkpoint-restore-systems-evolution-techniques-and-applications-in-ai-agents-9ulta2ppr#faq","mainEntity":[{"@type":"Question","name":"How does CRIU checkpoint a running container and what state does it preserve?","acceptedAnswer":{"@type":"Answer","text":"CRIU freezes all processes in the container using a freezer cgroup or ptrace, dumps their memory pages, CPU registers, open file descriptors, and OS resources (including TCP socket state via a kernel socket-repair API) to disk. The container runtime (e.g., Podman) additionally saves namespace configurations, cgroup state, and network stack metadata. On restore, namespaces are re-established first, then CRIU restores processes inside them, resuming open connections and in-memory caches exactly as they were. Teams migrating stateful services between hosts track CRIU compatibility and container runtime support on daily.dev."}},{"@type":"Question","name":"What is CRIUgpu and how does it checkpoint GPU workloads like deep learning training jobs?","acceptedAnswer":{"@type":"Answer","text":"CRIUgpu extends CRIU to support GPU-accelerated workloads by leveraging low-level driver APIs from NVIDIA and AMD that expose CUDA context, GPU memory contents, and kernel state. It coordinates GPU state extraction with CPU process checkpointing, allowing a running deep learning training job to be transparently snapshotted — including all GPU tensors — and later restored without any modifications to the training code. This enables GPU schedulers to preempt jobs and resume them later with zero lost progress. ML infrastructure engineers choosing between framework-level and system-level GPU checkpointing follow developments like CRIUgpu on daily.dev."}},{"@type":"Question","name":"What is the difference between stateless and stateful checkpoint restore for AI agents?","acceptedAnswer":{"@type":"Answer","text":"Stateful restore recreates the exact process memory image so an agent resumes mid-execution with no divergence — ideal for real-time or tightly-coupled multi-agent scenarios but requires identical hardware and OS environment. Stateless restore relaunches a fresh process and reloads serialized state (conversation history, model weights, agent memory) from files or a database — simpler and portable across versions and machines, but may miss transient in-memory variables and requires recomputation time. Hybrid approaches use stateful snapshots for fast local failover and stateless saves for cross-version durability. Developers designing fault-tolerant AI agent architectures weigh these trade-offs alongside emerging tooling news on daily.dev."}}]} ```