--- title: Faster inference: torch.compile vs TensorRT | daily.dev description: A comparison of torch.compile (PyTorch 2.0's JIT compiler) and NVIDIA's TensorRT for deep learning inference optimization. Benchmarks on LLama-7b, LLama-3-8b,... canonical: https://daily.dev/posts/faster-inference-torch-compile-vs-tensorrt-mqvivznnq twitter:card: summary_large_image twitter:site: @dailydotdev og:type: website og:site_name: daily.dev og:title: Faster inference: torch.compile vs TensorRT | daily.dev og:description: A comparison of torch.compile (PyTorch 2.0's JIT compiler) and NVIDIA's TensorRT for deep learning inference optimization. Benchmarks on LLama-7b, LLama-3-8b,... og:url: https://daily.dev/posts/faster-inference-torch-compile-vs-tensorrt-mqvivznnq og:image: https://api.daily.dev/og/posts/MqVIvznnq.png og:image:alt: Faster inference: torch.compile vs TensorRT og:image:width: 1200 og:image:height: 630 og:locale: en --- > ## Documentation Index > Fetch the complete documentation index at: https://daily.dev/llms.txt > Use this file to discover all available pages before exploring further. # Faster inference: torch.compile vs TensorRT **[Collabora Blog](https://daily.dev/sources/collabora)** · 4 min read · 0 upvotes · 0 comments ## Summary A comparison of torch.compile (PyTorch 2.0's JIT compiler) and NVIDIA's TensorRT for deep learning inference optimization. Benchmarks on LLama-7b, LLama-3-8b, Mistral-v0.1, phi-3, and phi-2 show torch.compile consistently matching or outperforming TensorRT in tokens-per-second throughput, with the gap most visible on smaller models. torch.compile wins on ease of use (single line of code, no ONNX conversion), hardware flexibility (CPU/GPU/TPU), and dynamic graph support. TensorRT retains advantages in static graph specialization, mixed-precision (FP16/INT8) support, and model serialization that avoids recompilation overhead on each inference session. The conclusion favors torch.compile for most PyTorch workflows unless deep NVIDIA ecosystem integration or TensorRT-specific features are required. ## Full article daily.dev links to this article rather than hosting it. Read it at the original source: ## Questions this post answers ### Is torch.compile faster than TensorRT for LLM inference? torch.compile matches or slightly outperforms TensorRT across LLama-7b, LLama-3-8b, Mistral-v0.1, phi-3, and phi-2 in tokens-per-second benchmarks. The performance gap is marginal for larger models but more noticeable for smaller ones like phi-2 and phi-3. Given that torch.compile also requires no ONNX conversion and works with a single line of code, it is the more practical choice for most PyTorch users. _ML engineers choosing between these tools for production LLM serving track real benchmark comparisons like this on daily.dev._ ### Does torch.compile require recompilation on every inference run unlike TensorRT? Yes. torch.compile recompiles the model at the start of each inference session because it does not support graph serialization. TensorRT avoids this by serializing the optimized model and reusing it across runs, resulting in lower startup latency. This is a meaningful trade-off when startup time matters in production deployments. _Teams shipping inference pipelines where cold-start latency matters find trade-off discussions like this on daily.dev._ ### What are the main steps required to use TensorRT compared to torch.compile? TensorRT requires exporting the model to ONNX format and configuring precision modes (FP16, INT8) before deployment, adding preprocessing complexity. torch.compile, by contrast, is applied with a single line — model = torch.compile(model) — and integrates directly into existing PyTorch training and inference workflows without format conversion. _Developers weighing deployment complexity against performance gains for NVIDIA GPU workloads follow these comparisons on daily.dev._ ## Similar posts on daily.dev - [Why Is PyTorch Compile So Fast: Kernel Fusion – PyTorch](https://daily.dev/posts/why-is-pytorch-compile-so-fast-kernel-fusion-pytorch-gq553uop8) · PyTorch · 0 upvotes · 0 comments - [Adaptive Inference in NVIDIA TensorRT for RTX Enables Automatic Optimization](https://daily.dev/posts/adaptive-inference-in-nvidia-tensorrt-for-rtx-enables-automatic-optimization-pmsq81agb) · NVIDIA Developer · 0 upvotes · 0 comments - [Train a Model Faster with torch.compile and Gradient Accumulation](https://daily.dev/posts/train-a-model-faster-with-torch-compile-and-gradient-accumulation-qve6vricc) · Machine Learning Mastery · 2 upvotes · 0 comments - [Why is pytorch compile so fast?](https://daily.dev/posts/why-is-pytorch-compile-so-fast--c9e6spmov) · Red Hat Developer · 0 upvotes · 0 comments - [Automating Inference Optimizations with NVIDIA TensorRT LLM AutoDeploy](https://daily.dev/posts/automating-inference-optimizations-with-nvidia-tensorrt-llm-autodeploy-qklxaunml) · NVIDIA Developer · 1 upvotes · 0 comments --- Tags: [#ai-inference](https://daily.dev/tags/ai-inference), [#deep-learning](https://daily.dev/tags/deep-learning), [#llm](https://daily.dev/tags/llm), [#pytorch](https://daily.dev/tags/pytorch), [#tensorrt](https://daily.dev/tags/tensorrt) [View this post on daily.dev](https://daily.dev/posts/faster-inference-torch-compile-vs-tensorrt-mqvivznnq) ```json {"@context":"https://schema.org","@graph":[{"@type":"Organization","@id":"https://daily.dev/#organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180},"sameAs":["https://twitter.com/dailydotdev","https://github.com/dailydotdev","https://www.linkedin.com/company/daily-dev-ltd"]},{"@type":"WebSite","@id":"https://daily.dev/#website","url":"https://daily.dev","name":"daily.dev","publisher":{"@id":"https://daily.dev/#organization"},"potentialAction":{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https://daily.dev/search?q={search_term_string}"},"query-input":"required name=search_term_string"}}]} {"@context":"https://schema.org","@type":"TechArticle","headline":"Faster inference: torch.compile vs TensorRT","url":"https://daily.dev/posts/faster-inference-torch-compile-vs-tensorrt-mqvivznnq","mainEntityOfPage":{"@type":"WebPage","@id":"https://daily.dev/posts/faster-inference-torch-compile-vs-tensorrt-mqvivznnq"},"datePublished":"2026-08-11T08:54:56.824Z","dateModified":"2026-10-14T02:27:14.294Z","description":"A comparison of torch.compile (PyTorch 2.0's JIT compiler) and NVIDIA's TensorRT for deep learning inference optimization. Benchmarks on LLama-7b, LLama-3-8b,...","image":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/82a4aaabbfcd42cbc247a248e2740dbc?_a=AQAEuop","thumbnailUrl":"https://media.daily.dev/image/upload/f_auto,q_auto/v1/posts/82a4aaabbfcd42cbc247a248e2740dbc?_a=AQAEuop","isAccessibleForFree":true,"articleSection":"Collabora Blog","inLanguage":"en","publisher":{"@type":"Organization","name":"daily.dev","url":"https://daily.dev","logo":{"@type":"ImageObject","url":"https://daily.dev/apple-touch-icon.png","width":180,"height":180}},"author":{"@type":"Organization","name":"Collabora Blog","logo":"https://media.daily.dev/image/upload/s--ZGZ0FjqS--/f_auto,q_auto/v1786438306/logos/collabora?_a=BAMAMicg0","url":"https://daily.dev/sources/collabora"},"commentCount":0,"discussionUrl":"https://daily.dev/posts/faster-inference-torch-compile-vs-tensorrt-mqvivznnq","interactionStatistic":[{"@type":"InteractionCounter","interactionType":{"@type":"LikeAction"},"userInteractionCount":0},{"@type":"InteractionCounter","interactionType":{"@type":"CommentAction"},"userInteractionCount":0}],"keywords":"ai-inference,deep-learning,llm,pytorch,tensorrt","timeRequired":"PT4M"} {"@context":"https://schema.org","@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://daily.dev"},{"@type":"ListItem","position":2,"name":"Collabora Blog","item":"https://daily.dev/sources/collabora"},{"@type":"ListItem","position":3,"name":"Faster inference: torch.compile vs TensorRT"}]} {"@context":"https://schema.org","@type":"FAQPage","@id":"https://daily.dev/posts/faster-inference-torch-compile-vs-tensorrt-mqvivznnq#faq","mainEntity":[{"@type":"Question","name":"Is torch.compile faster than TensorRT for LLM inference?","acceptedAnswer":{"@type":"Answer","text":"torch.compile matches or slightly outperforms TensorRT across LLama-7b, LLama-3-8b, Mistral-v0.1, phi-3, and phi-2 in tokens-per-second benchmarks. The performance gap is marginal for larger models but more noticeable for smaller ones like phi-2 and phi-3. Given that torch.compile also requires no ONNX conversion and works with a single line of code, it is the more practical choice for most PyTorch users. ML engineers choosing between these tools for production LLM serving track real benchmark comparisons like this on daily.dev."}},{"@type":"Question","name":"Does torch.compile require recompilation on every inference run unlike TensorRT?","acceptedAnswer":{"@type":"Answer","text":"Yes. torch.compile recompiles the model at the start of each inference session because it does not support graph serialization. TensorRT avoids this by serializing the optimized model and reusing it across runs, resulting in lower startup latency. This is a meaningful trade-off when startup time matters in production deployments. Teams shipping inference pipelines where cold-start latency matters find trade-off discussions like this on daily.dev."}},{"@type":"Question","name":"What are the main steps required to use TensorRT compared to torch.compile?","acceptedAnswer":{"@type":"Answer","text":"TensorRT requires exporting the model to ONNX format and configuring precision modes (FP16, INT8) before deployment, adding preprocessing complexity. torch.compile, by contrast, is applied with a single line — model = torch.compile(model) — and integrates directly into existing PyTorch training and inference workflows without format conversion. Developers weighing deployment complexity against performance gains for NVIDIA GPU workloads follow these comparisons on daily.dev."}}]} ```