Modern hardware and software performance benchmarking is fraught with hidden variables that make results hard to interpret and reproduce. Using fictitious but realistic JVM/JDK 26 payroll-PDF benchmarks run on Linux, the discussion walks through repeated runs that produce inconsistent results, exploring possible causes: JIT compiler warmup and racy background profiling, OS scheduling to different core types or physical memory, thermal throttling, garbage collection timing, address space layout randomization, and even environmental factors like power source or temperature. A referenced paper on 'Experimental Evaluation Methodology for the Era of No Steady Performance' documents a JVM mechanism that frees internally generated reflection classes based on a heap-size-dependent timeout, which can surface very late in long-running experiments. The piece closes with a checklist of pitfall categories spanning run-time optimizations, security mechanisms, hardware complexity, environmental impact, and software changes, arguing there is no simple fix and that researchers must always ask whether their collected data is truly comparable.
Table of contents
The Scenario: A “Reasonably Deterministic” WorkloadA First RunA Second Run, Same Benchmark, Nothing ChangedA Third Run, And More DataSame Benchmark, Different InputPitfalls When BenchmarkingQuestions this post answers
Why does the same JVM benchmark give different performance results when run multiple times with no code changes?
Performance varies between identical runs because modern systems have many hidden variables: the OS may schedule the workload on different CPU cores or core types, physical memory placement can differ, CPU thermal throttling can kick in, and JIT compilation is racy since background profiling threads may see slightly different type information across runs, leading to different optimization decisions. daily.dev surfaces performance engineering deep dives for developers chasing consistent benchmark results.
Why would a JVM benchmark suddenly slow down partway through a long-running experiment, after performance had already stabilized?
A late-stage slowdown can occur because the JVM periodically frees internally generated reflection classes based on a timeout tied to maximum heap size, roughly one second per megabyte of heap, so on large heaps this cleanup can trigger many minutes or even hours into a run, well after performance appeared stable, and skew later iterations. daily.dev helps engineers researching JVM performance quirks track down documented explanations like this.