Just-in-time compilation transformed how dynamic languages achieve performance, but early JIT systems faced a fundamental tradeoff: compile quickly with minimal optimization, or spend time generating highly optimized machine code. Modern runtimes solve this with tiered compilation, using multiple optimization levels that progressively improve performance as code proves itself hot.

The Cold Start Problem

When a JIT runtime starts executing code, it faces uncertainty. Most functions run infrequently or only once during initialization. Spending significant time optimizing rarely-executed code wastes compilation resources and delays startup. Early JIT implementations chose one of two extremes: either interpret everything slowly until profiles justified optimization, or eagerly compile everything and accept the compilation overhead.

Tiered compilation takes a middle path. Code begins execution at the lowest tier—often an interpreter or extremely fast baseline compiler—then migrates through progressively more optimized tiers as it accumulates execution counts and runtime profile data.

How Tiers Work

A typical modern runtime implements three or four distinct tiers. The interpreter tier executes bytecode directly with minimal overhead, collecting basic profiling data like function invocation counts and branch directions. When a function crosses an invocation threshold, it promotes to the baseline JIT tier.

The baseline compiler generates machine code quickly with almost no optimization. It produces code perhaps 10x faster than the interpreter runs it, but the compilation itself takes milliseconds rather than the seconds required for full optimization. Critically, the baseline tier instruments the generated code with counters and type feedback collectors.

Once a function demonstrates sustained hotness—executing thousands or tens of thousands of times—and accumulates sufficient type information, it promotes to the optimizing tier. This tier applies aggressive optimizations: inline caching, method inlining, escape analysis, loop unrolling, and speculative optimizations based on observed types.

Some runtimes add a fourth tier for sustained peak performance, applying even more expensive optimizations like auto-vectorization and advanced register allocation.

Profile-Guided Optimization at Runtime

The key insight behind tiering is that optimization decisions improve dramatically with runtime data. Static compilers guess which branches are likely and which types dominate. JIT tiers know for certain because they observed actual execution.

When the optimizing tier compiles a function, it has access to concrete type distributions from the baseline tier. If a polymorphic call site received only one type across thousands of invocations, the optimizer can speculate on that type, inline the method, and insert a guard to deoptimize if the assumption breaks. This speculative optimization based on runtime profiles often produces faster code than ahead-of-time compilers can generate.

Deoptimization and Invalidation

Tiered compilation requires bidirectional flow. Code not only promotes to higher tiers but occasionally demotes when speculative assumptions fail. If optimized code encounters a type it didn’t expect or a branch that invalidates inlining decisions, the runtime deoptimizes back to a lower tier.

The runtime must maintain enough metadata to reconstruct interpreter or baseline state from optimized code. When deoptimization occurs, execution transfers to the baseline version, which continues collecting new profiles. If the code stabilizes again, it may reoptimize with updated assumptions.

Compilation Budget and Latency

Tiering manages compilation as a limited resource. An optimizing compiler might spend 100ms generating code for a single hot function. In a latency-sensitive application with thousands of warm functions, optimizing everything serially would create unacceptable pause times or consume excessive CPU.

Runtimes use compilation queues with priority schemes. Functions enter the optimizing queue when they cross hotness thresholds, but only the hottest compile first. The baseline tier provides acceptable performance for warm-but-not-hot code, avoiding the need to optimize everything.

Some systems run compilation on background threads to avoid blocking execution. Others compile during garbage collection pauses when the application threads are already stopped.

Why This Matters

Tiered compilation fundamentally changes performance characteristics. Applications start quickly because minimal compilation happens upfront. Peak performance rivals or exceeds ahead-of-time compiled code because optimizations use actual runtime data. Memory overhead stays reasonable because only hot code receives expensive optimization.

This architecture explains why modern JavaScript engines, the JVM, .NET, and emerging runtimes like GraalVM all converged on similar tiering strategies. The tradeoff space strongly favors progressive optimization over any single compilation strategy.