Inline Caching: Speeding Up Dynamic Dispatch


Dynamic languages and object-oriented systems pay a runtime cost for flexibility. When you call a method on an object, the runtime must figure out which concrete implementation to invoke. This lookup happens millions of times per second in production applications, and even a few extra pointer dereferences add up. Inline caching turns these repeated lookups into fast paths by betting that the same code site will see the same types most of the time.

The dispatch problem

In statically typed compiled languages, method calls are often resolved at compile time or through a single vtable lookup. In dynamic languages like JavaScript, Python, or Ruby, the runtime has less information. A function that receives obj.toString() doesn’t know what obj is until the moment the call happens. The naive approach is to search the object’s properties, walk the prototype chain, or consult a method table on every single call. That works, but it’s slow.

Even in statically typed languages with polymorphism, virtual method dispatch requires dereferencing the object’s vtable pointer and indexing into it. For megamorphic call sites that see dozens of different types, this lookup becomes a bottleneck and defeats CPU branch prediction.

Monomorphic inline caches

An inline cache is a small piece of mutable code or data structure placed directly at a call site. The first time the call executes, the runtime performs the full lookup, then writes the result into the cache. On subsequent calls, the runtime checks whether the incoming object matches the cached type. If it does, the cached method address is used directly. If not, the cache is updated.

This is a speculative optimization. The assumption is that most call sites are monomorphic in practice: they see the same type over and over. Real-world programs tend to exhibit stable type patterns even in dynamic languages. A loop iterating over an array of user objects will almost always call the same render method, even though the language permits any object at that site.

When the assumption holds, the inline cache collapses what would be a multi-step lookup into a single type check and a direct jump. The CPU’s branch predictor learns the pattern, and the instruction cache stays hot because the same path executes repeatedly.

Polymorphic and megamorphic cases

Not all call sites are monomorphic. Some see two or three types regularly. Inline caching systems handle this by storing multiple entries. A polymorphic inline cache (PIC) is a small lookup table embedded at the call site, holding a few type-method pairs. The runtime checks each entry in sequence. This is still faster than a full property lookup, but slower than the monomorphic fast path.

When a call site sees too many types—often more than four or five—the cache is marked megamorphic and the system falls back to a generic dispatch routine. Megamorphic sites are a performance red flag. They defeat inline caching, stress the branch predictor, and often indicate poorly structured code or excessive use of heterogeneous collections.

JIT compilers use inline cache state to guide optimization. A monomorphic site can be inlined aggressively. A polymorphic site might be split into a guarded fast path and a slow fallback. A megamorphic site is left as an out-of-line call.

Invalidation and deoptimization

Inline caches depend on type stability, but programs evolve. When the shape of an object changes—adding a property, changing a prototype, or redefining a method—the inline cache must be invalidated. Some runtimes track dependencies between caches and object shapes, invalidating stale caches in bulk when a prototype is mutated. Others rely on version tags or shape transitions.

If the runtime has already compiled optimized machine code based on inline cache assumptions, that code may need to be discarded. This is deoptimization: the runtime bails out to a generic interpreter or recompiles with updated type information. Frequent deoptimization is costly, so systems try to stabilize hot code before committing to aggressive specialization.

Where it appears

V8 uses inline caches extensively for property access and method calls in JavaScript. The Smalltalk VM pioneered the technique decades ago. PyPy, LuaJIT, and Ruby’s YJIT all rely on inline caching to close the performance gap with statically typed languages. Even statically typed JIT runtimes like the JVM use inline caching for interface calls and virtual dispatch.

Inline caching isn’t magic, but it’s one of the most effective runtime optimizations ever developed. It turns the repeated cost of dynamic dispatch into a predictable branch, letting CPUs and compilers do what they do best: execute the same fast path over and over.