The genesis of Octane #
Diminishing returns and over-optimization #
instanceof operator was another example of an optimization tailored to a narrow set of Octane-specific cases that led to significant regressions in Node.js applications.
Another problem is that over time, small bugs in Octane become a target for optimizations themselves. For example, in the Box2DWeb benchmark, taking advantage of a bug where two objects were compared using the
>= operators gave a ~15% performance boost on Octane. Unfortunately, this optimization had no effect in the real world and complicates more general types of comparison optimizations. Octane sometimes even negatively penalizes real-world optimizations: engineers working on other VMs have noticed that Octane seems to penalize lazy parsing, a technique that helps most real websites load faster given the amount of dead code frequently found in the wild.
Beyond Octane and other synthetic benchmarks #
Given the plateau in scores across most JS VMs and the increasing conflict between optimizing for specific Octane benchmarks rather than implementing speedups for a broader range of real-world code, we believe that it is time to retire Octane as a recommended benchmark.
Check back here for more posts about improvements in our measurement methodology and new workloads that better represent real-world performance. We are excited to continue pursuing the performance that matters most to users and developers!