Editorial: To Benchmark, or Not to Benchmark?

You may have seen some news headlines recently about Google's plan to stop using the Octane JavaScript benchmark suite. If you don't know about this or haven't finished reading the title, let me briefly review it. Google launched Octane to replace the industry-standard SunSpider benchmark. Created by Apple's Safari team, SunSpider is one of the first JavaScript benchmarks.

SunSpider has two problems. First, it is based on microbenchmarks (think thousands of tests to create new arrays), which doesn't reflect real-world usage very accurately. Second, SunSpider's rankings take a lot of weight among browser manufacturers, leading some vendors to optimize their JavaScript engines for better benchmark scores rather than meeting the needs of actual programs. In some cases, these tweaks even cause production code to run slower than before! Octane focuses on trying to create tests that more accurately simulate actual workloads and becomes the standard for measuring JavaScript implementation. However, browser manufacturers have caught up again and we see optimizations for Octane testing. This is not to say that benchmarking is useless. Competition among browsers has indeed led to a significant improvement in JavaScript performance.

You might say, it's a little interesting, but what does this have to do with my daily work as a developer? Benchmarks are often cited when trying to convince people of the benefits of frame Y over frame X, and some people take these numbers very seriously. Last week, I noticed a new UI library called MoonJS is circulating widely on some news aggregators. MoonJS positioned itself as a "minimized, extremely fast" library and cited some benchmark data to support this statement. To be clear, I am not targeting MoonJS here. This focus on speed is very common, especially in UI libraries (for example, look at any React clone). However, as we see in the SunSpider and Octane examples, benchmarking can be misleading. Many modern JavaScript view libraries and frameworks use some form of virtual DOM to render output. In studying different implementations, Boris Kaul spent some time researching methods for measuring virtual DOM performance and found that VDOM performance is relatively easy to tune to achieve good results in the benchmark. His conclusion is: "When you choose a framework or library, don't use the numbers from any web framework benchmarks to make your decision."

When comparing based on the claimed speed of the library, there are other reasons to be cautious. It is important to remember that, like SunSpider, many benchmarks are microbenchmarks; they measure repetitive operations at scales that are unlikely to be achieved in the process of creating an application interface.

It is also worth considering the importance of speed to your specific use case. Building a simple CRUD application is unlikely to overwhelm any UI library, and the learning curve, available talent pool, and developer happiness are also important considerations. I've seen a lot of discussion in the past about whether Ruby is too slow to build web applications, but despite the faster options, there are still many applications written in Ruby and continue to be written in Ruby.

Speed ​​metrics can be misleading, but they may also be of limited use depending on what you build. As with all rules of thumb and good practice, it’s better to stop and think about how (or whether) it applies to your situation. I'd love to know your experience: Have you used software in practice that doesn't meet its benchmark statement? Have you built applications where speed differences are important? Please leave a message to tell me!

JavaScript Benchmark FAQ (FAQ)

What is the purpose of JavaScript benchmarking?

JavaScript benchmarking is a process that measures the performance of a specific code snippet or function. It helps developers understand how efficient their code is and find out where improvements are needed. By comparing the execution time of different code snippets, developers can choose the most efficient solution to meet their needs. Benchmarking is critical in JavaScript development because it directly affects the user experience, especially in terms of speed and responsiveness of web applications.

How does the SunSpider benchmarking tool work?

SunSpider is a popular JavaScript benchmarking tool developed by WebKit. It runs a series of tests on the JavaScript engine and measures the time it takes to complete each test. These tests cover all aspects of JavaScript, including control flow, string processing, and mathematical calculations. The shorter the total time, the better the performance of the JavaScript engine.

What is the difference between SunSpider and other benchmarking tools?

While all benchmarking tools aim to measure JavaScript performance, they vary in the type of tests run and the way results are calculated. SunSpider focuses on real-world use cases and avoids microbenchmarks that test only a single feature. Other tools such as jsben.ch and jsbench.me allow developers to create and run their own tests, providing greater flexibility.

How to explain the results of JavaScript benchmarks?

Benchmark results usually provide time measurements, which indicate how long it takes for a particular operation to complete. The shorter the time, the better the performance. However, interpreting these results requires understanding the context. For example, a few milliseconds of difference may not be important in the user interface, but may be critical in high-performance server applications.

Can I use benchmarks to compare different JavaScript engines?

Yes, benchmarking is a common way to compare the performance of different JavaScript engines. By running the same tests on different engines, you can understand their relative performance. However, remember that real-world performance is affected by many factors, and benchmark results are just a small part of the puzzle.

How to create my own benchmark?

Tools like jsben.ch and jsbench.me allow you to write and run your own JavaScript benchmarks. You can use these tools to test specific code snippets or compare different ways to solve the problem. When creating a benchmark, it is important to make the test as realistic as possible and run it multiple times to get accurate measurements.

What are the limitations of JavaScript benchmarking?

While benchmarking is a powerful tool, it also has its limitations. Creating real-life tests can be difficult, and the results can be affected by many factors, including specific hardware and software environments. Additionally, over-focusing on benchmark results can lead to over-optimization and developers spend too much time improving code that has little impact on overall performance.

What is the role of benchmarking in the development process?

Benchmarking is an important part of the development process as it helps developers identify performance bottlenecks and verify that their changes improve performance. However, it should not be the only tool to evaluate the quality of your code. Other factors such as readability, maintainability and functionality are also important.

Can benchmarks help improve the performance of my web application?

Yes, benchmarks can help you identify areas of code that cause your application to slow down. By optimizing these areas, you can improve the overall performance of your application. However, remember that performance is only one aspect of high-quality web applications. Usability, functionality and design are also important.

How often should I benchmark my code?

The frequency of the benchmark test depends on the nature of the project. For performance-critical applications, you may want to be benchmarked regularly, even after small changes. For less critical applications, benchmarking can be done less frequently, such as after major changes or before a new version is released.

The above is the detailed content of Editorial: To Benchmark, or Not to Benchmark?. For more information, please follow other related articles on the PHP Chinese website!