Java Virtual Machine (JVM) Performance Benchmarks with a primary focus on top-tier Just-In-Time (JIT) Compilers, such as C2 JIT, Graal JIT, and the Falcon JIT.
This repository contains various Java Virtual Machine (JVM) benchmarks with a primary focus on top-tier Just-In-Time (JIT) Compilers, such as C2 JIT, Graal JIT, and the Falcon JIT.
The benchmarks are structured in three distinct categories:
java.io
, java.nio
, java.net
, java.security
, java.util
, java.text
, java.time
, etc.) and the Java Development Kit (JDK) (e.g., jdk.incubator.vector
, etc.).All benchmarks are implemented using the Java Microbenchmark Harness (JMH) library.
Ionut Balosin
Florin Blanaru
The main objectives of this project are:
Each benchmark focuses on a specific execution pattern or task that could be fully optimized under ideal conditions (i.e., clean profiles). While some of these patterns might rarely appear directly in user programs, they can emerge after several optimizations, such as inlining high-level operations. Real-life applications can have varying conditions, making benchmarks not always a reliable predictor on a larger scale. Nonetheless, even though artificial benchmarks may not capture the complete truth, they can still offer valuable insights when properly implemented.
Out of Scope:
Current publications that rely on benchmarks from this repository can be found at:
JMH uses HotSpot-specific compiler hints to control the Just-in-Time (JIT) compiler.
For this reason, the fully supported JVMs are all HotSpot-based VMs, including vanilla OpenJDK and Oracle JDK builds. GraalVM is also supported. For more details, please refer to the compiler hints and supported VMs.
Using JMH Blackhole.consume() may dominate the costs, obscuring the results, in comparison to normal Java-style source code.
Starting with OpenJDK 17, the compiler supports blackholes (JDK-8259316). This optimization is available in HotSpot and the Graal compiler.
To ensure a fair comparison between OpenJDK 11 and OpenJDK 17, compiler blackholes should be manually disabled in the benchmarks.
The cost of Blackhole.consume()
is zero (the compiler will not emit any instructions for the call) when compiler blackholes are enabled and supported by the top-tier JIT compiler of the underlying JVM.
In the case where a benchmark-annotated method returns a value instead of consuming it via Blackhole.consume()
(e.g., in the case of non-void benchmark methods), JMH will wrap the return value of the benchmark method in a blackhole to prevent dead code elimination.
In this scenario, blackholes are generated by the test infrastructure even though there is no explicit use of them in the benchmarks.
Explicitly using
Blackhole.consume()
(in hot loops) can result in misleading benchmark results, especially when compiler blackholes are disabled.
When conducting benchmarking, it is advisable to disable potential sources of performance non-determinism. Below are the tuning configurations provided by the benchmark for specific operating systems.
The Linux tuning script configure-os-linux.sh performs the following actions:
isolcpus
or cgroups
)Note: These configurations are tested on Ubuntu 22.04 LTS (a Debian-based Linux distribution).
For further references, please check:
For macOS, the Linux tuning settings described above do not apply. For instance, Apple M1/M2 (ARM-based) chips do not have hyper-threading or turbo-boost mode, and disabling ASLR is more complex.
Due to these differences, the script configure-os-mac.sh does not enable any specific macOS tuning configurations.
Windows is not the primary focus of this benchmark, so the script configure-os-win.sh does not enable any specific Windows tuning configurations.
The table below summarizes the JVM distributions included in the benchmark. For transparency, we provide a brief explanation of why others are not supported.
JVM Distribution | Included | Build |
---|---|---|
OpenJDK HotSpot VM | Yes | Download |
GraalVM CE | Yes | Download |
Oracle GraalVM (1) |
Yes | Download |
Azul Prime VM | Yes (2) |
Download |
Eclipse OpenJ9 VM | No (3) |
NA |
Notes:
(1)
Oracle GraalVM was formerly known as GraalVM EE(2)
License restrictions might apply(3)
Please see the reasons belowIn the case of Azul Prime VM, please ensure that you have read and understood the license before publishing any benchmark results.
JMH may functionally work with the Eclipse OpenJ9 VM. However, none of the compiler hints will apply to Eclipse OpenJ9, potentially leading to different results (i.e., unfair advantage or disadvantage, depending on the test).
For more details, please refer to JMH with OpenJ9 and Mark Stoodley on Twitter.
Currently, Eclipse OpenJ9 is out of scope until a suitable alternative is identified.
At present, the benchmark is configured to work only with the latest JDK Long-Term Support (LTS) versions.
Version |
---|
11 - LTS |
17 - LTS |
21 - LTS |
If you need another LTS (or non-LTS) version, you will need to configure it manually.
After installing the JDK, you must update the JDK path in the configuration properties. Follow these steps:
Open the config.properties file.
Update the specific VM_HOME property for the JDK you intend to use. You don't need to update all of them, only the one you plan to use for compiling and running the benchmarks.
OPENJDK_HOTSPOT_VM_HOME="<path_to_jdk>"
GRAAL_VM_CE_HOME="<path_to_jdk>"
GRAAL_VM_EE_HOME="<path_to_jdk>"
AZUL_PRIME_VM_HOME="<path_to_jdk>"
Linux OS:
OPENJDK_HOTSPOT_VM_HOME="/usr/lib/jvm/openjdk-17.0.5"
Mac OS:
OPENJDK_HOTSPOT_VM_HOME="/Library/Java/JavaVirtualMachines/openjdk-17.0.5/Contents/Home"
Windows OS:
OPENJDK_HOTSPOT_VM_HOME="/c/Program_Dev/Java/openjdk-17.0.5"
The table below summarizes the top-tier JIT compilers targeted by these benchmarks.
JVM Distribution | Top-tier JIT Compiler |
---|---|
OpenJDK HotSpot VM | C2 JIT |
GraalVM CE | Graal JIT |
Oracle GraalVM | Graal JIT |
Azul Prime VM | Falcon JIT |
The benchmarks are organized into suites (i.e., benchmark suites). To run a benchmark suite on a specific JDK version, it requires a highly specific configuration. There are predefined benchmark suites provided in JSON configuration files for each supported JDK LTS version:
The benchmark suite will sequentially execute all the tests defined in the configuration file.
There are several reasons why such a custom configuration is necessary:
On the main branch, we adhere to a forward-only approach for newly created benchmarks that function with the latest LTS release. We do not intend to include a newly created benchmark in an older suite that has already been executed and published at the time of writing the benchmark. We are following a forward-only approach for newly created benchmarks.
Examples:
To avoid these inconsistencies on the main branch, we recommend switching to the appropriate JDK release branch when running on a previous JDK and initiating the suite from there.
We provide a baseline benchmark for the infrastructure, InfrastructureBaselineBenchmark, which can be used to assess the infrastructure overhead for the code being measured.
This benchmark evaluates the performance of empty methods both with and without explicit inlining. Additionally, it assesses the performance difference between returning an object and consuming it via blackholes. All of these mechanisms are utilized within the real suite of tests.
This benchmark is particularly valuable when comparing different JVMs and JDKs. It is recommended to run it before any other real benchmark to establish baseline performance metrics. If the results of the infrastructure baseline benchmark differ, it may not be meaningful to compare the results of other benchmarks across different JVMs and JDKs.
Running these commands will build the benchmark suite only, without executing any benchmarks.
For building and running the benchmark suite (e.g., via the ./run-benchmarks.sh
command), please refer to the next section.
./mvnw -P jdk$<jdk-version>_profile clean package
Replace <jdk-version>
with either 11, 17, or 21. If you omit specifying the profile, JDK profile 21 will be selected by default.
Examples:
./mvnw clean package
./mvnw -P jdk11_profile clean package
./mvnw -P jdk17_profile clean package
./mvnw -P jdk21_profile clean package
Running a benchmark suite triggers the complete setup process in a highly interactive manner, allowing the user to choose which steps to skip. The process includes the following:
The dry run mode simulates all the commands without altering any OS settings or executing benchmarks. We recommend using this as a preliminary check before running the benchmarks.
./run-benchmarks.sh --dry-run
Note: You should execute this command with sudo
to simulate the OS configuration settings. This is necessary, even in dry run mode, to access certain system configuration files that would otherwise be inaccessible. However, please note that it will not have any impact on the actual OS settings.
./run-benchmarks.sh | tee run-benchmarks.out
Note: Launch this command with sudo
to apply the OS configuration settings.
The benchmark results are saved under the results/jdk-$JDK_VERSION/$ARCH/jmh/$JVM_IDENTIFIER
directory.
To properly execute bash scripts on Windows there are a few alternatives:
The benchmark plot generation is based on R/ggplot2 that needs to be installed upfront.
To generate all benchmark plots corresponding to one <jdk-version>
and (optionally,) a specific <arch>
, run the below command:
./plot-benchmarks.sh <jdk-version> [<arch>]
If the <arch>
parameter is omitted, it is automatically detected based on the current system architecture.
Before generating the benchmarks, the script triggers a few additional steps:
results/jdk-$JDK_VERSION/$ARCH/geomean
directory.The benchmark plots are saved under the results/jdk-$JDK_VERSION/$ARCH/plot
directory.
If you are interested in contributing code or providing any form of support, including sponsorship, you are encouraged to do so by contacting us directly. You can contribute by sending a pull request, raising an issue with an attached patch, or by directly contacting us.
Please see the LICENSE file for full license.
JVM Performance Benchmarks
Copyright (C) 2019 - 2024 Ionut Balosin
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.