Direct Proxy
Acts as a transparent entity
between a client and a backend service
ESB Performance
Enterprise Service Bus Performance Benchmarking
ESB Performance Benchmarking - Round X - 01
ESB Performance Benchmarking - Round X - 01
1. Overview
This report is based on the Round X - 01 of Enterprise Service Bus performance benchmarking as a continuation to the previous seven rounds of benchmarking conducted since June 2007. Please refer to the write-ups for Round 7 and the previous rounds, for more information and the history of the benchmark.
In this performance benchmarking round, the latest release of AdroitLogic’s new integration solution, UltraESB-X-17.07 is compared against two of the leading ESBs in the industry which are referred as ESB-1 and ESB-2 throughout this report. The benchmarking test was conducted on an Amazon EC2 instance on August 15, 2017.
2. Disclaimer
ESB configurations were built by AdroitLogic, based on the public documentation, other articles, and guides of the vendors. An immense effort was put to configure ESBs to work under identical conditions as much as possible.
3. Test Scenarios
Four message flow scenarios were tested on HTTP transport over a set of concurrent users with different payload sizes. The scenarios tested are as follows.
Content Based Routing (CBR)
Routes a message with XML payload based on
XPath evaluation over the payload body
Header Based Routing (HBR)
Routes a message based on a
transport header value
XSLT Transformation
Performs an XSLT transformation
on a message with XML payload
The test was conducted over a variety of XML payload sizes to evaluate the ESBs' capabilities of handling small to large messages.
1 kB
5 kB
10 kB
40 kB
100 kB
In order to understand the stability of ESBs, different levels of concurrency have been introduced into the test.
20 concurrent users
40 concurrent users
80 concurrent users
160 concurrent users
320 concurrent users
640 concurrent users
For more information of the benchmark, please refer Performance Test Cases.
4. Test Environment
4.1 Amazon EC2 Environment
For this performance benchmarking, an Amazon EC2 compute-optimized c3.xlarge instance with 4 vCPUs, 7.5 GiB in memory and 40GiB of General Purpose SSD was used. The operating system which the test executed on, 64-bit Ubuntu 16.04.1 LTS with the kernel version of 4.4.0-79-generic, was first tuned up for better performance, as described in the Standalone Execution article. All the ESBs were allocated 2GB of Java heap memory, on 64 bit Oracle JDK 1.8.0_121-b13. A configured Apache Tomcat 7.0.56 instance was used as the backend server.
ESBs, Tomcat server and the load generating client program resided in the same machine. The backend server was started first, followed by the ESB under test, and finally the load generating program was started generating a load profile for all the scenarios over different concurrency levels with different payload sizes. A few iterations of warm up rounds were carried out before each test run to make sure that the JVM is warmed up beforehand. After completion of the test run for a particular ESB, the backend server was stopped, along with the ESB. This same procedure was followed again for the other ESBs, completing one iteration of the test. The benchmarking test was carried out for 3 such iterations to eliminate any discrepancy among the results.
Since all the traffic was only among the load generator, ESB, and the backend service, network isolation was not considered. The instance was configured to have a 4GiB RAM disk which was utilized by the UltraESB-X. All the ESBs were attempted to be configured as allowing approximately 500 worker threads, and a socket timeout of 120 seconds.
4.2 Results Worksheet
All the results extracted from this benchmarking test are summarized and publicly shared in a Google Spreadsheet.
5. Results and Observations
For each case, successful responses, failures and transactions per second (TPS) were measured. Average values of corresponding data were calculated to enrich the fairness of results. Unlike the previous test rounds, results for each payload size in each test scenario were considered separately in this round of benchmarking.
5.1 Throughput
Throughput is one of the most significant metrics in a performance intensive environment. Transactions per second (TPS) was measured to monitor the ability of ESBs to perform seamlessly in such an environment. Summarized TPS values for each scenario across all the concurrency levels, for each payload size, averaged over 3 iterations displays a trend as depicted in the figures.
Direct Proxy
Average throughput of 1kB payload
| Concurrent Users | Transactions per second | |||
|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | ||
| 20 | 6,174.15 | 4,920.45 | 5,378.61 | |
| 40 | 6,614.88 | 5,224.53 | 5,811.71 | |
| 80 | 6,803.60 | 5,471.91 | 6,150.13 | |
| 160 | 6,856.81 | 5,478.39 | 6,316.35 | |
| 320 | 6,684.07 | 5,393.68 | 6,410.74 | |
| 640 | 5,865.33 | 4,838.80 | 5,687.25 | |
Average throughput of 5kB payload
| Concurrent Users | Transactions per second | |||
|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | ||
| 20 | 5,624.01 | 4,635.01 | 4,930.78 | |
| 40 | 5,951.06 | 5,004.45 | 5,261.22 | |
| 80 | 6,216.71 | 5,205.72 | 5,508.98 | |
| 160 | 6,280.45 | 5,162.61 | 5,793.93 | |
| 320 | 6,098.71 | 5,035.87 | 5,932.43 | |
| 640 | 5,299.78 | 4,507.47 | 5,425.55 | |
Average throughput of 10kB payload
| Concurrent Users | Transactions per second | |||
|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | ||
| 20 | 4,568.46 | 3,818.66 | 3,553.29 | |
| 40 | 5,082.07 | 4,297.54 | 4,442.94 | |
| 80 | 5,355.87 | 4,415.24 | 4,589.95 | |
| 160 | 5,393.83 | 4,398.78 | 4,657.90 | |
| 320 | 5,409.04 | 4,328.21 | 4,707.31 | |
| 640 | 4,919.94 | 4,004.87 | 4,290.22 | |
Average throughput of 40kB payload
| Concurrent Users | Transactions per second | |||
|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | ||
| 20 | 3,320.99 | 2,662.92 | 2,261.00 | |
| 40 | 3,529.79 | 2,856.85 | 2,322.66 | |
| 80 | 3,706.11 | 2,879.42 | 2,235.65 | |
| 160 | 3,720.15 | 2,824.67 | 2,107.35 | |
| 320 | 3,688.13 | 2,811.22 | 1,969.69 | |
| 640 | 3,443.32 | 2,652.03 | 1,870.87 | |
Average throughput of 100kB payload
| Concurrent Users | Transactions per second | |||
|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | ||
| 20 | 2,116.56 | 1,643.15 | 1,219.77 | |
| 40 | 2,266.72 | 1,735.18 | 1,211.20 | |
| 80 | 2,317.27 | 1,736.76 | 1,109.19 | |
| 160 | 2,305.84 | 1,705.46 | 978.68 | |
| 320 | 2,275.59 | 1,660.96 | 972.74 | |
| 640 | 2,135.39 | 1,554.06 | 957.52 | |
Observations Summary
- UltraESB-X clearly outperforms the other ESBs across all the payload sizes under test.
- With the increase of payload size, UltraESB-X significantly increases the lead over the next best contender.
Content Based Routing
Average throughput of 1kB payload
| Concurrent Users | Transactions per second | |||
|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | ||
| 20 | 4,669.94 | 2,022.77 | 3,880.27 | |
| 40 | 5,034.94 | 2,069.91 | 4,345.64 | |
| 80 | 5,352.62 | 2,168.32 | 4,740.56 | |
| 160 | 5,535.95 | 2,238.38 | 5,080.64 | |
| 320 | 5,621.32 | 2,280.44 | 5,259.36 | |
| 640 | 5,240.49 | 2,192.39 | 4,904.30 | |
Average throughput of 5kB payload
| Concurrent Users | Transactions per second | |||
|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | ||
| 20 | 3,957.21 | 1,424.65 | 2,894.79 | |
| 40 | 4,323.83 | 1,440.52 | 3,263.33 | |
| 80 | 4,589.52 | 1,463.97 | 3,547.26 | |
| 160 | 4,736.92 | 1,486.92 | 3,725.29 | |
| 320 | 4,809.92 | 1,525.06 | 3,816.73 | |
| 640 | 4,390.81 | 1,487.13 | 3,574.32 | |
Average throughput of 10kB payload
| Concurrent Users | Transactions per second | |||
|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | ||
| 20 | 3,184.66 | 994.64 | 2,166.38 | |
| 40 | 3,521.28 | 998.32 | 2,413.75 | |
| 80 | 3,733.69 | 1,001.54 | 2,557.94 | |
| 160 | 3,876.91 | 1,023.53 | 2,592.70 | |
| 320 | 3,944.78 | 1,046.68 | 2,614.28 | |
| 640 | 3,688.02 | 1,032.86 | 2,506.25 | |
Average throughput of 40kB payload
| Concurrent Users | Transactions per second | |||
|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | ||
| 20 | 1,839.76 | 342.04 | 429.63 | |
| 40 | 2,006.37 | 328.37 | 476.82 | |
| 80 | 2,059.44 | 326.87 | 505.15 | |
| 160 | 2,083.34 | 323.82 | 514.47 | |
| 320 | 2,090.80 | 335.41 | 485.51 | |
| 640 | 1,996.40 | 332.95 | 381.57 | |
Average throughput of 100kB payload
| Concurrent Users | Transactions per second | |||
|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | ||
| 20 | 1,033.10 | 133.28 | 114.03 | |
| 40 | 1,063.85 | 127.42 | 113.68 | |
| 80 | 1,073.89 | 114.92 | 110.13 | |
| 160 | 1,070.85 | 111.14 | 109.16 | |
| 320 | 1,064.77 | 114.60 | 106.94 | |
| 640 | 1,043.01 | 114.85 | 89.75 | |
Observations Summary
- UltraESB-X clearly outperforms the other ESBs across all the payload sizes under test.
- With the increase of payload size, UltraESB-X significantly increases the lead over the next best contender.
Header Based Routing
Average throughput of 1kB payload
| Concurrent Users | Transactions per second | |||
|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | ||
| 20 | 6,071.20 | 2,955.87 | 5,358.50 | |
| 40 | 6,473.24 | 3,061.55 | 5,703.36 | |
| 80 | 6,773.37 | 3,221.11 | 5,985.75 | |
| 160 | 6,903.60 | 3,309.53 | 6,300.14 | |
| 320 | 6,843.33 | 3,384.17 | 6,458.16 | |
| 640 | 5,918.46 | 3,149.17 | 6,174.63 | |
Average throughput of 5kB payload
| Concurrent Users | Transactions per second | |||
|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | ||
| 20 | 5,618.94 | 2,360.48 | 5,024.91 | |
| 40 | 5,935.84 | 2,408.13 | 5,355.72 | |
| 80 | 6,303.66 | 2,505.70 | 5,588.37 | |
| 160 | 6,236.83 | 2,581.57 | 5,852.64 | |
| 320 | 6,183.98 | 2,614.43 | 6,019.53 | |
| 640 | 5,313.38 | 2,450.34 | 5,482.85 | |
Average throughput of 10kB payload
| Concurrent Users | Transactions per second | |||
|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | ||
| 20 | 4,764.00 | 1,801.21 | 4,162.28 | |
| 40 | 5,060.82 | 1,827.94 | 4,395.90 | |
| 80 | 5,337.81 | 1,861.78 | 4,531.93 | |
| 160 | 5,397.61 | 1,912.52 | 4,625.86 | |
| 320 | 5,335.58 | 1,955.07 | 4,673.19 | |
| 640 | 4,730.34 | 1,901.76 | 4,301.63 | |
Average throughput of 40kB payload
| Concurrent Users | Transactions per second | |||
|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | ||
| 20 | 3,315.82 | 792.38 | 2,272.88 | |
| 40 | 3,494.66 | 794.67 | 2,340.29 | |
| 80 | 3,692.63 | 798.64 | 2,205.39 | |
| 160 | 3,636.73 | 813.80 | 2,097.56 | |
| 320 | 3,652.78 | 829.20 | 2,015.23 | |
| 640 | 3,311.15 | 819.16 | 1,802.42 | |
Average throughput of 100kB payload
| Concurrent Users | Transactions per second | |||
|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | ||
| 20 | 2,046.79 | 363.86 | 1,231.78 | |
| 40 | 2,242.12 | 365.63 | 1,195.01 | |
| 80 | 2,282.80 | 361.29 | 1,085.82 | |
| 160 | 2,271.96 | 368.79 | 964.87 | |
| 320 | 2,236.87 | 374.74 | 979.95 | |
| 640 | 2,099.52 | 372.85 | 982.59 | |
Observations Summary
- UltraESB-X clearly outperforms the other ESBs across all the payload sizes under test.
- With the increase of payload size, UltraESB-X significantly increases the lead over the next best contender.
XSLT Transformation
Average throughput of 1kB payload
| Concurrent Users | Transactions per second | |||
|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | ||
| 20 | 2,922.51 | 2,973.04 | 3,062.80 | |
| 40 | 3,282.87 | 3,272.39 | 3,477.23 | |
| 80 | 3,565.60 | 3,348.90 | 3,841.69 | |
| 160 | 3,673.88 | 3,497.14 | 4,074.99 | |
| 320 | 3,729.35 | 3,596.50 | 4,170.35 | |
| 640 | 3,525.61 | 3,411.91 | 3,834.14 | |
Average throughput of 5kB payload
| Concurrent Users | Transactions per second | |||
|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | ||
| 20 | 1,960.42 | 2,071.99 | 2,186.50 | |
| 40 | 2,119.19 | 2,132.25 | 2,413.48 | |
| 80 | 2,201.79 | 2,157.79 | 2,558.99 | |
| 160 | 2,235.91 | 2,220.67 | 2,636.77 | |
| 320 | 2,259.35 | 2,304.36 | 2,691.65 | |
| 640 | 2,179.07 | 2,240.81 | 2,571.29 | |
Average throughput of 10kB payload
| Concurrent Users | Transactions per second | |||
|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | ||
| 20 | 1,363.29 | 1,435.38 | 1,566.53 | |
| 40 | 1,418.21 | 1,429.39 | 1,659.44 | |
| 80 | 1,447.27 | 1,421.31 | 1,713.95 | |
| 160 | 1,459.95 | 1,480.16 | 1,726.61 | |
| 320 | 1,475.85 | 1,550.60 | 1,730.08 | |
| 640 | 1,432.36 | 1,496.77 | 1,661.26 | |
Average throughput of 40kB payload
| Concurrent Users | Transactions per second | |||
|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | ||
| 20 | 474.63 | 487.30 | 298.32 | |
| 40 | 456.02 | 358.48 | 298.55 | |
| 80 | 441.68 | 345.27 | 311.74 | |
| 160 | 437.73 | 359.93 | 315.79 | |
| 320 | 438.40 | 379.08 | 320.06 | |
| 640 | 427.78 | 377.44 | 314.03 | |
Average throughput of 100kB payload
| Concurrent Users | Transactions per second | |||
|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | ||
| 20 | 176.00 | 152.54 | 113.40 | |
| 40 | 176.18 | 147.95 | 120.39 | |
| 80 | 176.28 | 146.79 | 125.98 | |
| 160 | 176.28 | 150.24 | 128.63 | |
| 320 | 175.31 | 154.87 | 129.67 | |
| 640 | 174.29 | 156.94 | 128.77 | |
Observations Summary
- UltraESB-X lags the performance to the ESB-2 for small payloads.
- However, with the increase of payload size, UltraESB-X starts performing better, and outperforms the other contenders for payloads higher than 10 kB.
5.2 Execution Observations
All the ESBs completed the full benchmark by handling 1,920,000 requests each, for a particular test scenario, over different concurrent user levels, with different payload sizes in each iteration. Total failures, errors or timeouts occurred for each ESB for different test scenarios were as follows.
Failures, errors or timeouts
for each scenario
| Test Scenarios | Number of requests | Total Errors | |||
|---|---|---|---|---|---|
| UltraESB-X | ESB-1 | ESB-2 | |||
| Direct Proxy | 5,760,000 | 0 | 0 | 2 | |
| Content Based Routing | 5,760,000 | 0 | 0 | 0 | |
| Header Based Routing | 5,760,000 | 0 | 0 | 1 | |
| XSLT Transformation | 5,760,000 | 0 | 0 | 0 | |
6. Conclusion
All the ESBs completed the full test round over three iterations. While UltraESB-X along with ESB-1 displayed high stability with no failures or errors, ESB-2 resulted in three errors in total. UltraESB-X outperformed the other ESBs across all the payloads and test scenarios except for XSLT transformation. However, even with the XSLT transformation scenario, when the payload gets larger than 10 kB, UltraESB-X starts dominating the performance.
Refer to the Google Spreadsheet, ESB Performance Benchmarking - Round X - 01 for complete results.
7. Authors
Chamath N. Kirinde, Senior Performance & Automation Engineer, AdroitLogic
For more information please contact us.
Email: info@adroitlogic.com
8. Notices
UltraESB, UltraESB-X and AdroitLogic are trademarks of AdroitLogic Private Ltd. All other product and company names and marks mentioned are the property of their respective owners and are mentioned for identification purposes only.
Publish date: 17th August, 2017
Note: Content of the article published on July 5th, 2017 has been updated by the latest benchmarking results.