ESB Performance Benchmarking - Round X - 01

1. Overview

This report is based on the Round X - 01 of Enterprise Service Bus performance benchmarking as a continuation to the previous seven rounds of benchmarking conducted since June 2007. Please refer to the write-ups for Round 7 and the previous rounds, for more information and the history of the benchmark.

In this performance benchmarking round, the latest release of AdroitLogic’s new integration solution, UltraESB-X-17.07 is compared against two of the leading ESBs in the industry which are referred as ESB-1 and ESB-2 throughout this report. The benchmarking test was conducted on an Amazon EC2 instance on August 15, 2017.

2. Disclaimer

ESB configurations were built by AdroitLogic, based on the public documentation, other articles, and guides of the vendors. An immense effort was put to configure ESBs to work under identical conditions as much as possible.

3. Test Scenarios

Four message flow scenarios were tested on HTTP transport over a set of concurrent users with different payload sizes. The scenarios tested are as follows.

Direct Proxy
Acts as a transparent entity between a client and a backend service
Content Based Routing (CBR)
Routes a message with XML payload based on XPath evaluation over the payload body
Header Based Routing (HBR)
Routes a message based on a transport header value
XSLT Transformation
Performs an XSLT transformation on a message with XML payload

The test was conducted over a variety of XML payload sizes to evaluate the ESBs' capabilities of handling small to large messages.

1 kB
5 kB
10 kB
40 kB
100 kB

In order to understand the stability of ESBs, different levels of concurrency have been introduced into the test.

20 concurrent users
40 concurrent users
80 concurrent users
160 concurrent users
320 concurrent users
640 concurrent users

For more information of the benchmark, please refer Performance Test Cases.

4. Test Environment

4.1 Amazon EC2 Environment

For this performance benchmarking, an Amazon EC2 compute-optimized c3.xlarge instance with 4 vCPUs, 7.5 GiB in memory and 40GiB of General Purpose SSD was used. The operating system which the test executed on, 64-bit Ubuntu 16.04.1 LTS with the kernel version of 4.4.0-79-generic, was first tuned up for better performance, as described in the Standalone Execution article. All the ESBs were allocated 2GB of Java heap memory, on 64 bit Oracle JDK 1.8.0_121-b13. A configured Apache Tomcat 7.0.56 instance was used as the backend server.

ESBs, Tomcat server and the load generating client program resided in the same machine. The backend server was started first, followed by the ESB under test, and finally the load generating program was started generating a load profile for all the scenarios over different concurrency levels with different payload sizes. A few iterations of warm up rounds were carried out before each test run to make sure that the JVM is warmed up beforehand. After completion of the test run for a particular ESB, the backend server was stopped, along with the ESB. This same procedure was followed again for the other ESBs, completing one iteration of the test. The benchmarking test was carried out for 3 such iterations to eliminate any discrepancy among the results.

Since all the traffic was only among the load generator, ESB, and the backend service, network isolation was not considered. The instance was configured to have a 4GiB RAM disk which was utilized by the UltraESB-X. All the ESBs were attempted to be configured as allowing approximately 500 worker threads, and a socket timeout of 120 seconds.

4.2 Results Worksheet

All the results extracted from this benchmarking test are summarized and publicly shared in a Google Spreadsheet.

5. Results and Observations

For each case, successful responses, failures and transactions per second (TPS) were measured. Average values of corresponding data were calculated to enrich the fairness of results. Unlike the previous test rounds, results for each payload size in each test scenario were considered separately in this round of benchmarking.

5.1 Throughput

Throughput is one of the most significant metrics in a performance intensive environment. Transactions per second (TPS) was measured to monitor the ability of ESBs to perform seamlessly in such an environment. Summarized TPS values for each scenario across all the concurrency levels, for each payload size, averaged over 3 iterations displays a trend as depicted in the figures.

Direct Proxy

Average throughput of 1kB payload

Concurrent Users Transactions per second
UltraESB-X ESB-1 ESB-2
20 6,174.15 4,920.45 5,378.61
40 6,614.88 5,224.53 5,811.71
80 6,803.60 5,471.91 6,150.13
160 6,856.81 5,478.39 6,316.35
320 6,684.07 5,393.68 6,410.74
640 5,865.33 4,838.80 5,687.25
Figure: Average transactions per second of direct message flow over various concurrent users of 1kB payload size

Average throughput of 5kB payload

Concurrent Users Transactions per second
UltraESB-X ESB-1 ESB-2
20 5,624.01 4,635.01 4,930.78
40 5,951.06 5,004.45 5,261.22
80 6,216.71 5,205.72 5,508.98
160 6,280.45 5,162.61 5,793.93
320 6,098.71 5,035.87 5,932.43
640 5,299.78 4,507.47 5,425.55
Figure: Average transactions per second of direct message flow over various concurrent users of 5kB payload size

Average throughput of 10kB payload

Concurrent Users Transactions per second
UltraESB-X ESB-1 ESB-2
20 4,568.46 3,818.66 3,553.29
40 5,082.07 4,297.54 4,442.94
80 5,355.87 4,415.24 4,589.95
160 5,393.83 4,398.78 4,657.90
320 5,409.04 4,328.21 4,707.31
640 4,919.94 4,004.87 4,290.22
Figure: Average transactions per second of direct message flow over various concurrent users of 10kB payload size

Average throughput of 40kB payload

Concurrent Users Transactions per second
UltraESB-X ESB-1 ESB-2
20 3,320.99 2,662.92 2,261.00
40 3,529.79 2,856.85 2,322.66
80 3,706.11 2,879.42 2,235.65
160 3,720.15 2,824.67 2,107.35
320 3,688.13 2,811.22 1,969.69
640 3,443.32 2,652.03 1,870.87
Figure: Average transactions per second of direct message flow over various concurrent users of 40kB payload size

Average throughput of 100kB payload

Concurrent Users Transactions per second
UltraESB-X ESB-1 ESB-2
20 2,116.56 1,643.15 1,219.77
40 2,266.72 1,735.18 1,211.20
80 2,317.27 1,736.76 1,109.19
160 2,305.84 1,705.46 978.68
320 2,275.59 1,660.96 972.74
640 2,135.39 1,554.06 957.52
Figure: Average transactions per second of direct message flow over various concurrent users of 100kB payload size

Observations Summary

  • UltraESB-X clearly outperforms the other ESBs across all the payload sizes under test.
  • With the increase of payload size, UltraESB-X significantly increases the lead over the next best contender.
Content Based Routing

Average throughput of 1kB payload

Concurrent Users Transactions per second
UltraESB-X ESB-1 ESB-2
20 4,669.94 2,022.77 3,880.27
40 5,034.94 2,069.91 4,345.64
80 5,352.62 2,168.32 4,740.56
160 5,535.95 2,238.38 5,080.64
320 5,621.32 2,280.44 5,259.36
640 5,240.49 2,192.39 4,904.30
Figure: Average transactions per second of content payload based routing flow over various concurrent users of 1kB payload size

Average throughput of 5kB payload

Concurrent Users Transactions per second
UltraESB-X ESB-1 ESB-2
20 3,957.21 1,424.65 2,894.79
40 4,323.83 1,440.52 3,263.33
80 4,589.52 1,463.97 3,547.26
160 4,736.92 1,486.92 3,725.29
320 4,809.92 1,525.06 3,816.73
640 4,390.81 1,487.13 3,574.32
Figure: Average transactions per second of content payload based routing flow over various concurrent users of 5kB payload size

Average throughput of 10kB payload

Concurrent Users Transactions per second
UltraESB-X ESB-1 ESB-2
20 3,184.66 994.64 2,166.38
40 3,521.28 998.32 2,413.75
80 3,733.69 1,001.54 2,557.94
160 3,876.91 1,023.53 2,592.70
320 3,944.78 1,046.68 2,614.28
640 3,688.02 1,032.86 2,506.25
Figure: Average transactions per second of content payload based routing flow over various concurrent users of 10kB payload size

Average throughput of 40kB payload

Concurrent Users Transactions per second
UltraESB-X ESB-1 ESB-2
20 1,839.76 342.04 429.63
40 2,006.37 328.37 476.82
80 2,059.44 326.87 505.15
160 2,083.34 323.82 514.47
320 2,090.80 335.41 485.51
640 1,996.40 332.95 381.57
Figure: Average transactions per second of content payload based routing flow over various concurrent users of 40kB payload size

Average throughput of 100kB payload

Concurrent Users Transactions per second
UltraESB-X ESB-1 ESB-2
20 1,033.10 133.28 114.03
40 1,063.85 127.42 113.68
80 1,073.89 114.92 110.13
160 1,070.85 111.14 109.16
320 1,064.77 114.60 106.94
640 1,043.01 114.85 89.75
Figure: Average transactions per second of content payload based routing flow over various concurrent users of 100kB payload size

Observations Summary

  • UltraESB-X clearly outperforms the other ESBs across all the payload sizes under test.
  • With the increase of payload size, UltraESB-X significantly increases the lead over the next best contender.
Header Based Routing

Average throughput of 1kB payload

Concurrent Users Transactions per second
UltraESB-X ESB-1 ESB-2
20 6,071.20 2,955.87 5,358.50
40 6,473.24 3,061.55 5,703.36
80 6,773.37 3,221.11 5,985.75
160 6,903.60 3,309.53 6,300.14
320 6,843.33 3,384.17 6,458.16
640 5,918.46 3,149.17 6,174.63
Figure: Average transactions per second of transport header based routing flow over various concurrent users of 1kB payload size

Average throughput of 5kB payload

Concurrent Users Transactions per second
UltraESB-X ESB-1 ESB-2
20 5,618.94 2,360.48 5,024.91
40 5,935.84 2,408.13 5,355.72
80 6,303.66 2,505.70 5,588.37
160 6,236.83 2,581.57 5,852.64
320 6,183.98 2,614.43 6,019.53
640 5,313.38 2,450.34 5,482.85
Figure: Average transactions per second of transport header based routing flow over various concurrent users of 5kB payload size

Average throughput of 10kB payload

Concurrent Users Transactions per second
UltraESB-X ESB-1 ESB-2
20 4,764.00 1,801.21 4,162.28
40 5,060.82 1,827.94 4,395.90
80 5,337.81 1,861.78 4,531.93
160 5,397.61 1,912.52 4,625.86
320 5,335.58 1,955.07 4,673.19
640 4,730.34 1,901.76 4,301.63
Figure: Average transactions per second of transport header based routing flow over various concurrent users of 10kB payload size

Average throughput of 40kB payload

Concurrent Users Transactions per second
UltraESB-X ESB-1 ESB-2
20 3,315.82 792.38 2,272.88
40 3,494.66 794.67 2,340.29
80 3,692.63 798.64 2,205.39
160 3,636.73 813.80 2,097.56
320 3,652.78 829.20 2,015.23
640 3,311.15 819.16 1,802.42
Figure: Average transactions per second of transport header based routing flow over various concurrent users of 40kB payload size

Average throughput of 100kB payload

Concurrent Users Transactions per second
UltraESB-X ESB-1 ESB-2
20 2,046.79 363.86 1,231.78
40 2,242.12 365.63 1,195.01
80 2,282.80 361.29 1,085.82
160 2,271.96 368.79 964.87
320 2,236.87 374.74 979.95
640 2,099.52 372.85 982.59
Figure: Average transactions per second of transport header based routing flow over various concurrent users of 100kB payload size

Observations Summary

  • UltraESB-X clearly outperforms the other ESBs across all the payload sizes under test.
  • With the increase of payload size, UltraESB-X significantly increases the lead over the next best contender.
XSLT Transformation

Average throughput of 1kB payload

Concurrent Users Transactions per second
UltraESB-X ESB-1 ESB-2
20 2,922.51 2,973.04 3,062.80
40 3,282.87 3,272.39 3,477.23
80 3,565.60 3,348.90 3,841.69
160 3,673.88 3,497.14 4,074.99
320 3,729.35 3,596.50 4,170.35
640 3,525.61 3,411.91 3,834.14
Figure: Average transactions per second of XSL transformation flow over various concurrent users of 1kB payload size

Average throughput of 5kB payload

Concurrent Users Transactions per second
UltraESB-X ESB-1 ESB-2
20 1,960.42 2,071.99 2,186.50
40 2,119.19 2,132.25 2,413.48
80 2,201.79 2,157.79 2,558.99
160 2,235.91 2,220.67 2,636.77
320 2,259.35 2,304.36 2,691.65
640 2,179.07 2,240.81 2,571.29
Figure: Average transactions per second of XSL transformation flow over various concurrent users of 5kB payload size

Average throughput of 10kB payload

Concurrent Users Transactions per second
UltraESB-X ESB-1 ESB-2
20 1,363.29 1,435.38 1,566.53
40 1,418.21 1,429.39 1,659.44
80 1,447.27 1,421.31 1,713.95
160 1,459.95 1,480.16 1,726.61
320 1,475.85 1,550.60 1,730.08
640 1,432.36 1,496.77 1,661.26
Figure: Average transactions per second of XSL transformation flow over various concurrent users of 10kB payload size

Average throughput of 40kB payload

Concurrent Users Transactions per second
UltraESB-X ESB-1 ESB-2
20 474.63 487.30 298.32
40 456.02 358.48 298.55
80 441.68 345.27 311.74
160 437.73 359.93 315.79
320 438.40 379.08 320.06
640 427.78 377.44 314.03
Figure: Average transactions per second of XSL transformation flow over various concurrent users of 40kB payload size

Average throughput of 100kB payload

Concurrent Users Transactions per second
UltraESB-X ESB-1 ESB-2
20 176.00 152.54 113.40
40 176.18 147.95 120.39
80 176.28 146.79 125.98
160 176.28 150.24 128.63
320 175.31 154.87 129.67
640 174.29 156.94 128.77
Figure: Average transactions per second of XSL transformation flow over various concurrent users of 100kB payload size

Observations Summary

  • UltraESB-X lags the performance to the ESB-2 for small payloads.
  • However, with the increase of payload size, UltraESB-X starts performing better, and outperforms the other contenders for payloads higher than 10 kB.

5.2 Execution Observations

All the ESBs completed the full benchmark by handling 1,920,000 requests each, for a particular test scenario, over different concurrent user levels, with different payload sizes in each iteration. Total failures, errors or timeouts occurred for each ESB for different test scenarios were as follows.

Failures, errors or timeouts for each scenario
Test Scenarios Number of requests Total Errors
UltraESB-X ESB-1 ESB-2
Direct Proxy 5,760,000 0 0 2
Content Based Routing 5,760,000 0 0 0
Header Based Routing 5,760,000 0 0 1
XSLT Transformation 5,760,000 0 0 0

6. Conclusion

All the ESBs completed the full test round over three iterations. While UltraESB-X along with ESB-1 displayed high stability with no failures or errors, ESB-2 resulted in three errors in total. UltraESB-X outperformed the other ESBs across all the payloads and test scenarios except for XSLT transformation. However, even with the XSLT transformation scenario, when the payload gets larger than 10 kB, UltraESB-X starts dominating the performance.

Refer to the Google Spreadsheet, ESB Performance Benchmarking - Round X - 01 for complete results.

7. Authors

Chamath N. Kirinde, Senior Performance & Automation Engineer, AdroitLogic

For more information please contact us.
Email: info@adroitlogic.com

8. Notices

UltraESB, UltraESB-X and AdroitLogic are trademarks of AdroitLogic Private Ltd. All other product and company names and marks mentioned are the property of their respective owners and are mentioned for identification purposes only.

Publish date: 17th August, 2017

Note: Content of the article published on July 5th, 2017 has been updated by the latest benchmarking results.
Contact Us