This site introduces the Performance Test Framework to benchmark Enterprise Service Bus [ESB] implementations. The framework is developed to be simple to use, fair, and includes some of the most commonly used scenarios with an ESB. The test suite includes all software, tools and scripts and does not depend on any external libraries or utilities.
Consult the Wikipedia article for a definition of an Enterprise Service Bus [ESB]
Enable performance comparison of ESBs under fair and identical conditions thus preventing apples-to-oranges comparisons
Publicly document the ESB Performance Test Framework scenarios and disclose all details thus providing for extension
Make the ESB Performance Test Suite which implements the framework and includes the load generation tools available FREE of charge
Make it easy to performance benchmark any ESB - from any vendor, by ensuring that the framework remains vendor agnostic
Make it easy to run the benchmark on local hardware or on Amazon EC2 (or equivalents)
The ESB Performance Test Suite ships as part of the UltraESB distribution (~40MB) from AdroitLogic. The complete download includes the following:
A high performance mock service to simulate a backend service
A high performance multi-threaded HTTP/S client to generate load
Sample messages randing from 512 bytes to 100K bytes - (Un-secured and WS-Secured)
Load test scripts to generate loads for 20, 40, 80, 160, 320, 640, 1280 and 2560 concurrent users
A graphical test utility ToolBox to help setup and test scenarios with any ESB implementation
Utilities to convert results into CSV format for analysis with a spreadsheet application such as Excel or OpenOffice
The ESB performance benchmark was conceived in 2005 by a team of engineers from the Virtusa Corporation, assigned on a staff augmented project for one of the largest Internet and media companies at the time ("client"). The team was lead by Asankha Perera, and included Jacqueline Thangarajah and Sandeep Vallabhaneni of Virtusa, and was directed by a senior technical architect of the client. The client required the development of a framework to measure the performance of a leading commercial ESB, against another traffic management appliance and an Apache proxy under identical conditions for direct proxy and XSLT test scenarios, under different levels of concurrency and payload size. Detailed information about this project and its output reports are covered by confidentiality and non-disclosure agreements.
The ESB Performance Test Framework was first published in June 2007 by the WSO2 ESB team members Asankha Perera, Ruwan Linton, Indika Kumara and Chathura Ekanayake along with the WSO2 technical leadership of Paul Fremantle and Sanjiva Weerawarana. WSO2 open sourced this work, allowing other vendors as well as end users to use the framework and/or extend it. It has since been revised and enhanced over the years - and used by ESB vendors such as Mule, BEA, WSO2 and AdroitLogic.
The ESB Performance Test Framework defines a test suite across some of the most common use cases for an ESB - and provides a ESB Performance Test Suite free of charge - so that any end user could use it to benchmark any ESB of choice - either open source or commercial. The Test suite itself is vendor agnostic, and the test cases should be supported by any production ready ESB publicly available.
Achieving Optimal ESB Performance - WSO2, March 2012
Run Mule… run faster - Mulesoft, January 2011
WSO2 ESB Performance Testing (New) - http://WSO2%20ESB%20Performance%20Testing%20%28New%29[WSO2, December 2010]
ESB Performance Testing Round 4 - AdroitLogic
OpenESB and ServiceMix in Comparison - predic8, September 2008
Mule Performance Test Results - MuleSoft, July 2008
ESB Performance Testing Round 3 - WSO2, June 2008
Setting the New Benchmark for Enteprise-class ESB Performance - BEA, March 2008
ESB Performance Testing Round 2 - WSO2, July 2007
ESB Performance Testing Round 1 - WSO2, June 2007