Framework - Objectives and History

This site introduces the Performance Test Framework to benchmark Enterprise Service Bus [ESB] implementations. The framework is developed to be simple to use, fair, and includes some of the most commonly used scenarios with an ESB. The test suite includes all software, tools and scripts and does not depend on any external libraries or utilities.

  • Consult the Wikipedia article for a definition of an Enterprise Service Bus [ESB]

1. Objectives

  • Enable performance comparison of ESBs under fair and identical conditions thus preventing apples-to-oranges comparisons

  • Publicly document the ESB Performance Test Framework scenarios and disclose all details thus providing for extension

  • Make the ESB Performance Test Suite which implements the framework and includes the load generation tools available FREE of charge

  • Make it easy to performance benchmark any ESB - from any vendor, by ensuring that the framework remains vendor agnostic

  • Make it easy to run the benchmark on local hardware or on Amazon EC2 (or equivalents)

The ESB Performance Test Suite ships as part of the UltraESB distribution (~40MB) from AdroitLogic. The complete download includes the following:

  • A high performance mock service to simulate a backend service

  • A high performance multi-threaded HTTP/S client to generate load

  • Sample messages randing from 512 bytes to 100K bytes - (Un-secured and WS-Secured)

  • Load test scripts to generate loads for 20, 40, 80, 160, 320, 640, 1280 and 2560 concurrent users

  • A graphical test utility ToolBox to help setup and test scenarios with any ESB implementation

  • Utilities to convert results into CSV format for analysis with a spreadsheet application such as Excel or OpenOffice

2. Conception

The ESB performance benchmark was conceived in 2005 by a team of engineers from the Virtusa Corporation, assigned on a staff augmented project for one of the largest Internet and media companies at the time ("client"). The team was lead by Asankha Perera, and included Jacqueline Thangarajah and Sandeep Vallabhaneni of Virtusa, and was directed by a senior technical architect of the client. The client required the development of a framework to measure the performance of a leading commercial ESB, against another traffic management appliance and an Apache proxy under identical conditions for direct proxy and XSLT test scenarios, under different levels of concurrency and payload size. Detailed information about this project and its output reports are covered by confidentiality and non-disclosure agreements.

3. History - Used since June 2007 by multiple vendors

The ESB Performance Test Framework was first published in June 2007 by the WSO2 ESB team members Asankha PereraRuwan LintonIndika Kumara and Chathura Ekanayake along with the WSO2 technical leadership of Paul Fremantle and Sanjiva Weerawarana. WSO2 open sourced this work, allowing other vendors as well as end users to use the framework and/or extend it. It has since been revised and enhanced over the years - and used by ESB vendors such as MuleBEAWSO2 and AdroitLogic.

4. A vendor neutral approach to ESB performance testing

The ESB Performance Test Framework defines a test suite across some of the most common use cases for an ESB - and provides a ESB Performance Test Suite free of charge - so that any end user could use it to benchmark any ESB of choice - either open source or commercial. The Test suite itself is vendor agnostic, and the test cases should be supported by any production ready ESB publicly available.

5. Previous Results and Claims Published on ESB Performance by different vendors

6. ESB Benchmarking resources published by others

Contact Us