ESB Performance

Objectives and History

This site introduces the Performance Test Framework to benchmark Enterprise Service Bus [ESB] implementations. The framework is developed to be simple to use, fair, and includes some of the most commonly used scenarios with an ESB. The test suite includes all software, tools and scripts and does not depend on any external libraries or utilities.

  •   Consult the Wikipedia article for a definition of an Enterprise Service Bus [ESB]

Objectives

  •   Enable performance comparison of ESBs under fair and identical conditions thus preventing apples-to-oranges comparisons
  •   Publicly document the ESB Performance Test Framework scenarios and disclose all details thus providing for extension
  •   Make the ESB Performance Test Suite which implements the framework and includes the load generation tools available FREE of charge
  •   Make it easy to performance benchmark any ESB - from any vendor, by ensuring that the framework remains vendor agnostic
  •   Make it easy to run the benchmark on local hardware or on Amazon EC2 (or equivalents)

The ESB Performance Test Suite ships as part of the UltraESB distribution (~35MB) from AdroitLogic. The complete download includes the following:

  •   A high performance mock service to simulate a backend service
  •   A high performance multi-threaded HTTP/S client to generate load
  •   Sample messages randing from 512 bytes to 100K bytes - (Un-secured and WS-Secured)
  •   Load test scripts to generate loads for 20, 40, 80, 160, 320, 640, 1280 and 2560 concurrent users
  •   A graphical test utility ToolBox to help setup and test scenarios with any ESB implementation
  •   Utilities to convert results into CSV format for analysis with a spreadsheet application such as Excel or OpenOffice

History - Used since June 2007 by multiple vendors

The ESB Performance Test Framework was originally developed in June 2007 by the WSO2 ESB team members Asankha Perera, Ruwan Linton, Indika Kumara and Chathura Ekanayake along with the WSO2 technical leadership of Paul Fremantle and Sanjiva Weerawarana. WSO2 open sourced this work, allowing other vendors as well as end users to use the framework and/or extend it. It has since been revised and enhanced over the years - and used by ESB vendors such as Mule, BEA, WSO2 and AdroitLogic.

A vendor neutral approach to ESB performance testing

The ESB Performance Test Framework defines a test suite across some of the most common use cases for an ESB - and provides a ESB Performance Test Suite free of charge - so that any end user could use it to benchmark any ESB of choice - either open source or commercial. The Test suite itself is vendor agnostic, and the test cases should be supported by any production ready ESB publicly available.

Previous Results and Claims Published on ESB Performance by different vendors

ESB Benchmarking resources published by others