1. The Scenarios Tested

The test scenarios are mainly around the HTTP protocol for SOAP/XML payloads - which incidentally makes up a large segment of messages passed through modern ESBs today. Currently four aspects are tested, namely - Direct proxy, Content Based Routing - Body, SOAP Header and Transport Header, XSLT transformation and WS-Security. Message sizes are varied from 512 bytes to 100K bytes, and the concurrent number of users from 20 to 2560 to evaluate each ESB under differing load levels. Refer below for detailed descriptions of each scenario.

1.1. Why test HTTP performance over JMS performance?

As stated earlier, XML or SOAP over HTTP/S makes up most of the messages passed around through an ESB today. REST use cases are increasingly becoming a considerable chunk as well. Most ESBs supports non-XML payloads such as Hessian, Text, HTML, binary etc as well as Fast Infoset support and Gzip compression of the output over HTTP/S as well. Typically ESBs support HTTP/S, JMS, Email (POP3/IMAP/SMTP), File, S/FTP, FTP, FTP/S, MLLP/S, TCP/S, FIX, Database etc and many others.

Some of the ESB’s that have been based over existing EAI or Messaging technologies sometimes may convert each message into a JMS or other messaging system based message, irrespective of the transport. This may give good performance for JMS or Messaging Oriented Middleware [MOM] system based use cases, while messages processed over other transports may suffer undesired overheads.

The performance of JMS or MOM infrastructure depends heavily on the form of persistence provided by the underlying technology. Thus, a vendor using in-memory persistence of messages may easily show better performance over another that used Database or File based persistence for recovery. Typically most ESBs available free of charge or as open source ESB’s - works with many JMS or MOM systems - that maybe usually be hosted externally, outside of the ESB - similarly to how a typical web application may use an external Database. Thus the performance of JMS or MOM transports will be based on the underlying JMS/MOM frameworks used - and will not convey the performance of the ESB itself - for handling the messaging load.

1.2. Message sizes and the number of concurrent users

The performance test uses messages of size:

  • 512 bytes

  • 1K bytes

  • 5K bytes

  • 10K bytes

  • 100K bytes

With the concurrent number of users:

  • 20 concurrent users

  • 40 concurrent users

  • 80 concurrent users

  • 160 concurrent users

  • 320 concurrent users

  • 640 concurrent users

  • 1280 concurrent users

  • 2560 concurrent users

Sample request message

<soapenv:Body>
<m:buyStocks xmlns:m="<a href="http://services.samples/xsd" class="external free"rel="nofollow">http://services.samples/xsd<;/a>">
<order><symbol>IBM</symbol><buyerID>asankha</buyerID><price>140.34</price><volume>2000</volume></order>
<order><symbol>MSFT</symbol><buyerID>ruwan</buyerID><price>23.56</price><volume>8030</volume></order>
.....
</m:buyStocks>
</soapenv:Body>

1.3. Direct Proxy Service

perf sc 1

This scenario will demonstrate the ability of the selected ESB to act as a virtualization layer for backend web services - similarly to how the Apache Web Servers or Hardware Load balancers were used in front of web backends to virtualize and balance load and provide failover capabilities. Although this scenario does not include load balancing, fail-over, security verification or schema validation etc, most of the ESBs does support these features. This scenario assumes the service to be a SOAP Web Service, and thus expects the ESB to present the WSDL of the service to its clients.

Requirements:

  • The Proxy service MUST be available over http at a URL of the form: http://localhost/:<port>/<path>/DirectProxy

  • The Proxy service MUST expose its WSDL at http at a URL of the form: http://localhost/:<port>/<path>/DirectProxy?wsdl

  • The Proxy service MAY present a custom error reply, if there was an error

1.4. The Content Based Routing (CBR) Proxy

perf sc 2

Typically this pattern is used in practice to route request messages to different endpoints, or handle them differently at the ESB based on the message attributes - such as the payload, or transport headers etc. This scenario is now presented in three variants:

  • CBR on a HTTP transport header

  • CBR on a SOAP Header

  • CBR on the SOAP Body / Payload

CBR on Transport Header

The first scenario evaluates the most optimal way to route messages, based on HTTP transport level headers. This technique is widely used when routing Hessian or other non-SOAP payloads, as is extremely efficient. The condition performs a simple String string comparison over a HTTP header.

CBR on SOAP Header

This scenario performs a routing decision on a SOAP header, as it could be better optimized than processing a large SOAP payload body. The condition performs an XPath evaluation over a SOAP header. This test scenario extracts a SOAP header specified as a simple String, and then performs a comparison against a given constant string.

CBR on SOAP Body

This scenario will demonstrate the ability of the selected ESB to route an XML payload based on the evaluation of an XPath expression over the payload body. The condition performs an XPath evaluation over a SOAP body. This test scenario expects the first order elements’ symbol to be equal to "IBM" to forward the request to the backend service. On a failure, its expected to return a SOAP fault to the client with an error message.

Requirements:

  • The Proxy service MUST be available over http at a URL of the form: http://localhost/:<port>/<path>/CBRProxy

  • The Proxy service MUST expose its WSDL at http at a URL of the form: http://localhost/:<port>/<path>/CBRProxy?wsdl

  • The Proxy service MUST present a custom error reply if the routing condition is not satisfied by the request

1.5. The XSL Transformation Proxy

perf sc 3

This scenario will demonstrate the ability of the selected ESB to transform XML payload messages (such as SOAP) between different schemas. In practice transformations are used to convert messages from XML to CSV or Text etc, or to change a request adhering to one version of a schema to a different version (for backwards compatibility of services/clients), or to create messages that are enriched with some attributes from the original request, and other attributes picked up during mediation of the message - for example from a Database. Typically ESB’s offer multiple options for transformations - including XSL, XQuery, Scripting language, Milyn etc. In future this scenario maybe extended with other scenarios.

This test expects the transformation to convert the original message to the format shown below - by reversing the element names. Thus, the Echo service backend will echo back this reverse message, which is transformed again to the original format and sent to the client.

<soapenv:Body>
<m:skcotSyub xmlns:m="<a href="http://services.samples/xsd" class="external free"rel="nofollow">http://services.samples/xsd<;/a>">
<redro><lobmys>IBM</lobmys><DIreyub>asankha</DIreyub><ecirp>140.34</ecirp><emulov>200000</emulov></redro>
<redro><lobmys>MSFT</lobmys><DIreyub>ruwan</DIreyub><ecirp>23.56</ecirp><emulov>803000</emulov></redro>
....
</m:skcotSyub>
</soapenv:Body>

Requirements:

  • The Proxy service MUST be available over http at a URL of the form: http://localhost/:<port>/<path>/XSLTProxy

  • The Proxy service MUST expose its WSDL at http at a URL of the form: http://localhost/:<port>/<path>/XSLTProxy?wsdl

  • The Proxy service MAY present a custom error reply, if there was an error

perf sc 4

1.6. WS-Security Proxy

perf sc 4

This scenario will demonstrate the ability of the selected ESB to act as a Security Gateway. Typically an ESB is used to validate and verify security credentials, and establish the authenticity of messages, before passing them through mediation and to internal services within a corporate environment. Although this test case considers WS-Security, many SOA deployments rely on SSL, HTTP Basic and Digest authentication and other mechanisms as well.

This test expects the client to send a timestamped, digitally signed and encrypted request, and verifies the message and forwards the actual payload without the WS-Security header to the backend service. Once the Echo service responds back with the same message, the ESB timestamps, digitally signs and encrypts the response and sends it back to the client.

The test suite has bundled a set of requests that wrap the original requests with a timestamp valid for 10 years. These requests are found with file names of the form "<size>_buyStocks_secure.xml". Thus, the Apache Bench style load generator can very efficiently push a load of secured messages for a valid performance test of the ESB. However, it was noted that the Mule ESB did not accept the 10 year valid messages due to the older creation date, and hence the current framework includes a utility to regenerate the secured requests before an execution run against Mule or other such ESBs that will not accept the pre-secured messages.

The messages are secured using the standard WS-Security interoperability test suite keystore, and are signed as the user "alice" and encrypted for user "bob", and the passwords for the keystore as well as "alice" and "bob" is "password" - as per the standard WS-Security interoperability test cases.

Requirements:

  • The Proxy service MUST be available over http at a URL of the form: http://localhost/:<port>/<path>/SecureProxy

  • The Proxy service MUST expose its WSDL at http at a URL of the form: http://localhost/:<port>/<path>/SecureProxy?wsdl

  • The Proxy service MAY present a custom error reply, if there was an error

Back to Home Page