TID: [] [WSO2 ESB] [2012-08-04 08:48:41,657] ERROR {org.apache.synapse.core.axis2.Axis2Sender} - Unexpected error sending message back {org.apache.synapse.core.axis2.Axis2Sender}
org.apache.axis2.AxisFault: Error in signature with X509Token
...
TID: [] [WSO2 ESB] [2012-08-04 08:48:49,276] ERROR {org.apache.synapse.transport.nhttp.ServerWorker} - Error processing POST request {org.apache.synapse.transport.nhttp.ServerWorker}
java.util.ConcurrentModificationException
...
TID: [] [WSO2 ESB] [2012-08-04 09:30:11,494] WARN {org.apache.synapse.commons.jmx.ThreadingView} - SYSTEM ALERT: 58.48484848484848% of the HttpClientWorker threads were in BLOCKED state during last minute! {org.apache.synapse.commons.jmx.ThreadingView}
TID: [] [WSO2 ESB] [2012-08-04 09:30:11,503] WARN {org.apache.synapse.commons.jmx.ThreadingView} - SYSTEM ALERT: 60.45454545454546% of the HttpClientWorker threads were in BLOCKED state during last minute! {org.apache.synapse.commons.jmx.ThreadingView}
...ESB Performance Testing - Round 6
Table of Contents
ESB Performance Testing - Round 6
|
AdroitLogic would like to state that we openly announced the execution of Round 6 about a month prior to the final execution of these tests, and requested any interested parties to submit configurations or updates for any of the ESBs to be tested. We are happy to state that we’ve received and incorporated such changes from parties who replied, and that we also enhanced the test cases and execution of Round 6 itself, based on suggestions we received from others. We’ve since received more updates from yet more parties after execution of Round 6, and we hope to include those changes into Round 7. We welcome any one to re-execute these tests and publish both the results obtained - and the corresponding AMI images publicly. We would also request any parties wanting to submit enhancements to make them available via a Mercurial pull request against the repository https://bitbucket.org/adroitlogic/esbperformance. We would gladly re-execute the tests once such changes are received, as we are committed to making this benchmark fair, transparent and open for enhancement, with input and feedback from others. |
1. Overview
This article is based on the 6th round of public ESB Performance benchmarking since June 2007. This performance framework has become the de-facto ESB Performance test suite, with multiple vendors using it to publish benchmark results in the past.
The significant changes in this round includes:
-
Using an asynchronous Tomcat backend service for better performance and for similarity with real world systems
-
Execution of three iterations for each ESB, to reduce any effects from the AWS execution environment variability for the final results
-
Extending the WS-Security test cases to include 10K messages
This round includes the above mentioned suggestions and improvements with the kind contributions of many, who helped improve the benchmark
Similar to the last round #5, this round includes:
-
The inclusion of 8 free and open source ESBs
-
The publication of a standalone Amazon EC2 AMI image of the complete test scenario and resources (ami-09ef4560)
-
The publication of the ESB Performance Benchmark resources project on BitBucket updated for round #6
2. Disclaimer
These configurations have been built by AdroitLogic, partly based on previously published resources and contributions submitted to the resources repository. The solutions we’ve come up with may have inherent limitations due to our limited understanding of some of the other ESBs and the limited time we had spent. Hence these configurations may not be optimal, and/or may not be optimally tuned for the scenarios tested
However, AdroitLogic has spent many days and nights to get the other ESBs to work under identical conditions as much as possible. We would be happy to receive feedback to improve the configurations used, and the tuning parameters for the different ESBs from the vendors and/or end users. We will include those updates into future rounds, and could be submitted directly to the Mercurial repository as a pull request.
3. Scenarios tested
-
Direct Proxy Service
-
Content Based Routing Proxy
-
on SOAP body payload
-
on SOAP header
-
on Transport Header
-
-
XSLT Transformation Proxy
-
WS-Security Proxy
Each of these scenarios are described in detail under Performance Test Cases
4. Performance Test Environment and Resources
4.1. Amazon EC2 AMI image
The performance test was carried out on an Amazon EC2 High CPU Extra Large (c1.xlarge) instance running the Public AMI "ami-09ef4560" based on the EBS backed official Ubuntu 10.10 image. This instance was a 64 bit instance and had 7GB of RAM with a 8GB EBS backed storage, 20 ECUs and 8 cores. All tests against each ESB was carried out on the same instance started on the 3rd of August 2012, and executed one after another. Network isolation was not required as all test traffic was only between the local machine hosting the load generator, ESB and backend service.
Each ESB was allocated 2GB of Java Heap memory. The 64 bit Oracle / Sun JDK 1.6.0_33-b04 was used with the unlimited strength policy files applied to allow WS-Security testing with larger key sizes. The instance had a 2GB RAM disk which was utilized by the UltraESB. We attempted to allow approximately 300 worker threads, and a socket timeout of 120 seconds for each ESB, but this was not possible for some ESB’s due to time and/or documentation limitations.
4.2. Performance scenario configurations for each ESB
The configurations used for each ESB are published on the BitBucket repository at https://bitbucket.org/adroitlogic/esbperformance. This includes a README.TXT file for each ESB, with basic instruction to configure each ESB from scratch, and to build and deploy the configurations.
4.3. Results Analysis Worksheet hosted on Google Docs
ESB Performance Testing Round 6 is a publicly accessible Google document hosting the summarized results.
5. Results
5.1. ESBs that failed to complete the benchmark
The following ESB’s did not complete the full benchmark successfully, and hence were excluded from the final analysis. Details of the errors can be found on the README for each ESB
-
Apache ServiceMix v4.3.0
-
Fuse ESB v4.4.1
-
JBoss ESB v4.11
-
Petals ESB v4.1
The rest of the ESBs were selected for the final analysis, which would execute the full benchmark three times, and use the best and average TPS for each scenario for a final analysis. This step was introduced in Round 6 to validate that the slight variability of performance on the Amazon EC2 environment will not significantly affect the overall result.
The following ESBs completed the full benchmark without any significant issues
-
Talend ESB SE v5.1.1
-
UltraESB v1.7.1 - Vanilla
-
UltraESB v1.7.1 - Enhanced
-
Mule CE v3.3.0
-
* encountered a few errors and failures during some of the individual tests)
-
Although the WSO2 ESB was selected for the final analysis based on the initial test run, it failed during the three iterations of the benchmark under load, and had to be retried
-
WSO2 ESB v4.0.3
-
The ESB was stuck for over 6.5 hours with 100% CPU utilization. Since we had selected this ESB for the final run initially, we re-attempted the analysis and the retry completed the test.
-
5.2. More details on the ESBs that failed during the benchmark under load.
5.2.1. WSO2 ESB v4.0.3 - more details
Exceptions seen on the logs during the load test were many and the complete log can be found at /home/ubuntu/esbs/wso2esb-4.0.3/repository/logs/crash/wso2carbon.log. Some snippets follows
After many hours (~6.5hours), the system was still fully blocked on CPU as per EC2 monitoring and top. As no progress was being made, the test was abandoned.
Tasks: 165 total, 1 running, 164 sleeping, 0 stopped, 0 zombie
Cpu(s): 91.5%us, 0.0%sy, 0.0%ni, 2.5%id, 0.0%wa, 0.0%hi, 0.0%si, 6.0%st
Mem: 7132352k total, 3899800k used, 3232552k free, 219356k buffers
Swap: 0k total, 0k used, 0k free, 467920k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7382 ubuntu 20 0 2821m 2.3g 15m S 791 33.7 2815:14 java
858 ubuntu 20 0 19276 1356 968 R 0 0.0 0:00.07 top
1 root 20 0 23744 1936 1300 S 0 0.0 0:00.43 initTrying out the Direct Proxy and Transport Header based routing Proxy scenarios with the WSO2 ESB server using its `Passthrough Transport' generated the following log messages. When the passthrough transport is being used, none of the other use cases can execute simultaneously on the WSO2 ESB, even over multiple different HTTP/S ports. Hence any use of the Passthrough Transport is limited only to simple proxying and/or transport header based routing.
[2012-08-05 06:34:52,764] WARN - SourceHandler Illegal incoming connection state: CLOSED . Possibly two send backs are happening for the same request
[2012-08-05 06:34:52,768] WARN - SourceHandler Trying to write response body while the handler is in an inconsistent state REQUEST_READY
[2012-08-05 06:34:52,770] WARN - PassThroughHttpListener System may be unstable: HTTP ListeningIOReactor encountered a runtime exception : null
java.lang.NullPointerException
at org.wso2.carbon.transport.passthru.SourceHandler.outputReady(SourceHandler.java:179)
...
[2012-08-05 06:34:52,778] WARN - SourceHandler Connection closed while writing the response: 127.0.0.1:51216->127.0.0.1:8280
...When we tried the WSO2 ESB again without the Passthrough Transport, Exceptions were seen on the logs as can be found at /home/ubuntu/esbs/wso2esb-4.0.3/repository/logs/wso2carbon.log from timestamp 2012-08-05 06:57:22 onwards. Some snippets follows
TID: [] [WSO2 ESB] [2012-08-05 07:40:40,570] WARN {org.apache.synapse.commons.jmx.ThreadingView} - SYSTEM ALERT: 60.15151515151514% of the HttpClientWorker threads were in BLOCKED state during last minute! {org.apache.synapse.commons.jmx.ThreadingView}
TID: [] [WSO2 ESB] [2012-08-05 07:40:40,577] WARN {org.apache.synapse.commons.jmx.ThreadingView} - SYSTEM ALERT: 60.30303030303028% of the HttpClientWorker threads were in BLOCKED state during last minute! {org.apache.synapse.commons.jmx.ThreadingView}
...
...
TID: [] [WSO2 ESB] [2012-08-05 08:12:28,825] ERROR {org.apache.synapse.core.axis2.Axis2Sender} - Unexpected error sending message back {org.apache.synapse.core.axis2.Axis2Sender}
org.apache.axis2.AxisFault: Error in signature with X509Token
at org.apache.rampart.handler.RampartSender.invoke(RampartSender.java:76)
...
Caused by: org.apache.rampart.RampartException: Error in signature with X509Token
...
Caused by: org.apache.ws.security.WSSecurityException: Signature creation failed; nested exception is:
...
... 22 more
Caused by: java.util.ConcurrentModificationException
...5.2.2. Apache ServiceMix v4.3.0 - more details
The smoke test itself failed, with the following error
13:09:57,003 | WARN | rovider-thread-9 | PhaseInterceptorChain | - - | Interceptor for {http://services.samples/xsd}HBRTransportEchoService has thrown exception, unwinding now
org.apache.cxf.interceptor.Fault
at org.apache.servicemix.cxfbc.interceptors.JbiOutWsdl1Interceptor.handleMessage(JbiOutWsdl1Interceptor.java:144)[156:servicemix-cxf-bc:2011.01.0]
at org.apache.servicemix.cxfbc.interceptors.JbiOutWsdl1Interceptor.handleMessage(JbiOutWsdl1Interceptor.java:54)[156:servicemix-cxf-bc:2011.01.0]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:255)[116:org.apache.cxf.bundle:2.3.2]
at org.apache.servicemix.cxfbc.CxfBcProvider.process(CxfBcProvider.java:247)[156:servicemix-cxf-bc:2011.01.0]
at org.apache.servicemix.common.AsyncBaseLifeCycle.doProcess(AsyncBaseLifeCycle.java:651)[121:servicemix-common:2011.01.0]
at org.apache.servicemix.common.AsyncBaseLifeCycle.processExchange(AsyncBaseLifeCycle.java:606)[121:servicemix-common:2011.01.0]
at org.apache.servicemix.common.AsyncBaseLifeCycle.processExchangeInTx(AsyncBaseLifeCycle.java:501)[121:servicemix-common:2011.01.0]
at org.apache.servicemix.common.AsyncBaseLifeCycle$2.run(AsyncBaseLifeCycle.java:370)[121:servicemix-common:2011.01.0]
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)[:1.6.0_33]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)[:1.6.0_33]
at java.lang.Thread.run(Thread.java:662)[:1.6.0_33]
Caused by: java.lang.NullPointerExceptionDuring the load test iteration with 20 concurrent users to the XSLTProxy, the ServiceMix ESB became almost unusable, this was still at the time the pre-load test warmup phase was executing. It was not really servicing requests, but had become extremely slow (as per a tcpdump) but the log files did not report any faults.
5.2.3. Fuse ESB v4.4.1 - more details
Unlike in Round #5, the security smoke tests passed. So we used the full test suite against the Fuse ESB. The Fuse ESB became unresponsive at 2560 concurrent users running the Direct Proxy scenario with 500 byte messages. As seen during Round #5, we again saw log messages like the following after the ESB was restarted after a failure.
Warning: The encoding 'UTF-8' is not supported by the Java runtime.
Warning: encoding "UTF-8" not supported, using UTF-8We noticed that in the Direct Proxy scenario, the SOAP headers were not proxied to the backend service. This could be due to a limitation in the configuration we tested, and could have been present for other scenarios as well. link:[] link:[]On the second run, the ESB became seemingly unresponsive after 160 concurrent users with 100K messages. A "top" showed the following output, but nothing was visible in the log files. Part of the visible thread dump on a "kill -3 10573" appears at the end of this file. However the test completed approximately after 3 hours. link:[]
top - 10:18:08 up 2:22, 5 users, load average: 14.64, 15.73, 11.29
Tasks: 173 total, 1 running, 172 sleeping, 0 stopped, 0 zombie
Cpu(s): 62.7%us, 3.9%sy, 0.0%ni, 32.4%id, 0.0%wa, 0.0%hi, 0.5%si, 0.5%st
Mem: 7132352k total, 3730196k used, 3402156k free, 59360k buffers
Swap: 0k total, 0k used, 0k free, 587884k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
10573 ubuntu 20 0 2710m 2.3g 16m S 599 33.8 157:09.97 java5.2.4. JBoss ESB v4.11 - more details
During CBR on payload testing, we noticed that the response for a 5K request was 368 - i.e. a SOAP fault, indicating the XPath expression failed. However manual testing with the SOA ToolBox confirmed that randomly the XPath evaluation would pass and fail for the requests, we also noticed that when the load test was running, all XPath expressions were failing. We aborted further testing after this issue since correct and consistent operation was a pre-requisite for the benchmark, over performance figures.
5.2.5. Petals ESB v4.1 - more details
The Petals ESB started to give errors from the smoke tests, and after processing 20 concurrent users with 100K messages, the system became unusable with a top output as follows:
top - 18:59:20 up 11:04, 4 users, load average: 39.65, 36.95, 30.43
Tasks: 169 total, 1 running, 168 sleeping, 0 stopped, 0 zombie
Cpu(s): 93.9%us, 0.6%sy, 0.0%ni, 4.2%id, 0.0%wa, 0.0%hi, 0.1%si, 1.2%st
Mem: 7132352k total, 4885072k used, 2247280k free, 222712k buffers
Swap: 0k total, 0k used, 0k free, 1369876k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
22714 ubuntu 20 0 2912m 2.4g 15m S 774 34.8 312:58.08 javaThe log file contained lots of error messages, similar to the following:
0 2012/08/01 18:30:17,448 GMT+0000 WARNING [Petals.Container.Components.petals-bc-soap] : Try #0. No JBI message exchange processor is available in the pool. Wait 259ms before next try.
0 2012/08/01 18:30:41,815 GMT+0000 WARNING [Petals.Container.Components.petals-bc-soap] : Catch an exception on the WS invocation : null
java.lang.NullPointerException
0 2012/08/01 18:30:42,270 GMT+0000 WARNING [Petals.Container.Components.petals-bc-soap] : Catch an exception on the WS invocation : The server localhost failed to respond with a valid HTTP response
org.apache.axis2.AxisFault: The server localhost failed to respond with a valid HTTP response
0 2012/08/01 18:55:22,376 GMT+0000 INFO [Petals.JBI-Messaging.PriorityOrdererModule] : Memory threshold reached, persist some pending exchange if any6. Observations
The following graph shows the summarized TPS for each scenario across message size and concurrency, taking the best of the three iterations for each ESB included into the final analysis. (The graph taking the Average of the three iterations showed an almost similar result, and can be found on the Google Worksheet)
| Mule-3.3.0 | Talend-SE-5.1.1 | UltraESB-1.7.1 Vanilla | UltraESB-1.7.1 Enhanced | WSO2-ESB-4.0.3 | WSO2-ESB-4.0.3 PassThrough | |
|---|---|---|---|---|---|---|
| Direct Proxy | 3,375 | 3,315 | 4,904 | 4,839 | 2,330 | 4,693 |
| CBR Proxy | 3,198 | 3,108 | 3,700 | 4,703 | 2,428 | |
| CBR SOAP Header Proxy | 1,458 | 3,185 | 3,841 | 5,063 | 2,567 | |
| CBR Transport Header Proxy | 2,262 | 3,751 | 5,394 | 5,523 | 2,764 | 5,795 |
| XSLT Proxy | 2,225 | 2,333 | 1,957 | 3,387 | 839 | |
| WS-Security Proxy | 458 | 534 | 604 | 603 | 463 | |
| Average | 2,163 | 2,704 | 3,400 | 4,020 | 1,899 |
To simplify the results into a further level, we computed the average TPS across each scenario benchmarked, and computed this for both the best case and the average case of the three iterations.
| Mule-3.3.0 | Talend-SE-5.1.1 | UltraESB-1.7.1-Vanilla | UltraESB-1.7.1-Enhanced | WSO2-ESB-4.0.3 | |
|---|---|---|---|---|---|
| Best Case | 2,163 | 2,704 | 3,400 | 4,020 | 1,899 |
| Average Case | 2,068 | 2,560 | 3,220 | 3,818 | 1,807 |
Refer to the Google Document at : *ESB Performance Testing Round 6 for complete results.
-
The UltraESB - Enhanced version had a very clear lead above all ESBs across all scenarios, followed up by the UltraESB - Vanilla version. The main difference of the "Enhanced" version is making the Saxon library (for XSLT performance) and the VTD XML and Fast VTD XML libraries (for CBR performance) available to the runtime. These libraries are easily downloadable by end users, although not shipped by default due to licensing issues related to these libraries.
-
Talend ESB had better performance across results over the Mule CE and WSO2 ESBs, and showed very good stability
-
WSO2 ESB failed to complete the first full benchmark execution running at full CPU utilization for over 6.5 hours, and had to be retried once more for the results shown above
-
Mule ESB showed a few failures and errors for some of the scenarios
-
Compared to Round #5, Talend ESB had optimized its performance by addressing the issues reported. The README contains instructions on how the Talend ESB was patched up to overcome the issues, and contains necessary steps to apply the patches that are freely available
-
Compared to Round #5, the Mule CE performance has improved over that of the WSO2 ESB.
-
Compared to Round #5, both the performance and the stability of the WSO2 ESB has suffered
7. Reproducing the results
Full details on reproducing these results on an Amazon EC2 instance can be found here
8. Authors
Asankha C. Perera, Founder and CTO AdroitLogic
Ruwan Linton, Director of Engineering, AdroitLogic
Email: info@adroitlogic.com
8.1. Contributors
We value all contributions received to the ESB Performance Benchmark project on BitBucket, and the suggestions for improvements submitted privately by many.
9. Notices
WSO2 and WSO2 ESB are trademarks of WSO2 Inc. MuleSoft and Mule ESB are trademarks of MuleSoft. Fuse ESB is a trademark of FuseSource, Corp. Petals is a trademark of EBM WebSourcing. JBoss is a trademark of Red Hat, Inc. Talend is a trademark of Talend. Apache ServiceMix is a trademark of the Apache Software Foundation. UltraESB and AdroitLogic are trademarks of AdroitLogic Private Ltd. All other product and company names and marks mentioned are the property of their respective owners and are mentioned for identification purposes only.