Execution - EC2 Execution of Round 6

The ESB Performance Benchmark Round 6 was executed on an Amazon EC2 node. To reproduce the results, you will require an Amazon EC2 account and will incur a small cost for the instance hours used.

1. Starting the node on EC2

From your web browser, navigate to https://console.aws.amazon.com/ec2/home and login with your EC2 account credentials to the AWS Console. Ensure that you are looking at the "N. Virginia" region (1). Navigate to the "EC2" tab, and from the left navigation pane, select "AMIs". From the right hand pane, select "Public Images" (2) from the Viewing drop down list, and specify the AMI ID "ami-09ef4560" (3) for filtering and press the enter key. The AMI will be shown in the results table.

aws ami selection

Right click the result row, and select "Launch Instance" or "Request Spot Instance" as you prefer, noting the slight difference in charges, and the possibility of the node being taken down during the test by Amazon, if your bid falls below the rate at the time of execution for a spot instance. As the instance type select "High-CPU Extra Large (c1.xlarge, 7GB).

The next set of options could be left at their defaults if your Amazon account is configured for EC2 use already - i.e. with a keypair, security group allowing SSH etc. For any help regarding these steps, refer the Amazon EC2 documentation or Help. Once you submit the instance request, navigate to the "Instances" from the left menu. You should "refresh" the list until you see your instance up and running. Once the new instance appears, right click on the row, and select "Connect"

Select the option to connect with a standalone SSH client, and open a SSH session to the new instance using your key. The command presented will look like the following "ssh -i <yourkeyfile.pem> ubuntu@ec2-107-22-142-4.compute-1.amazonaws.com" Note that the default login is set to "root@" needs to be changed to "ubuntu@" as shown in the example below:

ssh -i  adroitlogic.pem  ubuntu@ec2-107-22-142-4.compute-1.amazonaws.com

The key file name will change depending on your Amazon credentials. On login, you will be presented with the legal information about the AMI image. Open about 6 ssh sessions to the node for use as follows:

  • Session 1 - Run Echo Service - mock backend service used by the ESBs

  • Session 2 - Run the ESB of choice

  • Session 3 - Run the load generator script, redirecting output to a file

  • Session 4 - Monitor raw output of load generator

  • Session 5 - Monitor errors and failures via load generator output

  • Session 6 - Use to build ESB configurations and deploy / configure artifacts

2. Execute the Echo Service Back-end service

On the first SSH session, start the Tomcat server hosting the backend mock service, and check its state using the `netstat' command as shown below

ubuntu@domU-12-31-39-0B-C5-41:~$ cd backend-service/apache-tomcat-7.0.29/bin
  ubuntu@domU-12-31-39-0B-C5-41:~/backend-service/apache-tomcat-7.0.29/bin$ ./startup.sh
  Using CATALINA_BASE:   /home/ubuntu/backend-service/apache-tomcat-7.0.29
  Using CATALINA_HOME:   /home/ubuntu/backend-service/apache-tomcat-7.0.29
  Using CATALINA_TMPDIR: /home/ubuntu/backend-service/apache-tomcat-7.0.29/temp
  Using JRE_HOME:        /opt/jdk
  Using CLASSPATH:       /home/ubuntu/backend-service/apache-tomcat-7.0.29/bin/bootstrap.jar:/home/ubuntu/backend-service/apache-tomcat-7.0.29/bin/tomcat-juli.jar
  ubuntu@domU-12-31-39-0B-C5-41:~/backend-service/apache-tomcat-7.0.29/bin$ netstat -na | grep 9000
  tcp        0      0 0.0.0.0:9000            0.0.0.0:*               LISTEN

3. Start the ESB of choice

From the second SSH terminal, start your ESB of choice. For example, the UltraESB can be started as shown below. Wait for the last line shown below stating that the ESB started with the configuration.

ubuntu@domU-12-31-39-0B-C5-41:~/esbs/ultraesb-1.7.1/bin$ ./ultraesb.sh -sample 900
  Increasing heap memory to 2GB for performance test run...
  Starting AdroitLogic UltraESB ...
  Using JAVA_HOME  : /opt/jdk
  Using ULTRA_HOME: /home/ubuntu/esbs/ultraesb-1.7.1
  Reading configuration from : /home/ubuntu/esbs/ultraesb-1.7.1/conf/ | Current directory is : /home/ubuntu/esbs/ultraesb-1.7.1/.
  Root Configuration file : /home/ubuntu/esbs/ultraesb-1.7.1/samples/conf/ultra-sample-900.xml
  ...
  2012-08-07 15:58:35,834 [-] [main]  INFO ConfigurationImpl UltraESB/1.7.1 (GA) - localhost started with root configuration..
  2012-08-07 15:58:35,834 [-] [HttpNIOListener-http-8280]  INFO HttpNIOListener Starting NIO Listener : http-8280 on port : 8280 ...

Note: Refer to the Appendix for detailed instructions on using/starting each ESB

4. Start the Load Generator

The load generation scripts would slightly change for the selected ESB, as some scenarios are skipped for some ESBs. Additionally, JBoss and Petals has customized scripts. To run just a single request through the UltraESB (started above) you could use the "tiny.sh". Use this script (or a variant) to verify one or more scenarios.

ubuntu@domU-12-31-39-0B-C5-41:~$ cd client-scripts/
  ubuntu@domU-12-31-39-0B-C5-41:~/client-scripts$ ./tiny.sh http://localhost:8280/service
  Executing in directory /opt/ultraesb/lib/samples
  Warm-up...
  Running iteration 0

  Server Software:          UltraESB/1.7.1 (GA)
  Server Hostname:          localhost
  Server Port:              8280

  Document Path:            http://localhost:8280/service/DirectProxy
  Average Document Length:  1024 bytes

  Concurrency Level:        1
  Time taken for tests:     0.714896 seconds
  Complete requests:        1
  Failed requests:          0
  Write errors:             0
  Total transferred:        2048 bytes
  Requests per second:      1.40 [#/sec] (mean)
  Time per request:         714.896 [ms] (mean)
  Time per request:         714.896 [ms] (mean, across all concurrent requests)
  Transfer rate:            1.40 [Kbytes/sec] received
  1.40 kb/s sent
  2.80 kb/s total
  ...
  ..

Some ESBs would require freshly created WS-Security requests for the load test. To create these, simply execute the "recreate-secure-requests.sh" as follows when required.

ubuntu@domU-12-31-39-0B-C5-41:~/client-scripts$ ./recreate-secure-requests.sh
  log4j:WARN No appenders could be found for logger (org.adroitlogic.ultraesb.core.helper.XMLSupport).
  log4j:WARN Please initialize the log4j system properly.
  Recreating the secure request 500B_buyStocks_secure.xml from the plain request 500B_buyStocks.xml
  Recreated secure request written to : /opt/ultraesb-1.5.1-SNAPSHOT/samples/resources/requests/500B_buyStocks_secure.xml
  Recreating the secure request 1K_buyStocks_secure.xml from the plain request 1K_buyStocks.xml
  Recreated secure request written to : /opt/ultraesb-1.5.1-SNAPSHOT/samples/resources/requests/1K_buyStocks_secure.xml
  Recreating the secure request 5K_buyStocks_secure.xml from the plain request 5K_buyStocks.xml
  Recreated secure request written to : /opt/ultraesb-1.5.1-SNAPSHOT/samples/resources/requests/5K_buyStocks_secure.xml
  Recreating the secure request 10K_buyStocks_secure.xml from the plain request 10K_buyStocks.xml
  Recreated secure request written to : /opt/ultraesb-1.5.1-SNAPSHOT/samples/resources/requests/10K_buyStocks_secure.xml
  Recreating the secure request 100K_buyStocks_secure.xml from the plain request 100K_buyStocks.xml
  Recreated secure request written to : /opt/ultraesb-1.5.1-SNAPSHOT/samples/resources/requests/100K_buyStocks_secure.xml

The "smoketest.sh" is script would issue 1000 requests over 20 concurrent connections. You may use this (or a variant of it) as well to test a particular ESB before running the full load test. When you are ready to run the load test, execute the "loadtest.sh" script, and redirect the output to a text log file as follows:

ubuntu@domU-12-31-39-0B-C5-41:~/client-scripts$./loadtest.sh http://localhost:8280/service > ultraesb-1.7.1-vanilla.txt

5. Monitoring the raw output

Use the sessions # 4 and # 5 to monitor the output from the load generator

ubuntu@domU-12-31-39-0B-C5-41:~$ cd client-scripts/
  ubuntu@domU-12-31-39-0B-C5-41:~/client-scripts$ tail -f ultraesb-1.7.1-vanilla.txt

and

ubuntu@domU-12-31-39-0B-C5-41:~$ cd client-scripts/
  ubuntu@domU-12-31-39-0B-C5-41:~/client-scripts$ tail -f ultraesb-1.7.1-vanilla.txt | egrep '(Failed|errors)'

Monitor the output for any errors or issues - e.g. the ESB getting stuck during the load test. A single iteration of the load test could take around an hour or more to complete for one ESB. For the final analysis in Round 6, we used a script that executed the full load test three times sequentially.

6. Processing Results

Note that the scripts you use are actually symbolic links to an UltraESB installation placed at /opt/ultraesb. This is because the UltraESB ships the ESB performance benchmark suite with its distribution. Once you capture a full execution log, use the ab-to-csv.jar to convert the output into CSV format for analysis by spreadsheet packages.

ubuntu@domU-12-31-39-0B-C5-41:~/client-scripts$ java -jar /opt/ultraesb/samples/bin/ab-to-csv.jar ultraesb-1.7.1-vanilla.txt > ultraesb-1.7.1-vanilla.csv

Refer to the Google Spreadsheet we created for our analysis here.

Once you complete the performance test, be sure to terminate the EC2 instances - as otherwise, you would end up paying for the time the instance is running idle.

7. Appendix

7.1. Testing the UltraESB v1.7.1 - Vanilla

Refer the ~/resources/esbperformance/ultraesb/README.TXT file for full details

To start the ESB, execute "./ultraesb.sh -sample 900" from the "~/esbs/ultraesb-1.7.1/bin" director

ubuntu@domU-12-31-39-0B-C5-41:~/esbs/ultraesb-1.7.1/bin$ ./ultraesb.sh -sample 900

The services will be hosted under the URL : http://localhost:8280/service

To execute the load test, execute the load generator as follows from the "~/client-scripts" directory

./loadtest.sh http://localhost:8280/service > ultraesb-1.7.1-vanilla.txt

7.2. Testing the UltraESB v1.7.1 - Enhanced

To further enhance the XML processing performance of the UltraESB, the Saxon HE and VTD XML (GPL) libraries maybe made available by copying these JAR files into lib/custom folder, and restarting the UltraESB. Note that due to GPL licensing limitations, we do NOT ship these libraries with the UltraESB or the ESB Performance Benchmark EC2 AMI. An end user can freely download these libraries without limitation as follows:

e.g

$ cd ~/esbs/ultraesb-1.7.1/lib/custom
  $ wget http://downloads.adroitlogic.com/maven2/saxon/saxon-he/9.2/saxon-he-9.2.jar
  $ wget https://bitbucket.org/asankha/vtdfastxml/downloads/vtdfastxml-1.0.0.jar

Next download the VTD XML 2.10 distribution from http://sourceforge.net/projects/vtd-xml/files/ and extract it. And copy the JAR file into lib/custom

$ cd /tmp
  $ wget http://nchc.dl.sourceforge.net/project/vtd-xml/vtd-xml/ximpleware_2.10/ximpleware_2.10_java.zip
  $ unzip ximpleware_2.10_java.zip
  $ cp ximpleware_2.10_java/vtd-xml.jar /home/ubuntu/esbs/ultraesb-1.7.1/lib/custom/

Follow the same steps as for the "Vanilla" version after downloading the following three libraries to the ~/esbs/ultraesb-1.7.1/lib/custom directory

7.3. Testing the WSO2 ESB v4.0.3

Refer the ~/resources/esbperformance/wso2-esb/README.TXT file for full details to build from source and configure the ESB. 

To start the ESB, execute "./wso2server.sh" from the "~/esbs/wso2esb-4.0.3/bin" directory

ubuntu@domU-12-31-39-0B-C5-41:~/esbs/wso2esb-4.0.3/bin$ ./wso2server.sh

The services will be hosted under the URL : http://localhost:8280/services

To execute the load test, execute the load generator as follows from the "~/client-scripts" directory

./loadtest.sh http://localhost:8280/services > wso2-4.0.3.txt

7.4. Testing the Mule ESB CE v3.3.0

Refer the ~/resources/esbperformance/mule/README.TXT file for full details to build from source and configure the ESB. 

To start the ESB, execute "bin/mule" from the "~/esbs/mule-standalone-3.3.0" directory (Note: you must execute from the home/parent directory and not the bin directory)

ubuntu@domU-12-31-39-0B-C5-41:~$ cd esbs/mule-standalone-3.3.0/
  ubuntu@domU-12-31-39-0B-C5-41:~/esbs/mule-standalone-3.3.0$ bin/mule

The services will be hosted under the URL : http://localhost:8090/service

To execute the load test, execute the load generator as follows from the "~/client-scripts" directory

./loadtest.sh http://localhost:8090/service > mule-3.3.0.txt

7.5. Testing the Talend ESB SE 5.1.1

Refer the ~/resources/esbperformance/talend/README.TXT file for full details to build from source and configure the ESB. 

To start the ESB, execute "bin/tesb" from the "~/esbs/talend-esb-4.2/container" directory

ubuntu@domU-12-31-39-0B-C5-41:~$ cd esbs/talend-esb-se-5.1.1/container/
  ubuntu@domU-12-31-39-0B-C5-41:~/esbs/talend-esb-se-5.1.1/container$ ./bin/trun

The services will be hosted under the URL : http://localhost:9080/service
To execute the load test, execute the load generator as follows from the "~/client-scripts" directory

./loadtest.sh http://localhost:9080/service > talend-5.1.1.txt

7.6. Testing the JBoss ESB v4.11

Refer the ~/resources/esbperformance/jboss/README.txt file for full details to build from source and configure the ESB. 

To start the ESB, execute "run.sh" from the "~/esbs/jbossesb-server-4.11/bin" directory

ubuntu@domU-12-31-39-0B-C5-41:~$ cd esbs/jbossesb-server-4.11/bin
  ubuntu@domU-12-31-39-0B-C5-41:~/esbs/jbossesb-server-4.11/bin$ ./run.sh

The services will be hosted as follows:

./loadtest-jbossesb.sh > jbossesb-4.11.log

7.7. Testing the Apache ServiceMix ESB v4.3.0

Refer the ~/resources/esbperformance/servicemix/README.TXT file for full details to build from source and configure the ESB.

To start the ESB, execute "servicemix" from the "~/esbs/apache-servicemix-4.3.0/bin" directory

ubuntu@domU-12-31-39-0B-C5-41:~$ cd esbs/apache-servicemix-4.3.0/bin
  ubuntu@domU-12-31-39-0B-C5-41:~/esbs/apache-servicemix-4.3.0/bin$ ./servicemix

The services will be hosted under the URL : http://localhost:8192/service

To execute the load test, execute the load generator as follows from the "~/client-scripts" directory

./loadtest.sh http://localhost:8192/service > servicemix-4.3.0.txt

7.8. Testing the Fuse ESB v4.4.0

Refer the ~/resources/esbperformance/fuse/README.TXT file for full details to build from source and configure the ESB. 

To start the ESB, execute "servicemix" from the "~/esbs/apache-servicemix-4.4.1-fuse-07-11/bin" directory

ubuntu@domU-12-31-39-0B-C5-41:~$ cd esbs/apache-servicemix-4.4.1-fuse-07-11/bin
  ubuntu@domU-12-31-39-0B-C5-41:~/esbs/apache-servicemix-4.4.1-fuse-07-11/bin$ ./servicemix

The services will be hosted under the URL : http://localhost:8182/service

To execute the load test, execute the load generator as follows from the "~/client-scripts" directory

./loadtest.sh http://localhost:8182/service > fuse-4.4.1.txt

7.9. Testing the Petals ESB v4.1.0

Refer the ~/resources/esbperformance/petals/README.TXT file for full details to build from source and configure the ESB. 

To start the ESB, execute "petals-esb.sh" from the "~/esbs/petals-esb-distrib-4.1.0/esb/petals-esb-4.1/bin" directory

ubuntu@domU-12-31-39-0B-C5-41:~$ cd esbs/petals-esb-distrib-4.1.0/esb/petals-esb-4.1/bin/
  ubuntu@domU-12-31-39-0B-C5-41:~/esbs/petals-esb-distrib-4.1.0/esb/petals-esb-4.1/bin$ ./petals-esb.sh

The services will be hosted under:

To execute the load test, execute the load generator as follows from the "~/client-scripts" directory

./loadtest-petals.sh  > petals-4.1.0.txt
Contact Us