1. Starting the node on EC2

From your web browser, navigate to https://console.aws.amazon.com/ec2/home and login with your EC2 account credentials to the AWS Console. Navigate to the "EC2" tab. From its left navigation pane, and select "AMIs". From the right hand pane, select "Public Images" from the Viewing drop down list, and specify the AMI ID "ami-49428f20" for filtering and press the enter key. The AMI will be shown in the results table.

ec2 console

Right click the result row, and select "Launch Instance" or "Request Spot Instance" as you prefer, noting the slight difference in charges, and the possibility of the node being taken down during the test by Amazon, if your bid falls below the rate at the time of execution for a spot instance. We will use a spot instance with a bid price of $0.36 on the us-east-1c availability zone which where this bid price is generally safe as of this writing. As the instance type select "High-CPU Extra Large (c1.xlarge, 7GB) and specify your bid price.

The next set of options could be left at their defaults if your Amazon account is configured for EC2 use already - i.e. with a keypair, security group allowing SSH etc. For any help regarding these steps, refer the Amazon EC2 documentation or Help. Once you submit the spot instance request, navigate to the "Instances" from the left menu. You should "refresh" the list until you see your instance up and running. Once the new instance appears, right click on the row, and select "Connect"

ec2 connect

Then you will be presented with the SSH command to connect. This will look like the following "ssh -i adroitlogic.pem root@ec2-50-16-108-172.compute-1.amazonaws.com" Note that the default login is set to "root". To actually connect, replace this with "ubuntu" and issue an SSH command as follows:

ssh -i adroitlogic.pem ubuntu@ec2-50-16-108-172.compute-1.amazonaws.com

The key file name will change depending on your Amazon credentials. On login, you will be presented with the legal information about the AMI image. Open about 6 ssh sessions to the node for use as follows:

  • Session 1 - Run Echo Service - mock backend service used by the ESBs

  • Session 2 - Run the ESB of choice

  • Session 3 - Run the load generator script, redirecting output to a file

  • Session 4 - Monitor raw output of load generator

  • Session 5 - Monitor errors and failures via load generator output

  • Session 6 - Use to build ESB configurations and deploy / configure artifacts

2. Execute the Echo Service Back-end service

On the first SSH session, execute the echo service as shown below

ubuntu@domU-12-31-38-04-8C-71:~$ cd backend-service/
ubuntu@domU-12-31-38-04-8C-71:~/backend-service$ ./start-echo-service.sh
Starting AdroitLogic UltraESB ToolBox ...
Using JAVA_HOME  : /opt/jdk
Using ULTRA_HOME: /opt/ultraesb
Starting sample Echo server on port 9000...

3. Start the ESB of choice

The UltraESB can be started as shown below. Wait for the last line shown below stating that the ESB started with the configuration.

ubuntu@domU-12-31-38-04-8C-71:~$  cd esbs/ultraesb-1.6.0/bin
ubuntu@domU-12-31-38-04-8C-71:~/esbs/ultraesb-1.6.0/bin$ ./ultraesb.sh -sample 900
Increasing heap memory to 2GB for performance test run...
Starting AdroitLogic UltraESB ...
Using JAVA_HOME  : /opt/jdk
Using ULTRA_HOME: /home/ubuntu/esbs/ultraesb-1.6.0
Reading configuration from : /home/ubuntu/esbs/ultraesb-1.6.0/conf/ | Current directory is : /home/ubuntu/esbs/ultraesb-1.6.0/.
Root Configuration file : /home/ubuntu/esbs/ultraesb-1.6.0/samples/conf/ultra-sample-900.xml
...
2011-10-07 18:00:57,041 [-] [main]  INFO ServerManager Starting transport listeners
2011-10-07 18:00:57,054 [-] [main]  INFO ConfigurationImpl UltraESB/1.6.0 (GA) - localhost started with root configuration..

Note: Refer to the Appendix for detailed instructions on using/starting each ESB

4. Start the Load Generator

The load generation scripts would slightly change for the selected ESB, as some scenarios are skipped for some ESBs. Additionally, JBoss and Petals has customized scripts. To run just a single request through the UltraESB (started above) you could use the "tiny.sh". Use this script (or a variant) to verify one or more scenarios.

ubuntu@domU-12-31-38-04-8C-71:~$ cd client-scripts/
ubuntu@domU-12-31-38-04-8C-71:~/client-scripts$ ./tiny.sh http://localhost:8280/service
Executing in directory /opt/ultraesb/lib/samples
Warm-up...
Running iteration 0

Server Software:          UltraESB/1.6.0 (GA)
Server Hostname:          localhost
Server Port:              8280

Document Path:            http://localhost:8280/service/DirectProxy
Average Document Length:  1024 bytes

Concurrency Level:        1
Time taken for tests:     0.521454 seconds
Complete requests:        1
Failed requests:          0
Write errors:             0
Total transferred:        2048 bytes
Requests per second:      1.92 [#/sec] (mean)
Time per request:         521.454 [ms] (mean)
Time per request:         521.454 [ms] (mean, across all concurrent requests)
Transfer rate:            1.92 [Kbytes/sec] received
                          1.92 kb/s sent
                          3.84 kb/s total
...
..

Some ESBs would require freshly created WS-Security requests for the load test. To create these, simply execute the "recreate-secure-requests.sh" as follows when required.

ubuntu@domU-12-31-38-04-8C-71:~/client-scripts$ ./recreate-secure-requests.sh
log4j:WARN No appenders could be found for logger (org.adroitlogic.ultraesb.core.helper.XMLSupport).
log4j:WARN Please initialize the log4j system properly.
Recreating the secure request 500B_buyStocks_secure.xml from the plain request 500B_buyStocks.xml
Recreated secure request written to : /opt/ultraesb-1.5.1-SNAPSHOT/samples/resources/requests/500B_buyStocks_secure.xml
Recreating the secure request 1K_buyStocks_secure.xml from the plain request 1K_buyStocks.xml
Recreated secure request written to : /opt/ultraesb-1.5.1-SNAPSHOT/samples/resources/requests/1K_buyStocks_secure.xml
Recreating the secure request 5K_buyStocks_secure.xml from the plain request 5K_buyStocks.xml
Recreated secure request written to : /opt/ultraesb-1.5.1-SNAPSHOT/samples/resources/requests/5K_buyStocks_secure.xml
Recreating the secure request 10K_buyStocks_secure.xml from the plain request 10K_buyStocks.xml
Recreated secure request written to : /opt/ultraesb-1.5.1-SNAPSHOT/samples/resources/requests/10K_buyStocks_secure.xml
Recreating the secure request 100K_buyStocks_secure.xml from the plain request 100K_buyStocks.xml
Recreated secure request written to : /opt/ultraesb-1.5.1-SNAPSHOT/samples/resources/requests/100K_buyStocks_secure.xml

The "smoketest.sh" is script would issue 1000 requests over 20 concurrent connections. You may use this (or a variant of it) as well to test a particular ESB before running the full load test. When you are ready to run the load test, execute the "loadtest.sh" script, and redirect the output to a text log file as follows:

./loadtest.sh http://localhost:8280/service > ultraesb-1.6.0-vanilla.txt

5. Monitoring the raw output

Use the sessions # 4 and # 5 to monitor the output from the load generator

ubuntu@domU-12-31-38-04-8C-71:~$  cd client-scripts/
ubuntu@domU-12-31-38-04-8C-71:~/client-scripts$ tail -f ultraesb-1.6.0-vanilla.txt

and

ubuntu@domU-12-31-38-04-8C-71:~$ cd client-scripts/
ubuntu@domU-12-31-38-04-8C-71:~/client-scripts$ tail -f ultraesb-1.6.0-vanilla.txt | egrep '(Failed|errors)'

Monitor the output for any errors or issues - e.g. the ESB getting stuck during the load test. The complete load test could take around an hour or more to complete for one ESB.

6. Processing Results

Note that the scripts you use are actually symbolic links to an UltraESB installation placed at /opt/ultraesb. This is because the UltraESB ships the ESB performance benchmark suite with its distribution. Once you capture a full execution log, use the ab-to-csv.jar to convert the output into CSV format for analysis by spreadsheet packages.

ubuntu@domU-12-31-38-04-8C-71:~/client-scripts$ java -jar /opt/ultraesb/samples/bin/ab-to-csv.jar ultraesb-1.6.0-enhanced.txt > ultraesb-1.6.0-enhanced.csv

Refer to the Google Spreadsheet we created for our analysis here.

Once you complete the performance test, be sure to terminate the EC2 instances - as otherwise, you would end up paying for the time the instance is running idle.

7. Appendix

7.1. Testing the UltraESB v1.6.0 - Vanilla

Refer the ~/resources/esbperformance/ultraesb/README.TXT file for full details

To start the ESB, execute "./ultraesb.sh -sample 900" from the "~/esbs/ultraesb-1.6.0/bin" directory

ubuntu@domU-12-31-38-04-8C-71:~/esbs/ultraesb-1.6.0/bin$ ./ultraesb.sh -sample 900

The services will be hosted under the URL : http://localhost:8280/service

To execute the load test, execute the load generator as follows from the "~/client-scripts" directory

./loadtest.sh http://localhost:8280/service > ultraesb-1.6.0-vanilla.txt

7.2. Testing the UltraESB v1.6.0 - Enhanced

To further enhance the XML processing performance of the UltraESB, the Saxon HE and VTD XML (GPL) libraries maybe made available by copying these JAR files into lib/custom folder, and restarting the UltraESB. Note that due to GPL licensing limitations, we do NOT ship these libraries with the UltraESB or the ESB Performance Benchmark EC2 AMI. An end user can freely download these libraries without limitation as follows:

e.g

$ cd ~/esbs/ultraesb-1.6.0/lib/custom
$ wget http://downloads.adroitlogic.com/maven2/saxon/saxon-he/9.2/saxon-he-9.2.jar
$ wget https://bitbucket.org/asankha/vtdfastxml/downloads/vtdfastxml-1.0.0.jar

Next download the VTD XML 2.10 distribution from http://sourceforge.net/projects/vtd-xml/files/ and extract it. And copy the JAR file into lib/custom

$ cd /tmp
$ wget http://nchc.dl.sourceforge.net/project/vtd-xml/vtd-xml/ximpleware_2.10/ximpleware_2.10_java.zip
$ unzip ximpleware_2.10_java.zip
$ cp ximpleware_2.10_java/vtd-xml.jar /home/ubuntu/esbs/ultraesb-1.6.0/lib/custom/

Follow the same steps as for the "Vanilla" version after downloading the following three libraries to the ~/esbs/ultraesb-1.6.0/lib/custom directory

7.3. Testing the WSO2 ESB v4.0.0

Refer the ~/resources/esbperformance/wso2-esb/README.TXT file for full details to build from source and configure the ESB. 

To start the ESB, execute "./wso2server.sh" from the "~/esbs/wso2esb-4.0.0/bin" directory

ubuntu@domU-12-31-38-04-8C-71:~/esbs/wso2esb-4.0.0/bin$ ./wso2server.sh

The services will be hosted under the URL : http://localhost:8280/services

To execute the load test, execute the load generator as follows from the "~/client-scripts" directory

./loadtest.sh http://localhost:8280/services > wso2-4.0.0.txt

7.4. Testing the Mule ESB CE v3.2.0

Refer the ~/resources/esbperformance/mule/README.TXT file for full details to build from source and configure the ESB. 

To start the ESB, execute "bin/mule" from the "~/esbs/mule-standalone-3.2.0" directory (Note: you must execute from the home/parent directory and not the bin directory)

ubuntu@domU-12-31-38-04-8C-71:~$ cd ~/esbs/mule-standalone-3.2.0
ubuntu@domU-12-31-38-04-8C-71:~/esbs/mule-standalone-3.2.0$ bin/mule

The services will be hosted under the URL : http://localhost:8090/service

To execute the load test, execute the load generator as follows from the "~/client-scripts" directory

./loadtest.sh http://localhost:8090/service > mule-3.2.0.txt

7.5. Testing the Talend ESB

Refer the ~/resources/esbperformance/talend/README.TXT file for full details to build from source and configure the ESB. 

To start the ESB, execute "bin/tesb" from the "~/esbs/talend-esb-4.2/container" directory

ubuntu@domU-12-31-38-04-8C-71:~/esbs$ cd talend-esb-4.2/container/
ubuntu@domU-12-31-38-04-8C-71:~/esbs/talend-esb-4.2/container$ bin/tesb

The services will be hosted under the URL : http://localhost:9080/service
To execute the load test, execute the load generator as follows from the "~/client-scripts" directory

./loadtest.sh http://localhost:9080/service > talend-4.2.txt

7.6. Testing the JBoss ESB v4.10

Refer the ~/resources/esbperformance/jboss/README.txt file for full details to build from source and configure the ESB. 

To start the ESB, execute "run.sh" from the "~/esbs/jbossesb-server-4.10/bin" directory

ubuntu@domU-12-31-38-04-8C-71:~/esbs$ cd jbossesb-server-4.10/bin/
ubuntu@domU-12-31-38-04-8C-71:~/esbs/jbossesb-server-4.10/bin$ ./run.sh

The services will be hosted as follows:

./loadtest-jbossesb.sh > jbossesb-4.10.log

7.7. Testing the Apache ServiceMix ESB v4.3.0

Refer the ~/resources/esbperformance/servicemix/README.TXT file for full details to build from source and configure the ESB.

Although the AMI contains the ESB instances we’ve used, it is almost certain that you will not be able to restart the ServiceMix instance easily, as there is a dependency on first starting the Direct proxy scenario. The easiest way to overcome this is to create a temporary directory, and moce the artifacts from the ServiceMix "deploy" directory into it, start ServiceMix, and then first move the direct proxy artifact back to the deploy directory followed by the rest.

ubuntu@domU-12-31-38-04-8C-71:~$ cd esbs/apache-servicemix-4.3.0/deploy/
ubuntu@domU-12-31-38-04-8C-71:~/esbs/apache-servicemix-4.3.0/deploy$ mkdir /tmp/smd
ubuntu@domU-12-31-38-04-8C-71:~/esbs/apache-servicemix-4.3.0/deploy$ mv * /tmp/smd

To start the ESB, execute "servicemix" from the "~/esbs/apache-servicemix-4.3.0/bin" directory

ubuntu@domU-12-31-38-04-8C-71:~$ cd ~/esbs/apache-servicemix-4.3.0/bin
ubuntu@domU-12-31-38-04-8C-71:~/esbs/apache-servicemix-4.3.0/bin$ ./servicemix

Now copy the artifacts back to the deploy directory, starting from the proxy

ubuntu@domU-12-31-38-04-8C-71:~/esbs/apache-servicemix-4.3.0/deploy$ mv /tmp/smd/proxy-sa-1.0.zip .
ubuntu@domU-12-31-38-04-8C-71:~/esbs/apache-servicemix-4.3.0/deploy$ mv /tmp/smd/* .

The services will be hosted under the URL : http://localhost:8192/service

To execute the load test, execute the load generator as follows from the "~/client-scripts" directory

./loadtest.sh http://localhost:8192/service > servicemix-4.3.0.txt

7.8. Testing the Fuse ESB v4.4.0

Refer the ~/resources/esbperformance/fuse/README.TXT file for full details to build from source and configure the ESB. 

To start the ESB, execute "servicemix" from the "~/esbs/apache-servicemix-4.4.0-fuse-00-43/bin" directory

ubuntu@domU-12-31-38-04-8C-71:~$ cd ~/esbs/apache-servicemix-4.4.0-fuse-00-43/bin
ubuntu@domU-12-31-38-04-8C-71:~/esbs/apache-servicemix-4.4.0-fuse-00-43/bin$ ./servicemix

The services will be hosted under the URL : http://localhost:8182/service

To execute the load test, execute the load generator as follows from the "~/client-scripts" directory

./loadtest.sh http://localhost:8182/service > fuse-4.4.0.txt

7.9. Testing the Petals ESB v3.1.1

Refer the ~/resources/esbperformance/petals/README.TXT file for full details to build from source and configure the ESB. 

To start the ESB, execute "petals-esb console" from the "~/esbs/petals-platform-3.1.3/bin" directory

ubuntu@domU-12-31-38-04-8C-71:~$ cd ~/esbs/petals-platform-3.1.3/bin
ubuntu@domU-12-31-38-04-8C-71:~/esbs/petals-platform-3.1.3/bin$ ./petals-esb console

The services will be hosted under:

To execute the load test, execute the load generator as follows from the "~/client-scripts" directory

./loadtest-petals.sh  > petals-3.1.1.txt

Back to Home Page