YUI’sing Tsung to choose your CDN

An introduction to Tsung

Tsung is an open-source multi-protocol distributed load testing tool, written in …you guessed it erlang, by the folks at process-one ( they also contribute heavily to multiple erlang projects, and infact maintain ejabberd, etc ).

I built Tsung 1.3.0 from source on erlang R13A ( built from source, catch the earlier post on first impressions ) on an AMD 64-bit , 4 GB Ram machine running ubuntu. Beebole has a nice intro on how to setup Tsung. The only key hints I have - you can avoid tsung recorder , and straight away copy sample tsung config files from /usr/local/share/tsung/examples/http-simple.xml to $HOME/.tsung/tsung.xml , and edit/tweak them. Once set , running tsung start, should show the log directory that it references.

We needed to estimate and gauge how much load the quad-code servers at hover.in could take . To start off I ‘m simulating static file requests, and you can have a lot of control over what kind of requests you can to simulate .So I could have different set of user visit frequencies - like one set of requests at x frequency from M browser asking for url A, while another set of faster frequency y , form I browser asking for url B.

While it may take some time to figure out where you need to tweak you app/server ,since there are several areas that your LYME app can be optimized :

• kernel-polling , and other erlang VM optimizations
• size of log files , whether gzip is enabled in yaws.conf (through deflate = true )
  

  yaws with & without gzip ( deflate flag in yaws.conf against each server tag)

• max process’s set for the VM
• mnesia specific - the dump_log_time_threshold, table loading timeouts, etc
• yaws even has some other flags that you can insert into yaws.conf such as max_num_cached_files , max_size_cached_file, etc

So…. let’s go server some static files ! And what better way to test your load testing tool than to test it against various different content delivery networks out there. Now it’s a pity that I’m only running this on one node - will definitely work on using Tsung on a cluster soon.

Load-testing the YUI library across various networks

I decided to simulate that for a period of 10 minutes, a request would be made to a static file every 0.1.seconds!

<!-- tsung.xml -->
.
.
<load>
<!-- several arrival phases can be set: for each phase, you can set
       the mean inter-arrival time between new clients and the phase duration -->
       <arrivalphase phase="1" duration="10" unit="minute" >
            <users interarrival="0.1" unit="second"><users>
       </arrivalphase>
</load>
.
.

You can even give a combination of different sessions, of varying probability ( their summation being 100% ). But since i wanted to test only the single yahoo minified javascript file, a added a single session of 100% probability.

.
.
<sessions>
     <session name="hit1"  probability="100" type="ts_http" >
          <request><http url="http://full.path.to.my.static.file.com/ymin-2-4-1.js"
                   method="get" version="1.1" if_modified_since="Fri, 14 Jan 2009 02:043:31 GMT"></http>
         </request>
    </session>
</sessions>
.
.

The experiment

Our favourite javascript library has been the YUI hands down for the foresight, documentation and great developer community for the past many years. So we tried serving the same YUI javascript base file from 5 different networks, test their response times, effects of of hundreds of simultaneous users , and then to analyze and compare their performances, pro’s and con’s of :

1. Yaws server 1.77 ,running on erlang 12-B
2. Amazon S3
3. Amazon CloudFront
4. Yahoo’s YUI hosting
5. Google’s javascript library hosting ( ~little difference since google hosts only from 2.6.0 while the rest were 2.4.1 )

Results / Main statistics

The statistics that comes with tsung coupled with perl graphing magic, gives you some comprehensive statistics. In order not to be biased , I even took two trials of load -testing the yahoo yui library against four networks / or content delivery networks if you will, which we at hover have tested at some point. While firebug definitely gives you the end-user perspective , time to load, etc - you can have a look at the latency and network effects , and couple of stat’s that i’ve compiled ( tsung gives a much more detailed view btw) , and make your own judgements for yourself.

Simultaneous Users per second

1. hover yaws

   Amd 64-bit quad core machine ,4GB RAM ( extra un-used node that we’re removing as part of our see-how-much-you-can-get-out-of-one-machine scaling attempts ). We felt that for site-specific js/static files ,etc - we need’nt look to a CDN. The only thing you need to tweak around, ( and we seem to be getting loads of errors on yaws is that of possibly insufficinet sockets , when the load moves beyong 2-3 requests per second from our experiences. It infact increased when enabling gzip, but there may be other reasons. Anyhow tsung should help crash test our local dev servers long enough , and slow enough for us to figure out where the bottlenecks really are.

   

   hover yaws 1

   

   hover yaws 2

   ---

2. Amazon s3

   Reading through the dynamo paper gives some nice insights into their architecture, and we never had any issues with them. Cheap & effective . Only qualm is that they don’t gzip files ( but I hear that you can push gzipped content into S3 through few hacks )

   

   S3 1

   

   S3 2

   ---

3. Amazon CloudFront

   Our move to amazon cloudfront has had drastically good results with static file serving. One reason I might quit CloudFront though is that the files when pushed to S3, seem to be inconsistent across data centers. Although the lame way would be to keep giving new filenames, ( which by the way deleting from S3, doesnt delete from cloudfront) Simply not acceptable. ( version 1 from asia, version 2 from the US ,etc for the same file at the same time ) . We’ve beein doing upwards of 25 million GET requests from Cloudfront last month. Overall, Cloudfront steals the thunder - since it’s most likely the only edge network serving the static file among the other networks.

   

   Cloudfront 1

   

   CloudFront 2

   ---

4. yahoo’s free static content hosting

   They’ve been offering free hosting for something for various releases, and the free hosting is definitely worth it - be it for quick prototyping, or spanking new web apps.

   

   yahoo apis 1

   

   yahoo 2

   ---

5. google free static content hosting

   YUI was added recently to part of Google’s ajax api’s iniatives. The javascript needs to be lazy loaded using javascript though. Doubt that they’re edge networks.

   

   googleapis.com 1

   

   google 2

And another compilation of number of HTTP status responses per second , again two trials per network.

HTTP Status Response Times

Conclusion

What’s interesting is that the figures indicate that the Amazon servers score less on handling concurrent users , and more on combating latency. But that’s quite understandable considering that every S3 object needs to be fetched from atleast 2 nodes while reading , and a conflict resolution is done during the reads, not the writes. ( they try to avoid neglecting writes from any subsystem at any cost ). But the increased response times from google , yahoo and not to forget - our very own hover server running on yet another web server - could be justified by the lesser number of requests being served by the respective CDN’s when compared to Amazon. What also interesting to see was that the yahoo api servers seem to consistently give higher throughput from their YTS/1.17.8 server.

That said, static files are just half the story, I’d love to benchmark more libraries when we get the time, but next up might be benchmarking images , css , and more dynamic content - possibly a comparison of distributed hash stores, cache’s and the like.

~B