Fun with Dynamics CRM 2011 Performance Issues

Gayan Perera, 31 May 2012

As we near completion of a global CRM 4 to CRM 2011 upgrade rollout we hit a performance problem with CRM 2011.

Fun began when we couldn’t reproduce these performance issues in our development or test environments which ran on 2GHz 4GB RAM 160GB SSD machines.

Now that we have pinpointed the causes of the issue we’d like to share our results with the community so that anyone who comes across a similar issue can hopefully resolve it much faster.

Multiple factors aided the performance issue and when combined caused catastrophic failures. Factors were:

Riverbed WAN Accelerator

Unfortunately the default configuration of a Riverbed WAN accelerator does more harm than good. By default it strips authentication and strips the compression headers from requests. This tells the IIS server to send uncompressed content. Riverbed does this is to try and optimize/cache data.

What Riverbed doesn’t realize is that when this happens Internet Explorer doesn’t cache the data locally. For example: ribbon, grid and general JavaScript.

Let’s say you open an account in CRM for the first time, IE will make 90+ requests, and out of the 90 about 80 will be cached locally by IE. Therefor upon opening a different account you’ll only notice IE making 8-10 requests. However with the Riverbed you’ll see 25-40 requests made by IE every single time. To make matters worse ribbons, grids and general JavaScript failures occur.

Red: Default Riverbed settings (compression stripping + authentication stripping)
Green: NO compression stripping, NO authentication stripping

As you can see when the default Riverbed settings were applied; 8x more data went through the LAN. WAN traffic was about the same but the biggest issue when Riverbed was on auto/default is that it caused IE to not cache data locally and caused CRM 2011 to crash.

The fix: Turn OFF authentication stripping and compression stripping. See Riverbed settings below:

NeoLoad Test Configuration

Performance tests were done using NeoLoad. There were two issues with this. First was that the test was recorded inside the local network through a proxy. When the remote test agent ran the test it re-ran the pre-recorded test with the proxy settings that were used during the recording phase causing additional round trips which added extra seconds skewing the test results.

Second issue was that the test included new functionality introduced in CRM 2011, e.g.: Dashboards & Fetch Reports. The results were compared against CRM 4 which didn’t have these features therefor again skewing the results.

Hardware

The production environment runs on HP Blades Dual Quad Core 2.53GHz 96GB RAM IBM SAN with ~22x 15K SAS drives but there was an issue with that which the infrastructure team picked up. We found this out by running Roadkil Disk Speed. Some say that this util is not very effective on virtual machines but it’ll give a pretty good indication J

The Final Results  

As you can see from the above results, the best configuration is to have Riverbed ON without compression stripping and without authentication stripping.

*Image from http://livinteractive.com/wp-content/uploads/pulling-hair-out.jpg