There are a few phases that companies go through with analytics. The first phase is misunderstanding and frustration. Starting with no understanding of purpose or capability of analytics they move swiftly into paralysis by analysis; they’re drowning in so much data they don’t know what is relevant and what isn’t. Soon, an agency steps in and simplifies everything: pick 4-6 good KPIs and look at trends. Look at top-level numbers first, and then delve deeper only as needed. Companies get so good at this they can literally evaluate and improve their sites down to fractions of a percent, which can be big money when you’re talking about 9-figure sales. But there is a third phase: obsession, compulsion, and micro-analysis, where small quantifiable decisions are given exaggerated importance. This is a time to walk away from the charts, and many of us have been there.

Tonight, I attended the Atlanta Interactive Marketing Association (AiMA) monthly event. Tonight’s talk concerned the important of page layout and optimization to improve conversion rates. All three case studies had similarities. All presenters were from Fortune 500 companies — Delta, InterContinental Hotels Group, and Budget/Avis — who book hundreds of millions (billions actually) in revenues through their ecommerce sites.

The presenters all had unique problems and solid systems in terms of tools, procedures, and adequate staffing to monitor site changes. They had the right monitoring tools and diagnostics in place and know when to consult them for troubleshooting and gauging performance.  They also knew when to optimize their site and when to reach out to other areas, such as partner integration, knowing their customers needs and wants, and simply keeping in front of them. They all knew optimization is not a magic bullet, but a lot of folks make this mistake.

A lot of webmasters who have moved past paralysis and simple KPI’s engage in some form of split testing (A/B or multivariate) to tweak and test seemingly insignificant changes on a page. What works better, a red submit button or a blue one? Should the welcome line be bold? Do the words “act now!” or “Buy Now!” get a higher conversion rate? Tweaking element after element on the page to statistical significance reveals what works better, and marginal improvements add up. All 3 presenters had personal accounts of tweaking and re-tweaking page elements that might yield a 2% conversion rate improvement… which translated to an extra $20 million in annual revenues booked online.

What marketer wouldn’t love this! What company doesn’t need this? You make some seemingly insignifant changes, quantify them and see the needle move slightly, and you can take credit for moving the business forward. Who wants to leave extra money on the table?

I’m not attempting to debunk this. Analytics are a Godsend to those of us who remember the web when the only choice was page hits and server logs. Numbers don’t lie, they reveal the blantanly obvious but overlooked problems. They solve arguments between team members. And they help us keep our jobs by indisputably proving our worth to the company.

But you can definitely go too far. First, I am skeptical about how the industry vendors are training companies to write their scorecards. There’s a few different ways the current wisdom goes, but here’s a simple example: start with your primary web metric, such as sales dollars or leads. Calculate average sale (dollars) or average lead value; if you don’t know what an average lead is worth, ballpark it by taking total annual sales divided by total leads.

Let’s say your average sale (or average lead value) is $100. Your conversion rate was 1%, and you’ve increased it to 1.1%, an increase of 10% (nice job, by the way).  This is just for a single sale, and you have 10,000 site visitors a day. Web consultants like to extrapolate it out to see the total value of the percentage increase. Careful though, this isn’t a forecast of what you’ll actually sell. Ask the finance guy for that.  Here’s the web analyst’s simple extrapolation of your impact to the bottom line:

BEFORE: Conversion rate at 1% 
10,000 visitors  x  .01 (conversion rate) = 100 sales x ($100 avg. sale) = $10,000/day

AFTER: Conversion rate at 1.1% (as you can guess, the revenue should be 10% higher)
10,000 visitors  x  .011 (conversion rate) = 110 sales x ($100 avg. sale) = $11,000/day

By this simple extrapolation (again, not a forecast), you’ve increased revenue $1,000 a day, $30,000 a month, or $365,000 a year, assuming your average accounted for seasonality. Not bad for making a few minor text and color tweaks! Who can argue with 10% more efficient and 10% more money?

But not so fast. There are always times that an inefficient site loses sales opportunities, but it’s a poor assumption to take the 10% sales boost literally. Things aren’t so cut and dry. You might dramatically increase efficiency and your revenues may not budge. What if your website is terrible? Maybe you’re getting the sales to your call center. True, a site is cheaper than human operators, but now you’re talking cost-recovery, not increased sales. Or, your site conversion rate might go up, coupled with traffic going down. Perhaps your site is now so compelling that instead of visiting, leaving, visiting, leaving, visiting, converting, visitors now convert on the first visit. So you got the sale on the first try, but you didn’t actually get any more customers. Or maybe you sell a product or service on your own web site and it is also available on another website. If your site gains efficiency or captured sales, you might just be stealing sales from another site.

These are all extreme scenarios but they illustrate that there are many variables. A 5% conversion rate increase might not be adding 5% to your bottom line, though it’s almost certainly not hurting it. But what are you spending your time on?

One of my favorite search agencies has a miner’s pan of gold dust for its corporate identity. The metaphor is perfect for ecommerce as well: start with a coarse sieve and go after the big pieces first. When you find a good spot, keep panning and keeping using finer and finer sieves to strain out as much of the gold as you can capture. But realize that you’re seeing diminishing returns the longer you stay there. At some point, your competitors have moved up stream. Or maybe they dug a mine. You need to think about moving along as well.

You still need your idea people, your usability and human factors folks, and some good designers and developers.  Continue to chase the little stuff and constantly refine your site, but don’t forget to innovate either.