Through years of helping clients develop their testing programs, I've found that without a structured plan for website optimization testing, it’s easy to end up with meaningless data, wasted time and frustrated stakeholders.
The good news is that developing a test plan isn't that difficult. Although it requires careful thought and consideration, those are hallmarks of any worthwhile improvement effort.
You and your testing team should have a formal test plan ready to go before you do any preholiday optimization. Having a test plan accomplishes the following three goals:
- helps organize and prioritize the test;
- helps set expectations; and
- demonstrates to your stakeholders that you're serious about testing.
Having a test plan also helps you create a road map based on opportunity, risk and other considerations.
Test plan template
The following is a template that we provide our clients to assist with development of their testing plans:
What are you trying to test? What factors are you hypothesizing that could lift your key performance indicators (KPIs)? What influence do your headlines, graphics or copy have on the “Buy Now” button, for example? If you need suggestions about what to test, click here to view an article by Kim Ann King, SiteSpect's chief marketing officer, which ran in eM+C Weekly on Dec. 4, 2008.
Why are you running this test? Are you trying to increase certain user behavior, decrease it or simply improve how something looks? And how are your goals (e.g., marketing) related to your end users' goals, as well as to those of your overall business? Ideally, all of these goals are in alignment. Otherwise, you have conflicting interests that'll manifest through testing as lackluster results.
What success or failure criteria do you have in mind for this test? Will you be successful if you increase the number of clicks from the page by 10 percent, for example? If your site conversion rate declines by 10 percent, should you stop the test? Criteria like these not only help prioritize tests, but will be used mathematically to predict required time and sample sizes for each test.
What specific metrics, KPIs and visitor segments will you monitor during this test? Will you track clickthrough or bounce rates for first-time visitors who view the test page? Conversion rate across the site for all visitors?
What marketing initiatives or external factors are you aware of that could affect the test? During your tests, will you also run email promotions, new display ads or offline campaigns (e.g., newspaper or TV ads)?
What internal and external resources are required for this test? Will creative, web development or IT resources be required? Are outside agencies involved in generating creative for the test? Will sales or finance need to be involved in the approval process prior to launching the test? Who and what else are needed? If you're running a test, know where your dependencies lie.
What departments and key internal stakeholders are potentially impacted by this test? Will this test impact sales or lead generation, for example? Could test designs be construed as “off-brand,” impacting public relations and marketing efforts? What are your plans for communicating with these folks? You’ll also want to know who the most senior person in the organization is requesting the test. Is it you, your manager, your manager’s manager or the CEO? What are their expectations for information about the test before, during and after it's completed?
Ideally, when do you need test results? The timing of your test depends on many variables, including the scope of the test, your success criteria and other tests being run.
While these questions need to be tailored to your company, I've found this template serves as an excellent starting point to get an organization focused on the process of testing.
Author's note: This article is based on research conducted by Eric Peterson of Web Analytics Demystified, a web analytics research, consulting and publishing firm.
Eric J. Hansen is founder and CEO of SiteSpect, a multivariate testing and behavioral targeting platform for websites and the mobile web. Reach Eric at ehansen@sitespect.com.
All About Email Creative
