I went to an interesting conference yesterday hosted by Webtrends titled “Successful Site Optimisation with Webtrends”. The main topics were about multivariate testing, which doesn’t get most people’s pulse racing, but it was presented in an interesting way with lots of useful tips.
Multivariate testing is a natural next step up from A/B testing.
A/B testing is where you have two pages that do the same thing, and track which one gives better results, whether it’s increased registrations, higher click through rates or more revenue, etc. These two pages will run for a certain number of users or a time period, to give good comparisons. Think of it as having two large research group.
The problem with A/B testing is that these pages are usually fundamentally different, so it’s difficult to know which difference made users increase metrics on one page over another. Hence multivariate testing was born. Multivariate tweaks the ‘control’ page, subtly changing any of the following:
- Content (copy)
- Page section behaviours (making whole section clickable, or just an image, or just the text next to an image)
- Audience segmentation.
These multiple changes can run in parallel as long as they are measured correctly.
At the conference, Webtrends asked who in the audience show a different landing page for new users compared to returning users. It sounds obvious – if a user has already come to your site before, they probably know the basics, so show them more detail, or entice them to actually buy something on the second visit. Out of 60 people in the audience, only 4 companies said they show different pages for new and returning visitors, and one of those companies were MoneySupermarket.com who gave a case study later on.
To do multivariate testing, you need to develop a culture within the business to try different ideas. Most organisations want to design a single solution for web pages, and implement those. The concept of creating multiple designs, copy and user behaviours for each page is alien.
Multivariate testing also requires more people. At the minimum, you need the following:
- Optimisation analyst
- Creative specialist
- Web develop
As mentioned earlier, MoneySupermarket.com gave a case study of how they implement multivariate testing. Steve Willey from MoneySupermarket.com explained that they have 120 million visits a year, and 6.7 million individual customers. This gives them a large control group to test updates.
Steve spoke a lot about the optimisation culture within MoneySupermarket.com, which is vital to fight the subjective decisions that most organisations take with design and content. His key recommendations were:
- Set objectives
- Take small steps
- Educate the rest of the business
- Keep wowing stakeholders with results.
It took MoneySupermarket.com two years after the first A/B test before optimisation became cultural. The first test was to controversially remove a large amount of content from some pages which gave improved conversion results.
MoneySupermarket.com have an SEO expert in the optimisation team to ensure they don’t lose their top spot for most keywords.
For new products, they always A/B test before launch. They always try to avoid new launches without any data of how well it will perform. Sample sizes are usually around 10% of the audience, giving pretty quick results to MoneySupermarket.com.
One of the things I noticed in a number of the case studies were how calls to action (e.g. the text on links and buttons) are more subtle rather than direct. For instance, ‘Next step’ worked better than ‘Book now’ on the sites.
The next speaker was Tom Waterfall from Webtrends. He gave 11 Dos and Don’ts of Site Optimisation:
- Do constantly test. Results only last for a certain period of time (e.g. during a sales campaign, perhaps with above-the-line support)
- Do make sure you have the right resources. Research from Forrester shows that the right resources will make the testing more successful.
- Do keep it simple. This reduces the risk internally and externally. It helps keep the test shorter and easier to analyse. Copy tends to have the most profound impact on visitors, and adding a chevron (>) helps. You can go into infinite detail about Add to cart buttons.
- Don’t forget to sell yourself on every page. Keep adding security information, free returns/ delivery/ service/ etc., and publish information about the amount of content, customers, products or even ‘Likes’ that your brand or products have.
- Do make use of other technologies, including analytics, insight database, eye tracking and heat maps.
- Do think about the audience. Tom gave a great example screenshot when he was looking to buy a pair of [men’s] shoes and the recommendations on the page included some women’s clothes. So have some recommendation rules to prevent these simple mistakes. Have a think about the audience based on their location, or the traffic source (where the user came from), their previous behaviour on your site, and if possible, demographic data.
- Don’t put all your eggs in one basket. Break the changes down from A/B testing to smaller multivariate tests. Try to test one thing at a time (although you can have multiple tests at one time, just not all on the same page). In summary, make sure you can analyse the results from your tests. And test each language separately – what works in one language probably won’t work the same in another.
- Do plan ahead. Have a long term plan of tests. As soon as one test has finished, start the next test, hence the need for a team, with a pipeline.
- Do think radical. Firstly, the more radical the changes, the bigger response you’ll get from users. MoneySupermarket.com sometimes push (or ignore) the corporate brand guidelines because they guidelines are sometimes subjective, and Steve’s team can prove that bending or changing the rules will give improved revenue.
- Do consider future trends. Tom recommended everyone should be testing on mobile, tablets and web, and on the first two formats, companies should be testing app and browser versions. Companies that took future trends into consideration more than two years ago stopped designing mouse hover over effects and lightboxes which don’t work well on touch devices.
- Do have fun with testing. Tom’s examples were a little lacking (read: corny!) here, so I’ll give an example of one of our customers a couple of years ago. The customer produced a mobile app, some tactical social media implementations, and a YouTube channel. All three were the first implementations of their type for this client. Everyone within the Endava and client team said what they thought would be the most successful initiative (and why), which gave us more ideas for each project.
The morning was a really good session, so thank you to Webtrends and MoneySupermarket.com for sharing so many useful tips for website optimisation.
Image courtesy of Mccormicky