Digital Marketing | 2013

Optimizing Amex's Digital Marketing Strategy

How can we optimize American Express's site for customer acquisitions through multivariate split testing?

Client

American Express

The Challenge

Amex needed to optimize its digital marketing acquisitions channel, which included paid search landing pages, branded homepage, international card landing pages, and owned media microsites. This involved developing hypotheses for multivariate test ideas, mocking up new designs, running the tests on a proprietary Accenture tool (ADO), and analyzing the data for these large-scale tests. Our tests needed to optimize for the volume of card applications and estimated customer value as primary outcomes, with other digital marketing metrics like click-throughs and page views as secondary metrics.

Involvement

At Accenture, I was an Optimization Analyst on a retained team of 6 with American Express, charged with testing and optimizing Amex's owned digital properties across consumer and small business cards.

The Outcome

We gave structure to ADO's testing program, including developing a strategic framework to organize test hypotheses, leveraging lessons from previous tests, and writing Python scripts to assist with tag management and QA to speed up implementation. Ultimately, we ran over 12 multivariate tests and 8 A/B tests over an 18-month period, achieving an average 5-12% uplift on primary metrics across the tests, resulting in over $4 million in projected revenue uplift in acquired customer value.

How can Amex strategize, prioritize, and optimize their digital marketing for conversion?

As a digital marketing consultant at Accenture, i worked with American Express for over 18 months to develop their Test and Learn program across various business units (consumer credit cards, small business credit cards, and international partner cards), personally working on over 30 A/B and multivariate tests (MVTs) across their owned digital properties. The program was a mix of quick A/B tests and very extensive MVTs, of which the latter contributed not only towards optimizing pages for acquisition and lifetime value, but also offered insight into which elements were most effective in conversions and filtering for qualified users.

Roadmap of all tests done across the site from 2012-2014

We conducted a number of tests oriented around 6 general categories of changes: messaging, layout, creative, user journey, product offerings, and site functionality. I was in charge of generating hypotheses to test across all areas, and keeping track of the effectiveness of these tests through their conversion impact, cost to develop, and contribution to business goals. We found that messaging tests in particular were convenient to use and had a relatively larger impact, while changes in creative assets had relatively less needle-moving effects. High-effort tests around the layout or user journey would sometimes be very impactful, but relied on more complicated multivariate tests to see the results bear out in the data.

Example of test design: Paid Search landing page

On the landing page for users clicking through paid search ads, we tested 5 hypotheses on the page with 2 to 3 variations each, resulting in 108 potential versions of the site. Building a comprehensive multivariate test, rather than a simple A/B split test, was most effective for us to test our hypotheses, observe for interactions, and be able to quantitatively give a measure of confidence about which treatment would perform the best.

Hypotheses for Paid Search landing page design content

With each hypothesis, we defined an alternate version of the page, categorized based on type of change, level of effort to implement, our estimated impact, and priority of need for such a change.

As one example of the tests we did on the landing page, we wanted to test whether adding an additional content section of a selection quiz would increase user engagement and application completion.

User breakdown of visitor profiles to the paid search funnel

In addition to understanding the user's behavior on the landing page, we analyzed the visitor data of users on the landing page and roughly attributed them to three general profiles based on their intent, engagement behavior, and application behavior. This insight helped refine our client's paid search buys and their understanding of where a user was in his or her purchase funnel.

Plot of estimated uplifts

Finally, out of 108 variations, we plotted the performance of each of the variations and noted the highest performing versions based off the different KPIs considered. Depending on the goal, different treatments were most helpful. This finding was especially helpful when analyzing data for user segments we had pulled from their browsing behavior.

After a test was complete, we would work with the client to fully implement and hard-code the winning treatment, sometimes designating different treatments depending on user segment. This continuously iterative process not only produced the most optimized landing page, but provided a continuous source of valuable data on user segments and behavior.