BigBook-Experimentation-Main-ebook.pdf -

Please enable JavaScript to view the full PDF

10X MEDIA HIGH TECH RETAIL TRAVEL FINANCE NONPROFIT Table of Contents 03 Introduction 04 Retail Case Studies 42 Financial Services Case Studies 51 Travel Case Studies 68 B2B Case Studies 75 Media Case Studies 89 Nonprofit & Government Case Studies Introduction No matter what data you’ve gathered about your customers’ preferences and behaviors, they only tell you about past history, not how they will react to new products, interfaces, user flows or campaigns. Relying on past experience and intuition simply isn’t enough in today’s business climate. Experimentation platforms are a must-have, allowing faster, more focused innovation with more control and greater insight. Without experimentation, companies can’t innovate; they can only guess. Today’s experimentation platforms provide a convenient and safe way to test hypotheses in production, with real users and measurable results, replacing internal opinions and disputes with actual behavioral data. Innovators in Retail, Financial Services, Travel, Media, B2B, Nonprofit, Government and more are seeing big benefits from experimentation. Read more about how Optimizely is working with customers in all these industries to innovate and experiment. BIG Book of Experimentation 3 Retail Case Studies STUBHUB Validating product ideation through Optimizely “For StubHub, the value of Optimizely is heavily tied to the Full Stack solution[...] and the ability to have always-on testing where we’re always collecting data. Everybody now has a framework through which to ask a question. Experimentation really puts everyone on an equal playing field and allows them to stack their ideas up against other people’s ideas.” Michael Alley, Senior Product Manager, StubHub BIG Book of Experimentation 5 HP Driving innovation at HP through an experimentation center of excellence HP self-identifies as being in the early stages of their journey towards creating a true culture of experimentation throughout their organization. At the same time, they know that increasing the pace of innovation for their campaigns will be essential EXPERIMENT to ensuring peak performance for various digital business units that have started doing experimentation, including their B2C online store, HP.com website, B2B Subscription enrollment offers e-commerce organization, Instant Ink organization, and Customer Support site. HP’s Instant Ink subscription service experimented with different enrollment offers. KEY RESULT The experiment involved showing customers an option for a free trial and positioning the service as a printer feature. The experiment increased enrollment by 37%. $21 million HP has invested in creating a program around experimentation, building a more robust process for ideation, prioritization, development, execution, and post- Incremental revenue impact campaign analysis. HP shares learnings with business stakeholders throughout the organization in their weekly program reporting meeting, and are also planning to roll out a weekly newsletter to help share updates and findings across their teams. HP’s ultimate goal is to give every employee the chance to experiment and gather data around their ideas. As a result of their focus on experimentation, they’ve been able to run almost 500 campaigns and have driven an incremental $21 million in revenue with Optimizely. BIG Book of Experimentation 6 HP’s Instant Ink subscription page. BIG Book of Experimentation 7 “With these instant, early results, we’re planning to scale the use of Optimizely company-wide.” Neville Davey, Digital Experimentation Worldwide Program Lead, HP BIG Book of Experimentation 8 MISSGUIDED How experimentation and personalization are driving the fast-growing UK fashion brand forward Forward-thinking fashion retailer Missguided designs wearable ready-to-go outfits for today’s millennial woman. Founded in 2009, the Manchester-based brand is EXPERIMENT experiencing growth of 75% year-on-year. Premium service offering, personalized offers for VIP The firm has over 10,000 products available online at any one time and offers up to customers 1,000 new styles each week. It is known for its ingenious and fresh approach to both fashion and business. Everything Missguided creates is informed by its customer along with global influences like social media, street style, and popular culture. KEY RESULT Trusting that experimentation is a viable method of evaluating business models, 177% Missguided decided to test a new premium service that offers unlimited free next day delivery for a year. To manage the risk associated with the new service and to check its commercial value, this offering was first made available to a small group of customers. With Optimizely data showing a conversion uplift of 177%, ‘unicorn Conversion uplift delivery’ was soon rolled out to the entire customer base. 33% Relative increase in revenue through personalization BIG Book of Experimentation 9 Testing a new premium service ‘Unicorn Delivery’ with a small group of customers. BIG Book of Experimentation 10 Make it personal Delivering a personalized shopping experience to every visitor is contributing to Missguided’s growing customer base. Deriving insights from customer data and behavior allows the brand to tailor each customer journey. With data about what a customer has recently viewed or purchased on the website, what products they added to their wish list and if they have a VIP status, Missguided can segment its audiences and develop personalized offers and promotions. A particularly successful campaign was targeted at ‘rising star’ customers, those on the cusp of becoming Missguided VIPs based on the number and frequency of purchases as well as site visits. Targeting them with exclusive offers, such as buying three items to get 30% off, allowed Missguided to convert these customers to the company’s most valuable VIPs. BIG Book of Experimentation 11 “Missguided has an entrepreneurial approach and isn’t afraid to experiment with new ideas and offerings to drive the business forward. Working with Optimizely gives us enormous insights into our customers’ needs, desires and behaviors and allows us to adapt and evolve our approach fast to reap the commercial rewards.” Mark Leach, Head of E-Commerce, Missguided BIG Book of Experimentation 12 BLUE APRON Validating products faster with experimentation Blue Apron creates incredible experiences. Founded in 2012, Blue Apron is a consumer lifestyle brand that symbolizes a love of home cooking, excitement and discovery, and deep, meaningful human connections. EXPERIMENT Blue Apron’s core product is a meal experience that customers create with the Redesigned reactivation flow original recipes and fresh, seasonal ingredients that are included in every box. All recipes are accompanied by printed and digital content, including how‑to instructions and stories of suppliers and specialty ingredients. KEY RESULT At Blue Apron, a large part of the business is powered by scheduled back end jobs. 10x Order charging, turning digital orders into physical packages, creating shipping labels, choosing recipes for users and sending emails all happen offline. A client- side solution provided limited flexibility when it came to testing these experiences. Increase in experiments run per month after adopting Also, running experiments and measuring key KPIs required work from the analytics Optimizely X Full Stack team, and turnaround time could take anywhere from two to four weeks. This greatly impeded Blue Apron’s ability to act upon needed changes quickly. So while they continued to use Optimizely Web, they also required a solution specific to their server side use case. BIG Book of Experimentation 13 With Optimizely Full Stack, Blue Apron gets statistically valid experimental results on their server and mobile applications which helps them make faster, more informed business decisions. With Optimizely’s real-time results, Blue Apron was able to avoid launching a redesigned reactivation flow, which would have resulted in a significant revenue loss. Blue Apron can now quickly and easily build and set up new tests that were simply not possible before, running over 10x the number of tests per month. BIG Book of Experimentation 14 “We can now get an early signal from the Optimizely Results Dashboard within a few days, whereas before it would have taken weeks or longer for someone on our analytics team to pull the data and understand what was going on.” John Cline, Engineering Lead, Growth/Member Experience, Blue Apron BIG Book of Experimentation 15 AUTOSCOUT24 Taking the guesswork out of contact forms helps triple product revenue AutoScout24 is Europe’s largest online marketplace for new and used cars. Over the last three years, the company has built a testing program that allows them to address real user problems and make measurable improvements in conversion on EXPERIMENT their site. Contact form optimization For AutoScout24 all transactions start with an online contact form. It is a widely used part of the website that is crucial to their customers’ success. In a user research study, car shoppers said they were not completing the contact form because they KEY RESULT did not know how to communicate to the seller. 22% The team hypothesized that providing more pre-filled information in the form and decreasing the amount of original thinking (or effort) the shopper has to do would increase form submissions. They tested three variations of the form with different treatments. All 3 variations outperformed the original, but the variation with pre-filled Increase in contact form conversion text was the highest performer and increased contact form conversion by 22%. BIG Book of Experimentation 16 Original Variation The original version of the contact form: They tested a form pre-filled with text, blank fields and text boxes for a buyer to ‘Hello. I am interested in your vehicle. complete and send to the car seller. Please contact me. Kind regards.’ BIG Book of Experimentation 17 DELIVERY HERO Digital heroes in the food delivery market Delivery Hero is the world’s leading platform for takeaway orders over the Internet. The company, founded in 2011 and headquartered in Berlin, operates in more than 40 countries on 5 continents, for example in Germany with their Foodora, Lieferheld and pizza.de brands. EXPERIMENT The ambitious goal of Delivery Hero is to become and remain the market leader in all Homepage media test countries in which the company is represented. To achieve this goal, Delivery Hero must provide its customers with an excellent customer experience and continuously optimize it. Only with digital experiments is it possible to recognize customer needs, says Erin McLaine, Global Head of Conversion Rate Optimization at Delivery Hero. KEY RESULT 10% Delivery Hero’s culture encourages scrutinizing opinions and ideas, and checking them using scientific methods. At a conference on A/B-testing, a speaker claimed that background videos on homepages would only work for Airbnb and that it was not worth the effort of producing such videos. The Delivery Hero team took up the Increase in conversion rate using challenge to prove him wrong. a background video The marketing team produced a background video for the website and the optimization team tested it against a variety of background images. The result was surprisingly clear – the conversion rate was more than 10% higher with the video than with any of the pictures. The video resonated with their target audience. “It really was a great success and gave experimentation at Delivery Hero a real boost,” says Erin. BIG Book of Experimentation 18 “As much as you may think that you know your customer, the product and the impact new features would have, you are not always right. Optimizely helps us to understand what our customers really want.” Erin Mclaine, Head of Conversion Rate Optimization, Delivery Hero BIG Book of Experimentation 19 FSA STORE Visually simplifying the site navigation substantially increases revenue per visitor FSAstore.com is the only e-commerce company supplying household essentials that caters exclusively to the 35 million Americans who are covered by a flexible spending account (FSA). EXPERIMENT FSAstore.com recognized they had a challenge with getting customers through the Simplified navigation website’s customer purchase funnel. The team hypothesized that customers were getting distracted by too many options, especially on the site’s category pages. KEY RESULT They decided to test a simplified site experience by removing the dense sub-header in their site navigation. The improvement was unmistakable with a 53.8% increase in 53.8% revenue per visitor on category pages. Increase in revenue per visit BIG Book of Experimentation 20 The variation experience removed the top-level navigation from category pages. BIG Book of Experimentation 21 REVOLVE Engaging the multi-channel shopper As an innovative online-only retailer, REVOLVE constantly strives to provide the most relevant digital experience possible to engage customers and keep them coming back for more. In order to better fine-tune the experiences they provide mobile shoppers, the EXPERIMENT REVOLVE team ran a test to increase the number of shoppers using their mobile Driving app downloads app. They hypothesized that directing mobile web visitors to a splash page with aggressive promotion for REVOLVE’s mobile app would drive more app installs. The variation with the aggressive splash page increased app downloads directly KEY RESULT from the mobile site by an overwhelming 350%. With more users shopping from the 350% mobile app, REVOLVE can provide more relevant, personalized experiences to their audience that will drive purchases. Increase in mobile app downloads BIG Book of Experimentation 22 Original Variation BIG Book of Experimentation 23 “We did a significant site revamp, which led us to realize we wanted to invest in a program to make gradual, incremental updates along the way to continually improve and optimize the site.” Grace Hong, VP of Product and Design, REVOLVE BIG Book of Experimentation 24 TOMTOM User experience insights inspire tests that improve key metrics TomTom is a global leader in navigation and mapping products, based in the Netherlands. In addition to mapping software and devices, the company also makes GPS watches, action cameras and state-of-the-art fleet management solutions, EXPERIMENT used by millions of people around the globe every day. Product details page layout As the company has grown, e-commerce has become an increasingly important channel for the business. Like many retailers, TomTom has been focused on increasing sales online, where they have the ability to reach consumers all over the KEY RESULT world and where profit margins continue to increase. 16% One place TomTom has seen huge success with their experimentation is on their map product pages. In an effort to get more visitors from the product pages into the checkout flow, Dave tested two new variations of the price anchor at the bottom of Increase in add-to-cart each product page. The Control version requires two clicks, first for the selection and conversion rate then to add to cart. They thought removing a click might remove some of the friction from the process. BIG Book of Experimentation 25 Variation #1 showed the price options horizontally with one option highlighted in the center. Variation #2 stacked the same price options vertically. They hypothesized that these variations would improve conversion on the product pages and ultimately help to increase revenue from these products. The results were even better than expected. While running Variation #1 of the price anchor, TomTom saw a 16% increase in conversion rate on the product pages. This relatively simple change in the layout of the price options ended up having a huge impact on online revenue for the business. BIG Book of Experimentation 26 “People are starting to come to me and say ‘we want to test this’ and opinions are starting to fly around. It’s beginning to take hold that the data will tell the truth.” Dave Powell, E-Commerce Conversion Manager, TomTom BIG Book of Experimentation 27 WONDERBLY Targeting shoppers with reassuring messaging drives orders from Facebook visitors Wonderbly (formerly Lost My Name) began its journey in 2012 as a dedicated online only retailer specializing in personalized children’s books. Since July 2015 their goal EXPERIMENT has been to create the same exceptional and personalized experience achieved in Targeted messaging its books for each online visitor. One of the team’s most significant wins was targeted messaging for shoppers with 3-letter childrens names—reassuring them that “short names don’t mean short KEY RESULT stories.” This targeted messaging resulted in an 18.1% lift in orders from Facebook 18.1% visitors, their largest acquisition channel. Lift in orders BIG Book of Experimentation 28 BLU DOT Capturing rich customer data to drive onsite and in-store sales Blu Dot, a designer and maker of modern home furnishings, worked closely with Optimizely Solution Partner, Clearhead, to build and scale an optimization program that fuels their business. EXPERIMENT The team hypothesized that if they replaced the email modal capture with a “Free Optimizing information capture Catalog” request form that conversions to the catalog request would go up. The results were a staggering 124% lift in catalog requests, which not only gave them rich customer data including location, but facilitated online and offline sales through a KEY RESULT more supported sales process. 124% Lift in catalog requests BIG Book of Experimentation 29 Not only did the new modal increase conversions tremendously. It also provided Blu Dot with something equally as valuable—more specific information about their customers’ physical location. BIG Book of Experimentation 30 VITAMIX Creating a differentiated buyer experience for the world’s best blenders Vitamix, a 96-year-old, family-owned, American company, has long had a reputation for great performance, durability and reliability amongst commercial chefs and home users alike. EXPERIMENT Its approach to innovation doesn’t stop with the blenders and is essential to Personalized shopping cart delivering a world-class e-commerce experience. The company has embraced an experience experimentation mindset with the help of Optimizely Solutions Partner, Clearhead. In doing so, the Vitamix team has been able to continually improve their digital customer experiences, increase the ROI driven by Vitamix.com, and use data to KEY RESULT confidently inform UX decisions and designs. 13.7% Vitamix was seeing nearly 80% of shopping carts created by web visitors abandoned. To engage return visitors and encourage them to convert to checkout, Vitamix used Optimizely X Web Personalization to develop a personalized experience Increase in order conversion rate for each return visitor. The new experience added a ‘mini cart’ functionality to with personalized experience for Vitamix’s site, which showed a personal list of items for purchase and a brightly cart abandonment colored call-to-action. By delivering this personalized experience to return visitors, the team was able to improve order conversions for this audience by 13.7%. BIG Book of Experimentation 31 The ‘mini cart’ experience showed a personal list of items for purchase and featured a brightly colored call-to-action. BIG Book of Experimentation 32 “With Optimizely, we’ve demonstrated the value of delivering personalized experiences to our audience. We’re excited to continue iterating on what we’ve already learned with experimentation to deliver even more value for our business.” Laura Szarek, Director, Digital Marketing, Vitamix BIG Book of Experimentation 33 ELECTRONIC ARTS Removing promotional offer dramatically increases pre-order sales Electronic Arts Inc. (EA) is a leading global interactive entertainment software that was preparing to release a highly-anticipated version of one of their most popular games—SimCity. Maxis, the division of EA that produces SimCity, wanted to maximize EXPERIMENT online revenue from the launch. Promotional offer display The team opted to extend a promotional offer to customers to drive more pre- orders. The offer was prominently displayed on their website and through paid ad campaigns but was not successful at driving an increase in pre-orders. They KEY RESULT decided to test the placement of the pre-order offer. Surprisingly, they found that the page which removed the offer altogether drove 43.4% more revenue from the 43.4% product launch. Armed with this knowledge, the EA team replicated this win across several paid media channels. Increase in revenue BIG Book of Experimentation 34 “Optimizely has helped us learn a lot about our users— what’s working and what’s not. In the end, this translates into higher revenue for us.” Mike Burke, Senior Online Product Manager, Maxis (EA) BIG Book of Experimentation 35 BROOKS RUNNING Reducing cost of returns by experimenting with helpful size-shopper experience Brooks Running leveraged Optimizely to test out a helpful experience targeted at shoppers likely to return a pair of shoes. The free return shipping policy is a big draw for customers, but is a significant cost to the company. EXPERIMENT By targeting these shoppers and helping them find appropriate sizing prior to Return deflection experiment checkout, Brooks Running decreased their return rate by 80%. KEY RESULT 80% Decrease in return rate Brooks showed this pop-up message to shoppers who had 2 or more pairs of shoes in a cart that were half size apart. BIG Book of Experimentation 36 ROCKSBOX Improving member interactions with experiments and an insatiable appetite for feedback In a constantly changing retail landscape, subscription offerings stand out as a bright spot in e-commerce innovation. Rocksbox has pioneered changing how EXPERIMENT women shop for jewelry, leveraging a combination of personal stylists and AI-driven Offering positioning and recommendations. However, building relationships with prospective customers member experience and educating them about the how-tos of a fundamentally new shopping model presents a number of challenges. Erin O’Leary, VP of Marketing at Rocksbox, tackles ambitious growth goals by KEY RESULT focusing on a strategy that will move the needle in the company’s acquisition costs 99% and retention metrics. Through her iterative process of reshaping Rocksbox’s acquisition funnel and transforming their member feedback experience, she has been able to deliver show-stopping improvements to key business metrics and rally the broader company around an adventurous, innovative culture and formal Conversion lift from new experimentation goals. positioning The cost of customer acquisition is a near-universal marketer’s challenge. Since advertising costs are constantly susceptible to fluctuation, Erin focuses on controlling costs by increasing conversion rates once potential members reach Rocksbox’s site. “We have a cycle of analyzing site data and customer surveys, planning groups of experiments to solve problem areas of the funnel and then we prioritize and execute two-week experiments to tackle each part of the funnel,” Erin describes. BIG Book of Experimentation 37 “To make an impact quickly, focus on testing new ideas to validate them with data, or share a surprising result that wasn’t expected. Frame losing experiments as risks that were minimized while still optimizing for learning.” Erin O’Leary, VP of Marketing, Rocksbox BIG Book of Experimentation 38 TRUNK CLUB Testing and personalization play a critical role in driving impact Trunk Club is a subscription men’s clothing service focused on helping its customers build an amazing wardrobe, without a lot of work. Trunk Club uses Optimizely to provide customers with delightful, personalized experiences on the EXPERIMENT web and mobile. Sign up flow testing At Trunk Club, testing is built into the company’s DNA. They’ve been testing everything from their digital experiences to their product packaging since they launched in 2009. KEY RESULT “Whenever we’re curious, confused or in doubt, we run a test and let the data speak 133% for itself,” explains Mike Wolf, Trunk Club’s Product Design Lead. Mike wanted to find a way to spice up sign-up user experience and increase the Increase in conversions overall number of visitors that completed the sign-up process. This would lower Trunk Club’s cost per acquisition (CPA) and increase the number of leads each stylist received, ultimately growing the business’ bottom line. To achieve this, Mike decided to test a different type of sign-up flow. The new sign- up variation added more steps to the process, but it provided more focus on each question and more visuals. Mike hypothesized that this type of experience was more in-line with what visitors expected from their brand and therefore, would increase conversions. BIG Book of Experimentation 39 However, lengthening the onboarding process was a risk. Introducing more steps for new users increased the number of opportunities for them to bounce on a given page, and would potentially lead to fewer new customers for the business. Mike put the new six-step flow to the test to measure the impact it would have on conversions. When the results came in, there was a clear winner. The longer, more visual, sign-up flow increased conversions by 133%. “It blew away our expectations!” says Mike. “It actually led to a new problem…too many leads for our stylists! We weren’t expecting that.” One of six steps in the variation onboarding flow, which emphasized visuals and smaller, broken-out steps. BIG Book of Experimentation 40