10X 10X 0 0 0 0 Table of Contents 03 04 42 51 68 75 89 Introduction Retail Case Studies Financial Services Case Studies Travel Case Studies B2B Case Studies Media Case Studies Nonprofit & Government Case Studies 3 BIG Book of E x perimentation Introduction No matter what data you’ve gathered about your customers’ preferences and behaviors, they only tell you about past history, not how they will react to new products, interfaces, user flows or campaigns. Relying on past experience and intuition simply isn’t enough in today’s business climate. Experimentation platforms are a must-have, allowing faster, more focused innovation with more control and greater insight. Without experimentation, companies can’t innovate; they can only guess. Today’s experimentation platforms provide a convenient and safe way to test hypotheses in production, with real users and measurable results, replacing internal opinions and disputes with actual behavioral data. Innovators in Retail, Financial Services, Travel, Media, B2B, Nonprofit, Government and more are seeing big benefits from experimentation. Read more about how Optimizely is working with customers in all these industries to innovate and experiment. Retail Case Studies 5 BIG Book of E x perimentation STUBHUB Validating product ideation through Optimizely “For StubHub, the value of Optimizely is heavily tied to the Full Stack solution[...] and the ability to have always-on testing where we’re always collecting data. Everybody now has a framework through which to ask a question. Experimentation really puts everyone on an equal playing field and allows them to stack their ideas up against other people’s ideas.” Michael Alley, Senior Product Manager, StubHub 6 BIG Book of E x perimentation Incremental revenue impact KEY RESULT EXPERIMENT Subscription enrollment offers $21 million HP Driving innovation at HP through an experimentation center of excellence HP self-identifies as being in the early stages of their journey towards creating a true culture of experimentation throughout their organization. At the same time, they know that increasing the pace of innovation for their campaigns will be essential to ensuring peak performance for various digital business units that have started doing experimentation, including their B2C online store, HP.com website, B2B e-commerce organization, Instant Ink organization, and Customer Support site. HP’s Instant Ink subscription service experimented with different enrollment offers. The experiment involved showing customers an option for a free trial and positioning the service as a printer feature. The experiment increased enrollment by 37%. HP has invested in creating a program around experimentation, building a more robust process for ideation, prioritization, development, execution, and post- campaign analysis. HP shares learnings with business stakeholders throughout the organization in their weekly program reporting meeting, and are also planning to roll out a weekly newsletter to help share updates and findings across their teams. HP’s ultimate goal is to give every employee the chance to experiment and gather data around their ideas. As a result of their focus on experimentation, they’ve been able to run almost 500 campaigns and have driven an incremental $21 million in revenue with Optimizely. 7 BIG Book of E x perimentation HP’s Instant Ink subscription page. 8 BIG Book of E x perimentation “With these instant, early results, we’re planning to scale the use of Optimizely company-wide.” Neville Davey, Digital Experimentation Worldwide Program Lead, HP 9 BIG Book of E x perimentation EXPERIMENT Premium service offering, personalized offers for VIP customers MISSGUIDED How experimentation and personalization are driving the fast-growing UK fashion brand forward Forward-thinking fashion retailer Missguided designs wearable ready-to-go outfits for today’s millennial woman. Founded in 2009, the Manchester-based brand is experiencing growth of 75% year-on-year. The firm has over 10,000 products available online at any one time and offers up to 1,000 new styles each week. It is known for its ingenious and fresh approach to both fashion and business. Everything Missguided creates is informed by its customer along with global influences like social media, street style, and popular culture. Trusting that experimentation is a viable method of evaluating business models, Missguided decided to test a new premium service that offers unlimited free next day delivery for a year. To manage the risk associated with the new service and to check its commercial value, this offering was first made available to a small group of customers. With Optimizely data showing a conversion uplift of 177%, ‘unicorn delivery’ was soon rolled out to the entire customer base. Conversion uplift Relative increase in revenue through personalization KEY RESULT 177% 33% 10 BIG Book of E x perimentation Testing a new premium service ‘Unicorn Delivery’ with a small group of customers. 11 BIG Book of E x perimentation Make it personal Delivering a personalized shopping experience to every visitor is contributing to Missguided’s growing customer base. Deriving insights from customer data and behavior allows the brand to tailor each customer journey. With data about what a customer has recently viewed or purchased on the website, what products they added to their wish list and if they have a VIP status, Missguided can segment its audiences and develop personalized offers and promotions. A particularly successful campaign was targeted at ‘rising star’ customers, those on the cusp of becoming Missguided VIPs based on the number and frequency of purchases as well as site visits. Targeting them with exclusive offers, such as buying three items to get 30% off, allowed Missguided to convert these customers to the company’s most valuable VIPs. 12 BIG Book of E x perimentation “Missguided has an entrepreneurial approach and isn’t afraid to experiment with new ideas and offerings to drive the business forward. Working with Optimizely gives us enormous insights into our customers’ needs, desires and behaviors and allows us to adapt and evolve our approach fast to reap the commercial rewards.” Mark Leach, Head of E-Commerce, Missguided 13 BIG Book of E x perimentation BLUE APRON Validating products faster with experimentation Blue Apron creates incredible experiences. Founded in 2012, Blue Apron is a consumer lifestyle brand that symbolizes a love of home cooking, excitement and discovery, and deep, meaningful human connections. Blue Apron’s core product is a meal experience that customers create with the original recipes and fresh, seasonal ingredients that are included in every box. All recipes are accompanied by printed and digital content, including how-to instructions and stories of suppliers and specialty ingredients. At Blue Apron, a large part of the business is powered by scheduled back end jobs. Order charging, turning digital orders into physical packages, creating shipping labels, choosing recipes for users and sending emails all happen offline. A client- side solution provided limited flexibility when it came to testing these experiences. Also, running experiments and measuring key KPIs required work from the analytics team, and turnaround time could take anywhere from two to four weeks. This greatly impeded Blue Apron’s ability to act upon needed changes quickly. So while they continued to use Optimizely Web, they also required a solution specific to their server side use case. Increase in experiments run per month after adopting Optimizely X Full Stack KEY RESULT EXPERIMENT Redesigned reactivation flow 10x 14 BIG Book of E x perimentation With Optimizely Full Stack, Blue Apron gets statistically valid experimental results on their server and mobile applications which helps them make faster, more informed business decisions. With Optimizely’s real-time results, Blue Apron was able to avoid launching a redesigned reactivation flow, which would have resulted in a significant revenue loss. Blue Apron can now quickly and easily build and set up new tests that were simply not possible before, running over 10x the number of tests per month. 15 BIG Book of E x perimentation “We can now get an early signal from the Optimizely Results Dashboard within a few days, whereas before it would have taken weeks or longer for someone on our analytics team to pull the data and understand what was going on.” John Cline, Engineering Lead, Growth/Member Experience, Blue Apron 16 BIG Book of E x perimentation AUTOSCOUT24 Taking the guesswork out of contact forms helps triple product revenue AutoScout24 is Europe’s largest online marketplace for new and used cars. Over the last three years, the company has built a testing program that allows them to address real user problems and make measurable improvements in conversion on their site. For AutoScout24 all transactions start with an online contact form. It is a widely used part of the website that is crucial to their customers’ success. In a user research study, car shoppers said they were not completing the contact form because they did not know how to communicate to the seller. The team hypothesized that providing more pre-filled information in the form and decreasing the amount of original thinking (or effort) the shopper has to do would increase form submissions. They tested three variations of the form with different treatments. All 3 variations outperformed the original, but the variation with pre-filled text was the highest performer and increased contact form conversion by 22%. Increase in contact form conversion KEY RESULT EXPERIMENT Contact form optimization 22% 17 BIG Book of E x perimentation Original The original version of the contact form: blank fields and text boxes for a buyer to complete and send to the car seller. Variation They tested a form pre-filled with text, ‘Hello. I am interested in your vehicle. Please contact me. Kind regards.’ 18 BIG Book of E x perimentation DELIVERY HERO Digital heroes in the food delivery market Delivery Hero is the world’s leading platform for takeaway orders over the Internet. The company, founded in 2011 and headquartered in Berlin, operates in more than 40 countries on 5 continents, for example in Germany with their Foodora, Lieferheld and pizza.de brands. The ambitious goal of Delivery Hero is to become and remain the market leader in all countries in which the company is represented. To achieve this goal, Delivery Hero must provide its customers with an excellent customer experience and continuously optimize it. Only with digital experiments is it possible to recognize customer needs, says Erin McLaine, Global Head of Conversion Rate Optimization at Delivery Hero. Delivery Hero’s culture encourages scrutinizing opinions and ideas, and checking them using scientific methods. At a conference on A/B-testing, a speaker claimed that background videos on homepages would only work for Airbnb and that it was not worth the effort of producing such videos. The Delivery Hero team took up the challenge to prove him wrong. The marketing team produced a background video for the website and the optimization team tested it against a variety of background images. The result was surprisingly clear – the conversion rate was more than 10% higher with the video than with any of the pictures. The video resonated with their target audience. “It really was a great success and gave experimentation at Delivery Hero a real boost,” says Erin. Increase in conversion rate using a background video KEY RESULT EXPERIMENT Homepage media test 10% 19 BIG Book of E x perimentation “As much as you may think that you know your customer, the product and the impact new features would have, you are not always right. Optimizely helps us to understand what our customers really want.” Erin Mclaine, Head of Conversion Rate Optimization, Delivery Hero 20 BIG Book of E x perimentation Increase in revenue per visit KEY RESULT EXPERIMENT Simplified navigation 53.8% FSA STORE Visually simplifying the site navigation substantially increases revenue per visitor FSAstore.com is the only e-commerce company supplying household essentials that caters exclusively to the 35 million Americans who are covered by a flexible spending account (FSA). FSAstore.com recognized they had a challenge with getting customers through the website’s customer purchase funnel. The team hypothesized that customers were getting distracted by too many options, especially on the site’s category pages. They decided to test a simplified site experience by removing the dense sub-header in their site navigation. The improvement was unmistakable with a 53.8% increase in revenue per visitor on category pages.