OKAY, HERE’S WHAT WAS HAPPENING..
Our customers on Amazon, when they had a question about Amazon products, programs or orders, first turned to Amazon to get more information. Customers used the Amazon search bar, to search for information or actions they wanted to take, like “ start prime subscription”, “where is my order?”, “How to cancel my order”, “connect echo to wifi” etc anticipating for answers or a path towards it. However, customers were mostly disappointed by irrelevant product results that either left them to find the information by themselves by navigating through Amazon departments or customer support or forced them to seek this information outside Amazon.
Customers who often searched for “What are the benefits of prime?” or “How to connect Echo to wifi” on popular search engines were often redirected back to Amazon, but now was routed right to the answer. This meant that Amazon customers, who looked for information about Amazon programs or products on Amazon, had to first exit Amazon and come back to Amazon through an external service to quickly find the right information. This was a huge gap within our customer experiences and introduced a friction point and frustration for our customers.
Our vision in closing this customer gap, was to utilize Alexa’s ability to find contextual, insightful and unbiased answers for product questions and establish Alexa as smart assistant on Amazon Search.
To comply with my non-disclosure agreement, I have omitted and obfuscated confidential information in this case study and have only outlined released/shipped portion of our vision for Alexa.
MY ROLE
Design Owner/Lead Designer (2020 - Current): I partnered with with multiple research, product and engineering owners across verticals to strategize, conceptualize and execute a multi-year vision, to elevate Alexa into a product advisor who could answer customer questions typed into Amazon search bar on Amazon App and Amazon.com.
I worked with key partner teams and stakeholders including Amazon Search, Alexa AI and Web Info, Alexa Shopping, AI Research and Science teams to present vision, gain alignment and ship several phases of the vision throughout 2020 and 2021.
Introducing type based Alexa answers on Amazon Search
We started by diving deep into the problem space to understand the similarities and differences between voice based answers and text/search based answers. Is the change only in the modality and the model of interaction? Is there a difference in content or content consumption? How can we create a scalable and responsive QnA framework that could be used across all Alexa touch points irrespective of the medium.
What makes a good answer?
Categorizing Question Types and Shapes
To provide customers with the right actionable answer at any instance, we had to first understand the questions customers were asking or would have during their shopping journey. Questions can be classified based on the type of answer the customer is looking for, the type of product, the way customers phrase these questions, based on their shopping journey etc. The type of answer or what is considered as a “good answer” varies based on this.
For example, consider the question “How can I hang frame on wall without gap?”
A customer who might be trying to hang a painting on the wall maybe looking for a product/DIY solutions that would help them to neatly put up the painting without damage to wall
A customer who might have purchased/is looking for a SAMSUNG Frame TV, might be trying to understand how to mount the Samsung Frame on wall.
The right answer varies with customer intent(Informational or Shopping) , customer context (Have purchased item or not) . Alexa would need to disambiguate between various intents and put forward a confident choice to the customer. It was important to understand the question types and the nuances a question would have based on intent and journey stages.
2. Answer Sources and Alexa’s Confidence
When it comes to information, over the years customers have formed certain mental models around what type of source to trust and when. For various question shapes and various intents, do customers prefer different sources? Alexa answers a question by first analyzing the question and then finding the best answer by sorting through thousands of answers from various sources. For every question, answers may be
Factual data on Amazon owned products/services/subsidiaries from Alexa’s database
Information on products from product detail pages on Amazon
Information or answers from sellers or manufacturers
Information or answers from websites, publishers or reviewers on web
Information or answers from Amazon customers and Alexa communities
For a given question, how can Alexa make a confidence choice when multiple answers are available from multiple choices? How can Alexa be transparent to the customers about the specificity or precision of the answer, the confidence in the answer and the trustworthiness of the source ? We started by trying to understand the relationship between question shapes and answer sources.
RESPONSIVE AND CONFIGURABLE INFORMATION ARCHITECTURE
We designed and developed a core architecture with components that could be turned ON and OFF depending upon the question - answer classification.
CONFIGURABLE TEMPLATES
From the component system, we configured few templates that could be used for several different instances like High Confidence -Informational Intent, High Confidence-Navigational Intent, High Confident - Action Intent, Low Confidence-Shopping intent etc. This helped us to pick templates that would for a given question, provide to the customer the most helpful and actionable content.
PROJECT AND UX CHALLENGES
Building or shifting an existing customer behavior
A major challenge we faced and will continue to face is that customers are not used to searching questions on Amazon. Customers come to Amazon when they know what to buy. While we continue to invest in building this customer behavior, few solutions we experimented were (1) by targeting implicit questions ( “prime subscription”, “audible benefit”, “connect echo”) and providing answers along with fully formed questions (2) by adding QnA suggestions within Search autocomplete. (3) proactively suggesting answers found by Alexa for informational intent queries (“clean Keurig” may lead to products and a QnA result for “How to clean Keurig?”) (4) proactively suggesting a set of Frequently asked questions within search results for high consideration queries like “apple watch” , “tv”, “laptop” etc
Building customer trust with unbiased information
How can we ensure that customers can trust the answers they receive through QnA on Search, especially for subjective questions or questions where Amazon has competing products (Eg “Is prime worth it?”). We branded the QnA experience as Alexa and continued to build on the customer mental model that Alexa is a smart assistant who collects and analyzes information to put forward the best information. We brought in web content on to Amazon and built an extensive database that helped us to clearly attribute answers and always trace back answers to their sources. While these required shift in the frameworks and outlook of many key partner teams, and business perspectives, we felt that it was the right investment to build customer trust. We care continuing to build frameworks that can intelligently determine the most trustable answer for a given question shape an intent, backed by extensive user research
Building frameworks for answer generation
As a UX lead on this project, an ongoing challenge is the content itself. Presenting the content in an easy to consume and actionable format is only part of the problem, a big piece of it is figuring out what makes a good answer? Is there a formula or framework that can be created which could then be automated with the help of Science and Engneering teams? The data Alexa has on customer sentiments on the products sold on Amazon makes information on Amazon truly unique. How can we generate answers or content with these trends, facets, reviews, ratings and other information that could help our customers? While we might not have yet found the perfect solutions, user research informs our models and have we experiment with multiple answer frameworks before landing on a framework for a given question shape. This somestimes would mean that different question shapes may need different frameworks or models for answer generation and collaboration with different teams. We are continuing to tinker and perfect our framework to generate high quality answers for each question shape that takes into account customer intent and journey stage to determine the best answer.
