Overview

The Research Methods course in my HCI program centered around a corporate partnership project. Moe’s Southwest Grill is a restaurant chain that I had the pleasure to work with in attempting to answer the question “Can Moe’s utilize voice interaction?” Our team was excited about this project because it was so open-ended and provided us a lot of room for innovation and creativity. We also had to be prepared for the outcome to show that voice interaction wouldn’t work. After preliminary research we did come to this conclusion and had to figure out how to proceed through the rest of the semester. We met with our industry partners and decided that we would create a prototype for a feasible voice ordering system as well as some guidelines that they should follow if they ever were to create such a system.

Information Gathering

Questions for Moe’s Corporation

Objective:

We wanted to get some general background knowledge about Moe’s so that we could better understand our industry partner as well as our users.

Method Details:

Questions we asked included the following:

• Who is Moe?

• What are the main flows or tasks involved with a Moe’s visit?

• What is the average age of the Moe’s customer?

• How have Moe’s sales been in recent years?

• What are the most popular items?

• What is the percentage split between in-store, online, and app ordering?

• How does Moe’s identity influence the food style?

• Why do people come to Moe’s over other competing businesses?

Findings:

We got back some really interesting information. The most surprising of which being that Moe is not a person, but rather an acronym. It stands for Musicians, Outlaws, and Entertainers. We were really surprised to hear this since we had all been to a Moe’s a few times and had never known this. This suggested to us that Moe’s had a brand identity problem. They told us that they were in the midst of a re-brand to become a more Southwest location. They also sent us the personas that they use.

Competitive Analysis

Objective:

We wanted to know what companies have already implemented a voice user interface (VUI), how they did it, and how it has worked for them. Additionally, we wanted to find out how the human-computer dialog was structured in an existing service-centered VUI.

Justification:

Method Details:

• Literature review on existing systems

• Used the Starbucks Google Skill (a Google Home app) to create a Task Analysis

Findings:

• Starbucks, Dunkin’, Domino’s, and Denny’s are the main existing systems

• Dom from Domino’s is most successful

• Dom features catchphrases and fun personalization

Survey

Objective:

The main objective of this survey was to act as a recruitment tool and a screener for the customer interviews that we were planning in the next section of the project. However, we felt that while we had access to customers, we could use the opportunity to gain some more information about them, rather than sending out an additional information gathering survey.

Method Details:

• Distribution plan: link to survey in an email sent to Moe’s loyalty customers

• Expected a few hundred responses, ended up getting about 30

• Questions focused on demographics, qualities of Moe’s, using online and app ordering, and use of VUIs in general

Customer Interviews

Objective:

We had made four groups of people that we wanted to talk to (with some overlap):

1. Those who identify as having used voice technology

2. Those who do not identify as having used voice technology

3. Those that have used the Moe’s website

4. Those who have used the Moe’s app

Method Details:

• Conducted 4 interviews over the phone

• Incentivized with a $20 Moe’s gift card

• Questions asked were based on group, then all were asked general questions

• Individually conducted interview sessions were analyzed through group discussion

Findings:

After going through all of the information gathered in the interviews, we pulled out two main takeaways:

1. Some customers are skeptical and untrusting of voice assistant technology

2. There is not enough information known about these technologies for them to want to use it

Interview with Former Employee

Objective:

Now that we had some feedback and insight from the customers, we wanted to get the perspective of an employee. What is the process for an employee and is it very different than that of a customer? What are the interactions between employees and customers like?

Task Analysis

Objective:

We wanted to compare various ordering systems for Moe’s (in-store, website, and app) to see if the user flow was consistent, and if not, what was different. We also had heard from our corporate contact that use of their app was suffering because users felt that that it was clunky and required too many clicks. Through a task analysis we could look into this issue and potentially find the friction points.

Method Details:

• Completed Hierarchical task analysis for each of the three current Moe’s ordering flows; in-store, online, and in the app

• Order to be completed for each method was kept consistent to ensure consistency

• Order was not intended to be completed when using the website or the app, but by necessity was for in-store

Findings:

After completing each task analysis we compared them as a group and came to the following conclusions:

• In-store ordering follows a back-and-forth conversational structure while online and app ordering follows a form-based structure

• It is difficult to back out of an order in the app once the process has begun

• There is no way to cancel an accidental app purchase

• When paying for an order in the app, there is no confirmation for payment. This resulted in an accidental purchase while completing the task analysis.

Personas and Empathy Maps

After completing all of our initial research, we created 3 personas, each with a different view on voice technology, and mapped them to the personas that Moe’s had provided us. This allowed us to easily talk about our findings in the context of the three potential groups of users for our design. By mapping the personas we developed to the previous personas, we could ensure that not only would our clients understand the personas but also that they accurately represented our users. The personas we created are shown below.

We also created an empathy map for each of our personas. This allowed us to properly understand our users and to put ourselves in their shoes. You can see these below.

After synthesizing all of our data from each method presented above, we came to the conclusion that…

There is no evidence that Moe’s should implement a voice ordering system or that doing so would be successful.

Now what?

We meet with our stakeholders

As a team we discussed with our corporate partners the data that we had collected and the implications on the project. Through this discussion we decided on two main paths forward that we would pursue:

• Create a set of guidelines that a Moe’s VUI should follow based on our research 

• Prototype such a system - with the caveat that they should not use it immediately

We also learned in this meeting that a testing location had just opened to the public nearby that followed the new brand. So, after this meeting we met there with our professor and discussed what we could do going forward to ensure the best outcome for the two paths mentioned above. She suggested conducting a participatory design session in the testing location.

So we got the team together and conducted a…

Participatory Design Session

Timeline: 2 weeks to plan, execute, and analyze

Activities (discussed in more detail below):

• Explore

• Categorical Cards

• Roleplay

• Group Discussion

Explore:

Goals:

• Meant to introduce participants to the new Moe’s brand

• Provide an opportunity for discussion

Details:

• Participants given time to walk around and take notes

• Groups of participants formed to discuss their notes together

Categorical Cards

Goals:

• Meant to provide insight into potential voice interactions for Moe’s outside of ordering

• Provide information on what type of information users would want and/or expect

Details:

• Users paired and given 3 cards, each with a topic pertaining to a different area of the store (e.g. salsa bar, checkout counter, tables, etc.)

• One card chosen and a voice interaction was acted out based on the card

• Cards rotated and interactions repeated 3 times

• Audio recorded of each potential voice interaction for analysis later

• Researchers would only ask clarification questions so that other participants could interact and participate more

Roleplay:

Goals:

• Provide understanding on how users expect to interact with a VUI

• Learn how the users would expect the VUI to respond

• What are potential breaks in the system so that we can preemptively avoid them

• How do users expect the system to respond to an error

Details:

• Users put into different pairs and given a board with an image of Alexa on it

• One was to act as the voice of Alexa, while the other would be the customer

• Given specific items to order using the “VUI” along with a menu of available options

• Participants were encouraged to try to break the system

• Pairs switched roles so that any personality traits that potentially influenced the interaction would be spread out between the VUI and customer role

• Audio of each interaction recorded for analysis later

Group Discussion:

Goals:

• Obtain feedback on the new Moe’s branding

• Foster discussion of VUI systems in general

• General feedback on the activities and any remaining thoughts

Details:

• Researchers asked questions like “Did you have any new perspectives?” and “What are things you didn’t expect?”

• Discussed new Moe’s branding with industry partner - this made participants feel they were being heard by corporate as well as provided great and insightful information directly our industry partner

Analysis and Findings

In order to analyze all of the data we listened to recordings as a group and created an affinity map. Important information from the recordings were written onto Post-it notes and organized first by activity, then by theme or topic. We ended up with 10 categories. A view of the affinity map can be seen below.

Design

Wizard of Oz Prototype

Based on all of the research we had done, we created a list of guidelines that a VUI system for Moe’s should follow. Because of an NDA I cannot include them here, but below is an example interaction developed using those guidelines that I can show.

We developed a script for a potential voice interaction and then created the demo shown below using BotSociety. For a full demo including audio, click here.

Functioning Prototype

In order to improve on our WOZ prototype, we needed to define a clear information architecture flow that the system would follow. We created an information architecture chart (shown to the left) based on the task analyses we had done previously as well as some of the findings from our participatory design session.

Then we created a functioning Google Skill using the Google DialogFlow tool that followed the information architecture we had made. Be sure to start your interaction with “Hey Moe“. Prototype here

Evaluation

Expert Heuristic Evaluation

Objective:

With this method we wanted to achieve 6 main goals: get feedback on the 9 heuristics, test the overall functionality of the prototype, evaluate the ease of use among expert users, assess the value of the prototype, determine the learning curve for using a voice recognition ordering system, and evaluate the system’s error recovery features.

Method Details:

• 2 evaluators - experts in usability and sonification

• Each given 2 specific meal orders to complete using the system

• Asked to think-aloud while completing each task

• Provided with a heuristic worksheet based on Nielsen Norman 10 Heuristics

• One researcher facilitating, one taking notes

• Audio recorded for later analysis

Findings:

The quantitative ratings we collected are presented to the right. Themes that we pulled out of the audio recordings are as follows:

• The system is too verbose

• The system needs to prompt the user and guide them to a predictable response

• Assist users that may be confused or novice, but carefully consider cognitive load

• The system needs robust error recovery features

• Consider all the ways the user may perceive the system’s personality

• Where possible, expand the user’s freedom

• Shortcuts are always good

Moderated User Testing

Objective:

Through our moderated user tests we wanted to accomplish the following: test the overall functionality of the prototype, evaluate the ease of use among novice users, assess the value of the prototype, and determine the learning curve for using a voice recognition ordering system.

Method Details:

• Conducted in-person and remotely with 4 participants

• Given 2 specific orders to complete using the system

• Encouraged participants to ask questions and think aloud throughout the session

Findings:

• Users appreciate some level of polite personality

• The system does not feel time efficient

• The system broke when users did not know exactly how to respond to the prompt

• Users like the call and response format when they knew how to respond

• Users had the most success when they replied with few words

• Voice recognition accuracy was a problem throughout

Design Recommendations

Conclusions

What I learned

This project taught me a very valuable lesson, that researching a product before it is implemented is critical. If Moe’s had not invested in researching the voice interactive ordering system and went straight to implementation, they would have spent millions to discover that no one wanted to use it. I also learned that extensive and varied research is helpful to ensure you come to the correct conclusions. Had my team only used the initial survey or competitive analysis, we may have incorrectly concluded that a Moe’s VUI would have been a great investment for the company.

What I would do differently

One way to improve the initial survey would be to use multiple methods of survey distribution. We received less that a third of the responses to our survey than we anticipated and a larger sample size would have been more reliable. Additionally, if I were to do a project like this again I would like to be more creative with my design recommendations to help create a more innovative solution.