The Voicebot Mechanic

6 min read

PUBLISHED ON

14 August 2020

The Voicebot Mechanic

A mechanic fixing a Robot

Fixing the Bot

img

A mechanic fixing a Robot

Fixing the Bot

img

MY ROLE

Product Desgin, Wireframing, UI Design

TEAM

Ayaneshu, Amit (Manchi) - PM

TIMELINE

May 2020 to August 2020

OVERVIEW

Vernacular.ai is an Indian AI-First SaaS startup with two flagship products: VIVA (Vernacular Intelligent Voice Assistant) and VASR (Vernacular Automatic Speech Recognition). These products work together to automate call centers for clients. Vernacular's mission is to transform customer experiences through Voice AI.

This is a UX Case study of a project which I completed as part of a 6-day design sprint during my internship at Vernacular.ai, where I was asked to deliver a digital solution to speed up the process for testing voicebots.

We successfully reduced the average call time from 58 seconds to 33 seconds. This improvement has saved up to one hour per day for individuals testing the voice bots, allowing them to use this extra time more productively. Additionally, there has been a significant enhancement in the rate of data generation and collection.

MY ROLE

Product Desgin, Wireframing, UI Design

TEAM

Ayaneshu, Amit (Manchi) - PM

TIMELINE

May 2020 to August 2020

OVERVIEW

Vernacular.ai is an Indian AI-First SaaS startup with two flagship products: VIVA (Vernacular Intelligent Voice Assistant) and VASR (Vernacular Automatic Speech Recognition). These products work together to automate call centers for clients. Vernacular's mission is to transform customer experiences through Voice AI.

This is a UX Case study of a project which I completed as part of a 6-day design sprint during my internship at Vernacular.ai, where I was asked to deliver a digital solution to speed up the process for testing voicebots.

We successfully reduced the average call time from 58 seconds to 33 seconds. This improvement has saved up to one hour per day for individuals testing the voice bots, allowing them to use this extra time more productively. Additionally, there has been a significant enhancement in the rate of data generation and collection.

MY ROLE

Product Desgin, Wireframing, UI Design

TEAM

Ayaneshu, Amit (Manchi) - PM

TIMELINE

May 2020 to August 2020

OVERVIEW

Vernacular.ai is an Indian AI-First SaaS startup with two flagship products: VIVA (Vernacular Intelligent Voice Assistant) and VASR (Vernacular Automatic Speech Recognition). These products work together to automate call centers for clients. Vernacular's mission is to transform customer experiences through Voice AI.

This is a UX Case study of a project which I completed as part of a 6-day design sprint during my internship at Vernacular.ai, where I was asked to deliver a digital solution to speed up the process for testing voicebots.

We successfully reduced the average call time from 58 seconds to 33 seconds. This improvement has saved up to one hour per day for individuals testing the voice bots, allowing them to use this extra time more productively. Additionally, there has been a significant enhancement in the rate of data generation and collection.

About the Product

The Console serves as an all-encompassing solution for creating conversational flows, monitoring, and collecting data to enhance the performance of voice bots. Essentially, it is a web-based tool utilized by Vernacular.ai's team. Given that even a minor adjustment in the Voice bots ML model can yield multiple outcomes, testing becomes a crucial practice to ensure the integrity and functionality of the system.

What are Voicebots?

Voice bots are AI-powered software that enables callers to navigate interactive voice response (IVR) systems using their voice, typically employing natural language. This eliminates the need for callers to listen to menus and press corresponding numbers on their keypads. Instead, they engage with the IVR in a simplified simulation resembling a conversation with a live operator.

At Vernacular, the call flow and specific conversation states were tested by placing calls to a number assigned to a particular voice bot. A single caller conducted approximately 80–100 calls per day, with an average duration of around 58 seconds per call. Any faulty calls were identified and reported through another interface known as Call Reporting (which will be discussed in a separate article).

initial process of testing a voicebot

initial process of testing a voicebot

img

initial process of testing a voicebot

initial process of testing a voicebot

img

Problems faced by the users

The existing process was tedious and time-consuming. Regular testing of Voice Bots was imperative due to the potential for significant repercussions from even minor changes in the ML model (Butterfly effect). To facilitate testing, a single caller was required to make approximately 80–100 calls per day. Urgent simplification was needed, particularly with the frequent development of new conversational flows for various clients (organizations). The efficiency of testing and reporting faulty calls was on the decline.

The major problems identified included:

  1. High call frequency: Testing a specific conversation state necessitated repetitive calling and performing the same steps to reach and check that state.

  2. Long call duration (Average 120 seconds).

Sneak Peak into the Design process

Setting Projects Goals

  1. Remove telephony dependency.

  2. Faster testing and data generation.

  3. And reporting all the faulty conversation turns during the call.

Proposed Solution

Develop a web-based service within Console designed to simulate the calling experience. This service aims to enhance flexibility in testing call flow and specific conversational turns by allowing repetitive testing within a single call.

Defining Job Stories

Given that the product was exclusively for internal use, our users were readily accessible.

We opted for Job Stories instead of creating User Personas and User Stories. This decision was driven by the need for a quicker approach and a desire to delve deeper into Jobs Theory. Integrating this approach into a sprint allowed us to test and evaluate the pros and cons effectively.

  1. When I make a call, I want to loop into one conversational state so I can test it repeatedly.

  2. When I make a call, I want to loop into one conversational state so I can report the incorrect intents predicted by the ASR(Automatic Speech Recognization).

  3. When I’ve answered the voice bot, I want to know the predicted intent so I can report that conversational term there and then if it is faulty.

  4. When I make a call, I want to select the language and version for the voice bot so that I can conduct more tests.

These were the 4 Job Stories we listed to ideate the features for the MVP version.

Structure of a Job Story

Structure of a Job Story

img

Structure of a Job Story

Structure of a Job Story

img

Important Decision

We opted against using text input to communicate with the voice bot on the interface, as we aimed to avoid transitioning into a chatbot, which was not our intention. Instead, we chose to take voice as the input from the user and read the bot's textual output. Although the primary output of the bot was textual, there was an option to play the prompt as well.

The reasons behind this decision were:

  1. We wanted to replicate the on-call experience using an interface.

  2. Enabling the text input option might have led us to assume that the Automatic Speech Recognition (ASR) was functioning perfectly, potentially overlooking areas for improvement.

This decision proved instrumental in creating a new and faster user experience, allowing users to speak their queries and read the bot's prompt. Notably, it contributed to reducing the average call duration from 120 seconds to 90 seconds per call.

User Flow

The flow diagram below illustrates the user's journey during a call on the interface:

User Flow for Call

User Flow

img

User Flow for Call

User Flow

img

Shipping the MVP

Acknowledging time constraints and code limitations, the UI received less emphasis, although it certainly requires improvement. Despite this, the Sub-Testing service has become functional. The MVP version has been shipped, inviting users to engage, test, and provide feedback. This iterative process aims to further reduce friction in the overall experience.

Prototype

vid

Prototype

vid

Impact the changes created

Following the MVP version demo of the Sub-Testing interface, we received an excellent response. Team members who regularly tested voice bots began using this newly functional service within Console immediately.

Key statistics collected include:

  1. A reduction in the average call time from 58 seconds to 33 seconds.

  2. A saving of up to 1 hour per day for voice bot testers, enabling them to utilize this additional time more productively.

  3. Significant improvement in the rate of data generation and collection.

Final Thoughts & Learnings

We gained valuable hands-on experience using Job Stories. As a UX designer, the goal is always to deliver the best user experience, but in certain situations, prioritizing functionality and getting the product running takes precedence. This is an inherent aspect of Product Design.

Designing for an AI-based startup introduced a new set of challenges, particularly in dealing with the unpredictability of the voice bot's predictions, where outcomes varied even for the same input.

Collaborating closely with Amit Manchanda (Associate Product Manager at Vernacular.ai) proved to be an excellent experience. It allowed me to enhance my understanding of the broad spectrum of design, consider limitations, and pose the right questions to ensure a comprehensive approach.