USER RESEARCH: TYPES OF QUALITATIVE RESEARCH, ETHNOGRAPHIC INTERVIEWS

<< GOAL-DIRECTED DESIGN METHODOLOGIES: A PROCESS OVERVIEW, TYPES OF USERS

USER-CENTERED APPROACH, ETHNOGRAPHY FRAMEWORK >>

Human Computer Interaction (CS408)

Lecture

Lecture 19. User Research Part-I

Learning Goals

As the aim of this lecture is to introduce you the study of Human Computer

Interaction, so that after studying this you will be able to:

· Understand the difference in qualitative and quantitative research

· Discuss in detail qualitative research technique

The outcome of any design effort must ultimately be judged by how successfully it

meets the requirements of both the user and the organization that commissioned it. No

matter how skillful and creative the designer. If she does not have a clear and detailed

knowledge of the users she is designing for, what the constraints of the problem are,

and what business or organizational goals the design is hoping to achieve, she will

have little chance of success.

What and how questions like these are best answered by qualitative research, not

metrics or demographics (though these also have their purpose). There are many types

of qualitative research, each of which plays an important role in filling in a picture of

the design landscape of a product.

Qualitative versus Quantitative Research

Research is a word that most people associate with science and objectivity. This

association isn't incorrect, but it biases many people towards the notion that the only

valid sort of research is the kind that yields the supposed ultimate in objectivity:

quantitative data. The notion that numbers don't lie is prevalent in the business and

engineering communities, even though we all rationally understand that numbers--

especially numbers ascribed to human activities--can be manipulated or reinterpreted

at least as dramatically as words.

Data gathered by the hard sciences like physics is simply different from that gathered

on human activities: electrons don't have moods that vary from minute to minute, and

the tight controls physicists place on their experiments to isolate observed behaviors

are next to impossible in the social sciences. Any attempt to reduce human behavior

to statistics is likely to overlook important nuances, which though they might not

directly affect business plans, do make an enormous difference to the design of

products. Quantitative research can only answer questions about how much or how

many along a few reductive axes. Qualitative research can tell you about what, how

and why in rich, multivariate detail.

Social scientists have long realized that human behaviors are too complex and subject

to too many variables to rely solely on quantitative data to understand them. Usability

practitioners, borrowing techniques from anthropology and other social sciences, have

developed many alternative methods for gathering useful data on user behaviors to a

more pragmatic end: to help create products that better serve user need.

169

Human Computer Interaction (CS408)

The value of qualitative research

Qualitative research helps us understand the domain, context and constraints of a

product in different, more useful ways than quantitative research does. It also quickly

helps us identify patterns of behavior among users and potential users of a product

much more quickly and easily than would be possible with quantitative approaches. In

particular, qualitative research helps us understand:

· Existing products, and how they are used.

Potential users of new or existing products, and how they currently approach

activities and problems the new product design hopes to address

Technical, business, and environmental contexts--the domain--of the product

to be designed

Vocabulary and other social aspects of the domain in question

Qualitative research cab also help the progress of design projects by:

· Providing credibility and authority to the design team, because design

decisions can be traced to research results

· Uniting the team with a common understanding of domain issues and user

concerns

· Empowering management to make more informed decisions about product

design issues that would otherwise be based on guesswork or personal

preference

It is the experienced that qualitative method, in addition to the benefits described

above, tend to be faster, less expensive, more flexible, and more likely than their

quantitative counterparts t provide useful answers to impotent questions that leads t

superior design:

· What problems are people encountering with their current ways of doing what

the product hopes to do?

· Into what broader contexts in people's lives does the product fit and how?

· What are the basic goals people have in using the product, and what basic

tasks help people accomplish them?

Types of qualitative research

19.1

Social science and usability texts are full of methods and techniques for conducting

qualitative research. We will discuss following qualitative research techniques:

· Stakeholder interviews

· Subject matter expert (SME) interviews

· User and customer interviews

· User observation/ethnographic field studies

· Literature review

· Product/prototype and competitive audits

170

Human Computer Interaction (CS408)

Stakeholder interviews

Research for any new product design, through it must end with understanding the

user, should start by understanding the business and technical context in which the

product will be built. This is necessary not only to ensure a viable and feasible end

result, but also to provide a common language and understanding among the design

team, management, and engineering teams.

Stakeholders are any key members of the organization commissioning the design

work, and typically include managers and key contributors from engineering, sales,

product marketing, marketing communications, customer support, and usability. They

may also include similar people from other organizations in business partnership with

the commissioning organization, and executives. Interviews with stakeholders should

occur before any user research begins.

It is usually most effective to interview each stakeholder one-on-one, rather than in a

larger, cross-departmental group. A one-on-one setting promotes candor on the part of

the stakeholder, and ensure that individual views are not lost in a crowd. Interviews

need not last longer than about an hour, though follow-up meetings may be called for

if a particular stakeholder is identified as an exceptionally valuable source of

information.

The type of information that is important to gather from stakeholders includes:

· What is the preliminary vision of the product from each stakeholder

perspective? As in the fable of the blind men and the elephant, you may find

that each business department has a slightly different and slightly incomplete

perspective on the product to be designed. Part of the design approach must

therefore involve harmonizing these perspectives with those of users and

customers.

· What is the budget and schedule? The answer to this question often provides a

reality check on the scope of the design effort and provides a decision point

for management if user research indicates a greater scope is required.

· What are the technical constraints? Another important determinant of design

scope is a firm understanding of what is technically feasible given budget,

time, and technology.

· What are the business drivers? It is important for the design team to

understand what the business is trying to accomplish. This again leads to a

decision point, should user research indicate a conflict between business and

user needs. The design must, as much as possible, create a win-win situation

for users, customers, and providers of the product.

· What are the stakeholders' perceptions of the user? Stakeholders who have

relationships with users (such as customer support representative) may have

important insights on users that will help you to formulate your user research

plan. You may also find that there are significant disconnects between some

stakeholders' perceptions of their users and what you discover in your

research. This information can become an important decision point for

management later in the process.

Understanding these issues and their impact on design solutions helps you as a

designer to better serve your customer, as well as users of the product. Building

171

Human Computer Interaction (CS408)

consensus internally will help you to articulate issues that the business as a whole may

not identified, build internal consensus that is critical for decision making later in the

design process, and build credibility for your design team.

Subject matter expert (SME) interviews

Some stakeholders may also be subject matter experts (SMEs): experts on the domain

within which the product you are designing will operate. Most SMEs were users of

the product or its predecessors at one time, and may now be trainers, managers, or

consultants. Often they are experts hired by stakeholders, rather than stakeholders

themselves. Similar to stakeholders, SMEs can provide valuable perspective on a

product and its users, but designers should be careful to recognize that SMEs

represent a somewhat skewed perspective. Some points to consider about using SMEs

are:

· SMEs are expert users. Their long experience with a product or its domain

mean that they may have grown accustomed to current interactions. They may

also lean towards expert controls rather than interactions designed for

perpetual intermediate perspective. SMEs are often not current users of the

product, and may have more of a management perspective.

· SMEs are knowledgeable, but they aren't designers. They may have many

ideas on how to improve a product. Some of these may be valid and valuable,

but the most useful pieces of information to glean from these suggestions are

the causative problems that lead to their proposed solutions.

· SMEs are necessary in complex or specialized domains such as medical,

scientific, or financial services. If you are designing for a technical or

otherwise specialized domain, you will likely need some guidance from

SMEs, unless you are one yourself. Use SMEs to get information on complex

regulations and industry best practices. SME knowledge of user roles and

characteristics is critical for planning user research in complex domains.

· You will want access to SMEs throughout the design process. If your product

domain requires use of SMEs, you should be able to bring them in at different

stages of the design to help perform reality checks on design details. Make

sure that you secure this access in your early interviews.

User and customer interviews

It is easy to confuse users with customers. For consumer products, customers are

often the same as users, but in corporate or technical domain, users and customers

rarely describe the same sets of people. Although both groups should be interviewed,

each has its own perspective on the product that needs to be factored quite differently

into an eventual design.

Customers of a product are those people who make the decision to purchase it. For

consumer product, customers are frequently users of the product; although for

products aimed at children or teens, the customers are parents or other adult

supervisors of children. In the case of most enterprise or technical products, the

customer is someone very different from the user--often an IT manager--with

distinct goals and needs. It's important to understand customers and their goals in

order to make a product viable. It is also important to realize that customers seldom

172

Human Computer Interaction (CS408)

actually use the product themselves, and when they do, they use it quite differently

than the way their users do.

When interviewing customers, you will want to understand:

Their goals in purchasing the product

Their frustrations with current solutions

Their decision process for purchasing a product of the type you're designing

Their role in installation, maintenance, and management of the product

Domain related issues and vocabulary

Like SMEs, customers may have many opinions about how to improve the design of

the product. It is important to analyze these suggestions, as in the case of SMEs, to

determine what issues or problems underline the ideas offered, because better, more

integrated solutions become evident later in the design process.

Users of a product should be the main focus of the design effort. They are the people

(not their managers or support team) who are personally trying to accomplish

something with the product. Potential users are people who do not currently use the

product, but who are good candidates for using it in the future. A good set of user

interviews includes both current users (if the product already exists and is being

revised) and potential users (users of competitive products and non-automated

systems of appropriate). Information we are interested in learning from users includes:

· Problems and frustrations with the product (or analogous system if they are

potential users)

· The context of how the product fits into their lives or workflow: when, why,

and how the product is used, that is, patterns of user behavior with the product.

· Domain knowledge from a user perspective: what do users need to know to

accomplish their jobs

· A basic understanding of the users' current tasks: both those the product

requires and those it doesn't support

· A clear understanding of user goals: their motivations and expectations

concerning use of the product

User observation

Most people are incapable of accurately assessing their own behaviors, especially

outside of the context of their activities. It then follows that interviews performed

outside the context of the situations the designer hopes to document will yield less

complete and less accurate data. Basically, you can talk to users about how they think

they behave, or you can observe it first hand. The latter route provides superior

results.

Many usability professionals make use of technological aids such as audio or video

recorders to capture what users say and do. Care must be taken not to make these

technologies too obtrusive: otherwise the users will be distracted and behave

differently than they would off-tape.

Perhaps the most effective technique for gathering qualitative user data combines

interviews and observation, allowing the designer to ask clarifying questions and

direct inquiries about situations and behaviors they observe in real-time.

173

Human Computer Interaction (CS408)

Literature review

In parallel with stakeholder interviews, the design team should review any literature

pertaining to the product or its domain. This can and should include product

marketing plans, market research, technology specifications and white papers,

business and technical journal articles in the domain, competitive studies. Web

searches for related and competing products and news, usability study results and

metrics, and customer support data such as call center statistics.

The design team should collect this literature, use it as a basis for developing

questions to ask stakeholders and SMEs, and later use it to supply addition domain

knowledge and vocabulary, and to check against compiled user data.

Product and competitive audits

Also in parallel to stakeholder and SME interviews, it is often quite helpful for the

design team to examine any existing version or prototype of the product, as well as its

chief competitors. Doing so gives the design team a sense of the state of the art, and

provides fuel for questions during the interviews. The design team, ideally, should

engage in an informal heuristic or expert review of both the current and competitive

interfaces, comparing each against interaction and visual design principles. This

procedure both familiarizes the team with the strengths and limitations of what is

currently available to users, and provides a general idea of the current functional

scope of the product.

Ethnographic interviews

19.2

It has been experienced that combination of one-on-one interviews and work/lifestyle

observation is the most effective and efficient tool in a designer's arsenal for

gathering qualitative data about users and their goals. The technique of ethnographic

interviews is a combination of immersive observation and directed interview

technique.

Hugh Beyer and Karen Holtzblatt have pioneered an ethnographic interviewing

technique that they call contextual inquiry. Their method has, for good reason, rapidly

gained traction in the industry, and provides a sound basis for qualitative user

research.

Contextual inquiry

Contextual inquiry, according to Beyer and Holtzblatt, is based on a master-

apprentice model of learning: observing and asking questions of the users as if she is

the master craftsman and he interviews the new apprentice. Beyer and Holtzblatt also

enumerate four basic principles for engaging in ethnographic interview:

Context:

Rather than interviewing the user in a clean white room, it is important to interact

with and observe the user in their normal work environment, or whatever physical

context is appropriate for the product. Observing users as they perform activities and

questioning them in their own environment, filled with the artifacts they use each day,

can bring the all-important details of their behaviors to light.

174

Human Computer Interaction (CS408)

Partnership:

The interview and observation should take the tone of a collaborative exploration with

the user, alternating between observation of and discussion f its structure and details.

Interpretation:

Much of the work of the designer is reading between the lines of facts gathered about

user's behaviors, their environment, and what they say. These facts must be taken

together as a whole, and analyzed by the designer to uncover the design implications.

Interviewers must be careful, however, to avoid assumptions based on their own

interpretation of the facts without verifying these assumptions with users.

Focus:

Rather than coming to interviews with a set questionnaire or letting the interview

wander aimlessly, the designer needs to subtly direct the interview so as to capture

data relevant t design issues.

Improving on contextual inquiry

Contextual inquiry forms a solid theoretical foundation for quantitative research, but

as a specific method it has some limitations and inefficiencies. The following process

improvements result in a more highly leveraged research phase that better set the

stage for successful design:

· Shortening the interview process: contextual inquiry assumes full day

interviews with users. The authors have found that interviews as short as one

hour in duration are sufficient to gather the necessary user data, provided that

a sufficient number of interviews (about six well-selected users for each

hypothesized role or type) are scheduled. It is much easier and more effective

to find a diverse set of users who will consent to an hour with a designer than

it is to find users who will agree to spend an entire day.

· Using smaller design teams: Contextual inquiry assumes a large design

team that conducts multiple interviews in parallel, followed by debriefing

sessions in which the full team participates. Experiments shows that it is more

effective to conduct interviews sequentially with the same designers in each

interview. This allows the design team to remain small (two or three

designers), but even more important, it means that the entire team interacts

with all interviewed users directly, allowing the members to most effectively

analyzed and synthesized the user data.

· Identifying goals first: Contextual inquiry, as described by Beyer and

Holtzblatt, feeds a design process that is fundamentally task-focused. It is

proposed that ethnographic interviews first identify and prioritize user goals

before determining the tasks that relate to these goals.

· Looking beyond business contexts: the vocabulary of contextual inquiry

assumes a business product and a corporate environment. Ethnographic

interviews are also possible in consumer domains, though the focus of

questioning is somewhat different.

175

Table of Contents: