Picking the right informants in user research
A common challenge in software development is to identify and prioritize features in a feature system. Sometimes there are requirements, to begin with, but they are often defined as high-level specifications, and often technical. The important details of the functions and features as they are presented to and used by future users are typically vaguely defined and negotiable. In this specification, they are also underinformed in terms of what the future users need and want.
Identifying requirements or specifications of what users need to do their work successfully, or just what would improve existing user experiences is a very hard challenge within software engineering, human-computer interaction, and in practically any project developing software; it is also a very deep challenge because of fundamental issues with problem framing and the nature of problems. Some problems can be well-defined (given sufficient work, knowledge, and information), but people may be reluctant to do so because it requires effort, whereas others are hard to define or resist a single definition. Agile and iterative design approaches is very much a response to this challenge. We cannot specify the system to its fullest up front, so we need to be responsive and make changes as we know more.
Regardless of the challenge, projects need to progress and systems need to be implemented and delivered. Someone needs to drive the work of moving from loosely defined specifications to getting sufficient details to design and develop the right features in the right order. This is the sharp edge of the challenge -- we need to go get information, and the best way to do that is to go and ask people. This is the essence of user research.
Button and Sharrock (2021) propose a couple of maxims to guide doing user research. The first one is most relevant here: Keep Close to the Work: "The first rule of doing ethnographic work in workplaces is to be with the people who are doing the work". This means that if you have to build software for tracking and managing component quality in a manufacturing company, then you go talk to the people at the manufacturing company. If you are tasked with developing a system for archiving and handling documents within a public organization doing casework, then you go and talk to the case workers. If you are making scheduling applications for kindergartens or hospitals or an NGO organizing volunteer work you go and visit kindergartens, hospitals, and NGOs and talk to the people there (not all at once, as that would overload you and the project with potentially diverging information).
That will bring you halfway there. If you just go and spend time within the domain and talk to people, you will learn so much more than you bargained for and have enough to challenge ingrained assumptions that could potentially sabotage the project. The next question is, who do you talk to then? Here, I would like to introduce two analytical (theoretical) perspectives that might guide you. They can be applied individually or in combination.
As software developers or user researchers we are often outsiders and the people we want to learn from are insiders. Edward Relph has written about the outsider-insider challenge from a place perspective in "Place and Placelessness". Here the outsiders are people unfamiliar with a particular place and insiders are people familiar with a place. Familiarity is equally fluent -- you can be familiar with a place you have not visited through reading, media, and/or by visiting similar places. This suggests considering user research as engaging with a place where the activities of interest happen.
First advice: Go there -- you have to experience the place to understand and contextualize useful features.
Relph discusses how our familiarity with a place is a matter of insiderness, and consequently, our lack of understanding is a matter of being outside. As software developers or designers, we are very much outsiders to the particularities of a domain, practice, use, users etc.
Relph makes a continuum of outsiderness to insiderness that is relevant for us. From this we can identify where we are (likely outsiders) and whom we should talk to gain knowledge from an insider:
The first and last -- existential outsider/insider -- are not crucial for us. They merely point out that there is some knowledge and experiences that are so deep and existential that only those who experience it can experience it. You have experiences that nobody else can truly connect with, comprehend, and experience. And people have experiences that you will never be able to comprehend or experience.
Objective and incidental outsiders are two types of outsiderness where we do not understand the simple aspects of a practice, place, user experience, etc. If we visit a company that produces products or services that we have had very little knowledge of or interaction with, or maybe are even surprised to exist. Very few of us know the details of the procedures of the legal system, a heart transplant, or what happens to our trash once we throw it in the bin. Objective outsiderness is when we do not care -- it's just another job and surely a scheduling system for a kindergarten should not diverge that much from a scheduling system for a hospital. Incidental outsiderness is when we make the same assumptions unconsciously.
Second advice: Don't be an objective outsider -- embrace that we all are incidental outsiders in many situations.
Third advice: Avoid talking to outsiders about important practices and activities -- this might include parts of management and other stakeholders in the project.
Vicarious insider is a sort of middle ground between being an outsider and being an insider. On the one hand, vicarious insiders are curious outsiders who actively seek information about a particular domain or practice through written accounts, documents, media, and other sources of information. On the other hand, vicarious insiders try to get the knowledge they need through second-hand accounts, which only present partial information that is often outdated or does not reflect what happens in a particular practice. Relph compares it to reading travel books instead of traveling. I bet that a lot of software projects are plagued with vicarious insiders that know enough to navigate a domain, but always fail to get it just right because they only rely upon superficial sources of information.
Now we are getting to the folks that you should engage with to understand the work. The behavioral and empathic insiders. Both are people who know the domain, local practices, activities, people, and procedures. A behavioral insider, implied, knows how to behave and participate in the activities, but might lack deep relations and understanding. They know what to do, but not the local variations and history. We all know how to conduct ourselves and successfully navigate a library, but we might not know the local rules or who to talk to about a special subject in a particular library we have not been to before. The empathic insiders would have this kind of knowledge. We all experience being behavioral insiders when we switch education or job. We know how to conduct and perform, but it takes some time for us to get a sufficient understanding of the local practices that empathic and existential insiders have. More importantly, from our perspective as outsiders (or vicarious insiders), everyone in a domain are a behavioral insider. As outsiders, it is hard to identify empathic insiders.
Fourth advice: Talk to the behavioral insiders, but make it a priority to identify empathic insiders
Identifying and learning from the empathic insiders is often discussed as identifying and learning from lead users in design research and innovation studies. I use the empathic insiders simply because I like the framing better -- it is telling about what they know and their relation to practice. Lead user is forever tied to a product or technology, and marketing. So from now on think of lead users as empathic insiders with a product focus. The challenge is the same -- if we want to learn from the lead users, we need to identify who they are. With commercial products, one strategy is to data mine user data and social media, but for most software development dealing with a new system or a new domain or a new context, this is neither viable nor a good strategy.
How do we identify empathic insiders or lead users in a client organization, domain, or practice?
Within organizations and social groups -- we call them collectives in a recent publication -- some people are more familiar with the work, its history, the technologies, procedures, and the roles and politics of the organization. They are the empathic insiders in Relph's terminology, lead users in innovation studies, and more capable peers in activity theory. We all know people that would fit the role and description. We go to them for information, help with navigating a (new) organisation, and get the information we need to do our work. They are the people that are capable of answering questions about the informal structure and nature of tasks and activities; They are the people other people refer to when we ask questions: "You need to talk to Pete -- he's been here for a long time and was part of the restructuring back in 2010." In software engineering, they are the folks that know and remember obscure details about an old project and codebase because they were part of the initial whiteboard sessions and commits. You already know who they are for your organization -- we need to identify these people in the organizations and practices we develop software for.
Petrovsky, a social psychologist, talks about this behavior as referentiality. The collective (group) establishes a shared informal reference of someone within the group who is more knowledgeable about the activities and everything that entails. Sometimes this individual is recognized through an official role or position within the organization as a leader, mentor, coach, or something else, however, that is not given. This more-capable or knowledgeable peer might be hiding somewhere in the organization because of the informal and social nature of knowledge. They may not be a node in the official organizational diagram, but they will be important nodes in the social graph of an organization. Our research (and experiences from other contexts) show that you do this by asking the members of an organization, community, or collective who they go to if they need information, if the system breaks down, when in doubt, or when they need a second opinion.
Fifth advice: Ask people who they approach for advice and information within the organization around the central task and/or systems -- these are the lead users, empathic insiders, and more capable peers.
Up until now, I have ignored some of the challenges in approaching lead users in user research with client organizations and when developing larger systems. There are several facets to this challenge -- some are in the client organization, while others are on software engineering practices and how the development process is organized.
If the lead users "hide" in informal social relations and roles within the client organization, and if the project manager in the client organization thinks they are the lead user or have already lined behavioral insiders up as informants, it can be difficult to change the direction of and participants in the user research. This is something that is a challenge for research as well. I have gone through interviews with participants that were volunteered by their manager, while they clearly and self-acknowledged that they were not the best source of information. When possible, we should try to engage with this issue with the client organization, and if it fails, remember that a little goes a long way as Button and Sharrock say. Talking to the lead user is ideal, but we can learn enough to get us started from behavioral and even vicarious insiders. Then when we hit an information void on a particular issue, we can approach the client and illustrate the problem. Sometimes this can motivate more access. Proof of work and implications can often help commit resources.
Sixth advice: Take the information you can get and use it to illustrate where and why you need more precise and updated information.
One of the biggest challenges is the overreliance on existing assumptions and outdated knowledge within software projects. Because a lot of activities have superficial resemblances and we as humans tend to jump to conclusions, it is so tempting to rely on superficial understandings and ingrained assumptions. This is dangerous in combination with project organizations that are operating at a distance, with the wrong participants from the client organization, and with too many layers of abstraction, communication, and interfaces between the future users and the developers.
Seventh advice: Assume you act on partial information and recognize when and where you are an outsider!
Suchman talks about accountability and distance in her essay on "Located accountabilities in technology production". She points out that software engineering often works across organizational and disciplinary boundaries (think client organization and domain) and that introduces a distance that is maintained both by software engineering practices, clients limiting access to the users, and organizational issues in how software development is organized. For software development, Suchman's point is that design is done from "nowhere" with "detached intimacy". That is, from Relph's perspective, maintaining an objective outsiderness. Instead, Suchman argues that the only way to overcome this is to engage with the collective knowledge of the particular locations of design (software engineering project) and use (client organization and users). The collective knowledge as Suchman says is what Evans talks about in establishing a shared language in Domain Driven Design and what Participatory Design calls mutual learning. My addition would be the understanding that this learning process is what moves us from outsiders to partial insiders, and in that, the relations with the client organization and practitioners are of utmost importance.
Eighth and final advice: See any project you engage with as a learning process where you have to become an expert on the challenges, needs, and experiences of the client organization and whoever will end up using the software you are building!