Reading :: Streetlights and Shadows

Streetlights and Shadows: Searching for the Keys to Adaptive Decision Making
By Gary Klein

I just reviewed the methodology text that Klein coauthored; this book is a chance to see his methodological approach in action. Here, Klein focuses on how we make decisions in ambiguous situations. This question is actually quite hard to investigate in the lab, since "systematic errors aren't so serious outside the lab"; indeed, "reasoning strategies let us do many kinds of tasks without consciously or subconsciously performing calculations to perform an estimate" (p.59). So Klein turns to the scenarios that he always turns to in his popular/summary books: aircraft controllers and pilots, firefighters, NICU nurses, etc. (I would complain that he rehashes these scenarios too much across books, but I understand why he does so—they're all great illustrations, and the books use them to make related-but-different arguments to related-but-different audiences.)

Much of this book goes over principles that Klein addresses in his other books, so I'll just highlight a few standouts.

Klein points out that experts avoid data saturation by self-selecting which data to seek. That is, they know which data are most relevant and they shut out the extraneous data, making them more effective (p.133). In fact, he says, "there is never a right amount of information" and "we would be better off if we stopped worrying about getting the right amount of information and instead tried to see the meaning in the data that we do have" (p.135).

People need feedback—"feedback is essential for helping people become more skilled." But feedback itself isn't sufficient (p.165): outcome feedback (what was the result?) does not improve performance as much as process feedback (what were the cause-feedback relations in the performance?) (p.166).

Problems with emergent goals—so-called wicked problems (p.212)—include things such as business models (p.213). For such problems, "when facing wicked problems we have to re-define the goals as we try to reach them. ... No amount of thinking and analysis will make these goals well defined. In such cases, we are going to have to figure out the goals as we go along. The faster we can learn, the more successful we'll be" (p.223, his italics). Yet, he points out, many in this situation will instead "try to increase their control over events" and will thus "stumble into goal fixation" (p.223). In such situations, he advocates "Management by Discovery": "when we face complex conditions we should expect to revise and replace goals on the basis of what we learn" (p.224).

Overall, this book is readable and valuable. It's a little less valuable if you've read Klein's other books, since there's a lot of overlap, but his angle here is different—to dispel myths about decision making. If you're interested in how people make decisions in ambiguous situations (for instance, when entrepreneurs evaluate their business models), definitely pick it up.

Reading :: Working Minds

Working Minds: A Practitioner's Guide to Cognitive Task Analysis
By B. Crandall, G. Klein, and R.R. Hoffman

I've discussed Gary Klein's work before, and specifically how much I appreciate his attitude of trust and respect toward his participants. Klein's work focuses on how experienced professionals (such as firefighters, NICU nurses, and soldiers) make intuitive decisions in high-stakes, high-pressure environments.

To research such cases, Klein needed an ecological approach that allowed him to get at situated decision making in cases in which the participants couldn't necessarily articulate their assumptions, options, or triggers. At the same time, Klein couldn't just follow firefighters around—the events he wanted to study were just too rare, and when they happen, he didn't want his team to get in the way of rescue operations.

The approach that Klein and his partners developed for such cases is called cognitive task analysis (CTA), which "helps researchers understand how cognitive skills and strategies make it possible for people to act effectively and get things done," according to the back of this book. The book is, as the subtitle states, "A Practitioner's Guide to Cognitive Task Analysis." That is, it describes CTA and the situations in which it could be useful; it offers tools and strategies for performing CTA; and it discusses how CTA brings value to the participants. In this sense, it reminds me of Beyer and Holtzblatt's Contextual Design, a similar methodology book written by consultants for practitioners (although addressing different situations with a different methodological approach).

What struck me about Working Minds, though, was that the coauthors had developed a qualitative approach within psychology. As the authors note, in psychology and human factors, analysis typically happens quantitatively; students have little qualitative research training and use "preset plays" based on common statistical tests (p.107). "However, many CTA methods generate data that do not fit easily into standard statistical approaches" (p.107), and this is a problem since "quantification typically means stripping a body of data of its contextual links and decomposing it in order to assign numerical values" (p.108). At the same time, qualitative methods emerging from sociology, anthropology, and education tend to be focused on "topics that do not have a cognitive focus, such as analysis of social processes or attitudes surrounding terminal illness" (p.108).

Faced with this disjuncture, the authors set out to develop a suitable qualitative research approach for psychology's foci. Like many qualitative research approaches, this approach is not linear, with oscillations between structuring data and identifying meaning (p.110). It involves four main steps: preparation; structure data; discover meaning; identify/represent key findings (p.111). And the analysis involves creating "an audit trail that links raw data to eventual outcomes" (p.113). That is, it looks a lot like structured qualitative case study research.

In Chapter 8, the authors "introduce a level of cognitive phenomena—referred to as macrocognition— that emerges when we shift the focus to natural contexts. These are the types of cognition that CTA methods are uniquely designed to capture" (p.131). They discuss this level of cognition in terms of purpose, prior experience, situation, challenge, tools, team members, and organizational constraints (p.132). Macrocognition, they say later, is a "collection of cognitive processes and functions that characterize how people think in natural settings," as opposed to microcognition, which is "studied using carefully controlled methods and procedures" and is supposed to investigate basic, universal features (p.136). Think here of the contrast between Klein's contextualized field interviews and Kahnemann's word problems — or the contrast between laboratory measures of executive functions and ecologically valid measures. As the authors assert, "individuals make decisions but so do teams" and "decision making often depends on artifacts" (p.136). Cognitive activity, the authors assert (citing Hutchins), is "distributed across multiple agents as part of a stream of activity" (p.157).

Overall, I found this book to be rewarding. The authors have identified a need for a qualitative methodology in psychology, oriented to decision-making; they have drawn when appropriate from qualitative traditions in adjoining disciplines; but they have also recognized the differences between those methodological orientations and the one they need. They have carefully and responsibly developed and validated an approach that works for their objectives. And they have articulated it clearly and well—the book is well organized and easy to read. The result is a good intro for practitioners, but I think it would also be suitable for a methods class (with suitable framing). If you're interested in qualitative methodology, and especially if you're wondering why someone would pursue qualitative methods instead of quantitative ones, check it out.

(catching up)

I've been blogging much less regularly lately, about once a month. That's not a function of my reading so much as it is a function of my schedule: there's only so much time in the day, and the Wednesday mornings that I usually blog have been taken up with other things. Consequently, the books have been piling up.

Currently waiting to be blogged are:

• three books on decision-making (psychology)
• one classic book on human-computer interaction (based in anthropology)
• one book on wealth generation (business)
• a biography of a business leader

In addition, I'm reading a book describing a theory of the origin of language and in my Unread pile are books on posthumanism, the textual society, decision making, value, and sustainability. I'm hoping to clear my blogging backlog so I will be prepared to discuss those books as well. Stay tuned!