Readings :: Ethnographic Decision Tree Modeling

Ethnographic Decision Tree Modeling
By Christina H. Gladwin

I picked up this slim monograph the other week (93pp.) and read it in maybe one sitting. Part of the Qualitative Research Methods series, this 1989 book proposes a modeling strategy for interpreting ethnographic data on how people make decisions:

This method is called ethnographic decision tree modeling because it uses ethnographic fieldwork techniques to elicit from the decision makers themselves their decision criteria, which are then combined in the form of a decision tree, table, flowchart, or set of "if-then rules" or "expert systems" which can be programmed on the computer. (p.8)

The author emphasizes that decision criteria should be emic, but made explicit (p.9). She notes that "people do not rank order alternatives wholistically when they make a decision. They just choose one of several alternatives without ranking them" (p.10, citing Kahnemann & Tversky, Shoemaker, Quinn, and Arrow). Thus her approach is not linear, but rather context-sensitive, testing the interpretation of an observed behavior (p.11). Thus the ethnographic data collection can take from a few weeks up to two years, and model testing can take up to 6 months (p.13). These models, like model trains, are simplified (p.13).

When we get into the models, however, we find that there's a lot of craftwork and tacit knowledge involved. In an extended case study — how students decide whether to buy a meal plan at their dorm — she models the results of each interview, then combines them, telling us to "combine them in a logical fashion while preserving the ethnographic validity of each individual decision model" (p.39, her emphasis). Okay. The models also assume a lot about both decisions and justification, primarily by portraying branching pathways with yes/no decisions.

I like modeling qualitative data, so I can see value in this modeling process. It's a good way to aggregate messy data and see patterns that may not be visible in raw interviews. However, I do worry that these "model trains" may not take us very far: they attempt to rationalize decisions that are potentially much more complicated, and they appear to rely on untriangulated interviews, meaning that they represent what people say or recall or reconstruct about their decision making. 

Nevertheless, used judiciously, this modeling technique could be really helpful. I'll keep this book in mind in case I need to model decisions in the future. If you're interested in modeling qualitative data in this way, please do pick up this book.

Reading :: Working Minds

Working Minds: A Practitioner's Guide to Cognitive Task Analysis
By B. Crandall, G. Klein, and R.R. Hoffman

I've discussed Gary Klein's work before, and specifically how much I appreciate his attitude of trust and respect toward his participants. Klein's work focuses on how experienced professionals (such as firefighters, NICU nurses, and soldiers) make intuitive decisions in high-stakes, high-pressure environments.

To research such cases, Klein needed an ecological approach that allowed him to get at situated decision making in cases in which the participants couldn't necessarily articulate their assumptions, options, or triggers. At the same time, Klein couldn't just follow firefighters around—the events he wanted to study were just too rare, and when they happen, he didn't want his team to get in the way of rescue operations.

The approach that Klein and his partners developed for such cases is called cognitive task analysis (CTA), which "helps researchers understand how cognitive skills and strategies make it possible for people to act effectively and get things done," according to the back of this book. The book is, as the subtitle states, "A Practitioner's Guide to Cognitive Task Analysis." That is, it describes CTA and the situations in which it could be useful; it offers tools and strategies for performing CTA; and it discusses how CTA brings value to the participants. In this sense, it reminds me of Beyer and Holtzblatt's Contextual Design, a similar methodology book written by consultants for practitioners (although addressing different situations with a different methodological approach).

What struck me about Working Minds, though, was that the coauthors had developed a qualitative approach within psychology. As the authors note, in psychology and human factors, analysis typically happens quantitatively; students have little qualitative research training and use "preset plays" based on common statistical tests (p.107). "However, many CTA methods generate data that do not fit easily into standard statistical approaches" (p.107), and this is a problem since "quantification typically means stripping a body of data of its contextual links and decomposing it in order to assign numerical values" (p.108). At the same time, qualitative methods emerging from sociology, anthropology, and education tend to be focused on "topics that do not have a cognitive focus, such as analysis of social processes or attitudes surrounding terminal illness" (p.108).

Faced with this disjuncture, the authors set out to develop a suitable qualitative research approach for psychology's foci. Like many qualitative research approaches, this approach is not linear, with oscillations between structuring data and identifying meaning (p.110). It involves four main steps: preparation; structure data; discover meaning; identify/represent key findings (p.111). And the analysis involves creating "an audit trail that links raw data to eventual outcomes" (p.113). That is, it looks a lot like structured qualitative case study research.

In Chapter 8, the authors "introduce a level of cognitive phenomena—referred to as macrocognition— that emerges when we shift the focus to natural contexts. These are the types of cognition that CTA methods are uniquely designed to capture" (p.131). They discuss this level of cognition in terms of purpose, prior experience, situation, challenge, tools, team members, and organizational constraints (p.132). Macrocognition, they say later, is a "collection of cognitive processes and functions that characterize how people think in natural settings," as opposed to microcognition, which is "studied using carefully controlled methods and procedures" and is supposed to investigate basic, universal features (p.136). Think here of the contrast between Klein's contextualized field interviews and Kahnemann's word problems — or the contrast between laboratory measures of executive functions and ecologically valid measures. As the authors assert, "individuals make decisions but so do teams" and "decision making often depends on artifacts" (p.136). Cognitive activity, the authors assert (citing Hutchins), is "distributed across multiple agents as part of a stream of activity" (p.157).

Overall, I found this book to be rewarding. The authors have identified a need for a qualitative methodology in psychology, oriented to decision-making; they have drawn when appropriate from qualitative traditions in adjoining disciplines; but they have also recognized the differences between those methodological orientations and the one they need. They have carefully and responsibly developed and validated an approach that works for their objectives. And they have articulated it clearly and well—the book is well organized and easy to read. The result is a good intro for practitioners, but I think it would also be suitable for a methods class (with suitable framing). If you're interested in qualitative methodology, and especially if you're wondering why someone would pursue qualitative methods instead of quantitative ones, check it out.

Reading :: Institutional Ethnography

Institutional Ethnography: A Sociology for People
By Dorothy E. Smith

Dorothy Smith has been cited in a strain of work in professional communication studies dealing with ethnographic research. (I think the first reference I saw to her work was in one of Dorothy Winsor's articles.) Smith, a sociologist, published this 2005 book to explain and summarize institutional ethnography:

The aim of the sociology we call 'institutional ethnography' is to reorganize the social relations of knowledge of the social so that people can take that knowledge up as an extension of our ordinary knowledge of the local actualities of our lives. It is a method of inquiry into the social that proposes to enlarge the scope of what becomes visible from the site, mapping the relations that connect one local site to others. Like a map, it aims to be through and through indexical to the local sites of people's experiences, making visible how we are connected into the extended social relations of ruling and economy and their intersections. And though some of the work of inquiry must be technical, as mapmaking is, its product should be ordinarily accessible and usable, just as a well-made map is, to those on the terrain it maps. (p.29)

That is, IE takes a partnership orientation with its participants, like participatory action research (PAR).  Smith contrasts IE with what she characterizes as the typical mode of research in sociology, in which "everyday and local events" are interpreted "in terms of a framework originating in sociological and political economic discourse," a framework that displaces actual people: "Categories such as 'sociocultural differences,' 'social class,' and 'racial status' become the subjects" and "the potentially unruly (from the point of view of sociological discourse) actualities of people's everyday lives are selectively appropriated" (p.31; cf. Latour's similar argument).

In contrast, an IE "would begin in the actualities of the lives of some of those involved in the institutional process and focus on how these actualities were embedded in social relations, both those of the ruling and those of the economy" (p.31). So "research would begin by building accounts of their situation" and experiences; those accounts then define the ethnography's direction and further steps by identifying the "problematic." And Smith continues:

The institutional regime they confront would be explored from their perspective; their perspective and experience would organize the direction of the ethnographers' investigation. (p.31)

I pulled this quote because, reading the sentence within the paragraph, "they" and "their" seem to refer to "researchers" and "institutional ethnographers" in the previous sentences—but in the clause after the semicolon, "their" seems to be referring to the participants instead. I'm pretty sure Smith meant the latter interpretation.

In any case, "Institutional ethnography begins by locating a standpoint in an institutional order that provides the guiding perspective from which that order will be explored. It begins with some issues, concerns, or problems that are real for people and that are situated in their relationships to an institutional order" (p.32). From there, IE maps out social relations "so that the larger organization that enters into and shapes it becomes visible" (p.35).

Smith notes that IE may sound like Burawoy's extended case method. But the extended case method uses its investigation of people's lifeworlds to discover how the system objectively works—that is, it assumes a system that governs these individual lifeworlds. IE, on the other hand, has "no prior interpretive commitment"; "it means to find out just how people's doings in the everyday are articulated to and coordinated by extended social relations that are not visible from within any particular local setting and just how people are participating in those relations" (p.36). Ultimately, Smith sees IE as an alternate sociology, not just a methodology (p.50; see also pp.54-55). (Again, compare with Latour.)

Later in the book, Smith draws on Mead, Voloshinov, Vygotsky, and Luria to explore the relationship between consciousness and subjectivity (p.75). Interestingly, she characterizes the "text-reader conversation" as active, but "peculiar and unlike conversation in that the text remains the same no matter how many times it is read" (p.105)—I can see what she's trying to say, but I don't think this characterization is adequate within the Mead-Voloshinov-Vygotsky tradition. However, it underpins a later argument in the chapter on texts and institutions, where Smith claims that "it is the replicability of texts that substructs the ruling relations; replicability is a condition of their existence. ... Replicable and replicated texts are essential to the standardizing of work activities of all kinds across time and translocally. It is the constancy of the text that provides for standardization. ... Texts suture modes of social action organized extralocally to the local actualities of our necessarily embodied lives. Text-reader conversations are embedded in and organize local settings of work" (p.166).

In tying discourse to data, Smith later argues that when interviewing a participant, "I'm not interested in whether his account is an accurate telling of events; I am interested, rather, in his experience and how he tells it and in the traces of social relations and organization present in it" (p.138).

Later, Smith mentions grounded theory, charging that it tends to universalize the researcher's intuitions (p.160).

Overall, I found this book to be interesting, but not a revelation. It might be that Smith's earlier publications have impacted qualitative research methodology to the extent that I'm already familiar with her propositions before reading them—but much of this argument seems similar to work with which I'm already familiar, from Whyte to PAR to participatory design to Latour. And as I mentioned above, the characterization of texts seems lacking, especially the notion that texts (a) remain the same in a sociologically meaningful way and (b) in themselves standardize activities across time and space. I'm open to a weak form of that argument, but Smith seems to be pushing for a strong form that severs texts from ongoing meaningful activity.

Nevertheless, it's a solid book packed with details about IE's theory, methodology, and practice. If you're interested in qualitative research in general and IE in particular, I recommend it.

CCCC 2013 presentation

Here's my presentation for CCCC 2013, describing how I used social media to supplement more traditional qualitative research data in two recent projects.

Empirikom 2012: Analyzing computer-mediated communication in professional environments: An activity theory approach

Conf 2012-empirikom3 from spinuzzi

Here's yesterday's presentation to Empirikom 2012 in Aachen. I use the Semoptco case to illustrate three analytical constructs (genre ecologies, activity systems, activity networks) and how they can help us better understand CMC use in professional environments.

Aachen turns out to be a fantastic place, and my hosts have been extraordinarily kind.

How not to write fiction

Note: This blog post comes from a talk I presented at Texas Christian University on November 6, 2009.

How Not to Write Fiction: Style and Evidence in Qualitative Research Studies
So I no longer read fiction. I read perhaps one book of fiction a year.

It wasn't always that way. Growing up, I used to love reading fiction – especially, I admit, science fiction. In fact, the first long novel I read was Jules Verne's annotated 20,000 Leagues Under the Sea. I was intensely proud of having finished it – I was perhaps seven – and also really interested in the annotations, which taught me all about narwahls and light bulbs and so forth. Most of the science fiction I read wasn't so highbrow, though: Heinlein, Asimov, Niven, Pohl, Herbert, Burroughs (Edgar Rice, not William). I even went through a bad patch where I read Piers Anthony, which is the science fiction equivalent of drinking Keystone Light.

Don't get me wrong, I also read other genres. I read a lot of murder mysteries, mostly Agatha Christie. I read the Hardy Boys and Tom Swift. I read the Bible cover to cover at least three times by the time I finished high school. I read Solzhenitsyn's The Gulag Archipelago the summer before my senior year. And I read plenty of trash, such as Von Daniken's Chariots of the Gods. But science fiction was my passion, and I even thought that I might be interested in becoming a science fiction writer someday. Although I decided not to pursue that career in college, I kept reading fiction, at least a paperback or two a week, up to the point I started my PhD. Program at Iowa State University. And then, within a year or so, I stopped reading fiction almost entirely.

Why? It wasn't the course load. It wasn't that I thought I was suddenly too good to read fiction. It was simply that I had begun to read research studies, and I found that good research studies – the ones that were solidly grounded, well written, and intellectually curious – were more interesting than fiction to me.

Look at it this way. In fiction, you know the story is the author's construct. You know that it's often a reference to contemporary issues, sometimes an allegory, sometimes with a moral you have to intuit. You can scrutinize it to see how it's constructed. You can anticipate plot twists, sometimes far in advance, which really takes the fun out of it. You can pick holes and identify what's implausible. And although there's some pretty trippy science fiction out there, for the most part even the trippiest ideas are conventionalized enough to sell copies. Science fiction writers like to say that science fiction is not really about the future, it's about the present, and after a while you start to see the present concerns poking through the futuristic window dressing. It starts feeling preachy and contrived.

Compare that to, say, Edwin Hutchins' Cognition in the Wild, where he takes a concept as trippy as anything you see in science fiction – cognition isn't in your head, as everyone assumes, but instead it's distributed across people and artifacts – and he makes a solid case for it with compelling writing and stories that actually happened.

Compare it to Lev Vygotsky's studies in Mind and Society, where he argues that everyone has child development wrong – thought doesn't bubble out of children as language, language gets absorbed to produce thought – and he provides evidence with ingenious and bizarre experiments.

Compare it to Bonnie Nardi's Context and Consciousness, a collection of studies that is as compelling as any collection of short stories. (To me, anyway.)

These studies make claims that changed the way I saw everything, they backed them up with true stories and reproducible data, and they "showed their work" with methods sections – I could play along at home. And I had to: unlike my science fiction readings, these didn't give me the luxury of rejecting stories as implausible or suspending disbelief when they were well written. Either the theoretical framework held up and the methodology worked well, or not. And let's not forget that while fiction has a moral, research studies offer an implications section. Here's the lessons we learn; here's the work that must still be done; here's the ways in which actual people, including the real people in these stories, can benefit.

That, I decided, was the kind of study I wanted to write. Something useful, something interesting and trippy, something really readable in terms of style. Something like the studies that Bruno Latour writes. You've read Latour? He's a master stylist. Reading Latour is like drinking too much. Reading his books in the evening, you think, "this fellow is brilliant! Why isn't this clearer to his critics?" Then you wake the next day with a headache. "Ohhh, does that even make sense?" That's what I wanted to achieve – minus the headache.

A little side story here. A few years ago I was rereading some rhet-comp articles about research, and one casually mentioned that the researcher Steve Woolgar had gone as far as to make up a coauthor in one of his experimental ethnographies. That's odd, I thought. So I turned to the bibliography to see what ethnography the author was referencing. And there it was: Laboratory Life, by Steve Woolgar.

Laboratory Life, of course, was coauthored by Bruno Latour and Steve Woolgar. And it dawned on me that this author thought Latour was a figment of Woolgar's imagination! A fiction!

Style and Fiction: Who Killed Rex?
Okay, so here's a story of my own. In 2000, I found myself sitting in the Network Control Center of a regional telecommunications company in West Texas. I really didn't know what I was looking for, to be honest: I had talked this company into letting me study its employees over ten months in exchange for reports on the company's "communication problems." Whatever they turned out to me. So I started with Customer Service, moved to Customer Service Data Entry, and then to the NCC, simply to see how people were communicating with each other. I would spend two hours shadowing each individual, then observe them again a couple of weeks later, giving them an interview after the second observation. And I would pick up artifacts – printouts, sticky notes, anything that looked interesting and related.

The NCC was a big room with a screen on one wall, seats facing it on tiers, kind of like a small movie theater or a large Enterprise bridge. Specialists hot-desked in the NCC. And the phones were always ringing: the NCC was where you were transferred when you reported some sort of network problem, like lost or interrupted service, crosstalk on the line, and so forth. If your overhead line was snapped by a falling branch, if you accidentally dug up an underground cable with a backhoe, the incident would eventually make its way to the NCC. There, a specialist would create a trouble ticket, assign it to a technician – usually a technician working for the dominant telecomm provider in the area, which we'll call BigTel - and communicate with the technician until the problem was resolved.

The NCC was busy. I mean, our individual phone service is rarely interrupted, but strands of the telecomm network break all the time. So specialists would be working three tickets on the screen, typing up a fourth while talking to a customer about the fifth. It was quite incredible to watch.

So one day I was sitting in the NCC, waiting for all these incidents to form patterns, when a specialist, Nathaniel, leans over to the guy I'm shadowing, Donald. Nathaniel said:

"BigTel let somebody's dog out and it got run over. Nothing mentioned in the ticket about it."

Donald nodded and listed the tickets he was to work for that day.

A few minutes later, the NCC's assistant manager sternly told the story again, this time to the entire NCC.

"There was nothing about a dog on the ticket," he said. "You must note that."

So this intrigued me. I mentioned that I read murder mysteries as a kid, right? And although this wasn't a murder mystery per se, it was still a mystery. Of course I asked people about it. Long story short, treating this incident like a mystery helped me to understand how Telecorp communicated internally and externally. It intrigued me, just as mysteries had intrigued me when I was younger, just as research studies had intrigued me in grad school.

Eventually, I wrote it up for a collection being put together by my good friend Mark Zachry and my beloved former professor Charie Thralls.

And since I had thought of it as sort of a murder mystery, I wrote it up in those terms, making it as entertaining as possible, hoping that others would get the same sort of thrill reading this study that I had gotten reading Hutchins, Vygotsky, Nardi, and especially Latour. The editors and I were pleased with the result.

An Email and a Question
Then I got this email from a graduate student. I've redacted it, but you can get the gist.

Clay,

I'm currently taking a Qualitative Research Methods with [professor] at [university]. In our last class, we read your article "Who Killed Rex?" During our discussion, it was suggested that you fabricated your qualitative investigation of Telecorp in an effort to educate your readers ...

Fabricated!

Let's leave aside for a moment the fact that this graduate student has asked me if I have falsified research data, a serious ethical charge. Maybe he wasn't aware of how big a deal that would be. Instead, let's concentrate on the irony here. Ten years after I gave up on reading fiction because it wasn't interesting enough, I was being asked if I had written fiction. At least he hadn't accused me of being a figment of Steve Woolgar's imagination.

The answer was, of course, absolutely not. I have never fabricated or fictionalized research data. Besides being completely unethical, that would have missed the point. It would have taken all the fun out of it! How easy and how boring that would have been.

I told the grad student that the story of Rex was true, that I had the data to prove it, and that each claim had at least 2-3 different data points supporting it, data points coming from different data collection methods.

Write Like a Mystery Writer, Build the Case Like a Lawyer

I had written the case like a mystery author, but I had built the case like a lawyer. Specifically, I had sourced each claim by triangulating, and I made uncertainties clear through hedging. (This is something I just finished discussing with my undergraduate field methods class.)

There are at least two types of triangulation:

Triangulating across data types. In a qualitative study, you should have at least a few different types of data, and these should give you different views of the phenomenon. You can't rely on just one. If you just rely on observations, you'll only have your perspective - you will be able to describe what happened, but not why or how it connects with their goals. If you just rely on interviews, you get their perspective - their story - but sometimes people recall incorrectly or are just flat wrong. So as much as possible, you have to build a story by looking across the data to see what the different data types are telling you. In this case, people from the NCC were blaming the other units for the problem, and if I hadn't cross-checked their stories with the other data, I might have told a more simplistic, less accurate, less interesting story.

Triangulating across data instances. But I also had to make sure to triangulate across instances. It wasn't enough to, say, compare Nathaniel's statements with observations and artifacts pulled from his sessions. I also had to compare his statements against statements of others; his observations against observations of others; his artifacts - well you get the drift. Doing this allowed me to ask: how representative is this incident? How representative are these views? Am I talking to an oddball here, or does everyone see things this way? This sort of triangulation helps you to ensure that the most interesting or extreme views don't take over the argument.

Hedging. Finally, unlike a fiction writer, we don't get to take a God's-eye view. We don't see everything, and in fact John Law has some great examples of how he always seemed to be missing the action in his ethnography of a research center. So if we didn't see an incident, or get that interview describing it, or pick up the artifact that would confirm it, we have to fess up. This is ethos-building, it's confidence-building. Hedging is a sign of honesty and confidence, not weakness.

Some examples of hedging in "Who Killed Rex":

• "I did not witness the original complaint being filed, of course, but I did observe customer service clerks dealing with similar complaints."
• "We can't rule out Customer Service entirely, but their culpability seems less likely than Donald suggested."
• "I did not observe any sales reps asking about pets and locked gates. But ... sales reps had developed no self-regulative genres, no scripts or checklists or forms, to remind them to ask about pets and locked gates."

Being Accused of Writing Fiction
So much for building the case like a lawyer. Let's talk about writing like a mystery writer. As you can tell, this topic is dear to my heart, because I really, truly believe that the best studies can be as interesting and more world-changing - and certainly stranger - than fiction. How can you be accused of writing fiction?

Get excited about your work. When my undergrads conduct studies, I remind them that chances are, no one has ever looked at their site in this way, with this level of scrutiny, with these analytical tools. I tell them that they'll see a lot of things that their participants already know about - but they'll achieve a systematic overview and a depth of understanding that no one else ever has. That's a big deal. And it has transformative potential.

Celebrate when the data don't fit your preconceptions. When I started the Telecorp study, I was working within a standard theoretical framework, activity theory. Even my interview questions were constructed around the parts of an activity system. But soon I realized that the data were not fitting activity theory well. I was thrilled: that meant that my theoretical preconceptions were not shaping what I saw. I wasn't just repeating the story that other activity theorists had told me - I was actually recounting my own story, based on the data I had collected.

Let the evidence deepen your claims. As you triangulate, you'll find that your early hypotheses (claims) often won't be complex enough to account for the data. Great. Let the claims simmer a bit, working in the data, deciding which claims to abandon, which ones to hedge, which ones to make more specific. Often the early claims are not nearly as interesting as the ones that you've let simmer, just as a two-dimensional character isn't as interesting as a three-dimensional one.

Edit for style. Remember, build the case like a lawyer, but write like a mystery writer - or borrow from whatever genres interest you. Look how some of the more engaging studies handle the challenge. Me, I read a lot of Latour as I wrote this study, and some of his bombast and dramatism rubbed off on me. In the book, I also drew from all those readings during my youth, especially the Bible, which I quote and paraphrase and play with.

At its best, a qualitative study can be better than fiction. It can grab people, fascinate them, impact them, and improve their lives. And that's what I want to leave you with:

Make it better than fiction.

Participants can respond. Uh-oh.

This article about Jared Diamond being sued for libel should serve as a warning for qualitative researchers:

Two New Guinea tribesmen have filed a $10 million defamation lawsuit claiming Pulitzer Prize-winning author Jared Diamond wrote a New Yorker magazine article that falsely accused them of murder and other crimes.

Henep Isum Mandingo and Hup Daniel Wemp say in a single-page filing in Manhattan's state Supreme Court that Diamond's article published April 21, 2008, accused them "of serious criminal activity ... including murder."

The article was titled, "Vengeance Is Ours: What can tribal societies tell us about our need to get even?"

This incident is similar to, though not exactly the same as, the scenario I described in my chapter "The Genie’s Out of the Bottle: Leveraging Mobile and Wireless Technologies in Qualitative Research," published this year in Amy Kimme Hea's collection. There, I argued that although institutional research boards have historically been conceived as a way to protect participants from researchers' representations, social media mean that the danger is now bidirectional - participants can represent the researcher in damaging ways as well, and those representations could easily circulate more broadly than the researcher's. The nightmare scenario I described in that chapter was one in which the participants could openly contest (and ridicule) the researcher's representation, publishing their own competing accounts and evidence.

The obvious implication is that researchers must think seriously about confidence-building measures such as member checks, and even about bringing participants into the analysis, not as a matter of noblesse oblige but as a matter of self-protection.

Based on the linked article, Jared Diamond's situation sounds a bit different. He essentially accuses an interviewee of murder, and he uses the interviewee's real name - clearly not something that would be sanctioned by an institutional review board, since it could cause damage to the participant (and to the institution). Yet in other ways, the case carries a huge warning even for qualitative researchers following institutional guidelines. Diamond apparently didn't expect the participant to respond. And now the participant is not only responding in court, he is garnering considerable attention online.

The implications for methodology:

• Institutional review boards are your friends. Human subjects protocols are a contract between you and your institution; stand by those protocols and the institution will stand behind you.
  
• Methodology should include confidence-building measures, not as a matter of politeness or nicety, but as a matter of self-protection. You don't have to give away the farm by trying to achieve consensus, but you should be able to provide feedback loops and demonstrate how you'll take that feedback into account. That's especially true if you'll be using real names - a practice that is frowned upon by IRBs, but occasionally necessary.
  

Near UT? Contact me about having your workplace communication evaluated this summer.

Do you want a thorough evaluation of how people work and communicate in your organization, and a list of recommendations for improving it?

This summer, I'm teaching a class in which students will work in teams to do just that: enter organizations, observe people at work, interview them, and analyze the results in order to make solid, data-driven recommendations. Deliverables will include an interim report, a recommendation report, and prototype solutions.

So I'm looking for organizations (in the loose sense) that are

• near the UT campus
• relatively coherent (someone at the site can authorize the study)
• amenable to having small teams visit, observe, and conduct interviews
  

If this sounds interesting, drop me an email and we can chat further.

Administering "paternity tests" in qualitative research: or, creating a rigorous account of lineage that would convince Maury Povich's audience

So I got into an online discussion a while back on the question of whether coworking can trace its lineage back to other forms of loosely organized work. It's an open question, though I am skeptical. The comments on that post became really interesting, surfacing differences in assumptions and proofs. And the discussion got me thinking about how qualitative researchers establish such lineages -- an important discussion in itself, especially for people who do the type of work I do, examining workplaces, workplace development, and workplace contradictions.

Unfortunately it looks like the last four comments in the thread were lost. Rather than trying to reconstruct the conversation, I want to take the opportunity to explore the question: how can qualitative researchers establish developmental lineages? Or: How do we test for paternity? And to focus the discussion, how might a paternity test look for coworking? (The last question is not just abstract, since I'm planning to study coworking this summer and fall.)

Let's put the abstract question this way:

Does phenomenon B descend from phenomenon A? That is, did A develop into B? Did A cause B? Are they genetically related, with a specific lineage?

Here are three examples:

• Coworking (see the blog post). Does modern coworking trace its "lineage" back to pre-industrial forms of work? Is it descended from those forms of work, now reemerging? Or is it actually a different, unrelated phenomenon developing from more current work forms?


• Interface elements (See Tracing Genres through Organizations). Did elements of a modern interface descend from previous, pre-automation genres?


• Objectives (See Network). Can we trace current understandings of "universal service" back to previous understandings? Can we relate these to changes in markets and regulations?

How can we answer these questions?

• One common-sense impulse is to look for resemblance. (A and B have these things in common.) But resemblance isn't enough. The obvious rejoinder is to point out dissimilarities. (A and B also have these enormous differences.) After all, resemblance is often coincidental - and often in the eye of the beholder. (That's the tack I began to follow in my first comment on the coworking post.)
• Another common-sense impulse is to look for chronology. (B came after A.) But by itself, this is post hoc reasoning. I was born after JFK, but that doesn't mean JFK was my father. Mayan writing developed after Phonecian writing, but that doesn't mean that Mayan writing originated in Phonecia. Coworking came chronologically after other work that occurs among independent workers sharing a space, but that doesn't mean it descended from that work.

So we get a sense of why these two approaches are shaky. How do we get to a less shaky, more verifiable argument for paternity? Here's an analogy.

Analogy: Who's the Father?

Suppose you're watching a daytime talk show, like Jerry Springer or Maury Povich. The theme -- a recurring theme -- is: "My baby's father won't admit he's the father!"

You may be familiar with how this episode goes, because they all go basically the same way. The couple argues:

• The baby does/does not resemble the accused father. ("Look, he has his father's eyes.")
• The baby's conception does/does not fit the chronology. ("We got together and nine months later the baby was born.")

Oh, and the couple leans heavily on ethos. ("I would never do that." "I swear on my life that that never happened." "Believe me, I know!")

But the audience isn't credulous. They listen to these arguments for entertainment, and they have their favorites, but they know the question will be settled at the end of the episode -- with a paternity test. Without it, these arguments don't settle anything.

Conducting Paternity Tests in Qualitative Research

So, by themselves, resemblance and chronology don't constitute a paternity test. What does? How do we get to a rigorous explanation of paternity?

The question is important because research is itself an argument. And as with any other argument, you have to play the believing-doubting game in order to make that argument solid: you spend some time believing your emergent argument, then you doubt it and push yourself to find evidence that can turn back those doubts. A worthwhile audience will be skeptical -- and speaking as someone who reviews a lot of journal manuscripts, I am always skeptical of the methodology, no matter how banal the conclusions -- so you must be skeptical first and deepen your argument as much as you can. That skepticism, applied methodically, produces rigor. As my colleagues and I put it in a recent article, "Healthy research and a healthy disciplinary matrix for research involve developing a coherent and densely textured argument as a symbiotic cluster; it involves creating rhetorical rigor" (Fleckenstein et al. 2008, p.411).

So let's go back to my first book, Tracing Genres through Organizations. There, I was trying to establish the lineage of interface elements. So how did I test the paternity?

I went back to the basics: I triangulated different data sources in order to validate and deepen my analysis.

1. Artifacts. I started with the artifacts - the interface elements of GIS-ALAS (maps, menus, and dialog boxes), PC-ALAS (menus, dialog boxes), and Mainframe-ALAS (punchcards, printouts), as well as pre-automation forms and reports. I put these in chronological order and looked for similarities - but I went farther by rigorously examining dissimilarities. By doing this, I was able to
   
   • establish a set of unchanging and changing characteristics.
   • establish similarities with other preexisting artifacts (e.g., interface elements such as dialog boxes)
   • construct a reasonable story of how and why characteristics were mingled as new interfaces were developed (e.g., they had to break up this printed form's questions into two dialog boxes in order to make it work in a dialog box format)
   • establish not just a chronology or similarities, but a chain of custody in which characteristics were passed from one interface to the next.
   
   That gives us a decent story, but the story needs to be tested.


2. Documents. I then went to any documents I could find that could shed light on the transitions between interfaces. These included software manuals for the three computer programs, but also newsletter accounts, written records of the development, and in the case of GIS-ALAS, the thesis and the drafted dissertation of the developer. Doing this allowed me to
   
   • validate my reasonable story of the transitions (and in some cases, correct and deepen it)
   • gain insight into the specific decisions that led to adopting characteristics
   • validate the chain of custody of characteristics
   
   Okay, but I like to verify my verifications. So I took the obvious next step: I asked.


3. Interviews. Finally, I talked to people who had been involved in the activity and, when possible, the development of the different systems. Doing this allowed me to
   • gather more documents (see step 2), allowing me to further validate the record
   • gather unwritten history - although recollections are variable and always have to be triangulated, they can shed new light on how pieces of the documented history fit together
   • validate my reasonable story of the transitions (and in some cases, correct and deepen it)
   • gain insight into the specific decisions that led to adopting characteristics
   • validate the chain of custody of characteristics

At the end of the process, I had a verifiable story. Each part of the story, each transition - that is, each assertion I made about the lineage - was supported by at least two and usually three types of data, triangulated so that I could really pin down what happened. And the parts that I couldn't verify, I either left out or hedged so that readers knew where the weak parts of the chronology were. Based on that study, I was able to establish that some of the interface elements were actually hybrid genres, i.e., text types that could trace their lineage back to two or more separate texts originating in two or more separate activities.

In other words, I had a paternity test -- and more than that, I was able to establish the entire family tree.

Lineages and Rhizomes

Now, paternity tests and lineages are fairly restricted ways of thinking about these issues. Sometimes people enact work or organize themselves or use tools based on experiences that they idiosyncratically transfer from one focal point to an entirely unrelated one. Imagine, for instance, someone using a disused software manual as first base in an impromptu softball game -- or someone being unable to interpret a GIS-ALAS map properly because they think of the dots on the map as separate pushpins.

Issues like these are what pushed me toward looking at rhizomes in Network. Rhizomes are "anti-genealogies," as Deleuze and Guattari put it: they constitute associations, sometimes entirely idiosyncratic ones and sometimes ones that form interferences with each other. These interact with the "paternity tests" in definite and observable ways - they are often implicated in discoordinations and breakdowns, for instance - but they don't constitute lineages in themselves. (How can they? They're "anti-genealogies"). John Law does a great job of discussing rhizomes and the problems they pose for research in his book After Method, a book that has become very familiar to my grad students.

Rhizomes seem to completely destabilize lineages, and therefore "paternity tests." Think in terms of Ulmer's "chora," or resonations among entirely different meanings for the same word. Or in terms of "genre ecologies" in Tracing Genres through Organizations, in which a given text can represent a hybrid of two or more different genres, each with its own logic, assumptions, and associations. Or in terms of "splicing" in Network, in which whole activities are sutured together to form new, destabilized and dynamic ones. But in all these cases, unless they're completely tacit and idiosyncratic, these associations can be isolated and traced through some careful interview work and the right analytical technique -- grounded theory, for instance, excels at building a picture of a loose, slippery concept. That doesn't mean that everything can be simply reduced to a line of development, especially since qualities emerge from the interplay/hybridization/splicing among different lines of development. But it does mean that the careful researcher can still tease out these lines of development and build a case for each, separating the historical-developmental characteristics from the emergent/dynamic ones.

Giving Coworking a Paternity Test

So let's apply this approach to the case at hand. Can we give coworking a paternity test?

Sure. We can determine possible lines of development by looking at similarity and chronology, and we can verify those lines through our paternity test, careful triangulated research.

One obvious way would be to interview the coworkers. For instance, in I'm Outta Here, the authors interview people who were involved in the early coworking movement, turning up a number of precedents. These precedents don't stretch back to trades and guilds, but they do make definite connections to mid-20th century experiments in artist collectives. Interviews like these might at least establish some perceived lineage. Of course, you have to be careful to avoid feeding answers to your informants, who might be eager to see connections to older forms of work and who may express affinities or ideas rather than actual lines of development. "We're like a clan of nomads" is very different from "coworking can trace its lineage directly to nomadic clans."

If the interviews suggest a lineage, we could then try to establish a line of cultural development for that lineage. For instance, if a participant claims that he sees coworking as developing from artist colonies and collectives (I'm Outta Here p.20), how did that development work? Can we establish that an art collective turned into a coworking site, or its members migrated to that site, or its principles migrated to a professional organization to which workers belonged before they started their own coworking site? If a participant claims that coworking is the descendant of guild work, can we establish that guilds' tools, principles, or ideas survived in (say) workers' unions and were revived when union members became coworkers? Do these ideas, tools, or organizational structures have a traceable chain of custody? Or are they just similar, but developmentally separate, solutions to similar problems?

Finally, we could look at documents. Was the business plan of a coworking site influenced by a manifesto written by an art collective? Was the site's layout explicitly patterned on older studios?

If we can establish and triangulate evidence that allows you to clearly delineate these lines of development, their resonances and interferences, then we've successfully administered the paternity test.

Parting Thoughts

I'm really pushing here for an understanding of research as an argument, a rigorous argument that should emerge from the data and that should be hedged appropriately. That approach sometimes seems unnatural to us, especially for graduate students who are first beginning research: they know that research is unfamiliar to them, they're still working on mastering it, and they expect a lot of charity from their readers. That charity should be lent -- but only in the early stages, only by instructors who can mentor the beginning researchers and guide them into asking the proper questions. When research becomes "real," taken seriously, it also needs to be a solid enough argument to fly. And sometimes that means scaling back the scope of the research and its claims.

Certainly that doesn't mean cutting off speculation. But speculation should be clearly marked, heavily bracketed, and usually should appear in the Implications section, where the researcher can suggest it as a new research question.

But when you can, turn speculation into verification. Don't restrict yourself to saying "Don't you see his eyes? He looks just like his father!" Because you can expect your readers to be at least as smart and critical as Maury Povich's audience.

Rolling your own free, customized, free, multiplatform, and free qualitative data analysis tool. For free.

Qualitative data analysis tools are expensive. When I came to UT in 2001, I had the university spend $500 on one popular QDA tool, NVivo. It was going to change the way I did research. So I installed it, played around with it, was not impressed, and abandoned it.

Much more recently, I decided to try HyperResearch on the advice of a grad students from Education. Again, UT sprang for the $400 needed to buy it. I used it for two studies and again, I was not impressed: in some ways it was very limiting, particularly in terms of relating various types of data and coding. The interface was clunky.

And look: $900 spent for nothing.

But between those two times, I managed to analyze 89 sets of observations, 84 interviews, and assorted artifacts. This work followed me across three platforms (Linux, MacOSX, OpenZaurus), and it didn't involve an off-the-shelf qualitative research tool. I'm coming back to this solution for managing the data in my latest study, a study of collaboration and project management at high tech organizations. It offers better print formatting, more flexible data analysis, and multiple interfaces that can be chosen for the specific type of analysis or data entry. It's multiplatform. Fast. And it didn't cost me a dime.

So how do you save $400, $500, or even $900 on your next qualitative research project? It takes a little setup, but you can do it.

Needs
When you're analyzing qualitative data, you might have several different kinds of data. Here's the data types I regularly use:

• Interviews (audio recordings and transcriptions)
• Observations (transcribed field notes)
• Artifacts (usually digital photos or paper that can be scanned; I have also recorded ambient noise at sites.)

You might also use other data, such as system logging.

In addition, you typically have administrative data such as information on participants (I include first and last name, pseudonym, and title at minimum).

For each of these data, qualitative analysis includes coding. You can code in several different ways, but let's keep it simple and think of coding as free-form tagging.

So how do you make sense of all this? Let's start with some don'ts:

Don'ts
Don't use Excel or other spreadsheets. Spreadsheets only offer two dimensions, and that means you're very limited in how you analyze the data. You'll end up doing one of the following:

• Creating a spreadsheet for each datatype. So you'll have spreadsheets for observations, for interviews, etc. Since spreadsheets don't provide an intuitive or robust way to link data between spreadsheets, you'd have to do that connective work by hand.
• Creating a single spreadsheet into which all data go. This will involve tremendous redundancy, with several fields going empty in every entry -- and lots of redundant data, since you'll have to tag name and date for every entry.

Don't try to manage all this outside of a table. Sure, you could dump your data in a big Word file and use comments for tagging, and sure, you could search text and comments. But you lose a lot of granularity that way, as well as the ability to gain a top-level view (e.g., how many times did I use this code vs. that code?).

Don't store your data online. Several free web-based services offer great solutions. But your data will not be secure. In many cases, you simply won't be allowed to store your data on an unencrypted server that isn't administered by the university.

So that's what you don't do. Now here's what I do.

Overview of My System
I use a MySQL database to store the data, with a different database table for each kind of data. The first table to set up is the Participant table, with each participant receiving a key index number. Other tables are all indexed by that participant number, so I can join tables based on participant.

Each table has a CODES field where I can insert codes from a list. I keep the list of codes in a text editor and surround each one with asterisks like this:

**COMPANY_HISTORY**

The asterisks allow me to search across a table and pick up just the codes -- searching for "**COMPANY" picks up codes that start with that string, while searching for "COMPANY" might pick up uses of the actual word in interview or observational notes.

To analyze the data, I use several MySQL front ends, including YourSQL, CocoaMySQL, and phpMySQL. These front ends are all free, they afford different views of the data, but they all work on the same underlying data. The result is far more flexibility than I would get from an off-the-shelf QDA tool.

Limitations
Obviously, this solution isn't for everyone:

• You don't have to learn SQL, but learning just a little bit will make your life a lot easier.
• You may have a hard time storing files in your SQL database, depending on your front end. I typically store them on the hard drive and store filenames and metadata in the database.
• This method allows you to code by line, not by line portions or longer blocks.

How to Set it Up
The setup is not hard, but you'll need to be comfortable with uncertainty. Or get your system administrator to do it.

1. Download and install MySQL.
Go to mysql.com (or mysql.org) and download the free software. It has versions for several operating systems. The site also has a ton of documentation; keep a window open for installation.

2. Download one or more MySQL front ends.
Cruise on over to sourceforge.net and search for SQL. You should get a large list of SQL utilities and clients, some of which will be applicable, many of which won't be. I am using OSX, so I downloaded the following front ends:

• YourSQL
• CocoaMySQL
• phpMySQL (this one runs on your internal web server, so it works across platforms, just like MySQL. It will take some additional setup.)

3. Create a database.
Follow your MySQL installation instructions to set a root user and password. (You can set different user and permission levels, but if you're the only one using the database, why bother?)

Once you do this, run your front end (or one of them, if you downloaded several) and follow instructions to connect to MySQL. Then create a database. I suggest naming it something descriptive -- not "research". For instance, I named the database for my current project "research-pm" -- the same name I used for my tags in GMail, GDocs, and Remember the Milk for the same project.

4. Create a table for participants. Create rows.
Now you create tables within the database. MySQL is a relational database, which means that you can relate the tables in different ways once you have them set up. I typically make the participants table the "handle" for most of the rest of the database, since most of my analysis focuses on what individuals do and say. So we create that one first.

So what do you need to know about your participants? I usually put in the following information:

• pkey: a participants key. It's a unique integer that identifies the participant. When you refer to participants in other tables -- such as observational notes -- you can use that same number to designate the same participant in these other tables.
• lname: Participant's last name.
• fname: Participant's first name.
• fname_p: Pseudonym.
• position: text field for their job title (or similar information that might be relevant, such as profession).
• site: If the study includes multiple sites, use either a text string or a number to indicate each.
• Observation and interview dates: Depending on the data collected, you might or might not include these dates. Usually you can get these from querying the appropriate data tables.

Once you have roughed out the participant table, fill it out with information about each participant.

5. Create tables and rows for each kind of data you collect.
Each will be indexed to the participants table. For my current study, I created:

• observations
• interviews
• interviewfiles
• artifacts
• site notes

For each of these, create at least the following:

• key: The unique key for this piece of data. If it's from an observation, you might call this "okey," etc.
• pkey: This field links the individual to her or his data. If a given observation was of participant 1, you'd put a 1 here.
• date: The date you collected the data. If it's an observation, you might call it "obsdate," etc.
• text: The data itself. For instance, if you're filling in observational notes, "obstext" would contain perhaps a paragraph from your notes. If it's an interview, "inttext" would contain an answer or paragraph from the transcribed interview.
• codes: The codes you assign to this piece of data.
• notes: Any additional information you might want to insert that doesn't fit into the fields above. Sometimes I use this to make notes about further investigation, artifacts I should collect, or methodological issues.

6. For each table, fill out rows.
You can do this manually via one of the front ends. You'll find that each front end has advantages and disadvantages in terms of data entry.

If you don't mind learning a little SQL, you can take your raw data (say, observational notes or transcribed interview notes) and insert the appropriate SQL around them with some search and replace commands. Once you do that, you can plug the whole mess in as a single query and it'll update the table with that data. That's what I do. It's much faster as long as you're willing to spend half an hour learning the appropriate SQL command (INSERT).

7. Code the table.
Now that the data are in the tables, code each table. In this scheme, that means filling the "codes" field for each row of each table. Codes can come from your starter codes, open coding, axial coding, or all three. I typically put them all in the same field; you could differentiate them or place them in different fields if you think you need that level of complexity.

Note: If you code thousands of lines of data with a code (say, **WORKPLACE**) and then decide you really need to rename this code (say, to **WORK**), you can do a search-and-replace with the "update" command. See documentation for details.

Similarly, you can do autocoding with an "update" command. For instance, suppose you want to make sure that each mention of "msword" in the field notes is coded with **SOFTWARE_OFFICE**. You can use "update" to search for those incidents and code them appropriately. Brute-force coding can be tricky -- you risk false positives and broad-brush characterization of the data -- but depending on your data, it can also be very useful and gain a lot of traction quickly.

How to Search
Now that you've entered and coded the data, you can do simple and complex searches.

1. Simple searches within tables
These are searches within one table. For instance, suppose you want to find a mention of msword in your observational notes just so you can look up the context. Or you want to see how many interview notes are coded with **SOFTWARE_OFFICE**. I usually use these two tools:

Search-as-you-type (YourSQL)
I love search-as-you-type. The idea is that as you start typing the string, the results reduce. Eventually you have zeroed in on the data you want, even before you're done typing.

The advantage is that you get the results quickly. The disadvantage is that this method searches across all fields, so you might get false positives. Suppose you're looking for "software" in the observational notes, but you catch all instances of **SOFTWARE_OFFICE** in the codes.

Search by string (CocoaMySQL)
This method allows you to specify the field and the relationship before you search. So you might set "obskey=1" to catch all observations of participant 1, or "codes like "%**SOFTWARE_%" to catch all observations where the codes field includes a code starting with "**SOFTWARE_".

The advantage is that the search is fine-grained and focuses on just one field. The disadvantages are that (a) it's not as fast as search-as-you-type and (b) you can't set up searches that look in more than one field.

But if you want to set up more complex searches that go across tables, you'll have to learn a little more SQL.

2. Complex searches joining tables
Since MySQL is relational, you can link these tables you've set up, and the result is a much more powerful set of queries.

Here's an example from the Telecorp study that became my second book. I had the following tables:

• "workers"
• "interviews"

Now suppose I want to grab all interview notes for Customer Service workers that are coded ***JOB_DESCRIPTION***", then append the workers' first names and pseudonyms to them so I can remember who they are. I ran this query so that I could see how the many different CS workers understood their jobs, especially so that I could zero in on differences in those understandings.

That's too complex for the simple queries earlier. So I ran the simple SQL query. The names after dots (ex: workers.fname) are field names in the given table.

select workers.wkey, workers.fname, workers.fname_p, interviews.notes, interviews.codes from workers, interviews where ((workers.area='Customer Service') and (workers.wkey=interviews.wkey) and (interviews.codes like '%**JOB_DESCRIPTION**%'))

So we can get really specific searches that join the different tables and allow us to slice the data in different ways. I could have added further codes beyond job description, searched across additional areas, specified a date, etc. In fact, I did do all of these, and I occasionally joined three tables to yield really interesting connections among the different types of data.

Formulating these can be a pain, so I formulate them once, make sure they work, then save them. If I want to run it again with a different code, I copy and paste.

How to Print
One big problem with HyperResearch is that it does an appalling job printing data. In the system I've described, you could print in a number of ways. The best two are:

• Use phpMySQL to generate the table you want, then print from the browser.
• Use MySQL from the command line to dump the query into an HTML file.

As always, see the documentation.

Conclusion
So that's a lot to absorb, and I would have to write an entire tutorial to give you a more detailed idea of how to implement this system. Since I'm sort of busy with research, I won't do that. But don't hesitate to comment with specific questions.

You can't use Evernote for qualitative data analysis, even though it would be perfect

I mentioned a few days ago on my Twitter stream that qualitative data analysis tools are expensive and tended to lag other software, and that I preferred to build my own customized databases for QDA with MySQL and various front ends. More on that soon. But first, a few words about Evernote.

Evernote is described this way:

Evernote allows you to easily capture information in any environment using whatever device or platform you find most convenient, and makes this information accessible and searchable at any time, from anywhere.

Essentially, you install Evernote on all of your input devices, such as your laptop and iPhone, and capture the data you find most useful in the ways that you find most useful. So suppose that you see an interesting note on a whiteboard. You take a photo and it gets uploaded to Evernote. Text, audio, etc. all go to the same place and are accessible anywhere. Brilliant.

Whatever gets uploaded is "run ... through our recognition technology" and synchronized across all your input devices. Then you tag it and make notes. Tagging, of course, is a great way to code data. Now you have a vast, annotated, searchable database of heterogeneous data elements that can be accessed from anywhere and that is automatically synched (and therefore backed up). Perfect for qualitative data analysis, right?

Alas, no. Because any self-respecting institutional research board would balk at raw qualitative data being stored on a machine or server that is not owned and properly secured by the university. My IRB, for instance, specifies that data must be password-protected and stored on a hard drive that is encrypted at rest.

Hence I use a local, password-protected database running on an encrypted hard drive, and I set my screen saver to require a password for wakeup. I put audio, photos, etc. on the same hard drive. The drive gets backed up to a secure server. And anything I can't stick on the hard drive, such as physical collateral I pick up at the site, goes into a locked closet in my office.

How to perform card sorting

An article on this simple and inexpensive UI research technique.

Card sorting is a simple and effective method with which most of us are familiar. There are already some excellent resources on how to run a card sort and why you should do card sorting. This article, on the other hand, is a frank discussion of the lessons I’ve learned from running numerous card sorts over the years. By sharing these lessons learned along the way, I hope to enable others to dodge similar potholes when they venture down the card sorting path.

Card Sorting: Mistakes Made and Lessons Learned :: UXmatters

Blogged with Flock