Thursday, April 02, 2009

Administering "paternity tests" in qualitative research: or, creating a rigorous account of lineage that would convince Maury Povich's audience

So I got into an online discussion a while back on the question of whether coworking can trace its lineage back to other forms of loosely organized work. It's an open question, though I am skeptical. The comments on that post became really interesting, surfacing differences in assumptions and proofs. And the discussion got me thinking about how qualitative researchers establish such lineages -- an important discussion in itself, especially for people who do the type of work I do, examining workplaces, workplace development, and workplace contradictions.

Unfortunately it looks like the last four comments in the thread were lost. Rather than trying to reconstruct the conversation, I want to take the opportunity to explore the question: how can qualitative researchers establish developmental lineages? Or: How do we test for paternity? And to focus the discussion, how might a paternity test look for coworking? (The last question is not just abstract, since I'm planning to study coworking this summer and fall.)

Let's put the abstract question this way:
Does phenomenon B descend from phenomenon A? That is, did A develop into B? Did A cause B? Are they genetically related, with a specific lineage?
Here are three examples:
  • Coworking (see the blog post). Does modern coworking trace its "lineage" back to pre-industrial forms of work? Is it descended from those forms of work, now reemerging? Or is it actually a different, unrelated phenomenon developing from more current work forms?

  • Interface elements (See Tracing Genres through Organizations). Did elements of a modern interface descend from previous, pre-automation genres?

  • Objectives (See Network). Can we trace current understandings of "universal service" back to previous understandings? Can we relate these to changes in markets and regulations?
How can we answer these questions?
  • One common-sense impulse is to look for resemblance. (A and B have these things in common.) But resemblance isn't enough. The obvious rejoinder is to point out dissimilarities. (A and B also have these enormous differences.) After all, resemblance is often coincidental - and often in the eye of the beholder. (That's the tack I began to follow in my first comment on the coworking post.)
  • Another common-sense impulse is to look for chronology. (B came after A.) But by itself, this is post hoc reasoning. I was born after JFK, but that doesn't mean JFK was my father. Mayan writing developed after Phonecian writing, but that doesn't mean that Mayan writing originated in Phonecia. Coworking came chronologically after other work that occurs among independent workers sharing a space, but that doesn't mean it descended from that work.
So we get a sense of why these two approaches are shaky. How do we get to a less shaky, more verifiable argument for paternity? Here's an analogy.

Analogy: Who's the Father?

Suppose you're watching a daytime talk show, like Jerry Springer or Maury Povich. The theme -- a recurring theme -- is: "My baby's father won't admit he's the father!"

You may be familiar with how this episode goes, because they all go basically the same way. The couple argues:
  • The baby does/does not resemble the accused father. ("Look, he has his father's eyes.")
  • The baby's conception does/does not fit the chronology. ("We got together and nine months later the baby was born.")
Oh, and the couple leans heavily on ethos. ("I would never do that." "I swear on my life that that never happened." "Believe me, I know!")

But the audience isn't credulous. They listen to these arguments for entertainment, and they have their favorites, but they know the question will be settled at the end of the episode -- with a paternity test. Without it, these arguments don't settle anything.

Conducting Paternity Tests in Qualitative Research

So, by themselves, resemblance and chronology don't constitute a paternity test. What does? How do we get to a rigorous explanation of paternity?

The question is important because research is itself an argument. And as with any other argument, you have to play the believing-doubting game in order to make that argument solid: you spend some time believing your emergent argument, then you doubt it and push yourself to find evidence that can turn back those doubts. A worthwhile audience will be skeptical -- and speaking as someone who reviews a lot of journal manuscripts, I am always skeptical of the methodology, no matter how banal the conclusions -- so you must be skeptical first and deepen your argument as much as you can. That skepticism, applied methodically, produces rigor. As my colleagues and I put it in a recent article, "Healthy research and a healthy disciplinary matrix for research involve developing a coherent and densely textured argument as a symbiotic cluster; it involves creating rhetorical rigor" (Fleckenstein et al. 2008, p.411).

So let's go back to my first book, Tracing Genres through Organizations. There, I was trying to establish the lineage of interface elements. So how did I test the paternity?

I went back to the basics: I triangulated different data sources in order to validate and deepen my analysis.
  1. Artifacts. I started with the artifacts - the interface elements of GIS-ALAS (maps, menus, and dialog boxes), PC-ALAS (menus, dialog boxes), and Mainframe-ALAS (punchcards, printouts), as well as pre-automation forms and reports. I put these in chronological order and looked for similarities - but I went farther by rigorously examining dissimilarities. By doing this, I was able to
    • establish a set of unchanging and changing characteristics.
    • establish similarities with other preexisting artifacts (e.g., interface elements such as dialog boxes)
    • construct a reasonable story of how and why characteristics were mingled as new interfaces were developed (e.g., they had to break up this printed form's questions into two dialog boxes in order to make it work in a dialog box format)
    • establish not just a chronology or similarities, but a chain of custody in which characteristics were passed from one interface to the next.

    That gives us a decent story, but the story needs to be tested.

  2. Documents. I then went to any documents I could find that could shed light on the transitions between interfaces. These included software manuals for the three computer programs, but also newsletter accounts, written records of the development, and in the case of GIS-ALAS, the thesis and the drafted dissertation of the developer. Doing this allowed me to
    • validate my reasonable story of the transitions (and in some cases, correct and deepen it)
    • gain insight into the specific decisions that led to adopting characteristics
    • validate the chain of custody of characteristics

    Okay, but I like to verify my verifications. So I took the obvious next step: I asked.

  3. Interviews. Finally, I talked to people who had been involved in the activity and, when possible, the development of the different systems. Doing this allowed me to
    • gather more documents (see step 2), allowing me to further validate the record
    • gather unwritten history - although recollections are variable and always have to be triangulated, they can shed new light on how pieces of the documented history fit together
    • validate my reasonable story of the transitions (and in some cases, correct and deepen it)
    • gain insight into the specific decisions that led to adopting characteristics
    • validate the chain of custody of characteristics
At the end of the process, I had a verifiable story. Each part of the story, each transition - that is, each assertion I made about the lineage - was supported by at least two and usually three types of data, triangulated so that I could really pin down what happened. And the parts that I couldn't verify, I either left out or hedged so that readers knew where the weak parts of the chronology were. Based on that study, I was able to establish that some of the interface elements were actually hybrid genres, i.e., text types that could trace their lineage back to two or more separate texts originating in two or more separate activities.

In other words, I had a paternity test -- and more than that, I was able to establish the entire family tree.

Lineages and Rhizomes

Now, paternity tests and lineages are fairly restricted ways of thinking about these issues. Sometimes people enact work or organize themselves or use tools based on experiences that they idiosyncratically transfer from one focal point to an entirely unrelated one. Imagine, for instance, someone using a disused software manual as first base in an impromptu softball game -- or someone being unable to interpret a GIS-ALAS map properly because they think of the dots on the map as separate pushpins.

Issues like these are what pushed me toward looking at rhizomes in Network. Rhizomes are "anti-genealogies," as Deleuze and Guattari put it: they constitute associations, sometimes entirely idiosyncratic ones and sometimes ones that form interferences with each other. These interact with the "paternity tests" in definite and observable ways - they are often implicated in discoordinations and breakdowns, for instance - but they don't constitute lineages in themselves. (How can they? They're "anti-genealogies"). John Law does a great job of discussing rhizomes and the problems they pose for research in his book After Method, a book that has become very familiar to my grad students.

Rhizomes seem to completely destabilize lineages, and therefore "paternity tests." Think in terms of Ulmer's "chora," or resonations among entirely different meanings for the same word. Or in terms of "genre ecologies" in Tracing Genres through Organizations, in which a given text can represent a hybrid of two or more different genres, each with its own logic, assumptions, and associations. Or in terms of "splicing" in Network, in which whole activities are sutured together to form new, destabilized and dynamic ones. But in all these cases, unless they're completely tacit and idiosyncratic, these associations can be isolated and traced through some careful interview work and the right analytical technique -- grounded theory, for instance, excels at building a picture of a loose, slippery concept. That doesn't mean that everything can be simply reduced to a line of development, especially since qualities emerge from the interplay/hybridization/splicing among different lines of development. But it does mean that the careful researcher can still tease out these lines of development and build a case for each, separating the historical-developmental characteristics from the emergent/dynamic ones.

Giving Coworking a Paternity Test

So let's apply this approach to the case at hand. Can we give coworking a paternity test?

Sure. We can determine possible lines of development by looking at similarity and chronology, and we can verify those lines through our paternity test, careful triangulated research.

One obvious way would be to interview the coworkers. For instance, in I'm Outta Here, the authors interview people who were involved in the early coworking movement, turning up a number of precedents. These precedents don't stretch back to trades and guilds, but they do make definite connections to mid-20th century experiments in artist collectives. Interviews like these might at least establish some perceived lineage. Of course, you have to be careful to avoid feeding answers to your informants, who might be eager to see connections to older forms of work and who may express affinities or ideas rather than actual lines of development. "We're like a clan of nomads" is very different from "coworking can trace its lineage directly to nomadic clans."

If the interviews suggest a lineage, we could then try to establish a line of cultural development for that lineage. For instance, if a participant claims that he sees coworking as developing from artist colonies and collectives (I'm Outta Here p.20), how did that development work? Can we establish that an art collective turned into a coworking site, or its members migrated to that site, or its principles migrated to a professional organization to which workers belonged before they started their own coworking site? If a participant claims that coworking is the descendant of guild work, can we establish that guilds' tools, principles, or ideas survived in (say) workers' unions and were revived when union members became coworkers? Do these ideas, tools, or organizational structures have a traceable chain of custody? Or are they just similar, but developmentally separate, solutions to similar problems?

Finally, we could look at documents. Was the business plan of a coworking site influenced by a manifesto written by an art collective? Was the site's layout explicitly patterned on older studios?

If we can establish and triangulate evidence that allows you to clearly delineate these lines of development, their resonances and interferences, then we've successfully administered the paternity test.

Parting Thoughts

I'm really pushing here for an understanding of research as an argument, a rigorous argument that should emerge from the data and that should be hedged appropriately. That approach sometimes seems unnatural to us, especially for graduate students who are first beginning research: they know that research is unfamiliar to them, they're still working on mastering it, and they expect a lot of charity from their readers. That charity should be lent -- but only in the early stages, only by instructors who can mentor the beginning researchers and guide them into asking the proper questions. When research becomes "real," taken seriously, it also needs to be a solid enough argument to fly. And sometimes that means scaling back the scope of the research and its claims.

Certainly that doesn't mean cutting off speculation. But speculation should be clearly marked, heavily bracketed, and usually should appear in the Implications section, where the researcher can suggest it as a new research question.

But when you can, turn speculation into verification. Don't restrict yourself to saying "Don't you see his eyes? He looks just like his father!" Because you can expect your readers to be at least as smart and critical as Maury Povich's audience.


Douglas Walls, PhD said...

Not sure I have enough to respond to this entire post here but there are some ideas I do want to tackle. Certainly you have something talking about how similarity does not mean anything in genre beyond mere pattern recognition. If it did, I would have to believe that Frank Zappa caused the Toyota Truck symbol.

I think most research deals in probabilities. Good qualitative research let’s you know what the rules of those probabilities are during the methodology sections. Rigor comes from saying “here are the rules I am going to play by” and it is up to people reading to decide if you as a researcher have a) made honest rules or b) played by them. When you are talking about qualitative research you have to establish the probabilities up front and state them. Well, you don’t HAVE to, people get away with it all of the time by saying “teacher research” or “ethnographic” or “experimental design” etc. but there is so much variety between how those methods are deployed in qualitative field research that it doesn’t black box well. So you have to explain these things well enough to grab the audience on board.

Triangulation works because well because a researcher can point lots of stuff that is all pulling toward a conclusion. That increases the probability of claims in the heads of many people because triangulation is HARD to do. It is also AWESOME because it puts multiple data sets about something next to each other and, hopefully, lets a pattern emerge that makes sense given the rules outlined for the research.

I like your Maury analogy a lot. After all, why should "Look, he has his father's eyes" be worse evidence then the answer Maury has on his card? Maury’s card is completely black boxed! We have no IDEA how the answer gets on that card except to believe Maury when the show says it’s “the results of a paternity test” and that both parties have decided to abide by the results. Yet it works because everyone involved, the audience, potential and actual parents, Maury and the audience believe it to being meaningful.

Which, of course, leads to the problem you are talking about here. I am inclined to say who is the real daddy isn’t even the right question because who is functioning as the dad is more interesting to me. I think actor-networks account for thinking about how something called “lineages” might interact with rhizomes. They don’t even have to be grounded in some sort of actuality except to the extent that they impact and do things in that rhisomatic structure. It is a mistake to think about one species as an ecosystem. Ecological modeling of genre’s should account for the interplay between genres and their purposes and influences rather than causal relationships. After all, causal/parentage denotes that a) identifiable “separateness” which might just be a sampling error or b) that the “parent” one has stopped or ceased to be. Speaking broadly, I am more interested in the ecological model than an evolutionary one with missing links and a genetic truth. Genres as maps of social functions are really, really interesting but there is also a ganger of them becoming some sort of “indicator species” for a whole for the meaning making ecology (“workplace” “homelife” etc.) that they exist in. I think information/knowledge workers live in a world where genres move back and forth more and more through these different ecologies influencing each other in really interesting and dynamic ways. Tracings those influences seems exciting to me. Idiosyncratically, how things function where they are now and how cleaver accents shift and align the pieces they are given is far more interesting to me.

After all, just because I share 50% of my genetic material with a banana doesn’t mean the banana will . . . OK that sentence got away from me but you get the idea.

Clay Spinuzzi said...

Doug, great comments. I can tell that we are really going to abuse this Maury analogy, but here goes...

You suggest that "It is a mistake to think about one species as an ecosystem. Ecological modeling of genre’s should account for the interplay between genres and their purposes and influences rather than causal relationships." Yes, try that line on Maury's audience!

MAURY: "I am inclined to say who is the real daddy isn’t even the right question because who is functioning as the dad is more interesting to me."


DAD: "Do I have a legal and ethical obligation to take care of this child? Do I need to function as the dad? Who will pay child support? My colleague at work, Ralph, is bragging that he is the real father, undermining my reputation - should I just shrug and keep letting him do that? Should we invite my parents over for Thanksgiving, or Ralph's parents?"

MOM: "My husband has been running me down at church - should I just let him? He keeps saying he's going to leave - can't I anchor him with his commitments, locking him into the role of Dad? If he leaves, can't I at least ensure that he pays child support?"

PARENTS: "Maury, we came to you with a crisis. Don't tell us it doesn't matter!"

The parents don't simply come to Maury at random. They come when they perceive a crisis, one that doesn't just involve a paternity test but also threatens relationships (with each other, their families, their churches and coworkers) and identity (should the dad be a dad?). The crisis isn't just that of development but also interessement - not just the lineage of an individual factor, but also how the lineage or perceived lineage affects the current settlements that have defined the actors and locked them into place (to use the language of actor network theory).

I agree that it's not always useful to trace lineages. Maybe even most of the time. And a single lineage of a single actor is always being supplemented or crosswired or buffeted by others in a complex system or ecosystem.

But surely it's important sometimes. When people go onto Maury and dispute a paternity test, it's because they have reached some sort of crisis. Maybe their responsibilities are unclear, maybe their relationship has reached a turning point, maybe one of them is weighing a more fruitful partnership. Whatever precipitates it, these people think that establishing paternity is vital, and they know (or believe) Maury can perform a forensic examination that will settle the issue. In fact - to stretch the analogy - in qualitative reseach, "Maury" can't just rely on the card; he has to demonstrate how the paternity test works so that we, the audience, can reproduce it at home.

And here's another possible reason for performing a real-life paternity test: genetic predisposition. If the "father" is predisposed to (say) Mediterranean anemia, then the child will also have that predisposition. Does the child in fact carry those genes? If that predisposition is there, it influences our deliberations: we might detect issues and figure out how to deal with them if they come up again.

We tend to see the epideictic function on Maury - praise and blame - but the forensic and deliberative are also critical. And I think these are the case if we're looking at development of individual phenomena within a complex ecosystem, including but not limited to genres. Sure, they'll interact in complex systems, but they still have developmental trajectories and predispositions (developmentally speaking) and histories of interessement (rhizomatically speaking). When we see that the node map is covered seemingly at random with six-digit numbers, for instance, we want to know how this configuration arose - and when we contemplate designing a new system, we want a best guess at whether we should keep those numbers and how we should handle the way they integrate into the new system. Because we are really not going to start with a blank slate: the settlements (and here I'll switch back to the language of actor-network theory) are built on long histories of prior negotiations, and these folded histories are what give a system its stability. The actants aren't just clumping in the mixing bowl (yes, another metaphor); they're defined as actants, locked-for-now into relationships, through this process of interessement. The fact that they might dissolve to form other actors at some point doesn't mean that the current interessement doesn't matter - it's not arbitrary, it's just one of many possible interessements. The deadbeat could become a dad, the grandparents could become strangers, or alternately the "unfaithful" wife could be vindicated as faithful, and friends will boast that they knew it all along.

So what we're getting in this discussion, I think, is this tug-of-war between developmental and associative perspectives (i.e., weaving and splicing). Each is going to have its uses. Each might be the focus at some point. Each, I think, might have some occasion for forensic, epideictic, and deliberative investigations. It may not matter to Maury whether the baby has a genetic lineage, but it matters a lot to the couple and their circle of friends, as its effects reconfigure their relationships!

Sorry the comment is so long and muddled - if I had had more time, I would have written less :/

Douglas Walls, PhD said...

Normally I have a personal rule about engaging in metaphors where I have been cast as a daytime talk show host. With that exception in mind, I love everything about this part of your post.

Let me just say, for the record, that I would never tell someone who their father is doesn't matter he said trying not to alienate people with fathers or who are fathers. Happy early Fathers Day, get your Dad a card!

I think our metaphor is showing its threadbareness at this point. I thought parents = other genres? Are the parents writers then? Does it mean intellectual property or something?

What ever they are, I think this part is great:

DAD: "Do I have a legal and ethical obligation to take care of this child? Do I need to function as the dad? Who will pay child support? My colleague at work, Ralph, is bragging that he is the real father, undermining my reputation - should I just shrug and keep letting him do that? Should we invite my parents over for Thanksgiving, or Ralph's parents?"

MOM: "My husband has been running me down at church - should I just let him? He keeps saying he's going to leave - can't I anchor him with his commitments, locking him into the role of Dad? If he leaves, can't I at least ensure that he pays child support?"

Here we get at what being a father means different things to different people! Seeing how those notions of fatherhood intertwine and intersect, there is nothing random about it at all.. That would be VERY interesting to me. Coming up with a story about how MOM and DAD got to think that this is what “fatherhood-ness” means is interesting. I can certainly understand how this might fit into the associative/developmental dynamic but I think associative view is interested in the development of the idea . It is just interested in it from a different perspective. In this case, it would be interested in how those people have come to understand everything they have expressed in those statements. It would be interested in how the participants have come to align and understand histories. I think you are absolutely right when you describe that moment but it has little to do with an actual genetic code. Maury comes out and says DAD is not the father, all those important things up that each parent is talking about change in dynamic ways. Maury comes out and says DAD is the father, all those important things up that each parent is talking about change in dynamic ways.

The when-ness of what fatherhood means at this moment is important but it is also important because the series of moments of when-ness produce a pattern of how something changes over time. When accused of being one, DAD has his whole notion of Fatherhood change. His description of that change tells us a great deal about how fatherhood works for him at that moment.

You know, honestly, I would pay real money to hear the phrase “genetic disposition” in one of those Maury episodes. Sure that is important but only when mixed with environmental factors. In any biological organism genetic traces manifest and shape environmental concerns. Take sickle cell anemia and malaria for example. It isn't that the stabilized traits of genetic (or genres) aren't important or valuable to know, it is that what IS important about them shifts and changes over time. When I said it isn't important, what I really should have said is that understanding what is important at any given moment is the only way to figure out what genetic (consistently stabilized traits) markers should be looked for.

I think the great thing about qualitative research is its limited claims about knowledge. Yea if I came across saying that it was never ever useful, I was flat wrong. When you get to “The fact that they might dissolve to form other actors at some point doesn't mean that the current interessement doesn't matter - it's not arbitrary, it's just one of many possible interessements” I couldn't agree more! It is anything but arbitrary! What I am saying is that how the parents understand and make those connections is the whole point, well, at least the most interesting point for me as a researcher.

Tracing how certain elements of “something” elements of an activity system or accents move and influence is interesting and useful. But it is only useful to the extent you can make a case, not prove, for it in your research write up. Those cases have levels of probability so as you say it makes sense to group strong claims (ones with copious amounts of validity and evidence) on one section and weaker claims (where you are stretching or playing with worse odds against the House) in an implications section. Tracing the development of a thing is creating a convincing case based on probability. Developmental models want to stabilize the historical presidents of any given situation across contexts. Great! Very useful especially at the level of generalizations. That's how I've used it. But as you said in your post, research is primarily an argument about the way the world works and which part is working in that way. Just because I am not am more interested in one methodology research model because I think it should, according to its own rules, be better at being able to predict behavior (after all a trajectory should be able to be traced forwards as well as backwards) doesn't mean I think that sort of research isn't valid, has high degrees of rigor, or isn't useful given certain constraints that all empirical research must encounter. Constraints such as certain objects of inquiry or audience expectations about the nature of research claims, or *he said trying to bring it back to the posts claim about research* ideas about what counts as rigor in empirical research all shape what is useful knowledge.

If I wanted to be in a field where everybody was sure about what was going on and we all could generate clean answers to everything I would have gone into a science. I would much rather read good activity theory research than bad ANT research after all.

Hey, thanks for the invite to participate! I have go write some pesky document called a “Dissertation Prospectus” or something. Believe me, I would much rather do it in blog posts.
Hmmm. There is an idea.

Bill said...

You guys are about to go all "Cat's in the Cradle" on us here...

Oh, and Doug, Re: Blog posts as dissertations comments: No.