Karl Fast is an independent scholar, information architect, and futurist. He’s the co-author of Figure It Out: Getting From Information to Understanding. This is the second half of a two-part conversation about interaction and embodiment. If you haven’t done so already, please listen to part 1 before listening to this episode.

Show notes

Disclosure: I received Karl’s book for free as a previous Rosenfeld Media/Two Waves author.

Some show notes may include Amazon affiliate links. We get a small commission for purchases made through these links.

If you're enjoying the show, please rate or review us in Apple's podcast directory.

This episode's transcript was produced by an AI. If you notice any errors, please get in touch.

Read the transcript

Jorge: Karl, welcome back to the show.

Karl: Thanks for having me again!

Jorge: Well, I could not help but want to talk with you again, since you mentioned wanting to discuss rats and heroin, and that just sounded too intriguing to me at the tail end of our last conversation. So, why don’t we pick right back up there? What do you mean by rats on heroin?

Rats and heroin

Karl: So in the 1970s, there was a lot of concern about drug addiction in various parts of the society, especially in the US, and especially with the Vietnam War. And a lot of veterans — a lot of soldiers who were in Vietnam and who were exposed to heroin were… I think there was some estimate, something like 80 or 90% of all the soldiers who went over to Vietnam had used heroin at some point and had a serious… what we would now classify maybe as an addiction. I’m not sure if that number is correct. But there was this general widespread concern about this and what was going to happen when these people came back. I don’t think that this particular study was motivated directly by that, but certainly this was in the air.

And so, there was a Canadian psychologist named Bruce Alexander, and he was concerned about a lot of the studies that had been done around addiction and said, “well, these rats are stuck in like a small, cold cage, often socially isolated from all other rats. If I was in that kind of situation, maybe I’d like to take some heroin too!” So, he designed a study where he had a couple of different groups of rats. One group of rats, they were basically in the sort of normal lab cages that you would have. The other ones were in what are called “Rat Park.”

And you can actually look this up on Wikipedia. It’s called, “The Rat Park Experiments.” And they… they were like 200 times larger. There were about a dozen, 15 rats or so in there, both male and female. So, they had a completely different environment in which they were able to run around. They could play, there were all kinds of lovely things. There were things to keep them mentally stimulated. It was kind of like a rat heaven. And in both of these different groups, each of them, the rats were divided up into two different groups.

There were rats that were given water to drink, and the other rats had a choice between water and water that was sweetened, but also laced with morphine. What they found basically was that when they were in rat park — in rat heaven — very few of them would consistently go and choose the morphine. But the ones that were in the cages almost always did the morphine.

And there were several conclusions about this, but one interpretation was that social isolation was the major factor and that lack of socialization drove a lot of addiction. My understanding is that this is not… this has been criticized in a number of cases. But the broader one, the way that I’ve always interpreted this study, is that the role of the environment has a huge impact on stuff.

And when it came to the veterans, when they came home, a surprisingly small percentage of veterans came back to the US after the war and continued with their heroin addiction. So, there are studies with rats, there are findings with the actual veterans.

Embodiment and interaction

Karl: And to me, this raises this question that relates to what we were talking about last week. We had this idea of embodiment. And the big idea with embodiment here is that the mind is not the same thing as the brain. Our bodies, our tools, the space around us — how we move and act in the world — this is all part of our cognitive system; that our brain might be in the head, but our mind is embodied. Our mind extends out into the world. So, the systems and the tools, the information we have… all the things that we design that are exterior to the body, those should also be understood as part of the mind. They’re not just out there.

And that brings us back to the other main thing we talked about last week was this question of interaction. How do we interact with the world? Why do we interact with the world? And we introduced that distinction between pragmatic and epistemic actions. And we gave that example of moving the bishop, where you pick up the bishop, and you move it and put your finger on it, but then you move it back.

From a traditional perspective, we would say, “oh, you did two moves in the world. You’ve moved a piece, and then you pressed undo.” But there’s research, which says, “well, no, no, there’s actually real reasons why you would want to do that.” And the distinction is what they call pragmatic action and epistemic actions.

And the idea of a pragmatic action is you’re going to change the world in order to change the world. You want to bring about some change in the world. But an epistemic action is about changing the world to change your understanding of the world. To make thinking easier, or faster, or more reliable. So, we have these different ideas here. We have this idea that the environment plays a role in our mental wellbeing.

The environment plays a role in our ability to think. The way we interact with the world is part of how we think, in many cases. And what we talked about was… we went through a number of different examples, but the big idea is this I think has major implications for the things that we design in a world where we have a lot of information.

We have rich — increasingly rich — computational tools, powerful computational tools that don’t just sit in a desk, that are being embedded into every part of our world. And that can sense and interact and respond back to us in more meaningful and interesting ways.

Jorge: You mentioned in our previous conversation the opportunities inherent in the trajectories that we see in technology.

Karl: Yeah. So, especially I think with AI, but that’s an obvious one, right? Think about it this way: one of the things I argued last time is that we need to have a better conceptual toolkit when we’re designing, when we’re creating these different things. We need to think about how what we are making is not just out there, but is connected in a meaningful way to what our brains can do. So, we should think about certain definitions — certain words — and try to understand them and develop new concepts for how we talk about this.

So, for example, let’s start with some basic ones that I’ve already mentioned: action. I think of action is doing something. It’s a thing, done, it’s a gesture, it’s a movement. And that is distinct from reaction — what happens in response to a situation, an event, to an action. But I use the word interaction to not mean action and then reaction. You can think about it that way in terms of the timeframe, but I tend to think of interaction as the action coupled with the reaction.

So, you act in or on the world, that action combined with the response. Think about it as the difference between you can talk about, like, your coffee, and you can talk about the cream, but once you mix the two together, now they’re bound together and that’s different. So, we can think of them as separately, but we also need to learn to see them as connected together.

Now, language can be a little slippery. And I know that in my writing and in my speaking here, I have probably used… I’m sure that I have used the word action when I really meant, by those definitions, interaction. I tend to use them synonymously. I talked about epistemic action. Really, I think it should be epistemic interaction, but it’s a bit of a mouthful. But that’s an example of definitions.

Or another one would be “interactivity.” This is a word that gets tossed around kind of loosely and casually, often as a synonym with interaction. I think it’s best understood as a quality or attribute of an interaction. Some interactions are easy. Some are interactions are difficult. Sometimes easy is better. Not always, right? In learning, when interaction is too easy and quick — if learning is too easy — then the learning becomes shallow. You need a kind of certain friction. Or another example for interactivity: how difficult is it to articulate your intentions with a system?

So, you can talk about a gap between what you intend to do and your ability to articulate it. And can we, through design, close that gap? When do we want to close that gap? So, you might have the same interaction. But the interactivity or certain dimensions of that interactivity could be very different. And there could be good and bad around that.

Jorge: This brings to mind Bill Verplank diagram of interaction. I don’t know if you’re aware of that diagram.

Karl: Refresh my memory.

Jorge: Well, it’s a feedback loop between the world and some actor. And the actor senses the world. Verplank labels it “feel.” so, there’s this sensing of the world. The actor, through sensing, knows and then does. So that’s the other arrow. So, you sense, and you do. And this cycle of sensing and doing amounts to an interaction. And what I’m hearing you say is that… it’s not a binary thing, but there’s a gradient of interactions. So, some things are more interactive than others.

Karl: Yes. And you know so that loop, that idea here… this is where I think embodiment is trying to say, “it’s really complicated.” It’s a lot more complicated. It’s a lot more nuanced than we have tended to describe it.

You know, you take a class in cognitive psychology, you pick up a lot of different books around how the mind works, and you’re going to see — sometimes explicitly, sometimes implicitly — a simple loop around perception, cognition, and action. Perception is input, right? Information from the world that is input into the brain. The brain takes that and does computation on that. And then from that, we act on the world.

From the embodiment perspective, the simple and wrong way to think about that is that the cognition happens just in the head, right? Perception is input. Action is output. Cognition is what happens in the middle. And what embodiment is trying to say is, no. There is an element where that could be the case, and maybe often is the case. But it’s certainly far from the entire story. We need to redraw that picture.

Think of it as maybe a first order approximation of how the mind works. Now, we need to think about second and third order effects here. What are the other things that are going on here? And that’s what things, like epistemic actions, point to.

More broadly, it’s very common in design for people to have taken a lot of classes and to have studied aspects of perception. All graphic and visual design, color theory, like layout… you know, all of this is about changing the perceptual inputs. Controlling the perceptual inputs. All the aestheticism that we have is about the perceptual inputs are largely about that.

I’m much more interested in the other side of that, the interaction side of that loop. And how that is connected to the things that we create, and trying to unpack this in the light of what the science is telling us about embodiment.

Jorge: Would it be fair to say that the emphasis then is on the touch-points between these artifacts that we perceive and the people who are perceiving them?

Karl: Yes, and the way that those touch-points happen. And you know, the more I think about this… and I’ve been thinking about interaction for well over a decade. And you see this notion of interaction in so many different areas, in different fields.

Like we just talked about it with the example of rats and heroin and this connection to the environment. Another one from biology is, we have that old idea of the debate between nature or nurture. And most of us today have gone, “oh, well, it’s not about nature, it’s not about nurture. It’s nature and nurture, and some sort of balance between the two.” But biologists don’t believe that either. Not anymore. What a biologist would say is it’s more about how the organism interacts with the environment. Evolution is an interactive process.

An example related to that is genetics. In the mid-nineties, we had all this talk and this huge effort to decode the human genome, and the story that was told there was, “well, if we can just decode the human genome, then we’ll be able to read the book of life,” as though it was just information.

And genes were presented to us as a deterministic thing. You have a gene, this tells you how tall you’re going to be, this tells you what color your hair is going to be, it can determine intelligence, all of these things. But what we now know is that’s really not what’s going on. That is part of the story, but it’s much more complex than that.

And the big area in genetics is epigenetics. Epigenetics is how your behavior and your interaction with the environment shapes the way that the genes express themselves. So, maybe interaction will turn on a particular kind of gene, maybe it won’t. It depends on how the environment that you’re in really shapes this.

This leads us actually back to rats. There are studies that go back to the 1950s about baby rats and how their mothers lick them. There are what are called “high licking mothers” and “low licking mothers.” So, some rats get licked a lot when you’re a baby, and some mothers will not lick their babies a lot. And they do this to calm the rat down.

This changes the epigenetics: changes how the genes — especially around stress — and I believe it’s cortisol, but other things — express themselves. So, a rat that gets licked a lot is going to be a much calmer rat. It will not get stressed as much. A rat that is not licked a lot, won’t learn to do that.

And so, it’s genes will express themselves in a different way. And you might think, “well, it’s great to be calm, not so great to be anxious.” But then we come back to this problem of, well, what kind of environment is the rat in? If a rat is in a dangerous environment, it actually pays off, from an evolutionary perspective, to be anxious because you’re going to be more alert to threats. So, you have this really much more complex, interesting trade-off.

And interaction, to me, seems to drive all of this. The mother is licking the baby, that’s interaction. The rat grows up to be in a particular environment — grows up in one environment, then maybe migrates to a different environment — that connection between the environment is interaction.

All of this stuff again and again, we see interaction throughout everything. Think about another example, like programming. We talk about functions. And then we talk about objects. We talk about APIs… this, all the ways that programs interact with each other.

We gave an example in our previous conversation about why people talk with their hands and the importance of gestures. It’s not just a way of providing extra information to someone else. It also turns out that that physical movement has an inward-facing component that helps smooth our thoughts, or sort of like a cognitive grease. Epistemic actions. There’s another example. We had the example of chess, which comes from a study about how people play Tetris.

And the thing I’m trying to get at here is: I think interaction is fundamentally under-appreciated. We want to tell a story about the things. The things that we have. The objects, the artifacts, and the story that we want to tell might be about their perceptual qualities, their aesthetic properties, how beautiful something is, or the quality of which it is made. We might talk about its usability. We have various ways in which we talk about the objects. Or, from the cognitive side, we’ll talk about what the brain is capable of.

And I keep thinking from all the stuff that I see, this question of how we are interacting with those things: how we interact with things, how we interact with the environment, how we interact with other people. All of that seems to be really, really central. And my question has been, how do we come up with a better, richer — and from a design perspective — more useful way of talking about that?

Interaction design

Jorge: Most folks listening in will have heard of interaction design, and they probably associate it with things like the gestures and motions and… I’ve heard it described as though interaction deals with time. Whereas other disciplines deal with spatial relationships, interaction deals with time-based relationships.

Karl: Well, and I know if you read, say_ About Face_, the interaction design book by Alan Cooper. He defines interaction design in terms of behavior. But for my reading of behavior, I think of it as the way you act or conduct yourself often, especially towards others, or towards things. And it tends to be more of a macro definition. It’s a larger scale pattern. So, part of this is also a terminology question and looking at the things that we do at multiple levels.

There’s a nice paper by Cameron Seddick, who was my dissertation advisor actually many years ago, and he breaks interaction down roughly into, I think, four different levels. He talks about events. So, those are like the smallest-scale thing. This is something that actually changes and happens. An event would be like a click, a drag, pinch, scroll, tap, grab. Those types of things.

An interaction is something more like filtering or rearranging or annotating. That is comprised of — depending on how things are designed — one or more different kinds of events. Then from there above that, we have different kinds of tasks. And above that, we have activities. And so at each stage, we get a level of behavior that is broader and more comprehensive, as opposed to being narrower.

Activity theory, which is also another area which has strongly connected in many ways to embodiment, has a similar notion that interaction has multiple scales and dimensions. And also that things can move up and down, depending on our conscious level of control and experience. So, the classic example here would be learning to drive a car and especially learning how to drive say a manual transmission.

When you first learn, you have to be very clear about, okay, I am pressing the clutch. I am moving the gear shift. But as you get more skilled and experienced with that, that goes from a low-level set of actions to something that’s much broader. It’s easier and it’s more automatic. So, things can shift and change around. Again, we come back to this idea that… well, I’d say there are two reasons why I think interaction is not really as fully appreciated in design circles as it could be. Or maybe not appreciate is the right word… maybe a better phrase would be, talked about it in a way which allows us to fully leverage what interaction can do.

And one reason I think for that is that it is complicated. It is nuanced. And I think the other major reason is that we have this idea from cognitive science that is baked in pretty deep about what is this relationship between the world, and perception, and the brain or the mind, and action.

And we have tended to have a story from cognitive science, which is pretty simple: the one we talked about earlier, where perception provides information, the brain takes that and converts that information, which would then do mental computation. We call that thinking. And then action does the output. Perception, input. Action, output. And thinking happens in the brain. Embodiment is this big revolution in cognitive science, which is trying to really change that. But so much of the design world is based on traditional cognitive science from the 1960s, 70s, and 80s.

Jorge: What I’m hearing there is that it’s not just a mind acting in the world. It’s a mind acting on the world, so that the act of sensing and tweaking the thing that you sense, in interacting with the world, you gain an understanding of the world. And that is part of what cognition is. Is that a fair read on it?

Karl: I think so. The reason I talk about it as action and reaction combined, is because things run both ways simultaneously. And that always makes it much harder to grapple with.

Barbara Tversky is a longtime researcher at Stanford and wrote a book a couple of years ago, called Mind in Motion, which summarizes her lifetime of research in this area. And it’s very much a book about embodiment, and she talks about several laws of cognition. I don’t remember all of them, but the first two I think might be illuminative for this conversation. Her first law is that benefits always have costs. There is a cost to thinking just in the head. There is a cost to acting in the world. But when there’s a cost, there’s also benefits.

And when we think about interaction in this richer way of being action and reaction together, how we have this notion of the extended mind, and we think with and through the body and the world around us, what we’re talking about is how we’re shifting those cost-benefit trade-offs. Where the thinking happens is really important. A classic example here would be when you’re, say, multiplying two numbers. Most of us can multiply two times eight in our heads and get 16. But very few of us can multiply 28 times 342 in our heads and get a good answer.

So, we have moved the computation out into the world. We could do it on pencil and paper. We could do it with a slide rule. We could do it with an abacus. We could do it with a calculator. We could ask a voice assistant to do it, right? And so, where do we think about that unit of analysis? Do we think that the thinking is happening in the brain? Do we think that we’ve offloaded it out into the world? Or do we see the two things — what we have in the world, and what we have in our head — as combined, and see that as the cognitive system?

“Interactionism”

Jorge: You brought up in our last conversation this word “interactionism,” which I sense is a call for people who design interactions to be more conscious of what it is that we’re doing. And I’m wondering if you could unpack that term for folks.

Karl: So, I’ve been grappling for a word here which describes this notion that we would look at the world through this lens of interaction. We would not see interaction as, “well, I’ve pulled up whatever my favorite design tool is, and I’ve got a pallet of widgets on the left-hand side and I can drag and drop, and those are the interactions.” How do we begin to start seeing this as much more important?

So, interactionism in this sense here is a word, which I…. You know if you look up “ism,” it’s both a good word and a bad word, right? We all live in isms. We live in capitalism. Or liberalism. Or conservatism. We have lots of different ways around that. But what it really means is a doctrine, a perspective, a philosophy about how we look at things. What we see and the system in which operate.

And right now, I think a lot of times in design, we have certain isms that exist. Certainly if you come out of graphic design and visual design, historically, has really been a strong influence for user experience and design more broadly. And there, aestheticism is very strong.

Out of other kinds of design, we often have a very strong idea of the object of the thing. Usability engineering, and ergonomics, and human factors gave us a different kind of ism around the usability. UX itself, I think, is an ism, right? It is the experience that matters. And so, information architecture and other areas, have like sort of an information-ism, that we are focused on the information. Or maybe we could use the word structure. So, the structure-ism or structuralism.

Each of these words has positives and negatives. Each of them is loaded in different ways and have certain histories. Even the word interactionism has this. If you do a search for interactionism, you’ll find that it actually has at least two major meanings historically.

One comes out of philosophy, and philosophy of mind, going back all the way to René Descartes, who famously created the idea of mind-body dualism and said that there is the body and then the mind is somewhere else. And we have this mental space, which is not physical space, and that thinking happens… there’s this interaction between these two spaces, this mental space and the body and the world.

And that is one way that people talk about interactionism. Another one is a perspective in sociology, or it was called micro-sociology, that wants to look at and think about sociology entirely in terms of the interaction between individuals.

So, interactionism actually already is a word in at least two major areas. I’m interested in it simply as this perspective or this lens in which we really see this and begin to elevate it to a more prominent role, rather than thinking of it as, “Oh, well, I had an interaction here, and I sprinkled on some interaction there.” Or, “This interaction is very usable.”

Is it just philosophy?

Jorge: This is not something that you are going to encounter in a palette of components in a development environment.

Karl: Right. Yes.

Jorge: It’s more of a philosophical take on what it is that we’re doing here.

Karl: It is a philosophical take, but it’s also a philosophical take based on the way that the science of mind has been evolving and the new evidence that we have. So, it’s not purely philosophical.

Jorge: I say that with the caveat that I think that when a lot of folks hear the word “philosophy,” they turn off, right? They think, “well, this is not actionable.” But I think of a philosophy as a way of understanding what it is that you’re doing in the world.

Karl: Yes. And I think that anyone who has paid attention to robotics and artificial intelligence over the last five or six years, has realized that one of the major problems that the field has had has been ignoring philosophy. I mean, artificial intelligence and robotics has really revived philosophy in many ways. But I just don’t think it’s actually sort of revived it, but it’s made like all the people who are doing this philosophy of ethics is super important. Philosophy of mind is super important.

I recently read a book by Brian Christian called The Alignment Problem. And it’s all about this question of what’s known in AI as, how do we align human objectives with robotic objectives? How do we make sure that robots don’t go out of control? The infamous paperclip optimizer problem, right?

And you can read this book in a number of ways, but one of the most important ones is all of these people in computer science and robotics and AI going, “whoa, philosophy! Really important! Not a waste of time.” Deep, deep questions here that have completely changed the way that they have gone thinking about the technology and the assumptions that they have underneath it.

Jorge: And what I’m hearing here is that as designers of information systems or interactive systems — if we dare use that label now — we have to develop the awareness that what we’re making is not just a collection of things to interact with. In some ways, it is extensions of people’s minds. And much like the folks who are working on AI and robotics are taking the time out to think about what they’re doing in a broader light, that we should be doing the same. Is that a fair summary of what you’re saying here?

Karl: Yes, I think so. I would draw an analogy to another idea from biology — actually paleontology — of punctuated equilibrium. There was a question for a long time about how evolution happens and issues and questions with the fossil record because there seemed to be gaps in the fossil record and long periods of basically not much happens. And so, in the late sixties and early seventies, Stephen Jay Gould and Niles Eldridge proposed a theory… what they called, “punctuated equilibrium.”

You can think of it as a curve. Think of a chart where you’ve got an X and Y axis, where you have a rise in an uptick. Time is along the bottom. Rate of evolutionary changes on the Y axis. And you have a rise, and then it tapers off, and it plateaus, and then it goes up again for another period of time. And then we have a period of where it plateaus, and then you have another period of change, and then it plateaus.

So, evolutionary change is not a linear line. It’s not this sort of straight line up at a steady rate. You have periods of intense change, and then you have periods of relatively little change. And I think that this is one way to look at how design has evolved.

We had a period, I would argue, of very intense change through the late 90s and early 2000s with the advent of the web. You know, information architecture comes into its own, usability engineering becomes much more recognized and interaction design becomes a more widely recognized term. And user experience comes up and becomes more or less sort of this much more dominant term. But the early stages really are there, we have this period.

And then I would say that we had a long period where… long being relative, I guess, but we had a number of years where things were more stable. Then what happens is we get mobile, right? And especially by say 2010, along with this mobile, I feel like we had large organizations especially, went and said, “you know, all of this work that people in UX do, this is really important, and we’re going to stop hiring outside agencies and different groups to come in and consult and do this on a contract basis. We’re going to start building UX teams.”

And we’ve seen, over the last 10 years, this massive growth in UX as an industry. We have people who go get a degree in graphic design, computer science, or whatever, and then they do a 12-week bootcamp in UX, and now they’re a UX designer. We’ve been growing like this massive number of people in the field, and all of these people have been moving in-house. And so, we have this period of rapid change where we have a growth of the field, and we are having to integrate it into organizations where it’s now a really important part of it. And so, when I look at this, we see this period of change.

We’ve seen changes like this through technology. We had the PC revolution, then we had the web. And then we have mobile. Everyone in Silicon Valley is thinking about, well, what’s the next iteration of this because this tends to happen in periods of 12 to 15 years. What’s the next new major platform? And we can see this coming. We’re going to have another period of intense change.

And when I look at the trend lines, as opposed to the headlines, you can see some of them that. They seem fairly obvious: more and more information, more embedded computation, not just sitting on a desk or even in your pocket. We have pervasive sensors. We have intelligent response to what human beings are doing. The capacity for a greater and tighter connection between people and the digital world. We have more complex pattern matching. We are having machines now that — with robots and AI — they can do things at a scale that humans cannot do. Or an accuracy or a speed that they cannot do.

If we think about these as being separate, out there, aside from human abilities, I think we’re going to wind up missing a huge opportunity. I think design is going to be caught flat-footed. The book that Steven and I wrote is an early attempt, from a design perspective, strongly influenced by information architecture, but also coming from distributed cognition, to explore some of these ideas and do it in a way which could be helpful to designers.

And so, we break the book up into several sections. We talked about associations. We talk about visual representations. We talk about interaction, and we talk about something called coordination. And then we look towards the future. And in each of these, we explore some of the science. We give lots of examples. But we also try to provide frameworks for people, conceptual tools so that people can look at these different areas and begin this process of building it up.

We would not in any way think that this is a be all and end all, but we were consciously trying to provide people with a way to think about the problems that they have now and provide a way to think about them that is going to be more helpful in the future, given these trend lines.

Closing

Jorge: I already mentioned in the previous episode that it was one of my favorite reads of last year, and I strongly encourage folks to check out the book. And other than that, Karl, where are the best places for folks to follow up with you?

Karl: The best place to find me at the present is on Twitter. I’m @karlfast, and that’s K-A-R-L-F-A-S-T. I am not super active on Twitter, but if people ping me, I will respond. You can also look me up on LinkedIn. And again, if you message me there, I will respond.

Jorge: Thank you so much for being with us and being so generous with your time.

Karl: Pleasure to be here. Always nice talking with you.