Pierluigi Dalla Rosa is an interaction designer at Humane, the makers of the AI Pin. But that isn’t the focus of our conversation today. Instead, we discuss novel computer interfaces in general, and interactive environments in particular. This is a subject that’s near to my heart, as it points to exciting possibilities for future digital experiences.

Show notes

Some show notes may include Amazon affiliate links. We get a small commission for purchases made through these links.

If you're enjoying the show, please rate or review us in Apple's podcast directory.

This episode's transcript was produced by an AI. If you notice any errors, please get in touch.

Transcript

Jorge: Pier, welcome to the show.

Pier: I’m super excited to be here, Jorge. Thank you for having me.

Jorge: I’m very excited to have you here as well. I’ve known you for a little while. We’ve taught at CCA; we’ve actually taught together. And I’ve always been very impressed by both the depth of your thinking, the thinking that you bring to the teaching environment, but also by your work. And I was hoping that you’d tell us a little bit about that.

About Pier

Jorge: How do you introduce yourself?

Pier: I really struggle to define myself in a single way. I think professionally, I’m an interaction designer and also by trade. I studied cinema engineering, the technology behind movie-making. But after a short while in the industry, I decided to pivot and go into interaction design. More on a personal level, I think of myself as a mix of homo digitalis. I like to define myself as an in-between, a human and technology, has been the trait of all my life. I’ve been very interested in the medium of computation. But also, I’m a homo faber; I love to make things. I enjoy the process of shaping ideas into things that you can touch and play with. And then, there is the other part of me that is homo ludens. I love to play. So, I think this is to give you a little bit of what I feel I am: homo digitalis, homo ludens, and homo faber.

Jorge: That’s what I’m envisioning, a Venn diagram in my mind between making, playing, and digital stuff. Can you share with folks a little bit about how this Venn diagram has played out in your career? Because knowing your work, I can see how it takes advantage of the overlap between those three things, right?

Pier: Yeah, totally. As a teenager, I wanted to study computer science, but then I felt there was a dryness in just studying the technology and being a constructor, a builder only. I was interested in this intersection between humanities and the technical side of things. And so, cinema engineering felt like an avenue, and in the spectrum of possibilities of things that I could study, that felt like an interesting one in which I could give space to this intersection of humanity, creativity, artistic sensitivity, and, on the other side, technology and how we build things.

And I think interaction design was the natural evolution of that thought process and really recognized design and interaction design as very influential in the way they craft culture. They craft the world we live in. And I think computers have done that in a very profound and consequential way in the last thirty to forty years.

So, channeling my energy into interaction design has been something that I think has been in line with this kind of three Venn diagrams and intersection of them. After studying interaction design, I didn’t just choose to go into a digital agency and make websites or make apps.

And when I came out of school in 2009, the glory days of iPad and iPhone, I chose to go into a small studio that was building tangible objects, was working on product and product consultancy that was merging the physical and digital worlds. And I really found that intersection to be super interesting. And beyond products, we have worked a lot on interactive spaces. Interactive spaces have been one of the focus areas of my career in the last ten years. I’ve been teaching that with Joshua Walton, who has been working at Rockwell Lab and has been influential in creating the first generation of AR products. And that way of thinking - of seeing computation outside computers, outside the boundaries of the devices that have been given to us - is one of the things that I love to do. I love to think about how competition improves a space, improves an environment, and how it fits.

I have worked in that space, and then, at some point, I came to Silicon Valley to work on products, to work on startups. I worked on electric autonomous cars and saw how digital can be embedded inside the in-car experience. I worked at Apple, where we have been looking at expanding their landscape of home devices and home experiences. And now, I have been lately working on AI and hardware as well.

Introduction to Tramontana

Jorge: When I first became aware of your work, it was through something called Tramontana, and I was hoping that you would tell our listeners about it. What is Tramontana?

Pier: Tramontana is a project I worked on and, a project that has been very influential in the way that I’ve been thinking about computing. After working on interactive spaces and environments for some time, I realized that the way we’ve been building computers as personal devices forces us and biases us as technologies, artists, and designers to create things that are channeled through this idea of personal computing, with one person per device.

And so I wanted to break that mold. And this is what Tramontana is: a platform that allows people to have small touch points in a very simple way, using smartphones or small, cheap boards branded Tramontana, to create an interactive environment.

So you can enter a room and feel like there is an interactive system where you can play with it, and where multiple people can play with the same system. It doesn’t feel like everyone is biased towards having a single experience with one computer, but the computer is really alive and part of the environment. Tramontana is a platform for sketching, prototyping, and building interactive environments and spaces.

Jorge: That’s a really exciting idea, this idea of building a space that has computation inherent to the experience that you have in that environment. And, I’m even going to reflect it back to you, and also this comes from a little bit of experience using Tramontana.

The impression I have is that it’s a set of frameworks, libraries, and components that allow pretty much anyone to just start doing these things, right? So it’s, I, is it open source? I don’t remember. It’s just very open-ended, right?

Pier: It is open source. The person I worked with, Enrico Gueli, is a fantastic technologist based in Amsterdam. We have been crafting these apps for Android and iOS, and the libraries. And at some point, we realized that there was more benefit to open source the whole platform and let other people see what’s behind the scenes and be exposed to the technical ideas that make this possible, rather than try to monetize it or keep it closed source.

So at some point, we really decided it was time to open source it, and I think for the benefit of the industry overall. We want these ideas to spread, and we hope to bring a positive and fresh outlook to technology through interactive spaces. That’s the reason why we open source it, and we are really excited to see other people using it.

Jorge: It’s super exciting stuff, and for someone like myself who has a background in architecture, when I first saw it, I was like, “Oh my gosh, this is like a new material.” It adds this digital layer to the experiences that you create in a physical space.

And that’s actually part of the reason we are having this conversation. You recently published an article called An Exploration in Collective Interfaces, but I see it as a kind of call to a more collaborative computing experience in space somehow. I don’t know if that’s doing it justice, but can you tell us a bit about this idea of collaborative computing?

Pier: Absolutely.

Collective Interfaces and Collaborative Computing

Pier: So let me tell you first that this article has been crafted by a group of people. Recently, with some friends and former students, we created a collective of researchers. Without a better name, we call ourselves the Interaction Design Research Collective. And I’m going to just name the people that are in the group. So it’s not just my article, it is really a labor of love from different people that have collaborated in this collective: Vaidehi, Nitin, Mayur, Nikhila, Norris, Jack, and Isra.

So we came together and we’ve been thinking about what is missing from today’s narrative about computation and technology. And I think we have been really looking at the design space that I think has a lot of potential. This design field is a design field in which we have a shared space. We are many people inside this shared space, and we still want to have the benefits of digital that are, we can mention them like access to information, searching through databases, but also getting into a digital world and feeling part of that digital world. We did want to have all of these experiences, but we didn’t want to just be side by side next to each other looking at the screen without making eye contact, without using our full bodies. And so this is where this exploration started.

And as a design field, there is really little work done within. Tramontana is a step, a drop into the space. But we wanted to explore it a little bit more from a historical perspective. And so we did some research. We started to look into the papers and, you know, there are traces of these ideas all the way from Douglas Engelbart, the mother of all demos. Lauren and Rachel Ter in the nineties have been exploring this. So can we have collective experiences inside auditoriums, giving people paddles and have them playing pong in a collective way?

There is more research done in the first decade of the 2000s, in which we were looking at ubiquitous computing and this idea of calm computing that has been spawned by Mark Weiser in the nineties has been researched and brought forward. I think there is a group of Danish researchers that have been exploring things like room-aware and dynamic walls, interactive tables, communication chairs in which they planned furniture and technology in a way that it feels very integrated and very collective.

So we did this historical exploration and understanding what the traces of collective computing interactive spaces were. Then we looked at the state of the art, and then we started to brainstorm what we could do as a just small exploration to really explore the material. Just understanding what it means to have communal interfaces. What does it mean to have collective computing? And so this is really why we came together and built this project, and then made the article.

The Influence of Dynamicland

Jorge: You mentioned in the article that I just read, but you didn’t mention it just now, but you do mention it in the article: Brett Victor’s work in Dynamicland, right?

Pier: Absolutely. Yes. I think Brett Victor is one of the very influential thinkers on the way we think about computers today. And I think Dynamicland, for me, is probably the best embodiment of what I’m talking about. The thing that stands out to me in Dynamicland is that you are immersed in a dynamic environment without even knowing about it. You are seeing every surface as a potential interactive surface. And the space really takes advantage of the fact that there are multiple people. So you can sit next to each other, compare your programs, write programs together, walk around, and that is part of the interactive system and part of the input.

I really love the way they think about, or rather, I think there is a quote that says, “The computer of the future is not a product, but a place.” This is really much in line with what I’ve been thinking about, the ethos behind Tramontana. And I’ve been thinking a lot about how we can make Dynamicland even more accessible.

I think it’s an amazing project, and I would love to see it in every library and in every place in the world. I think it’s just such an interesting space from many different perspectives: a playful perspective, a potential learning perspective, and a computational perspective.

A few years ago, I actually gave a talk about museums as new agoras, as new places of congregation. This was in the context of social bonding and social discourse that is now happening so much online. So I’ve been very interested in bringing back the power of expressing opinions and having access to information, but in the same physical space.

And it used to happen like that at the local bars. I think having that experience in a place of culture is something that I’ve been speculating about, and that was part of what I expressed in this talk about museums as the new agora, where interactive technologies are there for many people to use, not just for one person per device.

Jorge: I think theme parks might also be a place like that, right? It’s one of the reasons I’m excited about the work that the folks at Disney and Universal are doing with their theme parks. But I want to circle back a little bit to Dynamicland because, first of all, I think most people listening in might not have heard of Dynamicland.

And I would like to describe it because, to your point, when I read the article that you and the collective put out, I kept thinking in my mind, “This is what I experienced in Dynamicland.” And since most people have probably not heard of it or definitely experienced it, it might be worth describing it.

And I’m going to give it a shot. I’m going to describe my experience of being there and then ask you to chime in. I’m going to speak of Dynamicland in the past tense because I believe the space has closed. There was a physical space in downtown Oakland, and you would walk up these stairs and enter these rooms - this set of rooms - that were augmented with computers. And the interaction with the computers happened through projectors that were projecting information onto surfaces in the room: tables, walls, that sort of thing. And input happened by means of cameras that were looking for particular things in the environment. There were also keyboards that you could use to type information.

The neat thing about this is that all these computers and projectors and stuff were networked together, running a bespoke operating system that was designed from the ground up to accommodate these kinds of more social interactions, where the interaction happens not through these small glass rectangles that we’re used to interacting with, but in physical space. For example, if you were looking at…

I’ll give an illustration for folks to try to make this idea come alive. One of the programs that was written in this operating system was a music-making program. Instead of having a dedicated interface like a music keyboard or something like that, there were things on tables that you could move around and interact with to make music. There were cameras that were looking for particular gestures as communicated through patterns on pieces of paper.

So it was very kind of sketchy… I was going to use the word “sketchy,” but I don’t mean it in the pejorative sense. It was like a sketch of a possible future interface for computers that was hugely exciting because you could change the code while you were there. So, what did I miss in that description?

Levels of Engagement

Pier: No, I think you described the experience of being there perfectly well. I think I will point out a few things. One is that an environment like Dynamicland, the thing that excites me a lot is that there are different levels at which you can engage. So you can go there, take a book, open it, and as you open it, different things are projected and augmenting the book that you’re reading. Someone can come and put a puppet on the book that you’re reading, changing the way you experience that book, the thing that you know.

If you want to engage even more and you don’t like the color of the character, you can take one of these augmented keyboards and change the code and say, “Oh, I want to change it to be, instead of an elephant. I want it to be a leopard and it’s going to be pink.” And you can do that in real time. There is this sense of tangibility and fluidity of the medium of computation that I think feels really special compared to the environments that we live in today, which are so safeguarded, where it’s so hard to know what is inside the machine.

Sometimes, I feel like the technology industry is trying to predict human intention and always serve the right thing. And we are really losing track of why we are seeing what we are seeing. And I think in Dynamicland, that was completely reversed. It’s like, “Why am I looking at you? Why never think of that as a surprise? The elephant is yellow, let me change it to red.” You just take an augmented keyboard, go into the code, and change it. And I think the way in which this has been thought about is in a way that a kid can do that.

So I think the, and I think the big leaps in computation have always been done with this ingenuity. It’s like, how do we allow people to really engage at the level in which they want, and if they want to go deeper, that’s there for you and is as simple as possible. Regarding your kind of sketchiness.

I want to just point out one thing I’ve been reflecting on – our computation has transitioned from being a cold medium, in a McLuhan way, to being a hot medium. We define cold media as things that are approachable and that have high participation, that collapse spaces and make space closer together. We can feel the excitement, they tribalize with us. It’s becoming a hot medium. It’s becoming a medium for consumption where the participation is low, it de-tribalizes, and it makes us feel so much further away from the computer itself, from other people.

Embodied Computing

Jorge: There’s this quote from Brian Eno that I like. I think he said this around 1995, so quite a while ago. He said, and I’m paraphrasing, that the problem with computers is that there isn’t enough Africa in them. I think what he meant by that is that computers, as we commonly think of them, whether it’s a laptop, a desktop, or a mobile device like a phone or a watch, have very limited channels of interaction. They have been designed for one person’s input, but that input is also very limited. Like typing with fingers or moving a mouse that controls a pointer on the screen. Touchscreen-based devices are a bit better since you can touch the screen, but you’re still constrained by the rectangular shape. As you mentioned before, it’s a very individualistic experience. An iPad, for example, is designed for one user at a time. I believe what Eno was getting at is that our bodies are capable of a much broader range of expressions. Just being in space, especially with other people, is a much richer experience than what can be conveyed through the narrow channel of a keyboard-mouse-touchscreen interface.

Pier: Yeah, you’re absolutely right. That’s part of what we have been trying to explore: how can we make computers sense and react to us, not just by using our tongue to flick things, but by jumping, moving our hands, and being present together. In the article, we discussed how we are hacking old controllers and attaching strings to our hands to create a more expressive input. By doing this, we can move our hands in 3D. It’s a very affordable solution. We take old controllers, attach a string, and get a very accurate positioning of our hands. Computer vision allows for this, but in terms of computation, it’s not as cost-effective. We wanted something that is easily accessible and allows for a more sketchy, crafty medium that encourages high participation and simplicity.

I feel like the idea of whole-body interaction is characteristic of interactive spaces and collective interfaces. As I’m speaking to you, I’m moving my hands, gesturing. These social cues are crucial for communication. I’m not saying that the computer should pick up on all of them, but it shouldn’t inhibit them either. When I sit in front of my computer, I feel inhibited. My hands are on the mouse and keyboard, and I’m sort of stuck there. Instead, I imagine being with other people, making eye contact, running behind or in front of them. That should be part of the input of the system.

Contrasting Augmented and Virtual Reality

Jorge: What’s your thinking – and when I say “your” I’m asking you as a representative of this collective group – what’s on the relationship between these augmented spaces and something like the Apple Vision Pro? These devices are meant to somehow augment your senses, rather than augment the space. They augment your senses and read your gestures as a way to layer digital information onto your experience of physical space.

Pier: Yeah, it’s a good question. The Vision Pro is a personal computer, first of all. It’s not a collective… It doesn’t really reflect what we are talking about. The input devices are different, and we are liberated from using a mouse and keyboard, even though a mouse and keyboard are great for certain kinds of work. I think the promise of the Vision Pro is to remove us from the environment that we are in and give us a very immersive, high-definition experience. It’s very hot as a medium, if you will. That’s the way I’m thinking about it in the McLuhan way. It’s too hot to touch, in a way.

The experiences that are channeled need to be even more perfect because they are so close to our eyes. They’re replacing our senses. I think the world that I imagine we are talking about is a colder environment, colder in the McLuhan sense, in which you can tape something on the wall and it becomes a button. You can draw something on a piece of paper, and now that is picked up by a computer. And if you turn that piece of paper, you are changing the dynamicity of that environment, you’re changing the behavior of the environment. You are adding little statues on your table. Those are the input devices. And by turning them, by moving them, you are influencing how that environment is behaving.

So, I think that we are talking about different things. In the Vision Pro, we are completely embedded, completely immersed in a digital environment. And although we are almost replacing our vision with this headset, it completely immerses us in the digital space. And I think that’s a kind of experience that has value and actually excites some people. But what we are talking about inside is this sense of presence, this sense of community, being together in the same space and really excited about that. So, I feel like they exist on different spectrums of the possibilities of where computing is going.

Jorge: Yeah, I can totally see the idea that it is designed as a personal computer only with an interface that kind of inverts your relationship with a computer. Somehow, you’re now inside the experience in a weird way. But I was thinking back to an experience that I had in Disney World, actually at a place called The Void, which was a virtual reality experience.

I was with a group of people. There were four of us. I think the group… I don’t think that place is open anymore, but there are similar spaces, right where the idea was that you go in with a group of four people, all wearing virtual reality equipment, into a space that has been configured where the surfaces of the space, the physical space, map to whatever you’re seeing in the virtual space. And there are cues like heat and, in some way, in a very strange way, I could see in my visual field through the device, I could see my teammates. They showed up as stormtroopers because it was a Star Wars experience.

There was a little bit of the collaborative thing, but it depended on all of us wearing these devices. Where I think that what you’re arguing for here is something where the computational interface is actually baked into the physical environment. That’s as opposed to requiring physical augmentation on your body.

Pier: Yeah, so in the article we mentioned this kind of experiences in which you have multiplayer games, and the video game industry has been thinking about collective or communal play in many different instances. We have, at some point, split screens and multiple controllers. And that has evolved all the way to what you are describing.

So we are fully immersed: we are wearing a haptic vest, we are holding an input device. Sometimes it’s a gun. Probably we have different types of inputs, can be a magic wand. And we are all sitting together in the same space, and we are completely replacing our senses, right? We are probably just walking into an empty, dark space. But that happens to be super exciting because we are mirroring the world in which we are, in which we have freedom of motion and we can use our hands to move these controllers. We are fully immersed in this mirror world, in this digital experience.

So I think that’s definitely one instance of how we can build shared environments. I found this to be a screaming use of technology. I think there is a place in which we can have communal experiences that are calm, that are about discourse, that are about creation together. I think they have a need for a different form that is not this kind of immersive virtual reality experience in which we are having these kind of action moments together.

But I think they’re super valuable and super interesting. Another example on the other side of the spectrum, if it is in the space of amusement parks, is Meow Wolf. I found that to be on the other spectrum of Disneyland in the way in which they are still using a narrative that guides you through spaces. And the spaces are augmented by digital experiences in some way. There are interactive experiences throughout the exhibition, but I think the thing that is interesting is that it is open-ended.

So I think that’s the other part of it. When you are in these middle worlds, a lot of it is about executing a task, killing the zombie, escaping the room. And it’s very biased towards a certain outcome. I am excited to build something like Dynamicland that is completely open-ended. I can open this book, and this book now is telling me this story. I can take an augmented keyboard and change a story into something else. I can sketch with you a new idea for a movie, and at the same time, I can just play a movie by just changing what the environment can do for me.

Pier: So I think that open-endedness and tasklessness. We talk a lot about tasklessness in the way we’ve been reflecting on this. We have three principles of collective interfaces that we’ve been exploring. And one is this idea of tasklessness and open-endedness, in which it’s too easy to just say, “okay, now you are the stormtrooper and this is what you can do, so go do it.” And I think we like to reverse that. Instead, you define what you can do, what you want to do.

And the beauty of that is that when you are with other people, the channel of communication is not the computer. It’s like I say to you, “I’m gonna be this character and I’m gonna design my character that way and I’m gonna be doing this.” And then, through the channel of human-to-human communication, you’re really crafting what that space does. So it’s about removing the friction and removing the constraints that we have in curated virtual experiences.

So that’s one thing that when I think of Dynamicland, it brings me a sense of happiness because I see that it has this open-endedness. I can go there and spend a million hours and still feel like it’s a new experience every time I go in.

Jorge: I love this distinction between… I’m gonna try to draw it out and name it, like the distinction between open-ended play of the sort that maybe a toddler would engage in with playing with blocks or something like that, versus more scripted play of the sort that you have when playing a big video game title or something where you’re on a happy path and you have these specific targets that you’re gonna meet or whatever.

The sense that I got from the article is that this collective that you’re working with is running experiments and it’s an ongoing thing. You did call out three emergent principles and I was hoping that you would share those real quick because it feels to me like this calls for new approaches to design in some way, because the material is so different. By adding this digital interactive layer to physical space, we’re entering realms that have not been as trodden as other design fields. And it was really useful for you to start sharing the principles that are coming up. So can you share those with us here?

Pier: Yes, absolutely.

Interactive Environment Design Principles

Pier: I think we have touched upon those, but I’m going to reiterate them. So the first one is no hierarchy. No hierarchy means that you are not going to have one person driving. It’s “oh, no, you are controlling the space and everyone has a secondary function.” And maybe that hierarchy can be passed around. So we want to create experiences where everyone can contribute in the same way.

And to explore that, the reason why we got to that is that we were. One of the experiments that we built was just to use these controllers that are full body tracking in 3D through strings that allow us to have a very cheap way to control… to get your hands tracked in 3D. And we said, what can we do with them? So we said, let’s do something very simple. We are actually scrolling a photo gallery, but we are doing it collectively.

So of course, there is one way of thinking, which is, “oh, now I’m controlling it, and now I passed the baton to you, and now you are controlling it.” But we found it is much more interesting to let everyone be able to control it at the same time. So up to four people can come and just say, “okay, I’m going to, I’m starting to go left and if you want to go right, then we are opposing each other and we are not going anywhere.” But that’s what sparks and what starts is something very interesting, which is this conversation.

And it’s a little bit what I was talking about before. It’s “oh, now the channel in which we are deciding where to go is something that we need to negotiate.” And the computer is not in the way. It’s very clear what we need to do. If you put your hand on the right side, you are adding force to move everything on the right side. And if you do the other on the other direction, you push things in the other direction. And in this way, I think the computer really disappears. There is no passing the baton, there is no interface. The interface is us talking about where we want to go.

And one other principle is tasklessness. That is what we were just talking about, the idea that there is open-endedness. It’s not that now there’s a task; now you are the stormtrooper and now you are the Jedi or now you are drawing and now you are doing something that is bound to a specific constraint. It is open-ended in the way in which I can choose what that is. So putting experiences not on a railroad track, but rather rewarding the fact that you can go around and explore and be where you want to be. We believe that this really grants the control of the system to the individuals, and I think that’s really important to the group. Instead of promoting me sitting in front of the computer and executing some data that has been there for me as rigid instructions.

And then the last principle that we’ve been thinking about is whole body interaction. So really we are in the same space and our bodies can do so many things. We can jump, we can move our hands. And of course, there are concerns about ergonomics that really need to inform this. But the way in which we express ourselves can be so much more, making full use of our bodies. So that’s something that I think will be super exciting to see. And it’s going to make us much happier when we can dance and jump and touch and actuate and fiddle with our whole bodies.

Closing

Jorge: This is super exciting stuff, at least to me. The idea that we can work with digital information beyond the bounds of a screen and with this more collaborative foundation is so rife with potential. And I want to thank you for the work that you’re doing, but also for coming here and sharing it with us. Where can folks follow up with you and find out more?

Pier: I’m on almost every social media, but I think if people want to follow me, they can find me on Instagram. P-I-E-R-D-R is my handle. Also, my website is P-I-E-R-D-R.com. If you’re interested in the work that I’m doing with this collective, I think we are IXD Research on Instagram. So IXD is short for Interaction Design, and IXD Research on Instagram is the channel that we are using to communicate and share our work.

Jorge: Fantastic. I’m going to include links to all of those things in the show notes. Thank you so much, Pier, for being with us.

Pier: This was really fun. And yeah, I’m looking forward to being here again.