My guest today is Carol Smith. Carol is a user experience researcher at the Software Engineering Institute at Carnegie Mellon University. Her focus is artificial intelligence, and prior to joining Carnegie Mellon, she worked for Uber’s Advanced Technology Group and IBM Watson. In this conversation, we discuss the benefits and limitations of artificial intelligence — and machine learning, more specifically – for our day-to-day information management.

Show notes

Show notes include Amazon affiliate links. We get a small commission for purchases made through these links.

If you're enjoying the show, please rate or review us in Apple's podcast directory:

This episode's transcript was produced by an AI. If you notice any errors, please get in touch.

Read the full transcript

Jorge: So Carol, welcome to the show.

Carol: Thank you. I'm happy to be here.

Jorge: Well, I'm happy to have you on the show. For folks who don't know you, can you please tell us about yourself?

Carol: Certainly. I am a user experience researcher, and I'm currently working at Carnegie Mellon University in the Software Engineering Institute, where we do work as well as general research on artificial intelligence and emerging technologies. And I've been working in user experience research generally for the past 18 years across a lot of different industries, working on different types of problems in healthcare and in finance and manufacturing and now, in... Specifically in software for a couple of government applications. I really enjoy working on more difficult problems and especially with artificial intelligence, so just so many ways these technologies can help and unfortunately hurt. So it's really an interesting time to be in this field.

Jorge: Artificial intelligence is a subject that I think a lot of folks have probably heard about or read about in the news. And I'm wondering if you can, just for the purposes of our conversation, tell us what you mean by "artificial intelligence."

Carol: Great question. Yeah. And it can mean a lot of different things to a lot of different people, but generally what I mean is a system that has been given a set of data, usually very narrow set of data it's using that information. And looking for patterns in it and trying to understand, and in a very computer-like way the connections are between the data and then it's using that information to decisions and to be successful in making the proper decisions. It's really trying to look for patterns, generally, and then use that information that's previously understood to be correct to new situations and to solve those situations with that older information that it has. So it's much like many of systems we've seen, like the chess games and things like that, where a computer has been trying to play the game. Those patterns and applying and potentially finding new patterns that we haven't thought of yet. So, for example, fairly recently, there was a game of Go, a computer system able to come up with new ways of playing that had never been seen before. And so that's a very exciting and new way that these systems can to apply their knowledge to problems.

Jorge: So let me read that back to you to see if I'm getting it straight. So there's a set of data, and AI is perhaps a set of algorithms that looks for patterns in that data.

Carol: Correct.

Jorge: To then make decisions. Is it making decisions on its own or is it helping the human make decisions?

Carol: Ideally, it's helping the human make decisions, but in some cases, we want the system to make decisions. So, for example, in that game of go, we would want the system to make its decisions on its own. In an example such as self-driving vehicles, we would want it to always stop at a red light, for example. We would not want to have to remind it to do that or to approve it. It's doing that. But there may be other situations, particularly ones where a person's life is at stake, their quality of life, their health, their reputation... All sorts of much more significant situations where we would definitely want the human to be responsible for that final decision, whatever it is. And so, it depends. But in a lot of cases, we do want these systems to be able to make some decisions for us. For example, I'm doing research across a set of documents. It would be very nice if the system could go ahead and pull papers that seem relevant to the research that I'm doing, and then suggest them to me for the works that I'm to do, versus me needing to say, "Yes, that one, that one, that one." And if articles it pulls aren't of interest to me, I could provide that feedback potentially to the system, and it would adjust what it's bringing to me.

Jorge: So to focus on that example. In that case, the data would be the text of the documents that you're trying to sit through. Is that correct?

Carol: Yeah, yeah. Or an abstract or, or just a title, depending on what you needed. Yeah.

Jorge: I think we've all heard the term "artificial intelligence," and we've also heard the term "machine learning." And I'm wondering if there's a technical distinction that's worth digging into.

Carol: For most people, there's not. Lately, people have been saying that the term artificial intelligence itself is kind of meaningless at this point because it means almost nothing specific. But machine learning is a specific [type] of artificial intelligence. And for the most part, it is artificial intelligence for most situations. And what that is, is a situation where you take data, and you train a system... There are a couple of different ways to have the system learn about the data, by creating models and then having it learn the information that's there and learn using those models. And then again, applying it to a situation. So that is the most common, and generally what you see in technology today.

Jorge: When you were introducing the subject, you talked about these algorithms being pointed to -- I think the phrase you used was "a narrow set of data." Does the breadth of the data base or the set of data that you're polling through, does the breadth of it matter to the quality of the results?

Carol: It definitely does. It's really, I think, surprising to people who are new to these systems, how narrow the data needs to be to it to get high levels of accuracy. So, for example, if you could imagine, I'm looking at a set of data about car repair, let's say. If you provide it with everything about every model of vehicle and every type of engine and all of that breadth, it's so much so that the system may begin to see connections that don't exist. And so by keeping it very narrow, so for example inaudible Honda Civic, and maybe even a particular model year, the system will learn a lot more about that set of data and be a lot more accurate. If it's too broad, it just can't see the patterns that are necessary for it to be confident and to provide the competence that we want. So, for example, most systems in artificial intelligence to get to 80 to 90% accuracy requires a huge amount of time and training work for that system to get to that level. And so to get, you know, into the nineties requires even more time and energy. And most of the systems are only going to get as good as a typical human in a lot of cases, and it may get better and better if you keep it more narrow. And that's one of the reasons why we don't have general knowledge with AI systems, and none of the existing AI systems, for example, have a third grader's understanding of the world. That just doesn't exist right now because there's not enough computing power, and the systems just need to be so narrow that that is just a huge leap or will be a huge leap in artificial intelligence in general for it to get to that point. So right now, we see systems that are more like narrow task systems, as you mentioned earlier, supporting humans in making decisions and helping us to understand a specific but not a more general set of information.

Jorge: You talked about accuracy and measuring the accuracy of the system. How is that done? How do you determine whether a system is accurate or not?

Carol: Yeah, that, that's a great question. And that's a pretty new area, and I'm not an expert in that at all. But generally what people do is they literally observe the system and see what kind of results are getting and determine... Have a more technical way of measuring it if it's a more technical area that can be measured in that way. It's one of the reasons why qualitative work is really difficult for an AI system to do. So, for example, while an AI system can do transcription. It can't necessarily understand that's happening within that transcript because there's a lot of additional language that's used by humans that not be applicable, and that may not be able to understand that wording.

Jorge: So I can imagine that in some problem spaces, the end result is easier for the human trainer to predict. So in a game like Go, that game can have a "winning state," right? Like where one of the players wins. And you can determine whether the machine has won. And I would imagine that that determines that the algorithm has somehow improved.

Carol: Exactly

Jorge: But I'm wondering for a problem like the one that you talked about earlier, where imagining an algorithm that is trying to help me sort through a large data base of academic papers... I'm imagining that it would be harder for me to determine what the right answers are, to measure that

Carol: Right right exactly. And that's why it gets. More difficult to be able to say how accurate it is until you have enough of a result batch if you will. So once you see a lot of results, you can start to say, "Oh, you know, most of the time it gave me the right content. Most of the time it's bringing any of the articles that I'm asking it for." So I would say it's gained a higher level of accuracy. Even the term accuracy may not be the right term to use in that instance. It may be more about relevancy or something along those lines. But regardless, thinking about, is the system performing at the level I expect? And is it providing me with the information most of the time or all the time ideally that I expect it to? And in the situation of the chess games and things like that, yeah, the more it is winning, the more it's making the right decisions in that situation, the better the system understanding and responding to the information in the environment. And that's part of it. It's just the response it has. It's not always a decision, but it is a response to the information in the environment.

Jorge: I'm guessing that there are also different degrees of -- I don't know if to call it like criticality -- towards getting the right outcome. It's very different to win a game of Go than it is to drive a car safely. Right?

Carol: Exactly, yeah.

Jorge: The downsides to losing at Go are not the same as the downsides to crashing the car and potentially killing someone.

Carol: Yeah, exactly. And that's why specifically the self-driving, that's such a complex system and will continue to be a really complex problem to solve whereas a game with rules is a relatively easy space for a system like this to work in. And so as you go into more critical areas -- so for example, healthcare would be near the top, along with self-driving -- those types of decisions are the ones where you really to be extremely confident before you allow a system to make those decisions outside of the testing environment. So in both of those much more critical situations you want to do a huge amount of testing to ensure that you really have the system that you think you have. Whereas with, for example, a shopping situation where the algorithms may be just determining what advertisements to show, or in a situation where the system is suggesting the articles that we were talking about, you've got more leeway. And as you said, it's not as... The risk is lower, and the humans are not going to necessarily be in any danger due to those decisions. So you can allow the system to be more independent and to make mistakes and to learn from that a... You can release it a little bit earlier and allow it to be making mistakes even when it's being in use versus having a much higher bar for testing and confirmation and validation.

Jorge: That makes me think that ongoing training might be an important part of the systems, no? You have to be able to tell it, "No, you're not doing it right now."

Carol: Right. Yeah. And especially as you add more information. So for example, in healthcare there are image systems that are quite good for example, recognizing eye disease is a relatively successful implementation of artificial intelligence where the system can compare eyes and scans of eyes and understand that this one doesn't look normal. You know, it can tell the difference between a healthy eye and an unhealthy eye. And if you were to introduce a different animal eye, for example, to that system, that would require the system to basically start all over again. Because even though it is an eye, it would be a completely different shape. Potentially, there'd be different features in that eye. And so that new information would require the system to learn potentially on thousands and thousands of images of that type of eye so that it could relearn how to do that same task in a different situation. And if you, for example, added a new set of papers to an existing database and they had new terms, you would need to teach the system those new terms so that it understood that those were part of this body of knowledge was looking at. So there's definitely a lot of situations where you need to be doing upkeep and maintenance on these systems so that they up to date in their understanding and then also retraining them potentially to understand this new information and incorporate it into its knowledge set.

 Jorge: I'm hearing you describe that, and thinking, oh boy, I think that in the future all of us are going to have to develop this kind of AI literacy, where we become cognizant of the fact that we are training algorithms all around us to do things for us.

Carol: For sure. Yeah. I would love for more people to really understand not only the potentials but also the limitations of these systems so that they're not so scary and so that people aren't intimidated by them, but rather how they can use them to their benefit and also what the risks are with using them. I think that AI literacy is really important because, as you just said, we're in a constant state of training these systems, whether we want to or not, they're all around us in many of the systems that we use every day. And the more we understand about that, the better informed we can be, and the better we can protect ourselves and protect other people.

Jorge: Many of the examples that you've been alluding to, healthcare, this idea of examining the person's eyes, self-driving vehicles, the one about the academic papers, these are all... I don't want to call them edge cases because they certainly have mainstream applications, but I don't see most folks who are listening to our conversation going out and training an AI to recognize eye disease on a day to day basis. Right? It's not something that we do all the time. But we are starting to run into these algorithms out in the wild doing stuff for us that does impact our day to day experience.

Carol: Right. Yeah, certainly there are of situations where we're seeing, smart calendars, for example, that recognize certain words and email and maybe add an item to your calendar. For better or worse we've seen some people using that for, negative situations for advertising and that sort of thing as well. We're seeing that certainly within social media and a variety of different ways of showing advertisements and things like that. And we see it within some of the work tools that we use on a regular basis. And sometimes it's hard to know, is it artificial intelligence? Is it creative writing, you know, programming? And there's a lot of gray there because as we progress in our Use of these systems, some things cease to be considered artificial intelligence anymore and they just become a tool. so it's an interesting, progression of these tools and, and these different technologies as they become more and more integrated into our lives, at some point, they become just a tool and not even seen necessarily as artificial intelligence. Many of the things that we're using every day are somewhere on that spectrum.

Jorge: When you say "just a tool," that's as far as our perception goes, right? What I'm hearing you say is that like it would lose the mystique of being "an AI."

Carol: Exactly, yeah. So, for example, many years ago there were tools that you could use to scan text into a computer system, and then that text would be recognized and would be editable in an editing system, in a tech system. And that was considered artificial intelligence for many years. And now, people just as tools and it's not as specific or as mysterious. And that's a big part of it is, as people understand these tools more and more, it's all just programming, it's not magic for the most part, and they can be understandable. And I think that's really an important key is that they do not have to be a black box. And that if they appear to be a black box, it's probably a bad sign. It should be understandable. It may be complex. It may be difficult for everyone to understand it, but clearly someone needs to know exactly how it's working and why is making the decisions or the changes or how it's responding to the environment and why it's doing that. And that type of understanding and control over that is really important for these systems and for humans.

Jorge: What I'm hearing there is a call for some degree of transparency in the interfaces that allow people to interact with the system Is that fair

Carol: Yes, definitely. Yeah, transparency is a huge piece of being able to trust for humans in these systems. And whenever we're met with a system that's not understandable, we tend to be less trusting of it -- with good reason, as we should be. And so a big piece is just helping people understand the purpose of the system, the limitations of the system, the data that it's using how it's using that data, how decisions are made, and being able to really see the rationale for that. It will help everyone to be able to not only trust but also just understand and more powerfully use these systems to their benefit.

Jorge: I feel compelled to share with you my tale of woe.

Carol: Oh no.

Jorge: Because I see it kind of as a harbinger of what could be a bleak future for users of these systems. And this has to do with my Gmail account. I think of my Gmail account as kind of a corpus of... A set of data that can have these algorithms pointed to it to do things like make recommendations. And I have the fortune/misfortune of having a pretty good Gmail address because I was an early user. So my Gmail address is my first initial last name at And I have a surname that is fairly common in parts of Latin America. I get a lot of people who must have my same first initial and same surname, signing up for services using my email address.

Carol: Oh no!

Jorge: They think that it's their own. And I've had Gmail give me recommendations for my upcoming trip to the Azores, or whatever. And it's like, well, I'm not going to the Azores, someone has reserved a hotel using my email address.

Carol: Right.

Jorge: And I can't opt out of it, you know?

Carol: Right.

Jorge: What I've learned about this is that -- I think -- back to your point about transparency, I think that if I delete those emails as opposed to archive them, it'll take them out of the training data set. But I don't know. I wish that there was a button in Gmail that said, "Ignore this one for the purposes of the AI."

Carol: Right. No, that'd be lovely. To your point, I don't know if it's coming through, so to speak, their system as it comes to you, and then it's always in, or if you can edit the body of knowledge that has. Yeah, that would be lovely if you could. And that's something that's really important, is to be able to take things out. So, for example, thinking about judicial decisions, about prison sentences and things like that, there's been a lot of problems with these types of systems not being fair and being very biased and racist and learning all of the horrible things that humans, at least in the United States, have been using for incarceration for generations. And if you can't remove that, if you can't take those previous bad decisions out of the system, that means that that system is always going to be broken. So there need to be ways, in the simple situations such as not having annoying things on your calendar, to much more significant decision-making where we need to be able to edit and to remove information that is erroneous or unwanted or biased or whatever is wrong with that information. We need to be able to remove it from these systems to keep us minimally less frustrated, and on the other end, safe and not subject to bad decision-making.

Jorge: On that note, I'm wondering, because you're immersed in this world -- it's your training and your job and everything -- leads you to understand information through this lens. And I'm wondering how this has affected the way that you manage your own information.

Carol: Yeah, I keep thinking that I will be more cautious and do things differently. But I also am just so busy. So I tend to not be as careful probably with things as I should be. But at the same time, I use, for example, dual-factor authentication on everything I can. So I do take many precautions with making sure that my information as protected as possible, but at the same time, I love using a lot of the tools out there. So I love using chatbot that I have in my home. I really love being able to just ask it to play NPR, or asking it to turn on some lights, or whatever it is. I really enjoy these types of tools, and I really like the idea that my email can add something to my calendar. That's really helpful. It doesn't always add it the way I want it to, and I'd like more control in that sense. But I find these tools to be so helpful in my day to day life. I've got kids, and I teach a class, and I work and got soccer games and a million things going on on any given day. I can't imagine having this busy of a life without these tools helping me day-to-day. That being said, I'm very cognizant of all of the danger that is potential with these systems and how much of my personal information is out there. I'm very protective of my kids' information and trying to keep them off of these systems for as long as possible. So it's a balance. It's a constant balance where I'm constantly trying to determine, is this still the system I want to be using? Should I be perhaps moving to a different system? Should I be not using the system that I believe to be more harmful? Trying to determine how to manage all that is a constant decision-making and evaluation process, and for people who are less familiar with these tools, I'm sure it's much more frightening and difficult.

Jorge: And what I'm hearing there, in the way that you've been describing it, is that it can be scary, and it can be something of a chore, but earlier you talked about the fact that these systems can help make decisions for you and there's an aspect to it where it would somehow alleviate some of your day-to-day chores.

Carol: Yeah. So mostly with regard to reminding me to do them. I don't know that it makes less work for me, but it does help me to remember to do things. So a reminder on my phone that I should leave soon to get to my next appointment, or automatically adding travel to my calendar means that I am going to see that on my calendar later, even if I didn't manually add it to myself. It'll be there, and it will be a nice reminder for myself as well as my husband, that "Oh, look, there's something on the calendar. I can't do that at that time because I'm going to be heading to the airport." And so those types of things, I'm not sure that they necessarily are doing huge amount of time-saving for me yet, but they are helping me stay more organized, which is really nice.

Jorge: I wish that we could keep talking about this. There's so much to cover, but we're nearing the end of our time together here. So Carol, where can folks find you?

Carol: Yeah, yeah. I'm on Twitter @carologic, and on LinkedIn and many other tools, and I'm always happy to talk about these types of things.

Jorge: I appreciate so much that you were able to talk with us about it today.

Carol: Yeah, my pleasure. Thank you for having me.