Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Episode 14: Color, Concepts, and Design: Guest Karen Schloss
19 Plays5 years ago
Our guest is Karen Schloss, who studies the way in which color is imbued with meaning through a lifetime of associations with objects (like bananas and fire trucks) and concepts (like love and politics). We discuss her research, including topics such as:
- What color should recycling bins be?
- A tool that can help designers use color-concept associations in their work
- The "blueberry problem" (why is is that blueberries aren't very blue?)
- How to market a blue banana
- What color heaven and hell should be
Links:
Dr. Karen Schloss's lab at the Wisconsin Institute for Discovery
Colorgorical: A color-concept tool
Our paper for discussion: "Color inference in visual communication: the meaning of colors in recycling"
Recommended
Transcript
Introduction to Dr. Karen Schloss and her research
00:00:00
Speaker
Welcome to Cog Nation. I'm Joe Hardy. And I'm Ralph Nelson. On today's episode, we talked with Dr. Karen Schloss, who's a vision scientist who specializes in perception of color.
00:00:14
Speaker
Dr. Schloss is coming to us directly from the University of Wisconsin at Madison, where she studies all kinds of interesting things about the relations between colors and concept. And she's going to talk to us today about some of her recent research and how we might use these kinds of color mappings to improve design in everyday life.
00:00:45
Speaker
Karen, and thanks a lot for being here. Thank you. Thanks so much for having me. Thanks, Karen. Thanks for coming on. Yeah, so let's just kind of dive right into it. What's the framework that you use for thinking about color and how we should treat color, especially in terms of design issues?
Understanding color perception and concept expectations
00:01:06
Speaker
Sure. So the general goal of our research is to understand how we can make visual communication more effective and efficient. And the focus is specifically on color. And the idea is that to interpret visualizations, so by visualizations, I mean things like graphs and maps and diagrams, people have to figure out how visual features, like color, map onto concepts. Now, it's easier to do this when the mappings match people's expectations.
00:01:35
Speaker
So let's say I show you a weather map with a storm coming. What are your expectations of what colors represent heavy rainfall versus light rainfall? And so to use this idea in design that it's easier for people when designs match people's expectations, we need to know what those expectations are.
00:01:55
Speaker
And so that's what a large aspect of my research is, is trying to understand how these expectations of how visual features map onto concepts, how those are formed in the first place, and then how they influence our judgments about the world. The first thing that would probably come to mind is, you know, red is fire engines, yellow is bananas. So that's the sort of thing we're talking about, correct?
00:02:16
Speaker
Yes, but to think about only one red and only one association with it is a really limited way to think about it. And I don't think it's how the mind works. So there's lots of different shades of red. There's light reds and dark reds. And a particular red will be more or less associated with different objects depending on the colors of those objects. So let's take
00:02:37
Speaker
A particular red, say the University of Wisconsin Badger Red. So that red is a very vivid kind of prototypical red. And that red would be associated with Wisconsin, but it's also associated probably with the store target and with blood and ripe roses, ripe roses, ripe strawberries and blooming roses. You could eat the roses I guess, ripe roses.
00:02:59
Speaker
But so you've got lots of different concepts that are associated with the same color. And I actually would go so far as to say that every single concept that we know about is associated with every single color to some degree. So now you said yellow and I think bananas or what was your yellow? Yeah, yellow and bananas.
Associations and communication through color
00:03:19
Speaker
So bananas are highly associated with yellow.
00:03:22
Speaker
probably the early experiences you've had with bananas were yellow. But as you experience more bananas, you learned about unripe bananas. So you might associate bananas with some yellowish greens. And you learned about rotten bananas. You would associate them with browns. And you can actually represent the degree to which you associate bananas with all possible colors with a number. So let's say my strongest association is with bananas and yellow, a very saturated vivid yellow. So let's call that a one. That's a super strong association.
00:03:51
Speaker
And let's say I don't associate bananas with purples at all, a particular purple. And so that would be a zero. And you can represent the association strength between all possible colors and the concept of banana as a number from zero to one. Purple would be a zero. Brown, like those brown spots on bananas might be four or five, depending on. And green bananas would be five or six, and then yellow would be up near the top.
00:04:19
Speaker
I think it seems like an important point here is that probably the associations, Karen, you mentioned this, the associations that you have depend on your own experiences. So, for example, I don't think I've ever seen the University of Wisconsin sign. Well, I've seen it, but I haven't seen it very many times and you've seen it probably many times.
00:04:41
Speaker
And so your association there would be much stronger than my association. For me, red doesn't necessarily have a strong association with Wisconsin versus for you, it does. Exactly. These associations are going to be, to some degree, personal because as we go about the world, we have our own personal experiences of co-occurrences between colors and concepts.
00:05:02
Speaker
But in other cases, they're going to be very ubiquitous. So we all associate clear sky with blue. And so a sunny day when there's not going to be rain is associated with a particular shade of blue or range of blues. And that's something that's common to all of us. So although there are these idiosyncrasies, which we can use to try to understand individual differences, there's enough commonality in the world that we can use color as a common language.
00:05:28
Speaker
And how much, when you think about it from that perspective, how special is color as a marker of concepts compared to, say, other visual representations, whether they be written words or other iconographic forms or things like that? Is there something very, very special about color, or is it just another kind of holder for concepts?
00:05:56
Speaker
That's a great question. So I think there are cases where color is unique and cases where it's like other visual features. And I've actually been starting to develop a case for the idea that colors are actually like words and sentences. So I guess let's start there. So your interpretation of a particular word in a sentence, so say the word rake might be different depending on if we're talking about
00:06:21
Speaker
a glass falling to the ground versus driving. And so you need context to be able to disambiguate the meaning of that word in a sentence and you will make different inferences about that word depending on the context of the other words. And colors are similar. So your interpretation of a particular color will depend on the other colors that are present and the other concepts that are active in your mind.
00:06:43
Speaker
So I think in that sense, we can talk about color as a tool for communication in a similar way as we can talk about words in that case. And this is an idea we're just starting to develop in the lab. So I'm sure there's going to be a lot of cases where this analogy breaks down. But it's an interesting way to think about color in this way. So the idea that you are forming associations
00:07:06
Speaker
throughout your lifetime between this visual feature of color and concepts is certainly not special to color. So we're forming associations between odors and concepts and shapes and concepts, really everything as we move about the world, we're forming these associations. I think a really special thing about color is that it's the only visual feature that's non spatial.
00:07:28
Speaker
So something like size and orientation and shape, those are all spatial properties, whereas color is not. And so what you can do is you can integrate color with these spatial properties, which is useful for design.
Color choices in practical design applications
00:07:42
Speaker
So let's say we're talking about creating a campus map.
00:07:45
Speaker
and we've got a bunch of buildings on the map. If I were to use size to code whether a building is for a dining hall or a dorm or an academic building, if I were to use size of the building on that map, you would make inferences about the size of these actual buildings. There's this direct correspondence between how things are represented in the map in terms of size and how big you think the things will be in the world. Similarly, if I were to use shape, you might expect some aspects of the building to have those properties.
00:08:14
Speaker
If I were to color the buildings, if the dorms were blue and the dining halls were, say, bread, I'm just making these up right now, I think you would be able to think of these as an abstract representation where you can still encode aspects of shape, like the shape of the building you're looking for and the size, and think of colors as kind of superficial property for the structure of the building, but that can communicate information.
00:08:37
Speaker
That's a good point. Sorry to jump in. In the example that you used, so if you color buildings, I forget which colors you had, so buildings blue and some other color for other features on a campus map. One thing about that is there seems to be an arbitrary relation between the color and the meaning that exists. From a design sense,
00:09:06
Speaker
you might think that it wouldn't matter what color you used, as long as it was something that was perceptible and different, so that you could tell a building apart from a road, apart from something else. So as long as you're consistent in what it means, it could be perfectly arbitrary. But I think what your research is pointing at is that it isn't really arbitrary. Is that right?
00:09:29
Speaker
Yes, that's beautifully said. So you could think of it as arbitrary because it can be this kind of superficial surface property in terms of function. But it's definitely not arbitrary in the way that people make inferences about what colors mean because we have these color concept associations that we build through our lifetime of experiences.
00:09:50
Speaker
But I think a lot of work in design, especially for information visualizations, has focused on making sure colors are different enough that you can tell them apart or that they're ordered in a systematic perceptual way without as much focus on the semantics. And that's where our work comes in. I think a great example of this is the coding of the recycling and trash bins that we all see everywhere.
00:10:19
Speaker
And you've done some interesting work in that area. I think that might be an interesting example to dive into a little bit. Sure. So some of the early motivation for this work was the experience, especially living in Berkeley, where you've got lots of different types of bins and restaurants. You go and you stand in front of these bins. You've got your food to throw away and compost. You've got
00:10:42
Speaker
Plastic, you've got a paper napkin and you're standing there in front of these bins and there's different coding systems for the bins. Sometimes there's color, sometimes they're shaped, sometimes there's both, sometimes they're labeled, sometimes they're not. And your task is to figure out how to sort these things. It can be like an intelligence test almost sometimes. It's very difficult. Yeah. And if you're in a giant hurry, please forgive me. Sometimes I would put things in the trash because it was too hard and I didn't want to get the recycling wrong.
00:11:07
Speaker
Because if you get the recycling wrong, they could ruin the recycling for everyone. So the default would then be to put things in the trash. So if you look in the recycling bins, you can see that a lot of people are making the same kinds of mistakes. Yeah. And this even happened. This happened to be the other day where it was something was shape coded. It was a I was holding something round and the hole was round. And I accidentally put the oh, I put a I put a coffee cup in the
00:11:29
Speaker
Please forgive me again. I put a coffee cup in the glass, in the glass and plastic bin where I worked because I was holding something round and the shape was round. And so there was this, this mapping that, so I did that again. Sorry, forgive me. So to get it out, but I couldn't anyway, that's a real life frustration. And it's different too. Like sometimes like, you know, like one time we'll have like the gray bin will be the trash.
00:11:58
Speaker
and the blue bin will be the recycling, and then the next town over will be the exact opposite. Exactly. And so this brings us to this really important issue of these many to one and one to many mappings. So you can have a single color that's associated with lots of things. We talked about this with the red being associated with roses and strawberries and Wisconsin and target and so on.
00:12:19
Speaker
And you can have lots of different colors that are associated with the same thing. So apples can be associated with reds and greens and yellows. And then we're talking about the Apple store. Then you've got silvers and matte black.
00:12:31
Speaker
And so this is both a challenge and an actual flexibility that means that we can use color for communication in really cool and sophisticated ways. So I had experienced these frustrations with recycling bins just in my daily life. And simultaneously, I was looking for a domain to study people's interpretations of colors and to see if we could encode colors in a way that people could interpret the meanings of them without any legends or labels to tell you the answer.
00:12:59
Speaker
Now, you might think that, well, why don't we just put signs on things? Why do we have to worry about this at all? Can't we just put words on recycling bins and tell you what they're for? But there's two issues. One is that there's evidence that suggests if the labels mismatch the colors, it's an incongruence and it's harder for people to interpret. They don't just ignore the color if there's a label there. What would be an example of that?
00:13:23
Speaker
So there's a study by Lin et al from 2013 where they looked at bar graphs of fruits. And so the task was to read the bar graph and say something about the relations of the bars. And so let's say banana was yellow and blueberry was blue, or in another case, banana was maybe green and blueberry was orange. And people are faster at interpreting the graphs when the colors match. So this is sort of like a stroup interference test where you automatically process that color and you just can't
00:13:53
Speaker
You can't avoid it. Yep, exactly. And so in some of the work from my lab, we looked at this with color maps. So people are faster. So the color maps, like on weather maps or in neuroimaging, you're mapping different magnitudes to color. So wind speed or rainfall or temperature to color.
00:14:10
Speaker
or brain activation. And so we find in the lab that people are faster at interpreting color maps when darker colors map to larger quantities. So it's a darkest more bias. And other people have identified the darkest more bias, but we show that when there's a legend that clearly specifies what the colors mean, people are faster if it matches their expectations of darkest more.
00:14:29
Speaker
And there's interesting perceptual intricacies of that, which we can talk about if you're interested in. But the main point here is that you can't just label things and think that color doesn't matter because people do have these associations, they have these expectations, and if the labels mismatch, then it's harder to process.
00:14:46
Speaker
And so coming into this recycling experiment, our question was for objects that can be any color. So for the bar graph, for example, yeah, bananas tend to be yellows and blueberries tend to be shades of blue. So in that case, it seems kind of straightforward in a stupy sort of way that you would get this interference. But what about objects that can be any color? So trash, anything can be trash. Paper can come in any color. Plastic can come in any color. Any food can be compost, especially when you're going to throw it away. That's a variety of colors as well.
00:15:15
Speaker
Will people, first of all, have systematic associations if we have them rate how much they associate colors with concepts? Will there even be some regularities there? Or is it just noise? And if they do have those associations, can you create a color coding system? So I show you a bunch of bins with no labels. Can I create ones that you can interpret which ones are for paper and glass and trash and so on without any labels?
00:15:39
Speaker
So in an initial experiment, we wanted to get these color concept associations. So we had people look at object names at the top of the screen. So the word paper would be printed at the top of the screen. And they saw each of a bunch of colors, just one at a time, as a color square below the name. And they rated how much they associated that color with the concept. So a particular shade of white, how much to associate this with paper, black, greens, and reds, and so on. So it was a bunch of different colors.
00:16:09
Speaker
And so from that, we got our color concept associations. And we found out that they were systematic. So whites
Challenges in color perception and digital design
00:16:15
Speaker
and like grays were associated with paper. Note that this is not the greens and blue greens that we see in the colors of paper cycling bins. So this is associating the object paper with these concepts. And we found that glass was associated similarly with whites and grays, but also some blue greens and greens.
00:16:35
Speaker
plastic, also similarly associated with whites and grays, and then compost and trash tended to be associated with blacks and browns and dark greens. And so on the one hand, this is great. We've got these associations. They're systematic across people. On the other hand- So I'm looking at this image right now with all of these associations, and it's a really beautiful image.
00:16:59
Speaker
And by the way, we'll put a link to this paper on the show notes so that you can see some of these pretty diagrams too. I have six objects here. So from paper all the way to trash. And you can see the color least associated with paper is kind of a bright red. And then it goes all the way up to light bluish and then white are the most associated with it. And then for metal, for example,
00:17:26
Speaker
kind of a grayish tone is the most associated with metal. So in terms of color space description, is there a continuum along which these things go so it looks systematic in a way, like it's sort of lighter colors, maybe almost pastel-y for some of these things, and then more saturated colors for compost and trash? Is there any systematic dimensions that this continues along?
00:17:55
Speaker
Cool. Yeah. So I think the systematicity I think is related to how the colors of these objects occur in the world. And so for paper, for example, you tend to have whites and pastels for paper in the world because you're printing black text on them and you want the text to show up.
00:18:13
Speaker
I think for glass, glass can come in any color, but you've got, I think clear glass is kind of the most prototypical and you've got that greenish tint to it. So I think of, basically you can think of color space, color space is a three dimensional space where the dimensions or the perceptual dimensions are hue. So like red, yellow, green, saturation, which are, or chroma, which is how pure the color is and lightness from dark to light. And you can think about,
00:18:41
Speaker
for a given concept, you can put a weight on every color in that space. And I think it's continuous. So if you've got a saturated yellow that's really associated with a particular concept, say, banana, you're going to get a gradation around that. And I think that's where the systematicity in color space comes from. So meaning that you would also have some association with more saturated, but possibly less associated with desaturated colors?
00:19:08
Speaker
No, I think it's a gradation around the particular color. So if you've got a supersaturated yellow and you've got a supersaturated blue that's supersaturated blue, although they're both saturated, they're very far apart in the color space, in the perceptual color space. So I think you're more likely to have a decline around there. But then you've got objects that are multiple colors. So like
00:19:27
Speaker
and apples, which are both red and green, which are very far. And they're relatively saturated, and they're very far apart in the color space. So I think it really depends on the colors of objects in the world, and then these gradations around them within color space. Yeah, I think also one thing that occurred to me when I was looking at this paper was the idea of when we talk as color scientists a lot about chromaticity space or perceptual color spaces,
00:19:52
Speaker
we think about things as they could be represented on a computer screen or even maybe on a piece of paper under very unified or normalized lighting conditions. But as I look at some of these objects and I think about them, especially as they would be organized or used in the real world, it makes me think about the fact that one of the most interesting aspects of the visual representation of color for metal and glass and plastic
00:20:21
Speaker
and paper is, for example, the specularity, how shiny or not shiny it is, how it reflects light from different angles. When I look at metal, for example, as the ratings here being kind of a gray color as being the most associated, it's probably not really gray, but more like a silver type of color that is not really easily represented on a computer screen.
00:20:51
Speaker
I mean we should say we know that so Karen used a very specific set of colors that have been used in a number of experiments before and are established as a clear set of reference colors that are carefully calibrated for the monitor and so you get the same in any experiment that you would use them for. Yeah I think the point that Joe brought up is a really interesting one which is that
00:21:17
Speaker
When you're talking about surface properties of objects in the world, they do have
00:21:22
Speaker
specularities and shadows and transparency and reflections. They're very complicated. Then when you're talking about mapping that to a single solid patch of homogenous color, how does that correspondence go? That's why it's helpful to get humans to make these ratings. Recently, we've been looking at automatizing that through image statistics. But the question is, if we're going to use a solid patch for visual communication, what are the colors that you want to use?
00:21:51
Speaker
So like in a bar graph, for example, or a weather map or something like that, where it is not going to have those highlights and specularities, what are the colors that map onto the concepts in that sense? And so there's going to be some correspondence between the colors of objects in the world, but there's going to be some abstraction as well. Does that make sense? No, absolutely. Absolutely. Thinking about, for example, like metal, if you were going to
00:22:17
Speaker
get really designy with your like metal bin for recycling, you might want to paint that was kind of shiny. So I guess it would be different if you were like painting the bin versus if you had like a sign, right? If you had a sign that was just like a color, that would be one thing. But if you were actually painting the bin itself, which could have some additional elements to it might be
00:22:39
Speaker
There's more information you could even convey, I guess. Yeah, I think that's a really interesting point, which is that color is just one aspect of this, but then there's other visual features you could use as well and other visual features that are surface properties. And so then how does the exact same base color change in its associations depending on whether it's shiny or not? So like,
00:23:01
Speaker
If you like a particular to get gross for a second, a particular shade of brown might look like chocolate or it could look like other gross things. And you could probably vary that depending on the kinds of surface properties like shadows and highlights. And I think that's going to be a really important aspect of of people's interpretations of colors in the world and potential for visual communication that we're not focusing on at all right now. We're specifically looking at color. But I think there's really interesting questions there.
00:23:27
Speaker
Yeah, cool. Well, it's good. I mean, we were talking at the beginning about before we started recording about how much we agree or disagree on what color is. So this is an interesting area because it sounds to me like maybe you would say that silver is not a color distinct from gray. Oh.
00:23:49
Speaker
That's complicated because, okay, so I usually get the opposite question, which is, well, white and black aren't colors. And I'll say, well, of course they are because they're in the three-dimensional. No, white and black are definitely colors, 100%. Well, of course, no, I'm saying. But so anything in color space is a color and white and black are in that space. But what you're saying is that you can have the exact same point in color space, but then you vary these other surface properties of it. And then are they then different colors, even though they might fall?
00:24:18
Speaker
Although, actually, hold on. So it wouldn't be the exact same point. I'm going to back up. So it wouldn't be the exact same point in color space. There would be multiple points in color space because you would have these highlights. And so basically, you would be getting a constellation of points in color space. It wouldn't be just one particular shade of gray. You would have lighter grays and darker grays. And that's what would give you silver. So when I say silver and gray or different colors, I
00:24:42
Speaker
Nobody's asked me that before. The layperson, he says, yeah, of course. If I'm going to pick crayons out of the crayon box, I can identify the silver one and the gray one, and they're two different crayons. Well, that's an interesting term. It's interesting to consider whether that's another dimension to be added onto color space that
00:25:04
Speaker
I mean, are there linguistic terminology? Is there linguistic terminology that indicates? Because when you say silver, I think, well, I can think of silver as, yeah, something that you pull out of a crayon box. But I also think that it does have this specularity to it. When you're looking at the actual, you know, the actual element of silver, it's got, especially this polished up, it looks different than what you can produce from crayons, right?
Adapting color associations in design contexts
00:25:31
Speaker
Is that additional dimension that like you say surface properties of visual surface properties other than color? Well, I think that I think we're not going to answer the question entirely right now, but it's interesting. I'm just trying to think of it. So let's say they certainly get different color words to describe them.
00:25:51
Speaker
Right. The perceptual experience of them is different. You can get a gray that is one single point in color space to get the perceptive silver. You probably have to bring in multiple points in color space to be able to get the highlights and shadows. And so there are. I think about this because my, okay. So my daughter's favorite color is yellow, pretty straightforward. And there's a really, no, this is always, it's been that way for.
00:26:20
Speaker
She has had a ducky since she was zero. That I do. This is a perfect example of your previous work on color preference, that you get color preference from objects that you like that are of that color. But my son's favorite color is gold and shiny. That's amazing. That's awesome color. Which is, I mean, it's a modifier on just gold, right?
00:26:48
Speaker
It's not exactly the same. It does have this other element to it. Right. But yeah, exactly. Yellow and gold are different colors. And you might say that the gold object, if you just brought out your spectrophotometer and pointed it at the gold, it would give you the same chromaticity coordinates that a yellow would give you. But it doesn't look like the same color. I mean, it feels natural to call it a different color. And we do in fact call it a different color.
00:27:17
Speaker
But I guess we don't have to get stuck in that, so maybe we should get back to recycling bins. We were talking about how this relationship between the different objects that go into the bins and the bins themselves having this overloaded relationship because there could be several different types of objects that want to have the same color representing them. Exactly.
00:27:42
Speaker
And so I had these ratings. Well, I had the question of, okay, so given that all of, so plastic and glass and paper all have really similar associations, how could you possibly figure out what colors to use? Because there are these confusions. So I went to my collaborator who, I think this is the beginning of our collaboration, Laurent Lessard, who's a professor in electrical and computer engineering at Wisconsin and also in the Wisconsin Institute for Discovery.
00:28:08
Speaker
And I came to him and I was like, I have these data, these ratings, and I want to be able to figure out how to color these bins so that people can interpret them. And he was like, oh, well, this is a basic assignment problem in optimization. I was like, okay, good, tell me more. And he was like, this is like a really basic, basic, simple problem that we solve where you've got a bunch of things in category A and a bunch of things in category B, and you need to figure out how to map them onto each other in, um, to optimize some metrics.
00:28:33
Speaker
So the example he gave me was swimmers in a relay race. So let's say you've got 20 swimmers on a team and there's four strokes in a relay race and you need to figure out which of those 20 swimmers you should assign to each one of the four strokes.
00:28:46
Speaker
And you might say, well, obviously you would just pick the fastest swimmer for each stroke and just assign them that way. But that's not always the optimal way to get the fastest race overall. Because sometimes you're better off at assigning a swimmer to a little bit of a less good or less, not their best stroke, because there's someone else who has another one that's really, really fast. And then that's the optimal assignment. And so this is an issue in
00:29:08
Speaker
swimming teams, but also in shipping, delivering, and just all different areas. This is just a general problem. And so we can approach this recycling problem in the exact same way. So we have these data, these color concept associations, and we can figure out ways to optimally assign the colors to these concepts, so these recycling concepts. But the question is, what do you optimize?
00:29:32
Speaker
because you define some sort of merit function, which is the thing that you want to maximize or minimize. But what is that in this case? And so this was an interesting problem. So one thing you might want to optimize is just the association strength between the colors and the concept. So put paper with its most strongly associated color, put plastic with its most strongly associated color with the rule that you can't have the same color that gets assigned to two different concepts.
00:29:57
Speaker
So that's one thing you can maximize. But another thing is you can also try to maximize association strength while also minimizing confusibility. And by that, we mean bringing in colors that are similarly associated with different concepts. So there's a trade-off there. Do you make the association strengths as strong as possible? Or do you allow for weaker associations but to avoid those confusible colors like grays, which are really associated with paper and plastic and glass?
00:30:24
Speaker
So as I'm looking at the associations with paper and plastic. So the second solution that you're talking about to avoid confusibility might take, you know, say white for paper, and then instead of taking a slightly darker white, slightly grayish color, you might move and get something that's more perceptually distinguishable.
00:30:54
Speaker
It's not just pretty exceptionally distinguishable. It's distinguishable in the associations. So something that's more distinguishable. Okay. So, and as I'm looking and I, it's hard to describe this, I guess, but as I'm looking at it, so there, so the first couple of colors for both paper and plastic are gray scale. And then there are some light blues and then there's some differentiation in that paper is more closely associated with yellow.
00:31:22
Speaker
And then plastic continues with darker colors or blues. So at that point of difference there, you would really be able to tell the difference if you used white for one and another strongly associated thing, but something that's differently associated for plastic. Right, and so it turns out that the best color for plastic under that kind of balanced merit score was red.
00:31:52
Speaker
Now, red is really weakly associated with all of these concepts. So with paper, glass, trash, metal, it's very weakly associated with all of them. But it's more associated with plastic than any of the other ones. It's super not associated with paper. It's super not associated with paper or any of the other ones, but it's a little associated with plastic. And so the model ends up picking red for plastic because that's a color that differentiates the plastic from the other concepts.
00:32:21
Speaker
So even though it's a really weak associate, that's what ends up picking. And so then when we show people these colored bins and we say, okay, which one's for paper, which one's for plastic and so on, when we use this balanced merit function, so it's avoiding confusibility, but having weaker associations, people are better at interpreting and they can do it reliably. They can figure out which bin is for what kind of object based on the color alone, whereas there's much more confusion. They can't do the task nearly as well as if we try to optimize just the overall association strength.
00:32:52
Speaker
So it's as though you've got a piece of paper and you don't immediately associate it. Well, let's see. Okay. Let me say that again. You've got a piece of plastic. You don't immediately associate and say that goes in the red bin for sure, but you know it doesn't go in the other bins. And you're much more likely if you have a piece of paper and a piece of plastic to put them in the correct bins, even though you're not as
00:33:19
Speaker
positive about how those bins are associated with the object. Yeah, how the colors are associated. Yeah, it's hard to use so many. It's hard to start using so many terms and make this understandable. But yeah, but I guess people were not only were they more able to make these correct, so-called correct associations, but or correct choices, but they were also faster, right? They were able to make these choices faster.
00:33:49
Speaker
Yeah, they're faster if they match their association strengths more strongly. That seems really important when you're talking about a practical application like this recycling bin thing. Because as we were talking about earlier in the show, if it takes you more than one beat to figure out which bin to put something in, it might go in the wrong bin. Yep. So yeah, I think it's super important from a practical perspective to get that right.
00:34:17
Speaker
From that perspective, what are you thinking about the application of this to the world? How generalizable are these mappings? I don't know where you collected the initial data for this one.
Tools and innovations in color semantics
00:34:35
Speaker
But if you took this and put it in another school or another city in either the same country or a different country, however far you want to think about going, how
00:34:47
Speaker
different or similar do you think these relationships would be? So I think the general principle should generalize. But if the associations are different, then you'll get different mappings. So the idea that you want to avoid confusion in the way that we did that, I think that should work in different locations, different countries. But if the associations are different, so if you're in a country where
00:35:13
Speaker
paper tends to be way more often yellow, or if you're working with lawyers who use yellow legal pads, and they much more, say hypothetically, they much more strongly associate paper with yellow than with white, then the optimal mappings are going to be different because it's optimizing these association strengths. I will say that we collected the association ratings at Brown University, and then we did the recycling bin interpretation tasks at Wisconsin. So at least there's some difference there.
00:35:43
Speaker
But there should be cultural differences to the extent that there are differences in these color concept associations. So if you're a designer, wouldn't it be great to have a set of these associations at your disposal for each color and each concept that you wanted to test? And I guess, yeah, you would have to have some cultural specificity.
00:36:14
Speaker
Have you thought about having a practical tool for people to use and designers to reference?
00:36:22
Speaker
Yeah, so we've been working for a wireless tool called Color Goracle, which you can find online. And this tool tries to balance perceptual discriminability and aesthetics. And there's no semantics in it at all. So you want colors that are different enough that you can tell them apart and you want it to look pretty. And so this tool will generate palettes for you. And what I would love to do would be to build in the semantic component too. So not only
00:36:50
Speaker
Can I tell the colors apart and I like them, but I also am making a design for recycling bins and I can enter in paper and plastic and it will tell me what colors I should use.
00:37:01
Speaker
And the problem is how do you estimate these color concept associations? So from this recycling sample we just discussed, you don't want to just know the most strongest associate or even the top couple strongest associates. You need the full range because sometimes you get this case like the red plastic example where you want a weak associate and that's what people will be able to interpret. Some previous people have looked at
00:37:24
Speaker
automating, extracting colors from images. And we have a project that we just did on developing that approach. So how you can take images from Google images and estimate the color concept associations from those and what sort of information we need to be able to do that. Because it takes a long time to get people to make ratings for a bunch of different colors for all possible concepts.
00:37:46
Speaker
So the question is, how can we automate this so we can get this giant database of color concept associations that then you can know what your colors are and concepts are and then do this optimization over them. Out of the top 50 images that I got for
00:38:00
Speaker
Mango, what are the colors in these images? And so we talked about this color concept association space at the beginning where for every single color you can put a weight, you can assign a number from zero to one for how much you associate that color with that concept. And we can estimate that from images. And does there seem to be some relation between your carefully gathered subjects doing this task and then what you would get in Google images?
00:38:28
Speaker
Yeah, so there's some direct relations and then some extrapolation that we need to do. So this is some recent work with Raghini Rathore and Laurent Lessard and Zach Lagan here in Wisconsin.
00:38:39
Speaker
And so in an initial attempt, we just looked to see, so we had a set of 58 colors that we defined and then we said, okay, how much of each one of these colors is in the images of different fruits? So we tested mangoes and strawberries and blueberries, 12 different fruits and avocados. And so our first question was, okay, how well can we estimate people's color concept association ratings that we got from our carefully calibrated study in the lab?
00:39:06
Speaker
with these image extraction methods from people images. And in our initial approach, we found that it worked for some fruits like mangoes, but it really didn't work for blueberries. So we affectionately call this the blueberry problem now. What is the problem with blueberries? Blueberries are a big problem. Oh, no. I love blueberries. I do too, especially when they're big. So the issue is that blueberries aren't
00:39:36
Speaker
Really, they're not blue. Yeah. So if you look at such as the distribution of colors that are in the image, the colors are they're very dark or desaturated bluish greens, but like a prototypical like blue, blue, blueberries just aren't that color. But if you ask people to make ratings of how much they associate colors with blueberries, they rate those colors very highly. Well, now there's not that much blue in the natural world. So.
00:40:02
Speaker
I guess maybe there aren't many other candidates for something called a blueberry. Why do we call blueberry? They're almost black if they're really super ripe and they're kind of purpley. A lot of times they're green. But if you were given the set of basic color terms, and this actually becomes really relevant to the research, if you were given the set of basic color terms, so red, yellow, green, blue, black, gray, white, pink, brown, orange, I think that's all of them. Is that 13? There's 11.
00:40:32
Speaker
And if I wasn't 11 in English, right? Yes. Okay. In English. So if I gave you those terms and I said, okay, here's a blueberry, which of these one labels do you assign to it? You probably would take blue, maybe black, but you wouldn't pick gray. They're not really green. So, so it's not that they're not blue. It's just that these very vivid and prototypical blues are not in blueberries yet people associate them with blueberries.
00:40:56
Speaker
So we face this problem of we've got this input from the images and it's not matching people's associations very well. This was not a problem for mangoes. They're fine. And for other fruits that were fine, but for blueberries in particular, this was a problem. You think it would be a problem if it was, if they were called like sour berries instead of blueberries.
00:41:15
Speaker
So I, you know, you might think it was the name, but then for orange, oranges, this worked out very well. And so our two endpoints actually of where the image extraction worked really well and where it didn't were two fruits that have color names in them, oranges and blueberries. But, but they may end up being very essential to this. So what we did was we said, okay, how can we figure out a way to get all these other blues into the model as getting weight, even though they are not in the images. And so what we did.
00:41:41
Speaker
was we used a category algorithm that other people had produced where you can feed an image through it, and it will tag each one of the pixels with its most likely basic color category. So red, yellow, green, blue, there's 11 terms that we just described. And that model was constructed from careful psychophysics data, where they figured out where category boundaries were. What we did then is we said, OK, we get our image, say of blueberries, and we get a pixel.
00:42:08
Speaker
And in addition to counting that pixel as being a particular part of color space, we also say, OK, what was its color term? And if it's given a blue, that pixel counts towards all other blues.
00:42:20
Speaker
So basically, we're coding color categories into our model. And so then the question is, if you're trying to predict these human judgments, what features or dimensions will the model use? And it does end up using this color category dimension in addition to these pixel input dimensions. And so, of course, we train our model on one set of fruits, and we test it on a different set of fruits using cross-validation to make sure we're not overfitting.
00:42:46
Speaker
And we find that we can solve the blueberry problem this way by breaking in color categories. So coming into this, our model for how this whole thing works is that you move about the world with new statistics of how colors co-occur with concepts. And then you just build these association spaces in your mind based on this input.
00:43:11
Speaker
And so now what we think happens is not only are you taking this input to inform these association spaces, but you actually are also categorizing the colors and then you're extrapolating to all other colors within a category. And then that influences your associations as well. And so just by trying to estimate these associations from images, it actually informed our theoretical framework for how we think
00:43:32
Speaker
humans are actually forming these color concept associations in the first place. So I have all kinds of other new experiments to follow up on to really test this model to see how well it actually predicts people's judgments. Very cool. Very cool. Well, this might be a good chance to take a break and then we can come back and maybe talk about some more big picture applications and future directions.
00:44:07
Speaker
Okay, and we're back and apologies about the poor sound quality on my end. I seem to have fixed that now, so I think I should sound a little better. So we're back with Karen Schloss. And one of the things I wanted to get to at some point is talking about your experience with using virtual reality as a research tool. So maybe you could expound on some of the uses that you've had with it.
00:44:32
Speaker
Sure. So we've both used virtual reality as a research tool, and we're also using it to develop new approaches to bringing technology into the classroom. So I can briefly talk about it as a research tool, but I think the main focus of my lab right now is actually the developing for the classroom. So as a research tool, we used VR to understand how people
00:44:55
Speaker
interpret colors in an emergency evacuation situation. So this is work with Max Knitader and Bill Warren at Brown. And the question was, if you are in a situation where there are different colored signs and you're in an emergency, which sign do you think is the correct one for the emergency evacuation door? This is an issue because there are tunnels, especially in Europe, where there's
00:45:21
Speaker
a lot of different sign colors, illuminated signs that signal different information. Some are call boxes, some are exits. And it can be really dangerous in a fire, especially when there's smoke and you can't see the symbols on the signs, you can't read the words, but you can see the colors. It can be really dangerous if you don't know which is the right one for the emergency evacuation. And so what we did in the study is we presented people with a room in virtual reality. So they're actually in a room that is rendered to look like the testing room.
00:45:50
Speaker
They stand at the back of the room and there's two signs in front. And one of them is one color, another is another color. And they're asked to just walk towards the door they think is the right one in the emergency. So the alarm starts playing and they walk towards the door. And the signs were just vertical bars and they were all pairwise combinations of blue and green and red and white and yellow and magenta. In green, I said that.
00:46:16
Speaker
Green's important, so I said it twice. So in previous work looking at signage and evacuations, these were done in regions where the exit signs were green. And so in those previous studies, they found that people tended to, like if they're navigating a maze and they have to figure out which sign is the one for the exit, just like in a computer simulation, people tended to pick the green one.
00:46:41
Speaker
But it wasn't clear if that's because that's the color of science in their environment or because there's this association with green and exitness. So we are fortunate enough to do this study in Rhode Island where exit signs are red. So we could dissociate these hypotheses. So one hypothesis is that
00:46:56
Speaker
you think the exit sign is the color of exit signs in your local environment. So that would mean that people should walk more towards the red door. Or it could be that we associate green with things like go and safety of red, so things like stop and danger. So you might be more likely to approach the green door, even though your local environment has red signs. So wait, sorry, in Rhode Island, I went to Brown University also,
00:47:21
Speaker
as an undergrad. So I know that everything is a little bit off in Rhode Island. But we're saying exercise like on the highway or or on doors or building. OK. Is that every everywhere in Rhode Island is like that? Yeah. Yeah. Yeah. Yeah. I could confirm that. And so in the experiment, we first had people do this walking task and they got all pairwise of signs and they tended to walk more towards the green door robustly.
00:47:51
Speaker
way more than the red door. But then afterwards, we asked them what color are the exit signs in this building? And they could all say red. And we said, what color should exit signs be? And people tended to say red. We said, what color are exit signs where you grew up? Most people said red. Yet just before that, in this simulated emergency, they walked way more towards the green door than the red door.
00:48:14
Speaker
So what this suggests is that people are, there's this dissociation between what people do and how they interpret colors when they're actually interacting with the world and how they respond to a questionnaire. It suggests that creating these real world design situations is important for understanding how people interpret visual features like color in design. And so,
00:48:40
Speaker
So that's a really powerful use of virtual reality for research is you can actually simulate these real world situations and you can manipulate the colors or other visual properties or sound properties quite easily through the interface. And then you can keep everything else constant and just manipulate that exact one thing that you want to do. Yeah. Yeah. So it's a really exciting direction for research, I think, to be able to do that.
00:49:05
Speaker
So this is an interesting, I hadn't thought about this, but so if you're, if exit signs are red and, but you're used to in traffic, red being stop, um, sort of like a mixed signal that sometimes red means stop and sometimes red means go. Yeah. Yeah. That's why I was also thinking about the highway, right? You know, like highway signs tend to be green.
00:49:31
Speaker
And, you know, it's an exit sign on the highway. So quote unquote would be would be green. Good point. Yes. But exit kind of means a different thing. It's like that. I don't know. It would necessarily be translated in other languages as exit in the same way. Right. Like it's like a different activity getting off the highway versus leaving a building. I don't know. Is it? Maybe some my.
00:49:55
Speaker
Association might be avoid going out that exit door if it's something that's going to cause an alarm. But if it's somewhere where you're trying to direct people to go, like after a movie or something, then. Oh, that's a good point, right? It makes sense. Those would be different. Those would be different purposes. Right. In an emergency, the exit is the way you want to go. The emergency exit is the way you want to go. But most of the time you don't want to go there. And maybe that's why people are this is a potentially
00:50:25
Speaker
life-saving kind of thing because if you can prevent that hesitation from going towards the red sign, then you could get people out faster and less chaos. Yeah, exactly. Maybe it needs to be able to switch between red and green. Yeah, there has been some work on dynamic signs where they can
00:50:50
Speaker
change depending on. So if it's a door you shouldn't, because there could be some doors you shouldn't use in an emergency if they're not safe, because maybe that's where the fire is. So then they can put up other lines through the exit sign and be like, don't use this one. Right. I guess then you worry about the power goes out, as it probably would. How effective is the dynamic sign? The default setting is probably very important in that case. Another question I have about some of this is,
00:51:19
Speaker
To what extent do you think of these color concept associations as something that people are aware of? And to what extent is it just going on in the background or sort of an implicit thing that you might not necessarily always be aware where your associations come from? Yeah, I think it's both. I think we are explicitly told about specific colors of things, especially when we're little, bananas are yellow.
00:51:48
Speaker
But I think we are constantly monitoring the statistics and picking up on them in our environment. So we do have some preliminary work from my lab suggesting that people are sensitive to these distributions just through laboratory exposure to co-occurrences between colors and concepts. So I think it's both, but I think for the most part, we're not, at least other people, I kind of do. But I think for the most part, people aren't going around the world and being like,
00:52:14
Speaker
Stop sign, red, tree leaves, green, flowers. I don't think we're labeling things all the time. And I don't think we're paying attention to these things a lot of the time, but I do think we are detecting them. You could still be registering them in some way. Yes. Even if it's not necessarily conscious, it'll be an association that gets registered and sort of filed for future use, but it could be or it may not be consciously aware.
00:52:41
Speaker
Yeah, and there's really interesting questions about whether it's the first early experiences that shape these associations or the more recent ones. So we know from the odors literature that early memories with particular odors are the things that seem to dominate your odor concept or odor memory associations. But you could imagine that if you have a lot of experience with, say,
00:53:05
Speaker
yellow bananas when you're little. And let's say you went to a hypothetical new country where bananas are now blue. Should you really, and you're trying to figure out whether a banana is ripe or not. This is kind of an odd example, but you're trying to figure out whether bananas are ripe or not. And in this new example, bright blue bananas are ripe and dark blue bananas are not, let's say. So are you, but you live the first say 20 years of your life in a planet where bananas are yellow. And now let's say we went to a new planet where bananas are blue.
00:53:32
Speaker
Are you better off holding onto these original associations or should you let them go and more emphasize or more heavily weight the newer ones?
00:53:42
Speaker
And so those are interesting directions that we're going in the lab right now is trying to understand these temporal aspects of it. And preliminary evidence suggests it's actually the more recent experiences that matter, which sounds adaptive, right? Because it should be the things that are most relevant to you at a given moment in your life that are the things that should influence these associations as opposed to these old or these interpretations, as opposed to these old associations that may or may not be relevant anymore. So from a marketing perspective, the
00:54:09
Speaker
market could absorb a new blue banana if that's what we needed because people could adapt to it relatively over the short term. In other words, you wouldn't have older people that are absolutely stuck with the concept of a yellow banana and they could just never get over it. They could maybe get used to a blue banana in a few years. I don't know. That's kind of a different question. I think was it Heinz tried to introduce a new catch-up color?
00:54:38
Speaker
That did not go well. That sounds bad. Yeah. Yeah. I guess it may take a few years to, um, well, as we all know, the Cavendish banana is possibly on its way out. So we may have to prepare ourselves for a strange, strange future of differently colored bananas. I mean, if you really were to get rid of all of the old associations altogether and really a hundred percent associate blue bananas with ripe, awesome bananas, then yeah, maybe.
00:55:06
Speaker
Maybe you need a period of time with no bananas. Is it, is that the key? I don't know. Cause if the associations are still there, if they don't, if the question is, do those associations dissolve with no input? And I don't know. Or gradual, maybe it could be a gradual shift from yellow to blue. Yeah, maybe. Although it would be a pretty gross color in between for awhile.
00:55:27
Speaker
That would be a little weird color face or distribution. So it could be that you still have all the yellow and blue bananas, but you slowly introduce the blue ones in between. Because then if you go through green or gray in the middle, that's a whole nother set of colors. Yeah. Yeah. You're just going to lose people with that. I'm kind of open to this point. I'm excited about the blue bananas. This is this is a good direction. I think we should we should we should think about this. Yeah, you can tell we're both. I think we're all excited about blue bananas now.
00:55:58
Speaker
Speaking of excitement, what areas of research, Karen, are you excited about? Yes, I think this idea of categorization influencing color concept associations, as I mentioned with the blueberry problem we talked about a little while ago, that's opened all these really interesting new research questions about how these color concept associations are formed in the first place. My previous work was largely actually on color preferences, so understanding why people
00:56:28
Speaker
like the colors they like and how that's influenced by color concept associations. And that's work I had done mostly with Steve Palmer back in Berkeley. And so in that work, we kind of took for granted that people have these color concept associations. And then we were really interested in how they influence people's judgments about the world. So how they influence their color preferences. And then in my more recent work, how these associations influence people's interpretations of colors in information visualization.
00:56:53
Speaker
But in all of this work so far, I've been really thinking of color concept associations as just the input. And it exists. It comes from experiences in the world. And that's the input. And then all the interesting things happens after that with these kinds of inferences we do based on these associations. But now I've been trying to understand where these color concept associations really come from. What kind of experiences in the world matter more or less? Is it the early ones or the later ones? And now this idea of this top-down categorization that might influence
00:57:21
Speaker
the associations we form, I think is really exciting directions that we're going in. And that's from a kind of theoretical perspective. And then from a practical perspective, really working towards being able to integrate semantics into something like color work, where once we know these associations and we can understand them,
00:57:39
Speaker
then you can just query particular concepts and it can automatically generate graphs or maps or other kinds of designs that are easy for people to interpret. So I think the ultimate goal, as I mentioned at the beginning, is learning how to make visual communication more efficient and effective. And so tackling some of these practical aspects of it and then the underlying theoretical work we need to do to really understand how to do that.
00:58:02
Speaker
One of the really interesting areas here is the idea that people that work on design have maybe thought of the meaning of colors before, but just not in a systematic data-driven kind of way. So any art book will say red means this, or green means this, or sort of talk about feelings associated with different colors, and talk about it in a sort of loose sense. But this really, it gets a
00:58:29
Speaker
better handle on what's actually going on and maybe even points to some surprising results that you wouldn't necessarily know before looking at it at this level of detail. Yeah, exactly. And I think that it's just the way of talking. You hit it right on the head. The idea of red means this and green means this is really problematic. And this is what you see in a lot of like popular pop websites on quote color psychology or
00:58:54
Speaker
or in design and also even in some of the psychological research, if you treat color as these color categories. So there's red and there's green and there's blue and that's it. But the particular red really matters. The particular blue really matters. And it depends on, again, like these experiences in the world.
00:59:15
Speaker
But also more abstract associations and we don't really know where those come from. So like color emotion associations, we still don't completely know. We don't know where they come from or why we have them. And part of the problem is that people would say things like, Oh, well, yellow is happy and blue is that. And the work that we've done in my lab, we actually controlled for luminance and we controlled for chroma. So how colorful the colors are. And at that point when the colors are light, especially yellow is no happier than blue. Hmm.
00:59:45
Speaker
If you're going to try to understand where these things come from, but you're describing the phenomenon in a way that's not quite right, then we're trying to answer the wrong questions. That's super interesting. From a design perspective as thinking about building experiences in technology that people can interact with and use easily, I think getting these things right is super important. One of the things that comes up a lot in design of technology especially is
01:00:15
Speaker
the idea of a certain language that develops within a community. For example, on the web, there are buttons take certain colors, and this is probably not necessarily based on anything inherent to the color itself or even a person's perception of the color, but it's somehow something that's built up over time just by people developing this as a community, this language of associations.
01:00:42
Speaker
I think it's super interesting though to think about how you can manipulate that or work with that and actually either optimize for that or work around that based on people's other associations outside of that little world that you're working in from a design perspective. Yeah, for sure. Especially if you're introducing new naming conventions or new systems, how can you do that in a way that builds on people's prior associations maybe, but also can communicate this new information?
01:01:11
Speaker
I think this raises another interesting question that we haven't really started studying in the lab, but I want to, which is how the use of color and design can actually hinder communication. So so far we've been focusing on. OK, if you get these associations right, it can make it easier to interpret. But if you look at like the slides that I use in class when I'm teaching or when I give talks, I don't like it. Certainly I do very little color coding of text. I use color to communicate information about colors or or say color photographs,
01:01:41
Speaker
Once people get excited about this idea of color coding, they can kind of maybe take it too far and then it becomes overwhelming and too many colors and too much input. And so I think this question of how we could use colors to help maybe where using colors or visual features for coding might hinder, I think is a really important thing to understand.
01:02:05
Speaker
All right, so one last and important question, I think. So we often talk about future dystopian possibilities. And I guess the relevant bit here is if you were to design heaven and hell, what color would they be? That is an outstanding question, Dr. Nelson.
01:02:33
Speaker
So the premise of my answer would be that, so people like colors that remind them of positive experiences and things that they like. And they just like colors that remind them of things they dislike. This comes from the ecological valence theory. And so if you were to design the colors of heaven,
01:02:51
Speaker
I think you would do it and how you would do it for an individual person. So you give people their own personal habit or their own personal. That's a good answer. Yeah. And you would have, because we're omniscient, we would know about all of their experiences in their entire lifetime. And in heaven, we would show them the colors that reminded them of all of their happy and wonderful memories. And in health, we would do the opposite. So I guess use colors that remind them of all the negative things.
01:03:20
Speaker
And if we look at average data that even for the most part work out cross culturally we would use the colors of biological wastes and rotting fruits so dark yellows and browns for hell if that would remind people of the negative things but if they were people who
01:03:39
Speaker
worked on a farm and with manure and that was the source of life and nutrients that would actually be their heaven color. So I think really focusing on the individual and their own previous experiences would be central. So you could form a general idea of what it would look like for most people, but really, if you want to get at the best hell or the best heaven, you've got to tailor it to the person. Yes, I think that's right.
01:04:06
Speaker
Karen, thank you so much for being on the show. Really enjoyed the conversation and look forward to hearing more about your research in the future. Thanks so much. It was great to talk with you both.