Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Episode #173: Niklas Elmqvist
185 Plays5 years ago
Niklas Elmqvist is a full professor in the iSchool at University of Maryland, College Park where he directs the Human-Computer Interaction Laboratory (HCIL). His research area is information visualization, human-computer interaction, and visual analytics. He was elevated to the rank...
The post Episode #173: Niklas Elmqvist appeared first on PolicyViz.
Recommended
Transcript
Introduction and Welcome
00:00:11
Speaker
Welcome back to the Policy Vis podcast. I'm your host, John Schwabisch. I hope you and your friends and your family are all well, healthy and safe in these strange times. And I appreciate you coming back and listening in to the podcast.
Introduction of Nicholas Elmfist
00:00:23
Speaker
And on this week's episode, I'm very excited to have Nicholas Elmfist, who's a full professor at the College of Information Studies at the University of Maryland and College Park, Maryland, which is just a short drive away from my home here in Northern Virginia.
00:00:35
Speaker
I reached out to Nicholas because he does some really great work at the University of Maryland. And in particular, we talk about his work on accessibility, which as you probably know, has been a subject of a couple of different interviews and episodes over the last few months. And we also talk about his work on animation and data visualization.
00:00:54
Speaker
So we sit down and talk about all these different issues and it's a great conversation and I hope you will enjoy it.
John's New Book Announcement
00:01:00
Speaker
Before I get to that, I just wanted to let you know that my new book, Better Data Visualizations is finally out for pre-order on Amazon. The book is hopefully going to come out this fall. It's all done.
00:01:12
Speaker
We're in the last stages of finishing the proofs and getting everything laid out. So I'm very excited for this book. It's a book that I had put off writing for many years and finally sat down and was able to pull this thing together. And I'm really excited about what I was able to put into it and the ground I was able to cover in just one book. So I'm very excited for that. If you're interested, please do head over to Amazon and take a look at it and maybe submit your pre-order. I put the link in the show notes.
00:01:42
Speaker
So again, I hope everyone's well in these strange times of the COVID pandemic and I hope you are staying safe and staying healthy and staying inside and taking care of yourselves and friends and neighbors and loved ones.
Nicholas' Background and Interests
00:01:55
Speaker
So on to this week's interview with Nicholas Elmfist. I hope you will enjoy it and here we go.
00:02:05
Speaker
Hi, Nicholas. How are you? How are you doing? Pretty good. How are you? I'm doing fine. I'm doing fine. A little bit of a rainy day today, but that's okay. Get to just look out into the world. As I sit here working, just get to look beyond my computer into the world. Well, thanks for taking time out of your day and coming on the show. There's a couple strands of your research that I'm interested in focusing on and chatting about. And so maybe for folks who aren't familiar with your work, you can talk about
00:02:30
Speaker
your background and the lab over there at the University of Maryland. And then I can pester you with questions. Sure. So I'm Nicholas Amprus. I'm a professor at the University of Maryland in the iSchool Information Studies, Informatics, and also the director of the Human Computer Interaction Laboratory, the HCL, which is actually the oldest HCI lab, I'm told, in North America. We were funded in 1983 by Vensho Neiderman.
00:02:58
Speaker
And I'm the sixth or seventh director, depending on how you count. And just like Ben, who founded the lab, I'm also a data visualization researcher. And I teach also on data visualization, data science, and actually also a little bit of game design recently. I'm a gamer as well.
00:03:18
Speaker
Oh, interesting. Interesting. Okay. So we can, I didn't know that. So we can talk about that a little bit too. Um, do you want to talk a little bit about your, um, maybe the history of your, of your research and how the threads of sort of woven back and forth and what you're working on now?
Research on Accessibility in Data Viz
00:03:33
Speaker
Well, I am for lack of a better word, uh, data visualization generalist, which means I think I work across many different things. It might mean I just have a hard time focusing on one thing.
00:03:45
Speaker
Another way to put it is I'm a full stack data visualization person. Maybe that's nicer. But that means my work ranges from low level graphical perception stuff, which I'll talk a little bit about today, all the way to devices and technologies, hardware for data visualization, even straying into pure human computer interaction research. I've done work on pointing and selections and even
00:04:14
Speaker
electronic readers and all kinds of things for those fields. So it's really wide. It's an interesting mix. I guess I have a hard time nailing down exactly what I'm passionate about. I'm passionate about everything.
00:04:28
Speaker
Well, there are two particular strands of research that I'm interested in. One is your work on accessibility in data visualization and the other is on your work on animation. Accessibility has been a topic on the show the last few weeks. So I think that would be nice to continue the discussion. The last few folks I've spoken with are more on the practitioner side of things. So I'm curious about your research on accessibility and then we can talk about animation. Maybe just give us a quick roundup of your accessibility work.
00:04:57
Speaker
So it's interesting that this is happening because it seems like a lot of things are coming together. I've been listening to those recent episodes where there's been accessibility in this podcast and I think it's exciting. And it's curious how it happens because in our case, obviously, these are well-documented situations and a lot of compelling research questions in how do we make data universally accessible.
Innovative Methods for Accessibility
00:05:20
Speaker
But in my case, it was a bit of a journey of discovery because
00:05:24
Speaker
about a year and a half ago, I was told that a student from University of Maryland was enrolling in my data visualization class. And this student was basically legally blind. It was very jarring to us because we had not had this situation before. But of course, it may seem like a paradox. But of course, thinking about it, even a blind person
00:05:48
Speaker
deals with physical space. They have to navigate 3D worlds. They're very familiar with shapes, just as we are. So it shouldn't be a surprise. But much of Dataviz has basically neglected to focus on this population. In our case, we came up with a very low-tech solution to make it possible for the student to take the course. Because, of course, he's not learning just how to read Dataviz, but also
00:06:14
Speaker
how to create them if you're taking a course like mine. So we had to come up with a solution with just a metal board and magnets, where the assistant would arrange those magnet on the board so that match was on the slides, and he could explain and even create his own visualizations. Well, interesting. And then this has led to a big new research agenda in my lab, where that two students who are working on various things from physicalization, turning
00:06:42
Speaker
data into physical forms so that you can feel them in your hands and fingertips or even your body, but also more traditional things like sound using audio and even some recent work where we're using smell.
00:06:57
Speaker
Oh, interesting. I love the fact, by the way, that this research track was inspired at least by a real life example or real life challenge that you had to overcome. I mean, I think that's like a great way that research starts and just being able to help a student better understand the topic. But can you talk about this research and
00:07:18
Speaker
Part of the thing that I find interesting about this discussion of accessibility is the difference between when we make static graphs where most people say, well, let's use different types of colors. But the big thing is alt text. Let's add that alt text. And then the interactive side of things of data visualization where I'm not sure people have thought through a lot of the issues. So I'm curious about that. Absolutely. I think there are many things that what you touched upon that is relevant to what we do.
00:07:47
Speaker
Overall, the interesting thing, and I know it's been noted before, when we do improvements to help this population, and of course, this is not a small population, it's important to think of this, because there's something like 300 million individuals that have visual impairments in the world. And 40 million of them are totally blind. And here in the US, it's somewhere between seven and 15. We've been working with the National Federation for the Blind in Baltimore. And it's exciting to see, you know, many of these
00:08:17
Speaker
improvements like audiobooks came about because blind people wanted to read too. But of course, now millions of people who are not blind use them all the time. And that's this curb cut effect that I heard discussed before, where improvements for universal accessibility will help people who are not necessarily in a wheelchair, but are temporarily under disadvantage.
Challenges and Solutions in Alt Text for Data
00:08:43
Speaker
Right. And that's true here, too. I mean, you mentioned all text. And all texts are, are, are great. But the problem is, a lot of the visualizations on the web, if you Google images, for a chart search, for example, you'll get lots and lots of images that don't have all text. They are just pixel maps with charts, like bar charts, or pie charts, or something where the pixels themselves don't carry the data.
00:09:09
Speaker
And a screen reader will not know what to do with those because there's no alt text. And the screen reader can't just turn the pixels into natural language, of course. So in a project that we did about two years ago, we used machine learning to translate these images into shapes so that we could then recover the data from. So at the very least, we were able to replace an image of a bar chart or pie chart or line chart or something.
00:09:39
Speaker
into the data table where it was created, where that type of data is usually not available. So that's a first step. But of course, there should be more ways to do this. You mentioned interaction. In one of the projects that a student of mine is working on right now, he is working on building a little robot, a three wheeled robot, looks like a triangle slice of your palm. And it has a handle that you could put your hand hand on and that handle can turn and it can vibrate.
00:10:09
Speaker
And the point of this device is that you could put it down. It's wireless. You put it down on a surface, a flat surface, and you connect using a Bluetooth or Wi-Fi connection to your phone or your laptop. And then you activate the device and you grab it and it will start moving to describe a shape. And you can feel, for example, I don't know, the stock market value of Google over time or the temperature around the world changing with seasonality or something.
00:10:38
Speaker
And of course, since these engines on this robot are back drivable, we can also use it for interaction. So you can basically use that thing to explore a space as well. I mean, obviously there's a long road ahead here, but do those sorts of devices, do you see them working differently in a mobile environment than in a desktop environment?
Mobile Accessibility Devices
00:11:01
Speaker
Yeah, so that's absolutely true.
00:11:04
Speaker
Many of the accessibility devices out there are not necessarily mobile. A braille, a refreshable braille display, which can be really costly that turn text into braille characters. There are mobile versions of those, but they're small. They only maybe show 10 characters versus bigger ones. So that's certainly something that you have to keep in mind.
00:11:28
Speaker
this particular robot that we're working on, we're trying to make it mobile so that of course you can have it in your pocket and put it down on any purpose. The thing that has happened that we've also been looking into is beyond that is screen readers with smartphones being so ubiquitous where they're everywhere has really revolutionized information access for blind people where they would use screen readers on their smartphone all the time.
00:11:55
Speaker
Right. Cause we're on it all the time. What about, have you thought about or done any work on other types of impairments? So I think, I think, I mean, I think we would probably both agree that, that in the data viz or info viz community, the first thing we think about, I would say most people probably think about our vision.
00:12:15
Speaker
impairments or difficulties, right? But there are all other sorts of impairments or limit people's ability to access information.
Research on Sensory Substitution Techniques
00:12:23
Speaker
Absolutely. So if you thought about any, I mean, I don't know if you've thought about these or started any research on these, but I am curious to hear how, you know, the sorts of things that you may be thinking about. Yeah, I mean, it's true.
00:12:39
Speaker
that for vision, it happens to be a particularly, I don't know, big elephant in the room for us database folks, because we realize so much on the magic aspects of vision. And we're seeing its praises all the time. But we don't recognize that not everyone has full use of their vision. In terms of other types of disabilities or impairments, we haven't
00:13:06
Speaker
are in my group, we haven't looked at any of those beyond beyond vision, except, you know, like I said, some of these research approaches are based on sensory substitution, which means, of course, you replace vision with another sense. And that type of philosophy, of course, can be applied. Let's say if you're deaf and you don't have use of your ears, you could use the same general approach of sensory substitution.
00:13:35
Speaker
And one of those that might be relevant is this, I mentioned early on, this use of smell, which sounds a little like a joke because, of course, computer interfaces don't typically smell. It's very an underused sensory channel. But again, here's the curb cut effect because there are situations where
00:13:58
Speaker
you cannot see, you know, you're blind perhaps, but you cannot, there are other situations where you cannot look. So let's say you're a fully sighted person, you're driving your car, you can't spend time looking at the screen. And then using sound or using, let's say, smell in this case, could be useful. And I'm sure that could also be applied, like I said, to a deaf person or potentially someone with cognitive impairments, because smell is such a
00:14:26
Speaker
primal type of sense. It's tied so strongly into memory. And the project that we did, we have built three prototypes of all factory displays. That's a technical name for a smell interface, essentially. Just like a screen is a visual display, it generates colored pixels, an all factory display generates smells. So we built several prototypes of these mobile ones, as well as tabletop ones.
00:14:53
Speaker
in the most recent one is a big device that has 24 bottles of essential oils and little ultrasonic diffusers that we can turn on and off. And that just like a humidifier at home, we can generate and mix and blend smells and then send them into that basically the nostrils of the user. We haven't used them that much as a replacement, but more as a compliment. But there are certainly situations where you could try to
00:15:20
Speaker
replace the vision instead of complementing it.
00:15:26
Speaker
Right. Wow. That is amazing.
Animation and Gestalt Principles in Data Viz
00:15:30
Speaker
So let's see if I can do this cool segue here. So we've talked about accessibility with static visualizations and interactive visualizations. But somewhere in the middle, somewhere are animated visualizations. And you've done some interesting work on animation. And so maybe that wasn't a great segue. I don't know. But anyway.
00:15:55
Speaker
You've done some interesting work on animation and in the paper, at least the paper that I've read and will link to on the show notes. What I like about the framing of that paper is you bring it all the way back to Gestalt principles.
00:16:10
Speaker
which I don't know if a lot of people in the database field think of it that way, right? We think of maybe, you know, preattentive principles or principles when it comes to static things that color or things are grouped together. But there are these principles about animation and motion. And so I like the way that that framing of that paper. So I was hoping you could sort of explain that framing and then talk about the actual research. Yeah, it was interesting to do a little bit of a journey
00:16:40
Speaker
back in time when we wrote this paper because graphical perception in general, I mentioned earlier, is this interesting intersection of vision science and perceptual psychology on the one hand and data visualization on the other hand, where we're trying to figure out how can we build visualizations that match how our vision system works and how can we figure out how to make it better.
00:17:04
Speaker
And a lot of the seminal work, as you know, in graphical perception is relatively recent. I mean, it's people like Jett Burton, cartographer, I mean, some time ago, but still not that long ago. And then Bill Cleveland and McGill did work on graphical perception and for statistical visualizations. But if you rewind a little further, you'll see that a lot of the work that was done on early work on visual perception was in the early 1900s.
00:17:32
Speaker
And it was called the Berlin School in Germany of Experimental Psychology. And they eventually came up with this notion of gestalt psychology, which is a theory of mind talking about how the whole is bigger than the parts and how that works in terms of the things we see. And essentially they came up with maybe something like five or six so-called gestalt laws that say how
00:18:01
Speaker
we humans group elements we see in our field of view into whole components. So things like proximity. When we have several things close to each other, we tend to think of them as a group. And that's commonly seen, you know, you have a scatter plot of dots. And if they're grouped together, you think of them, oh, those are the certain cluster of behavior. And then you have things like similarity, where two objects
00:18:30
Speaker
that have similar visual appearance, same color or same shape, you tend to group them together. And then there's additional ones. The thing that we were interested in this particular study that you mentioned was the law of common fate, which is the only gestalt psychology law that deals with things changing over time. So basically it says if several elements
00:18:58
Speaker
are behaving in a similar way, we humans tend to think of them as having the same or common fate, and then we group them together. Commonly, we use this idea for animation. So elements that move in the exact direction and speed tend to be grouped. But what we looked at in this particular study was whether this idea of changing together
00:19:25
Speaker
applied not just to animation or at least not not just to motion elements moving in the same direction and speed, but also to dynamic behavior like whether they grew together or shrunk together.
00:19:38
Speaker
or whether they changed color together. So as we are looking at an animated bar chart, for example, let's take the bar chart race. Right? Yeah. Yeah. So so as we view those, we see them moving together as a singular group. Yeah. Yeah. So so well, they have to move together in the same rate. So if you have a big
00:20:01
Speaker
a bar chart, and you have two bars that even if they're not next to each other, if they have the same behavior, so they grow about the same clip, then that the notion is that that you think of as as two elements at the same time. Another example is, sorry, one group that moved together as one. Another example is Hans Rosling's animated scatter plots or gapminder, where, you know, he talks about demographics, how countries move and increase in economic status.
00:20:31
Speaker
And if you squint, even if those types of visualizations are really confusing, you know, hundreds of dots moving together, you tend to see trends where a cluster of these countries, even if they're a little scattered, happen to have the same economic growth, they tend to stick together and become more of a unit in your mind's eye. Right.
00:20:56
Speaker
So let me ask you this question, because there are probably people who are listening to this discussion and thinking, oh, that's interesting. And I can see how that's true when I look at the Rosling bubble plots move, how this cluster, I kind of see this cluster of dots moving. But I wonder if people are also thinking,
00:21:18
Speaker
What does this mean for me as a creator of visualizations? So if someone were to say to you, OK, I'm a I'm an interactive data is developer, you know, how do I take the findings from your paper and apply it to my work? Yeah, it's a great question because our study is relatively basic. So there's clearly some explanation needed to say, how do you apply it in practice? I think the basic finding from our study
00:21:48
Speaker
was confirming something that I think many of us already knew. Sometimes that's, you know, research becomes. And in this case, it is that animation is extremely powerful. When we had elements moving in our study, that overshadowed everything. So elements that move together are much more tightly visually grouped than elements that change size or change color, or even are close together in space.
00:22:19
Speaker
So animation is maybe the strongest visual cue you can use in a data visualization, which means that you need to use it with caution. I know that's what Uncle Ben says, with great power comes great responsibility. So be careful with your data visualizations whenever you use animation, because it's going to grab people's eye. That's the clear thing.
00:22:43
Speaker
Yeah, no, that's really important. And just as a parenthetical, when you said that's what Uncle Ben says, my actual first thought was not Spider-Man. My first thought was that's what you guys call Ben Schneiderman, but maybe that's a...
00:22:58
Speaker
That's true. OK, so this is great. So because I think when I talk to students about static visualizations and ask them to identify the things that draw their attention, the big one that stands out is color. But when we move from a static visualization to an interactive or animated world, we may need to be thinking about how the motion may supersede some of these
00:23:25
Speaker
other characteristics that in a static world are the things that pop out to us. Yes, absolutely. The other finding that we uncovered was that we were able to confirm that the original formulation of this law of common fate, where they basically said in a much more generous way that anything that changes together will be perceived as having a common fate. Whereas in data visualization practice,
00:23:55
Speaker
Many of us have taken this to mean that things are animated and moved together. That's when the law of common fate applies. But our study was able to show that it's actually not just animation when things move together in the same speed and direction, but also when they change color together or when they change size together. So the grouping strength was stronger, not as strong as moving together, but still strong enough compared to all of the other grouping variables.
00:24:25
Speaker
So that means that you could, for example, if you wanted to use a data visualization and use animation to some degree, but not as strongly as having elements move together, you could have them change color together or change size together. That would be another cue that you could use. So we're adding to the arsenal of data visualization designers.
00:24:46
Speaker
Right. Right. Great.
Upcoming Symposium Announcement
00:24:48
Speaker
So one last thing is on the Maryland human computer interaction lab. You have a symposium coming up and I wanted to give you a couple of minutes to talk about that and where people can find speakers and more information about it. I've attended in the past and it's, and it's a great, it's a really great event. Yes. We've done this for 36 years. So it's been since the beginning of the lab. So we're longer running than many other data visualization conferences in the field.
00:25:15
Speaker
So yes, our annual symposium is happening at the end of May. Right now, we're trying to figure out exactly how to proceed with it. It's probably going to be pre-recorded talks with a bunch of popular science style blog posts about the research. All of the research is going to be about work that has happened in the last year in the research lab. A lot of it is students presenting. I'll be giving a keynote speaking about
00:25:45
Speaker
data visualization for the blind. So a lot of this you heard today will be expanded upon and explained. But there will be also interesting other talks given by my fellow faculty members and students. So it's a full day thing. Usually, as I said, we meet in person. Of course, now we are rethinking and reorganizing this. But still, I encourage you to keep an eye out. All of the information is available on the HCL website. So that will be hcl.umd.edu.
00:26:15
Speaker
That's great. Yeah, I will post a link to that so people can take
Conclusion and Farewell
00:26:19
Speaker
a look. It is a great event and with pre-recorded lectures and blog posts, maybe there can be a different type of communication between the speakers and the audience. So that's great.
00:26:31
Speaker
Well, Nicholas, thanks for coming on the podcast. It's been great chatting with you. I love this work and I'll look forward to, at the very least, watching your keynote address because I'm interested to see what other thoughts and more research you guys are doing. So thanks for coming on the show. It's been great. Absolutely. Thank you.
00:26:51
Speaker
And thanks for everyone for tuning into this week's episode. I hope you enjoyed it. I hope you learned a little bit about how to make your data visualization accessible and how to think about animating your data visualizations or at least reading animated data visualizations when you see them out there in the wild. So again, stay safe, stay healthy. And until next time, this has been the PolicyViz podcast. Thanks so much for listening.