Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Episode 37: Dr. Joshua Miele: The Blind Arduino Project
30 Plays3 years ago
Joshua Miele is a blind scientist and inventor living in the Bay Area. Amongst his many initiatives is the Blind Arduino Project, where participants learn how to navigate a world of maker electronics that are really designed for the sighted. He talks to Joe and Rolf about his work to bring accessible and useful tools to the blind community. He teaches blind soldering classes, and he has worked to make braille maps easily accessible. He discusses what technology in recent years has helped the blind, and what hasn't.
Blind Arduino Blog: http://blarbl.blogspot.com/
Josh's Handle: @BerkeleyBlink
Mailing list for the Bay Area blind arduino monthly meeting: babamm@groups.io
Special Guest: Joshua Miele.
Recommended
Transcript
Introduction to Dr. Josh Mealy
00:00:09
Speaker
Welcome to Cog Nation. I'm Joe Hardy. And I'm Rolf Nelson. On today's episode, joining us is Dr. Josh Mealy, who's a blind inventor, the community leader, and a cartographer who's worked with tactile representations for the blind. Great to have you with us today, Josh. Thanks for being with us. Thanks, Rolf. It's a pleasure. Always happy to join you guys. This is one of my favorite podcasts.
00:00:32
Speaker
Wonderful to hear.
Academic and Professional Background
00:00:34
Speaker
So Josh, you got your PhD at Berkeley, just like Joe and I. So this is how we know each other. Fortuitous, yes. To get that out of the way. And since getting your PhD, you've worked at a couple of different places and done all kinds of interesting projects. Do you want to talk a little bit about some of the things you've been involved in?
00:00:53
Speaker
Sure i'll give a quick quick overview, I mean I and when you say a couple places you're kind of exactly right i've. I left after grad school I got a postdoc at the Smith kettle well I research institute in San Francisco, which is a vision research research nonprofit sort of I call it a para academic.
00:01:12
Speaker
institution. It's sort of a think tankish place where you don't have any students but everybody is sort of publishing and getting grants and and there are postdocs and stuff like that. So I had it I started with a postdoc there and over the next 20 years basically went through a junior scientist and you know scientist and director of you know I had a bunch of projects and was a principal investigator there for for many years until I
00:01:41
Speaker
moved to, made the jump to industry and I am currently a principal researcher at Amazon Lab 126, the devices group at Amazon that does the, you know, the tablets and the fire TVs and the echoes and stuff like that. So, but I'm not here, you know, today I'm probably not going to talk too much about Amazon stuff because I didn't actually get their permission to, you know, to talk about stuff. So, but there's plenty of other stuff to talk about.
Commitment to Accessibility in Technology
00:02:09
Speaker
As you mentioned, I'm blind. I consider blindness to be a significant part of my identity. So a blind dad, a blind scientist, a blind inventor, et cetera. And my work is totally directed at developing cool stuff for blind people that makes our lives easier, better, more fun, more equitable, et cetera. You've mentioned maps.
00:02:37
Speaker
Lots of work on accessible maps, both auditory and tactile. I've done a lot of work on accessible video description technologies and techniques for making video more accessible to blind people. And there's other wayfinding stuff like GPS, mobile tools in addition to the tactile and auditory maps. And the other thing that is a significant part of what I've been working on is
00:03:06
Speaker
what I call the blind Arduino project, which is, I don't know how to, I don't know how to characterize it, but it's like, it's basically a project to work on getting blind and visually impaired people involved in building stuff with microprocessors, both to just for fun and to build confidence and also to build the tools that blind people often need that you simply can't buy off the shelf. So, so there's, there's, there's,
00:03:36
Speaker
That's kind of a quick overview of the kinds of things I find interesting and work on pretty regularly. No. Okay.
The Blind Arduino Project
00:03:45
Speaker
I'm interested because Joe and I are both, you know, we can, we can geek out on Arduinos and love tinkering around you. You luckily get to put the, put inventor in your title so you can call yourself an inventor, which is something I'd always dreamed of. So how can you say anything more about how the blind Arduino project works? Cause that sounds really fascinating.
00:04:06
Speaker
Yeah, I mean, there are two main components to it. I mean, one is a lot of the resources that are out there that encourage people to become makers and to sort of get into STEM through electronics making, a lot of those resources are very visual. You go on a website and you'll see there's a little write up and then there's a JPEG of a circuit diagram and it's like, okay, that's a blocker right there.
00:04:34
Speaker
Same thing with just finding your way around on an Arduino board. If you want to know where pin 2 is, most of the time that's communicated through a picture. So a lot of the one significant portion of the project is really devoted to developing
00:04:57
Speaker
accessible documentation and sort of helping encourage other people to think about the accessibility of their documentation. It's not just, you know, just the documentation that needs to be accessible, the Arduino
00:05:10
Speaker
IDE on Windows, the IDE for those of you non-nerds out there, that's the... We don't have any non-nerd listeners. Maybe Arduino is even something that we should introduce. Arduino is a microprocessor platform. It's open source. It basically is a little
00:05:31
Speaker
a circuit board that has a computer on it and lots of places to plug wires in and depending on what those wires are attached to whether they're buttons or sensors or switches or displays or motors you can tell the computer to do various things so you know you can tell it to run a motor when you push a button or you can tell it to turn on a fan when the when the temperature sensor reaches a particular
00:05:58
Speaker
level, or you can tell it to turn on the motors that are attached to wheels and scoot to the left when it sees something in the camera field of view on the right. So all kinds of robotics-y things that you can do with an Arduino, and they're really cheap, too. So they're cheap and easy to play with, and there's lots of documentation, and people are using them all over the world to teach kids about science and engineering.
00:06:24
Speaker
As often happens, blind kids are really underrepresented in those fields because a lot of the time, you know, most of the time sighted people think, oh, well, how is a blind kid going to do this? There's no way. You know, these wires are all colored. These numbers are very small. I can barely see them on the circuit board. How is a blind kid going to do it?
00:06:43
Speaker
And of course, those are sort of false barriers there's their alternatives to colors of wires there, as I've been mentioning, you know the fact that the numbers written on the circuit boards like you just need to you know so what I have you know descriptions of Arduino boards that tell you.
00:07:01
Speaker
based on the tactile landmarks on the board, what the socket numbers are and stuff like that. So that's the Arduino and then the IDE is this integrated development environment, which is like the word processor on your computer that you use to write the code that you load onto the Arduino that is the program that it runs. And the IDE on Windows is not accessible. You can't use it with a screen reader. So not only do we need to tell people sort of where the,
00:07:31
Speaker
You know how to find their way around on the hardware, we have to tell people how to do these tips and tricks about like using the command line environment to debug and upload their your Arduino programs called sketches so um so it's a it's a whole.
00:07:47
Speaker
world of documentation that needs like it's not just about making the regular documentation accessible it's about sort of documenting the work arounds and the the back doors that blind people need to know about in order to do the stuff at all.
00:08:03
Speaker
And then the other, the other part of the blind Arduino project, you know, so part of it is just like hey let's let's get more blind people into stand let's get more blind people into doing the things that you know that they want to do if there's a blind kid that wants to be a scientist I want them to have the same
00:08:20
Speaker
experience the same, you know, sixth grade robotics team experiences as their sighted friends. And if
Workshops and Technical Independence
00:08:26
Speaker
they decide they don't want to be a scientist, I want it to be because they don't want to do it, not because they didn't have the right opportunity. So thinking back on that, of course, you've faced lots of hurdles in every area, but are there particular things that you think of that you would like people not to go through today?
00:08:44
Speaker
Well, I mean, yeah, I mean, of course, it's it's sort of autobiographical. I mean, I was lucky and I had really supportive parents and teachers and was given like a lot of opportunities to succeed. I was always really interested in science and math and I was always supported by the grown ups around me. And but I did find myself in lots of classes, you know, where my parents were like,
00:09:09
Speaker
damn right you can take that, you know, electronics class, you know, but I would get in there and the teacher would be like, I don't really know how to help you with this, you know, and so like, so part of my hope is part of what we're doing is to try to raise, you know, provide resources and, and communicate to teachers so they know about this stuff and, and I also
00:09:34
Speaker
I don't know, Heathkit, I'm a little older than you guys, Heathkit may have been before. I remember that the electronics assembly things, you'd get the components and you'd solder them together. Exactly. And I always wanted to do those myself, but the documentation was always in print. It was always like the printed circuit boards, there were no tactile markings on them. And so I would buy them and basically force my mom
00:10:05
Speaker
to like read all that stuff to me. And you know, it helped me in like, you know, I, you know, my mom had, you know,
00:10:14
Speaker
I learned how to solder as an adult, but as a kid, even my mom had some limits and didn't want me to solder. So she learned to solder so that she could build these things with me. But blind people can solder. And if she had known that, she would have let me solder. She just didn't know. And part of what I do now as part of the blind Arduino thing is I teach blind soldering workshops, or at least I
00:10:42
Speaker
used to before the pandemic. And so, you know, that's, and you know, that's a real, that's a real, you know, a blind, a blind person with a soldering iron, that's like a blind person with a chainsaw, makes people nervous, you know, but it really says something, right? It's really, it's really the fact is that like, if you know what you're doing, you know, you can you can cut your own firewood, you can
00:11:09
Speaker
You can solder your own projects. You can barbecue your own steaks. Oh, this was not advocating for no soldering. This is advocating for more chainsaws. I didn't mention the blind chainsaw project, but no, I'm just saying. I mean, the tools are, there are lots of dangerous tools out there in the world, but if you know what you're doing and if you're careful, really, you don't need to be able to see.
00:11:39
Speaker
need to have appropriate safety precautions so that you don't hurt yourself or someone else. And that's the same thing is true of a sighted person. Sighted people often don't realize that there's an enormous amount of information
00:11:55
Speaker
in the world available that's not visual, but because you guys are vision scientists. I mean, you've claimed 30% of the cortical real estate, right? And it's pretty tough to ignore visual
Challenges and Stereotypes in Accessibility
00:12:11
Speaker
input. I mean, when visual input dominates. And so sighted people often think
00:12:17
Speaker
that because they use vision to do something, that's the only way to do it. But it's definitely not true. And I'm not saying that there's nothing that a blind person can't do. I think they're, I'm not necessarily calling for blind lifeguards or blind fighter pilots, but there's plenty of stuff that blind people can do that really is just a matter of technique. And then that other stuff like fighter pilots,
00:12:45
Speaker
you know, if you had a cockpit that had all the right accessible instrumentation, then I'd say, let's go. But the whole thing about, you know, soldering is that it really speaks to people. It's like once when blind and sighted people realize, oh my God, blind people can solder, like it's not a thing that they can't do. It's kind of transformative in the way people think about
00:13:13
Speaker
what they can do. So it's not just like this STEM thing. It's sort of a self-actualization kind of self-expansion kind of thing. And for sighted people, when they realized that they're blind people that saw that they're like, wow, maybe there's a whole bunch of other stuff that blind people can do that I didn't realize about. So it's a multi-dimensional project with lots of social and technical aspects to it.
00:13:42
Speaker
The other thing that the blind Arduino project is trying to do is there are no like you can't just go to Amazon and buy a talking multimeter or that was one of the things that I was thinking of too that's and that's that seems like an electronics equipment that just comes designed for sighted people.
00:14:00
Speaker
I mean you can they have built them over the years like you can buy one on eBay if you're lucky because they have existed at various times but nobody's manufacturing one right now and and that's like just the basics right you definitely can't buy an accessible oscilloscope anywhere so a big part of the blind Arduino project is like let's let's design
00:14:25
Speaker
let's design accessible test equipment so that blind makers actually have access to the things, the tools they need so they can build their own tools to do the stuff they want to do. So that's sort of the first part of the project is educational
Designing Affordable Braille Tech
00:14:41
Speaker
and sort of consciousness raising and on a bunch of different levels. And the second part is just very practical. Give people designs and help them design
00:14:52
Speaker
and build the tools they need, whether it's whether it's an oscilloscope or a GPS thing that does something that nobody else is doing or any kind of tool that you might need that you can't buy off the shelf because you're, you know, because you're a population of one. I imagine I imagine you have a collection of some pretty great hacked together stuff.
00:15:16
Speaker
It's actually surprisingly not true. I have a bunch of very basic tools that I keep, like a multimeter and a continuity tester. I'm in the midst of designing and building an oscilloscope, which I'm really excited about. But I also, I mean, there are a lot of just sort of basic things that I do, like we use
00:15:40
Speaker
You know, a lot of the time for sighted people they would just, you know, attach a, attach a visual display and and print stuff to it right whether it's graphics or text and there's no cheap way to do that for Braille Braille refreshable Braille is really expensive and hard to get.
00:15:59
Speaker
And part of what we want to do is keep stuff really cheap. So one of the things that I've developed and sort of published a little bit about is this thing I call the servo meter, which is using a servo as a tactile gauge, as a quantitative output. A servo for anybody who doesn't know is a motor, sort of like a windshield wiper motor where it can swing an arm
00:16:28
Speaker
180 degrees from one direction to another direction. But the cool thing about a servo is that it knows where it is. So you can tell it to go to a certain position, like, you know, 45 degrees. And then if you push on it, if you push it out of away from 45 degrees, it knows that it's being pushed and it pushes pushes back. So it's really great for tactile gauges and meters because they're really strong and, you know, and you can put tactile markings around the
00:16:57
Speaker
you know, around the perimeter, like a, like a, whatchamacallit, what's that protractor kind of thing. So that, you know, when it's pointing to 38 degrees, you can feel where the tackle markings are and know that that's you know, that that corresponds to 12.2 volts or whatever. So I use, I use servos a lot for output and, and also sonification use various, various auditory output.
00:17:24
Speaker
And you can use vibrating motors so that you can do Morse code output and stuff like that. Morse code is a little bit
Community and Innovation in Accessibility
00:17:33
Speaker
janky. Most people don't really know Morse code, and it's kind of a pain. Yeah.
00:17:39
Speaker
But yeah, man, that's the quick overview of that kind of stuff. That's really cool, man. I guess one question would be, you said that the pandemic is kind of hurting some of the getting together stuff, but maybe be curious to like, for people who are listening who may be interested in getting involved in that, if they're, say a few words about how someone might be able to get involved in this project. I do have a little mailing list.
00:18:09
Speaker
It's a local Bay Area list, but obviously since it's an email list, anybody can join. And this is a place where, you know, before the pandemic, we would sort of advertise our workshops and our meetups and our get togethers. And since the pandemic happened, I haven't really been too active on it, but it's something that I'm about to sort of spin up actively again. So that mailing list is called
00:18:40
Speaker
BABAMM, the Bay Area Blind Arduino, Bay Area Blind Arduino Monthly Meetup, BABAM, BABAMM at Groups.io. So BABAM with two M's, BABAMM at Groups.io. And that's just an open, it's a list anybody can join. And generally speaking, when there are interesting things going on,
00:19:09
Speaker
We'll publish them on that list. Also, I've got a fairly inactive blog, but by looking at the blog, people can definitely sort of see what the idea is and the blog is, it's blindarduino.blogspot.com.
00:19:29
Speaker
So yeah, so that's, and it's, you know, my, I'm really, I'm really tired of not doing this stuff because, you know, obviously it's really hands on, you know, but I'm getting to the point where if we can't start doing stuff
00:19:46
Speaker
in person pretty soon, then I'll, I have to figure out how to do this remotely. The remote part is hard because I always, to these meetups, I always bring all the hardware, right? Like I always bring all these Arduinos and cables and, and, you know, sensors and like, you know, give people a chance to
00:20:05
Speaker
to play with them. And then, you know, so for for a virtual thing, I'm gonna have to like send people a list of things to order and then people have to start shelling out money and it's just it gets it gets. So what are some interesting? Have you seen any interesting projects come out of it or any interesting ideas? Or do you think of it mostly as just a playground?
00:20:30
Speaker
It's mostly a playground. There are a couple of interesting ideas that have come out of it, but mostly it's the obvious stuff like the accessible oscilloscope, right? Or I ran a workshop a few years ago, we designed and built a multimodal multimeter where the multimeter had auditory, tactile, and speech output.
00:21:01
Speaker
One of the coolest things that I'm really interested in seeing if we can do is there's, you know, blind people, you know, we've got phones that are, you know, a blind person can use an iPhone or an Android and they can use a laptop, you know, with accessible software on it and stuff. But there's no equivalent to like a little black book that you can just
00:21:28
Speaker
grab and jot notes into and there's a whole sort of world of products out there for blind people called braille note takers and they have a sort of a braille keyboard on them and usually a refreshable braille or speech output and they're just they're really just for taking notes and they're really they're small and they're
00:21:50
Speaker
The first ones were really just for taking notes. And sort of as computers got more and more capable, people started putting more and more stuff into them until they became little computers in their own right. So this is something that would have, instead of any sort of monitor, it would just have raiseable Braille that could be adjusted. Or speech. Yeah, or text to speech. Or text to speech, too. OK. Yep.
00:22:20
Speaker
And so at this point, and because these little devices are so specialized, these braille note takers are so specialized, they're pretty expensive. They're sold, you know, they're thousands of dollars apiece. And they've, you know, they're based on Android and they've got, you know, they've got web browsers and calculators and databases and word processors. And it's just more complicated than it really needs to be. And a lot of
00:22:49
Speaker
cases. And so one of the things I'd really like to do is design a an open source Braille note taker that's like uses only off the shelf parts is really cheap easy to easy to make easy to modify and can be used as a simple Braille note taker by anybody in the world because you know for most you know I mean we're here in the US you know we have access to
00:23:19
Speaker
You know, if you're blind in the US, then if you're going to school, you've got access to resources to help you buy equipment. If
Advancements and Challenges in Accessibility Technology
00:23:27
Speaker
you are, you know, employed, you have resources there. But if you're just, if you're somebody who doesn't have a job and doesn't, you know, if you're not in the US or if you're in a developing country,
00:23:41
Speaker
getting hold of this kind of technology is really hard. So the idea of making something that's open source and easy to build from readily available parts is very appealing on a social level. So again, you never get far from the social justice angle of this. It's always because blind people are
00:24:08
Speaker
Let me put it this way. Most blind people don't have PhDs and work for Amazon. That's not the common path. And so most blind people live in poverty and don't have access to the kind of resources to get the tools they want or need. And yet,
00:24:35
Speaker
Blind people aren't stupid and they've got plenty of ability to think through and fix the problems they can. And so if there's a Braille note taker that's open source that people could just build or have a friend build for them, that would be a huge thing, really helpful to a lot of people all over the world.
00:24:58
Speaker
Thinking of technology and just technology in general, you know, when you were growing up, you didn't have a lot of, you know, a lot of the kinds of things that are available now. What's the difference between someone blind growing up now versus in the 1940s when you were a kid?
00:25:19
Speaker
Right. So I grew up in the 70s and 80s. And I mean, the the difference for blind people is the same as the difference for sighted people, I think, like, from a maker perspective, like you look at the kind of the kinds of things that you can now build with, you can build something in an afternoon with an Arduino for like 50 bucks that would have taken, you know, millions of dollars
00:25:44
Speaker
and a team of scientists in the eighties, right? It's just crazy what the technology off the shelf technology can do now. In terms of access to information for blind people, everything used to be on paper, but it's kind of one of these things where the level, a rising tide
00:26:11
Speaker
That's all boats, but the boats that are sinking before are still sinking after the tide rises, you know? So yeah, there's like, for blind people with the internet and with the technology we have, it's a lot easier to get access to some kinds of information, but there are still lots of types of information and lots of areas where accessibility to that kind of information or those, or capabilities is just like, doesn't, you know, just isn't possible
00:26:41
Speaker
And it's not because it's not possible. It's because the people that designed that particular experience weren't thinking of blind people when they made it. And that's changing too, right? In a lot of the, you know, we're now more conscious of accessibility and inclusion than we've ever been before. So things are getting better and there are now laws
00:27:07
Speaker
that require certain types of accessibility that there weren't before. So things are getting better, but it's still a pain in the ass to be blind for anybody. Yeah. Maybe you want to talk a little bit about some of the maps work that you've done and what that's like and how that supports people getting around in urban environments and
00:27:32
Speaker
Yeah sure yeah so I mean you know a lot of the time people think of maps as being you know maps or graphics as being really visual but they're not they're spatial right so there's spatial information is information that when it's the the content is based on the relative positions of different things in that
00:27:55
Speaker
in that representation
Tactile and Auditory Representations for the Blind
00:27:56
Speaker
so a map or you know a map is one kind of spatial representation or spatial data and so you can draw that with you know with ink that you can see with your eyes or you can you can draw it with lines that you can feel with your hands and tactile perception is quite different in many ways from visual perception but the
00:28:17
Speaker
But it all goes into, you know, both methods enable you to build a cognitive map of what is being shown. So for a long time, there was actually like data from scientists.
00:28:32
Speaker
who studied blind, sighted scientists, who studied blind people and said, blind people can't understand tackle maps. Like this is stuff done in the 60s or so, right? And if they do study tackle maps, then they should use one finger to trace things. And it's like,
00:28:53
Speaker
I knew that was wrong. I knew that that was definitely not true, because I use tactile maps. I enjoy them. I know that my friends use tactile maps who are blind. And I used in high school, I was lucky enough to have a transcriber who would transcribe all of my calculus graphics, like all of my geometry graphics. So I didn't just have the braille, I didn't just have the verbal
00:29:23
Speaker
part of it, I had the graphics to go along with the geometry, the trig, and the calculus. And so I knew that these kinds of graphics are super useful, along with maps and other types of spatial data. And from a practical sense, we all knew it. And I also knew that there was no way that using one finger to explore that was the most efficient way to do it. I use lots of fingers in both hands.
00:29:53
Speaker
to explore and yeah, if you wanna sort of get down and dirty and understand one particular outline, you would probably trace it with a finger, but that's just one method. So I was really irritated by that data being out there in the world, that information, you know, sort of, so people were sort of thinking, oh, blind people don't, why bother? Blind people can't use maps. Why should we bother to improve map resources for blind people?
00:30:22
Speaker
And it was kind of a chicken and egg problem. I couldn't really get funding to do maps research because people thought it was a dead end. One of my first postdoc projects, I took a tool I had built. You guys might remember this. When we were in grad school, I built this toolkit for MATLAB to do sonification output and
00:30:47
Speaker
tactile up, but I have a Braille embosser that can do tactile graphics. It can basically just, it's like a Braille printer hooked up to my computer and it can do Braille or it can do raised line drawings. So one time out here, so sonification, so if you're talking about an image, do you want to say something about what that would be?
00:31:10
Speaker
Yeah, so sonification for images is not so I don't consider that to be its best use sonification sonification is usually
00:31:21
Speaker
best for single variable or multivariable data representations. So the classic sonification is using frequency for the y-axis and time for the x-axis or possibly left-right positioning auditorily for the x-axis. And so you can trace a curve with using frequency for vertical and using time or
00:31:49
Speaker
left right space for the horizontal and so that's that's a great way to do it you can also do you can also like if you've got multiple variables you can map that to energies in the frequency spectrum so like you know the the sort of the timbre of your uh of the white noise that you're listening to might change based on uh maybe there's uh
00:32:14
Speaker
There are different ways of doing this and it's definitely still more of an art than a science. There's a lot of psychoacoustics that needs to go into complex
00:32:28
Speaker
auditory representations because you don't want one part of the representation masking other parts of the representation and so on. So simple sonification is still sort of the most common thing. The most available sonification tool is from SAS, the big data company. They make a plugin for, I think for the Chrome browser, which is called the
00:32:57
Speaker
the SAS sonifier or something. Anyway, you can probably Google it, but anybody can plug it in. And if you've created data with SAS, you can actually turn on the sonification module and be able to use a bunch of different types of auditory displays, usually. You can take it just from a spreadsheet file or something. I believe so, yeah. Yeah.
00:33:26
Speaker
So it's pretty cool. When I was thinking of sonification, I was first thinking of, you sound like you're not as hip to this, but the sonification of an entire pictorial image. There is a way to do that.
00:33:42
Speaker
So this gets into the whole realm of sensory substitution. Right. And I think it's interesting to consider what the differences are, say, between hearing and vision to transform one into the other.
00:33:59
Speaker
Yeah. Yeah. So the way, there are a couple of different ways that have been tried to do sensory substitution from vision to audition. And they mostly have to do with breaking an image into vertical strips from top to bottom.
00:34:19
Speaker
and assigning frequency bands to the vertical parts of the strip. So the bottom part of the strip is going to have low frequencies and the top part of the strip is going to have high frequencies. And what you'll hear is sort of a chord or a
00:34:37
Speaker
or a you know or a noise distribution as it stands over as it scans over the image and as it yeah so it'll break it breaks the image into vertical strips from left to right and then we'll play for you know
00:34:52
Speaker
will play those sometimes just in sequence and sometimes in stereo. But the problem with that and so if you the more the more pixel density or darkness or you know color saturation you have in a particular
00:35:08
Speaker
portion of the strip, the more sound energy there will be in that presentation. The more sound energy there will be for that frequency in that strip, right? So there's a thing called VOIC, which is you can, you know, you can look at, you can Google that and check it out. It's by a guy named Peter Meyer.
00:35:32
Speaker
He has been a very strong advocate for people to use this thing as a as a tool in their day to day world, and it is true that if you have like.
00:35:45
Speaker
a picture of a house, like, you know, if you're looking at the front of a house and it's got a peaked roof and it's got a chimney and it's got maybe a tree next to it. And if you listen to the sonification of that image using the VOIC, which uses the scheme that I was just describing, you know, you can sort of hear that whole thing. You can hear the roof line going up, you can hear the chimney, you can hear the tree, but it doesn't sound
00:36:14
Speaker
like a house. It doesn't give you the percept. And as soon as you change the camera angle, you're screwed. Because the geometry is really key to that whole thing. And so his whole, based on neuroplasticity, from Paul Bacharita and Michael
00:36:41
Speaker
Um, you know, your, your, your buddy, Joe, Mike Mursnick, the, the early, the folks in the sixties and seventies thought like, you know what? We can, we can make people hear pictures. Like we can actually substitute one thing for the other, but no matter how much you wish that were true, it's just not going to happen. And so the people that are advocates for this kind of thing, they're very optimistic. They're very well intentioned, but, but the science isn't.
00:37:09
Speaker
there. And what do you think the fundamental I mean, when I think about it, I can see, as you described, say seeing a house from different viewpoints. I mean, that's a huge problem in object recognition is how to see this how to recognize the same thing from different viewpoints, because it's like you say, it's totally different. You think that's just the nature of the auditory input that you're getting because it's so temporally organized,
00:37:34
Speaker
Well, it's temporarily and spatial. I mean, there's, there's plenty of spatial organization, but it's just, it is, I mean, the, the auditory system has a whole mechanism, you know, we've got plenty of stuff in there for object recognition, right. Or, you know, or source identification, but it's not, it's, it's certainly not organized like, like that. Right. It's not in these.
00:37:57
Speaker
vertical strips. That's like, I think that, you know, the idea of using sound to represent visual stuff in the world is a great idea, but it has to be, you know, you have to tailor it to what the auditory system is good at. And so if you want somebody to hear a house, you've kind of got to
00:38:21
Speaker
I think your best bet is to do a lot of computer vision processing, recognize that it's a house, and provide some other auditory representation that there's a house there. The sort of, the pixel for pixel representation into the auditory is not how
00:38:42
Speaker
the auditory system is going to be able to process that into an object into a, into an image or into a scene. I think the hope was that, that it would work sort of like machine learning where you just do enough, put enough data and eventually, eventually it'll just sort of magically sort of work. And that's what, that's what people, that's what, you know, Peter Meyer and, and Paul Bocky Rita always said, right? But
00:39:08
Speaker
But frankly, I ain't got that kind of time, you know? Right. Right. Well, I was wondering, I mean, I was I was curious as to, you know, if you've listened to it a few times, whether it just sounds like garbage to you or or. Well, I've listened to it. I've listened to it plenty. And I mean, sort of their their ask is for blind people to like where, you know, where, you know,
00:39:32
Speaker
use this thing you know eight hours a day you know for years in hopes that you would sort of eventually have this eureka i can see moment and it's like you know what that's actually more about their desire to
00:39:47
Speaker
their messianic
Practical Technology and Mobility for the Blind
00:39:48
Speaker
desires than about what's worth my time. It's actually much more worth my time to learn to get damn good at using a cane and being able to hear what's out there in the world with my own excellent auditory system than to try to substitute some electronic
00:40:08
Speaker
visual it's just it's it the priorities are all screwed up well yeah and it sounds like you mentioned earlier that you know there's so much information and sounds that are coming at you that i mean just processing that is plenty exactly and and that's not to say that there's no place for
00:40:27
Speaker
for visual information in blind wayfinding. There definitely is, but what you're already getting from the auditory world is so is really rich and you need to pay attention to it. So you definitely don't want, you don't want to be distracted. Yeah. You don't want to hear all this, you know, all this artificial noise sort of masking the real world. But you know, like I said, if, if I had a, um, if I had, you know, a camera that I could,
00:40:55
Speaker
point to the left before I cross the street to say, you know, there's still a car coming at about half a block away.
00:41:06
Speaker
If it could do that fast enough, that would be information worth having. But it has to say, there's a car coming from a block away, not some ambiguous noise that comes out of it. The less of a cognitive load there is. When you roll for about to cross the street, you probably always forget to look. It happens. But if you looked, right?
00:41:32
Speaker
how much like, you know, the monkey power necessary to identify that car coming at you is really low. Like there's a very low cognitive, cognitive load associated with that. I don't want to have to do a lot of extra cognitive work. Like I'd rather focus on the things that the cues that are available to me and maybe wait an extra 10 seconds at the curb, figuring stuff out then, then have, um,
00:42:02
Speaker
you know, then have to sort of really concentrate on this one thing and use those 10 seconds for that. You know what I mean? Yes, I do. Yeah, that makes a lot of sense. And I mean, just thinking about back to like your stuff, Josh, and like, I mean, back to what I really like about a lot of the stuff that you showed me over the years that you've been working on is it is all very, very much focused on like, let's put stuff out into the world that people can really use.
00:42:27
Speaker
Let's take technology at its where it is today. What can we do that actually helps people today? And I mean, that's where I thought when I visited you at Smith Kettlewell and you were kind of going through some of the stuff you were doing with the maps, it just seemed really, really compelling from that perspective.
00:42:47
Speaker
Well, and maybe I should just quickly finish telling that, that map story. So basically when I was doing this, you know, when I first, I mentioned the, the, the MATLAB sonification and tactile representation toolkit. Cause I developed that in grad school for my own use. So cause I needed to be able to look at my data and stuff. And so I took that toolkit and basically turned it into a,
00:43:12
Speaker
tactile map toolkit. I built, I connected it up to a janky GIS that I wrote myself and wrote a program that could basically take street information, street data and print out a nicely formatted Braille street map or a tactile street map of anywhere in the US. And that was like the first, that was like totally revolutionary because for a blind person to get access to a tactile map
00:43:41
Speaker
somebody would have to drive for them. There was no way to do this. This was maybe 2003, 2005. There was just no way to get access to street
00:43:56
Speaker
maps of anywhere you wanted. So this was like this program that for the first time made it possible for anybody to get a street map of anywhere they wanted in the country. And then, you know, later we expand like it's this so it's called T map the tactile maps automated production.
00:44:13
Speaker
system, TMAP, and I, in the last few years, it's been sort of rebooted and relaunched and the San Francisco Lighthouse now runs it as a service and can give tackle naps to anybody who wants them. And lots of people want them because guess what? Those people in the 60s were wrong. Blind people do benefit from tackle naps and they can read them. And actually, the one fingered stuff was also something I was able to make
00:44:42
Speaker
a small contribution to debunking. I had a postdoc named Valerie Morash for a couple of years, and she was an amazing collaborator, really just a methodological genius, like just one of the most capable and brilliant perceptual scientists I've ever met. She's not with us anymore. She died in a really tragic
00:45:08
Speaker
accident a few years ago. But she and I and a few others published in Perception this great work of hers on using multiple fingers and multiple hands to explore tactile maps. And she conclusively showed that more fingers and more hands is better through beautifully designed, beautifully counterbalanced, very, very rigorous science.
00:45:36
Speaker
Like, so we don't need to worry. She, she, it's solved. Yeah. Cool. Well, maybe, uh, this would be a good time to bring up the question that we always, uh, have to ask, which is, we have to ask, which is how is with the technology that you're inventing, how will this technology contribute or bring about the Robo apocalypse that destroys humanity?
00:46:04
Speaker
Now, we just want you to think about this, Josh, because there are side effects to everything. Even Arduino projects and Braille maps could have a devastating consequence. So surely you've given this some deep thought. What evil are you unleashing on the world? That's a good question. You know, I personally think that one of the
00:46:30
Speaker
I mean, you know, it's not me personally unleashing it, but accessibility and sort of the issues around disability access are very interesting. And I think that privacy issues have a lot of relevance for blind people.
00:46:49
Speaker
Our ability to communicate and have two-way video from pretty much anywhere has really changed the way blind people do stuff. There are now services, both paid and free, where a blind person can, through an app on their phone, basically call a sighted assistant and get help with something.
00:47:10
Speaker
and it can be anything from what's the address on this envelope to what are the cooking instructions on this package or what's the off button on this thermostat or how do I, you know, so things like that. Or it can be, you know, I'm lost, I'm looking for the CVS, can you see it around here somewhere? Or I just dropped my diamond earring
00:47:36
Speaker
on the carpet, can you help me find it? You know, these sorts of things are all possibilities. And people are using this kind of thing to do very personal stuff. Like, I mean, you know, is there blood in my stool? Is the, you know, did my, is my pregnancy test pink or, you know, you know, whatever the color might be. Do I, do I have this, does, does this sore have a
00:48:05
Speaker
You know what what color is the effluent from this sore on my you know body. So things like that are like things that a sighted person doesn't need any help with and they're things that blind people.
00:48:20
Speaker
would have gone to their doctor for or gotten help from a close friend or something like that. But now we're reaching out to strangers with cameras to do these sorts of things. And I think that we have not yet really sort of grappled properly with the privacy issues around these
00:48:40
Speaker
These services and another semi apocalyptic issue is, you know, we always have been talking about self driving cars as being the thing that blind people really could benefit from and believe me I'm.
00:48:56
Speaker
I'm looking forward to it. But Lyft and Uber, when they came along, were game changers for blind people. And they will, I think, continue to be an amazing resource for getting
00:49:11
Speaker
to the places you want to go because before that existed blind people had to use public transportation or rely on a friend or maybe use like you know some kind of in some cases use you know paratransit or something like that but but getting from place to place has always been a huge challenge for blind people
00:49:34
Speaker
And Lyft, the TNCs, the transportation network companies like Lyft and Uber, made a huge shift in what people could do. But at the same time, and I don't mean to sound like an old fart, but
00:49:53
Speaker
uh i'm damn good at getting around with a cane because i grew up having to do that like if i wanted to do my own thing i had to get there i had to do my own thing by getting there myself and with the reduction on the importance of use of uh you know independent use of public transit now a blind person can get pretty much anywhere they want if they could use a mobile phone and that is
00:50:22
Speaker
That is
The Role of AI in Accessibility
00:50:23
Speaker
sobering because it's not just about being able to use a cane and get from place to place in the outdoors. It's sort of about independence in general. And so there are definite concerns about people's ability to maintain the skills they need in order to be independent in a world where there are more and more opportunities for their hand to be held
00:50:48
Speaker
and escorted or assisted in one way or another. And, you know, everybody, you know, everybody says, you know, kids today, right? I mean, it's like I think every generation since, you know, since we climbed down out of trees has been weaker and less capable than the generation before, or at least that's how parents have probably always seen it.
00:51:11
Speaker
But this is a concern. It is worrisome that blind people will have fewer skills going forward because the technology allows us to be lazier, to be less independently capable of doing the things. So if the driverless cars don't come along, if Uber and Lyft evaporate or otherwise fail, there are going to be people who
00:51:40
Speaker
are going to be seriously impacted by that because they haven't developed skills or they've built their lives around expecting those services to be there. Like, for example, you know, I live in a place where I have great access to public transit because I moved here before Lyft and Uber existed, but if I were buying a house for the first time and I
00:52:04
Speaker
assumed Lyft and Uber would always be there, I could buy a house wherever I want, right? I could live wherever I want. And if they go away, all of a sudden, that really changes the way I live. But I think, unfortunately, your technology that you're developing is not going to contribute too much to the Robopocalypse.
00:52:23
Speaker
Well, I mean, the AIs, right, the AIs are always going to be useful, you know, in terms of, you know, if I've got a camera and I want good information about what's around me, that's not just going to be computer vision. Computer vision is going to be really good at maybe parsing the scene into objects.
00:52:42
Speaker
But if I want to know what's important, if I want something to be able to give me information about the relevant pieces of that scene, that's going to get more and more AI-ish. And so there
Closing Remarks and Appreciation
00:52:55
Speaker
are huge possibilities for AI to get into the accessibility space. And I think that accessibility may be a great excuse for the availability of
00:53:12
Speaker
AIs. I foresee that when we start legislating AIs and their capabilities, one of the levers that the pro AI side will use will be accessibility, for sure.
00:53:27
Speaker
Well, Josh, thank you so much for being on the show and always a pleasure to talk to you, especially about the technology and the stuff that you invent and the stuff that you're working on is really just very cool and very inspiring. So thanks a lot for joining us on the show. Thanks, Joe. It's always really fun to get to hang out with you guys. I miss you, and this is a great opportunity to throw some ideas around. This is fantastic. Thanks a lot, Josh. Thank you, Rolf. Lovely to hear your voice.