Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
/software development: navigating complexities at MSFT
41 Plays5 months ago
On this episode, we dig into the (maybe not so) unique challenges and opportunities of building cutting-edge technology at scale for Microsoft. Nat Pollock, Principal Software Engineer at Microsoft, joins Aaron and James this week, to chat about navigating the complexities of software development at a global tech giant.
Recommended
Transcript
AI: A New Frontier for PC Interaction
00:00:00
Speaker
For me especially coming from the touch and pen background where those those were each new exciting ways to interact with your PC. And so for us AI is just another even newer more exciting way to interact with your PC.
00:00:12
Speaker
And so we're we're just really trying hard to i have a circle and brainstorm and zero in on what is the most compelling user scenario for for how to have AI help you interact with your PC. There's definitely still a lot of questions and searching for what is the right product market fit. And so we're just really trying to figure out what is the right way to get that into users hands in a way that they actually want.
Introducing the Hosts and Guest
00:00:41
Speaker
Welcome to the forward slash where we lean into the future of it. t I'm your host Aaron Chesney with my beautiful co host James Carmen. Today we have with us Matt Pollack who is a principal software engineer on the windows AI team at Microsoft. He's a career Microsoft and play with which includes three internships all beginning in 2009.
00:01:03
Speaker
He's worked on several projects over the years, including content services, touchpad inking, and animations, and most recently, AI. He also has worked on something called paper airplanes, which to me sounds more like a hobby than a profession. Nat lives in Seattle with his wife, daughter, and dog, and they have a new daughter on the way. Welcome to the podcast, Nat, and congratulations on your expanding family. Thank you. But before we get started,
00:01:32
Speaker
I have an aside for you. My computer is always running cold. Do you think it's because I always have my windows open?
Tech Support Tales at Microsoft
00:01:40
Speaker
It's always running cold. oh that That's such a bad joke.
00:01:46
Speaker
will You almost got him to to try and solve the problem. I started brainstorming right there. You don't get that a lot do you? I'm sure with like family and like it's bad enough for us software developers. Hey, can you fix my printer? But you work like for the company who makes the operating system like that's probably I i get it all the time. Yeah. Oh, yeah. It must be just unbearable.
Microsoft vs. Smaller Companies: Development Insights
00:02:15
Speaker
So today we wanted to talk about how
00:02:19
Speaker
you know, is speaking of software development, how software development is the same or different than Microsoft as compared to, you know, smaller companies that aren't like flagship companies. Yeah, I mean, I can, since I've only been at Microsoft, I kind of only know the details of how it's done internally, but I can kind of think about the things that I think are probably different. Like one would be, like we we just have so much code and so much old code. um And so there's so many people working on one thing that all needs to come together in the end. um So there's this this big system of, we call them branches, but basically individual builds of windows that each team has to
00:03:02
Speaker
Like each team would own own their own little build and we make our changes and build it, make sure it works. And then those merge up another level when the managers decide it's ready. And there's anywhere from this four to seven levels of those all coming together in the end. Yeah, I can't imagine. I mean, so like their windows one came out in 1985. So you're talking about what is that 40 years now? Just about. Yeah. Yeah.
00:03:28
Speaker
of, of software. i I can't imagine that having a code base that old and still trying to make things work. I'm sure a lot of that has been replaced over the years.
The Windows Build Process
00:03:38
Speaker
Um, you know, like the, uh, the old eight character name limit, I was so happy to see that would go. Wait, that's gone now. I have to, I can stop this. Yeah, you can, you can use spaces too. Thank goodness. However, don't try and rename something with just a capitalization change.
00:03:58
Speaker
Oh yeah, that'll confuse git too. Yeah. there's there's ah There's a few systems out there that still have an issue. It's like, well, I changed the capitalization on it. It's a different file. No, it's not.
00:04:10
Speaker
Well, what's, what's it like? So you guys, you mentioned like you guys have your own built. So the different like teams within the windows, you know, broader windows team, you all have your own kind of build. Does, do you like have a physical device that you run the build on? Is it virtual machines? Like what, you know, as you're tinkering around, how how do you kick the tires on? Yeah, no. So there, there's a whole, whole system. There's a whole team that their only job is to make sure all these build servers are running and like There's no errors with the builds. They they set up like the schedule for the builds. The way it typically works is there's one build every night for each branch. um So like my team, our build comes out in the morning, anywhere from 9 to noon. um And so like if you had a bunch of changes, you really want to test. in So when you when you're developing your changes, you're obviously applying them and testing them before you check them in. um But you always want to make sure that things work in the actual build.
00:05:01
Speaker
um So yeah, so there's a whole, that's one of the the cool things that working out of big companies. we We have a whole team managing this whole server farm basically um to make sure everything's running. But the build drops like say in the morning, right? And you get a new build of of windows. how Like how do you make that come to life? Is that you're all all you're testing on virtual machines or do you physically put it on a box somewhere or is that done? Like how how do you do that?
00:05:29
Speaker
It's super variable depending on what you're working on. Even when i like but I worked on touch, you can't really use a virtual machine to test the performance of touch, right? So we would we would be applying those to our physical devices. we had have I have like probably seven service devices here now that I've tested over the very over the years. um And so i would like if I had something new, I probably wouldn't update every morning. Some people would update every morning. but Um, I found it to be a little bit too kind of consuming, but, but yeah, when I would need, when I think there would be enough changes or something I need to test, I would, um, go through the upgrade process. We've got a pretty streamlined there's scripts and you you type a few things, you wait an hour and you have a new, a new windows build installed on your device.
00:06:06
Speaker
um But now lately when I've been doing the some of the UX for the Recall app, that was quicker to do on a VM because you could just like run a VM in the background, have it ready whenever while you're working. um and Those didn't need any extreme performance. like Some of the stuff we were doing needed needed performance testing, but stuff that just like I'm moving a button around here, that kind of stuff, those are really easy to do on a VM. so it's just yeah It really depends on what you're doing.
00:06:33
Speaker
There's, we've gotten into a lot of internal debates about what is the right way and um we kind of realized there is there is no one right way. So for those those playing along at home, a VM is a virtual machine, which is basically a computer inside a computer. ah yeah where yeah basically it's Which is the premise of how cloud works, isn't it? is just ah Instead of having large you know server farms where you have
00:07:04
Speaker
Thousands of computers actually running. You have a fewer big computers that are segmented up into virtual machines. At least that's my understanding. cloud that's That's what I think of when I think of the cloud. um So that's ah really interesting. You mentioned something in in that answer ah called recall.
Windows Recall: Enhancing User Memory with AI
00:07:30
Speaker
what What is this recall thing? Yeah, so i I don't know. The official name is Recall for Windows or Windows Recall. um You may have seen some some bad news about it or some good news, depending on what you read. But the the gist of it is it's it's a way to help you remember everything you've done and get back to everything you've done on your machine on your Windows machine. So in the background, it's capturing what you're doing, running a bunch of AI models on them. And then you can open the Recall app and you can search, be like, hey, what was the What was the meeting I was doing with Aaron and James and like it would it would show you that meeting you could click on the link and get right back into it. um So I, I initially worked on that I did a bunch of the early brainstorming prototyping played with a few of the AI pipelines. And then as we get closer to shipping, I own the UX for the timeline. So I didn't I didn't mention that but there's a
00:08:19
Speaker
When you launch the recall app at the top, there's a timeline and I'm doing my hand motions. so You guys can't see those ah at the top. you You can see the timeline. You can click, click on the timeline and then you can drag back through all the history of the things you've done on your machine. So that sounds very big brotherly in kind of scary. Like I can, I can say it's like I knew it. They're watching us all the time. um So how do you, ah how do you,
00:08:49
Speaker
Make sure that the users that are using recall are that that their information isn't leaking out to the government or you know the the secret societies that they're worried about, you know, storming their their properties.
Security in Windows Recall
00:09:05
Speaker
Yeah, no, it's ah it's a really good question. That was the the basis of a lot of the critical feedback we got when we first announced in April or whenever that was. And so we did a ton a ton of work, like some of the craziest amount of work in a short amount of time I've seen in my career at Microsoft over the past that six-ish months or so um to really secure it. So the the the gist of what we do is there's this technology in Windows called an Enclave.
00:09:29
Speaker
um And it's similar to a VM. It's it's basically a really, really pared down VM. um So it's running on the machine. Everything inside that that VM is encrypted. It's locked down. And so all the information that we collect from your machine is stored in that enclave.
00:09:46
Speaker
And the only way to get that information out is by going through a Windows Hello prompt. So I don't know if you guys are familiar with Windows Hello, but it's the authentication method that was introduced somewhat recently. I don't know the history of it, but you it's really required to have a special TPM chip, security chip on your device and the Windows Hello, it can either do face authentication. I think they have fingerprint authentication option.
00:10:07
Speaker
um But you're required to have one of those two, so it's proof of presence. So your data can only ever be unlocked in red out of this special enclave if you're physically there, in theory.
00:10:18
Speaker
um And there's some other protections like the data locks automatically after 15 minutes so you can't like have someone come back later and try to hack into your data. Even if you had a like a hacker somehow plant something on your device, they can't access that data for longer than 15 minutes um unless you like continuously log in with your face.
00:10:38
Speaker
And they even we even added some stuff which might be interesting to talk about more that because I was involved in that to make sure that a hacker can't like just take everything all at once. You can only make query to the database every so often. So that's ah that's a gist of some of the protections. Because yeah, it's definitely very scary to think about. So we did a ton a ton of work to try to mitigate that.
00:10:58
Speaker
What's it like from your like from your perspective? I know you're you're probably proud of the work you do daily there at Microsoft and you're <unk> working on really cool stuff. And then you see that in the press and it's like, they're trying to steal my data and steal my information. and You're like, no, I'm not. I'm just doing really cool stuff. like How does that feel? What is that perspective like?
Navigating Press Coverage and User Responsibility
00:11:14
Speaker
That's that's a unique perspective.
00:11:16
Speaker
Yeah, no, I think, I mean, I think we kind of, there's, I see both sides of it. Like I see that there's a lot of valid feedback and then I also understand that it's the press's job to make things exciting and interesting for people to read and and maybe blow it out of proportion a little bit. um So I could, I can see both sides and I, I think we were talking about what's it like to work at Microsoft as a big company. Like that is something that's, I feel is unique and exciting, but also stressful is our user base is, as I was looking at, it's a, it's a billion users. wow If you look at all the windows users. that's a lot And so that's basically everyone. And so you can't, you can't say like, oh, well this, this will work for this subset of users because if you have to kind of, you have a responsibility to all of your users. I imagine like backwards compatibility is a, is a,
00:12:02
Speaker
big headache too, since you're, since you have to go out to the masses and you have to work on all kinds of different hardware. I imagine that just as all kinds of headaches. Yeah. There's so many business strategy discussions that I'm not anywhere close to about like the, for, for this recall product specifically, we decided to, it would only be on these new devices that have this special NPU AI chip. Um,
00:12:29
Speaker
but yeah I'm sure there were um a million discussions about, is that the right decision? Are we locking out customers that might be able to get it? um it's It's above my pay grade. and i want I keep trying to understand more about it. It's it's fun to learn about the business side. but yeah so you Were you in on the discussion when they came up with the idea for this whole recall thing? like Please tell me it's based on the Black Mirror episode. Did you guys see that one? where like yeah Your brain is, everything you see is being recorded at all times and like the guy's like, I'm ready to get on the plane. Like, okay, let me, let me thumb through the past few days. Have you done anything wrong or whatever? they like Whoa. Okay. That's crazy. Yeah. We, we talk about that all the time. Yeah. We try, we try to prevent that. Yeah. That's pretty scary stuff.
00:13:13
Speaker
great though what What are you talking about? Cause I have no idea. No, it's on Netflix, black mirror. So we're not sponsored by Netflix. Um, and they're like, yeah, no kidding.
00:13:30
Speaker
know joe rogan yeah this episode brought to you by Um, it's, uh, I would say it's like the modern day. Uh, what was the,
00:13:45
Speaker
Twilight Zone kind of a thing, right? But it's more like cool tech oriented and it's it's pretty interesting. There's some really, really cool episodes. I don't remember what that one was called, but it was something about. like that That was the grain, right? there's a It's a like a little grain you have in your ear and your brain. record It just records all your memories and you can replay them. and it's Yeah. I have to check that one out. The premise of the whole show is that it's like the a dark take on all the new technology, basically, right? Yeah. of it Yeah. There's, there's some great ones, uh, great episodes. You'll be, you wouldn't really like it Aaron. Yeah. It sounds right up my alley. Um, so speaking of strange things, I have to know what is this thing in your bio about paper airplanes?
00:14:28
Speaker
Yeah, so that was when i my third internship. I came back right when we were releasing Windows 8. So I don't know if you or your listeners remember, but Windows 8 was the big touch focus. Everything's going to be a touch device. We're replacing the start menu with a giant full screen touch.
00:14:44
Speaker
touch menu. um But part of that release was there's a new Windows Store for Windows apps and they wanted every intern that summer was tasked with creating an app to be in the new app store. um and The only stipulation was the app had to be somehow related to like the the Windows team that you were positioned on. And so that summer I had ended up on the um like devices and printing team within Windows. And so our mission was to figure out how do we write a Windows touch app that also needs printing.
00:15:14
Speaker
um And so the the most fun really the only thing we could come up with why why someone want to print something anymore ah was paper airplanes. And so we designed this app where you could ah scribble like use touch and pen or whatever you want to draw ink all over ah a 3D model of a paper airplane.
00:15:32
Speaker
And then we would deconstruct it so that when you printed it out and folded it back up, it would look the way that it did in your 3D model. um And so it was a really fun project, like using a lot of touch, doing 3D modeling. It was all in JavaScript for some reason. That was when um they were trying to bring a lot of the web developers into the Windows ecosystem. So we made sure that all of our APIs would work for JavaScript. um So i had it wasn't all me. I had another guy who he he was really smart. He did a really good job of the 3D modeling part. I was in charge of all the the touch drawing and that kind of stuff.
00:16:02
Speaker
um But yeah, we got we got it done, it worked. I remember one of my, one really fun thing was when we, we had a, I guess we had a pitch process to some of the leadership and when we initially pitched it, one of the the higher up leads was like, basically said it was impossible and we wouldn't be able to finish it in time and so it was, it was fun to like prove him wrong. Oh, I love that. I wish I could remember, I wish I could remember who that i wish i could remember who that was because I should, I could go talk to him now and make fun of him. In your face, Bill.
00:16:27
Speaker
So there's a similarity between the, the big corporations and the little shops. So there's always that one naysayer that you're like, Oh yeah, well, I'm going to and keys with, yeah with, with angst.
00:16:45
Speaker
Yeah. Yeah. i I don't think it's still out on the app store. I should, I should look, but I don't even know. We probably didn't get any downloads, but it was, it was a really fun internship. That's really cool. be fun to tinker with for sure. Yeah, we should we should have a we should run a contest. Send us your plane so when it flies the furthest will send you something. So what was it?
00:17:09
Speaker
you know like we We usually, you know, normal us mere mortal software engineers. We kind of get to stand on the shoulders of giants ah of of the operating system and all those things are like already built. Like I don't have to think about touch interface really ah a lot with what I do, but you do, right? So what was what was probably the biggest like, I don't know, aha moment for you when you're switching from like, okay, somebody's dealing with a mouse to to dealing with
Challenges in Touch and Inking Technology
00:17:35
Speaker
touching the screen. What was what was that for you? how How did that change your like how you view world. Yeah, no, I think the the biggest difference by far that is like still keeps coming up these days is it's you can't really get away with any um performance lags for touch or inking because it's when your finger is on the screen. So it's very, very visible if something's lagging behind your finger or lagging behind your pen. Whereas your mouse is it's your hands down here on the on the table and your mouse is on the screen. So the disconnect isn't as obvious.
00:18:05
Speaker
um And so it it ends up creating a ton of really interesting and challenging problems. One of the things I worked on was this product called Direct Manipulation Demonip, um which was part of the Windows 8 release and and we continued working on it for a long time after that. um But the the goal of it was any time you put your finger down and start moving it, we want that to respond on screen immediately with motion and very, very sticky motion to your finger.
00:18:34
Speaker
But for things like in the web, JavaScript can be running and they can do whatever they want with the input and they might be lagging. And so we need to somehow move, somehow he responded to that input by moving things on the screen in the background while the app or the browser or whatever is processing that input and doing what they want with it.
00:18:51
Speaker
And so it creates a lot of communication challenges, a lot of like synchronization challenges. um Sometimes you have to decide, like you have to create some contracts between the app or the browser and the OS about what motion is allowed, what motion is not allowed, so that the system can start moving before you even responded to the input. um Or even things like um like, I put my finger down, what is it what is my finger on top of? What should be moving?
00:19:17
Speaker
It's ideally we wouldn't even have to ask the app because that's too slow. Um, but so in some cases the app is going to tell us, okay, this is the thing that I want to move. You guys can move it while I finish loading the rest of my content or those kinds of things. So I don't know if you've ever used a touchscreen and you see, and you start scrolling and you see like, oh, the content isn't loaded yet. That's because the system is moving the page while the app is, while the content is slower and and loading in the background. That's really interesting because I, you know, and that's something that we,
00:19:44
Speaker
mere morals deal with all the time is those, those dependencies between like the load of of the software and and doing that, especially since like the invent of, of the internet.
00:20:02
Speaker
We have always been fighting against load times in trying to manage load times. And it's, it's interesting to hear that, you know, even, even with the finger swiping stuff in all of that, it's still all about how do we manage load times? So that's, that's pretty cool. With, um, you mentioned inking and I want to, I want to circle back to that.
00:20:30
Speaker
I know inking from a comic book reference like when you're when you're trying comic book images, you go back through and you have your inking pass, which is where they add the color and the darker lines. What is inking um for touch experience? Yeah, so you could think of it as like pen input, um but we call it inking because it's a It's a different um paradigm. like you can You can use your pen as ah as a stylus where you like just tap with a mouse or double click or drag stuff around, um but the the more powerful way to use your pen is like by drawing, inking, handwrit using handwriting or drawing. um and so the That is an even more
00:21:17
Speaker
um important in terms of performance because you have an expectation of like, when you draw with a pen on paper, like it's immediate. you you And you also have some of that tactile feedback, which we've experimented with adding, based with pen vibration, those kinds of things. um so So that's what's meant by inking. It's a different way of interacting with your machine. Well, you mentioned I have like, when you when you draw on paper, like as soon as I move my pen, there's ink left behind automatically, that's how we're used to.
00:21:45
Speaker
Do you guys find this to be true? Whenever a server at a restaurant gives you a pen, that's never the case. like that My pen never works. How is that? Never mind. I don't know. There's something happening to me last night. I have many suspicions on that one. I think it's the paper.
00:22:03
Speaker
Because if you think about the mechanism of a ballpoint pen, that ball has to roll, but they give you that slippery receipt paper. So of the ball doesn't have enough friction to roll across the paper. I've thought about this in depth because i they irritate the hell out of me.
00:22:20
Speaker
Yeah, all the time. And I blame the paper. They should probably all have Microsoft services and me they could use my inking. I have, I have seen some restaurants that have moved to a tablet system. Um, I don't, I don't know their OS, but, um, that have done that where they, you know, basically give you like something you to just sign the screen. And that's like, the only thing weird about using a pen on screen is that, and you mentioned the tactile feel is that It's slicker, you know, it's that that glasses is slit in it feels like you're you're writing underwater or something like that. It just doesn't it doesn't have the same feels like if you have like a graphic yeah like a ah pad.
00:23:08
Speaker
um that has more of a tactile feel to it, you know, where the stylus kind of grabs and it feels more papery. Yeah. One of the things they experimented with, I guess that they didn't get very much usage, but at one point they were providing like four different pen tips with the surface pen. and So you could use the one that had the the scratchiness that you liked, but they, they stopped doing that. I guess maybe, maybe it wasn't a hit. Oh, is that why they had four? I just thought they were a replacement tips.
00:23:35
Speaker
yeah think That was a documentation issue. that just be like Why do I have four different tips? I i don't know. So is there, with with the inking or the pen input, and there's gotta be some interaction like with the screen itself, i can it or is all the pressure built into the pen, like it with pressure sensitivity, like I wanna draw a thick line or whatever, or you just have even a thing, I guess? That's a question, I'm not sure I know the answer. I wanna say that the pressure sensors are all in the pen um and that the ah screen is just responsible for locating the pen, but I ah honestly don't know.
00:24:15
Speaker
Yeah, it would be hard. Like, how does it deal with a touch? I think there because ah it feels spring loaded when you're using it. So I do believe that the pressure sensitivity is in the pen as opposed to like I have a um ah Wacom tablet that I use for some other stuff. And that doesn't actually Yeah, it it also has like a spring loaded stylus. So a lot of that pressure comes from that. i I don't know. There may be pressure sensors in the pad too, but having it in the pen allows you to move it across and and go and flow without having like all like a giant sensor grid across your touchscreen.
00:25:00
Speaker
So I see that you did touch pad too, um, as, as well as like the inking with the touch pad, cause touch pad drives me nuts. Um, and I know I it's, it's probably my own shortcomings. How do you figure out like what gesture should be and how do you, you know, like recognizing those different things and in processing, and where what are the expectations there with those types of things?
00:25:30
Speaker
Yeah, it's a good question. I think the touchpad gestures has been a little bit of a Wild West in terms of we kind of, some people like one thing, some people like another thing, we try some things. um The RPM team and our design team kind of came up with, in the map patch at least, they kind of standardized on like, well, I think it's pretty standard. Two fingers is always going to be scrolling, scrolling and zooming, similar to two finger touch, right?
00:25:55
Speaker
um but Once you get past two fingers, then it that's when it gets kind of crazy. um So what what we found what we kind of decided was three fingers is going to be like window manipulation. So three fingers swiping left and right is going to switch between your active windows. Three fingers swiping up and down is going to minimize windows.
00:26:11
Speaker
And then we decided four fingers is for switching between virtual desktops. um i don't think I don't think a lot of people even know about the virtual desktops feature. um So probably even less people know about virtual desktops plus four finger gesture switching. But it's it's actually really, really powerful for doing um development on a laptop because you can have your remote desktop on one virtual desktop and you use four fingers to switch back and forth between your local device and your and your remote device. no But i I would imagine the number of users who have discovered that combination of those three things to be very small. I may have done it by accident once.
00:26:47
Speaker
yeah it it Because you know that's that's my issue with the touchpad is always center underneath my hands and I like accidentally sleep something and my mouse is over in my field and I'm like, what the heck's going on?
00:27:00
Speaker
Uh, but if I wanted to be a gestures master, I want to be, you know, I want to be the super user of the touchpad. Where could I go to find out what gestures are recognizable and, and what they do. Yeah. In the, in the windows settings.
00:27:20
Speaker
um window, yeah there's a whole section on touchpad and touchpad gestures, and you can even change them to be you can customize them the way you want. and it ah It tells you what the what all the defaults are. That's cool. I i had to stop myself from digging into the settings as you were saying.
00:27:38
Speaker
They should have like a little game that you have to do the all the different gestures for controlling your like a little platform or or something. you know that That would be a cool way to teach you the gestures. like Yeah. there We have a lot a lot of debates about which things are worth teaching tips and which things aren't because ah you could end up with a million teaching tips for every individual little feature. I made a paper clip.
00:27:59
Speaker
you know it's ah think
00:28:07
Speaker
For those of you who have no idea what we're talking about, he's referring to Clippy, the old paperclip helper assistant that used to be in, well, that's just Office, right? That wasn't one of those songs. There's definitely a Clippy, a Clippy gif that shows up in chat every couple of days or so about something or other.
00:28:26
Speaker
We use it to we we have a member of our team has just these phenomenal excel skills and we are always like the clippee do that for you like there's no way you figured that out on your own that's gotta be clippee you know. So talking about doing doing things for you. What's your.
00:28:49
Speaker
AI experience been as of late with being on the AI team. like what are you How are you leveraging AI at an OS level? I mean, we've been talking about how to do it at an application level and using it you know for development as a tool, but how is that different for and an OS level?
00:29:08
Speaker
Yeah, I think for me, especially coming from the touch and pen background, where those those were each new exciting ways to interact with you with your PC. And so for us, AI is just another even newer, more exciting way to interact with your PC. And so we're we're just really trying hard to i like circle and brainstorm and try to and zero in on what is the the most user like what is the most compelling user scenario for for how to help have AI help you interact with your PC.
00:29:34
Speaker
So that's where Recall was one of the attempts of like maybe AI can help you remember or get back to or summarize the things that you've been doing on your PC. Copilot is another strategy of like maybe AI can sit with you while you're working and and be there to answer questions immediate questions or help you drive your PC. um And so we're I think we're there's definitely still a lot of questions and searching for what is the right product market fit, I hear a lot. um And so we're we're we're just really Yeah, really trying to figure out how is the best way because we know we all know it's powerful. But what is what is the right way to get that into users hands in a way that they actually want? Well, one I think that would be really good is, you know, if you could use AI to as part of the troubleshooter, or, or something like that. Obviously, there's already a lot of information out there, I would think that's a huge
00:30:34
Speaker
model to work off of, especially like when you go out to forums and all these other things to be able to have AI assisted optimization or fine tuning of your system. And, ah you know, based on what you have, because I imagine that the number of computer configurations you guys have to support is infinite basically. Right. Like I build my own system and I can't imagine, I still amaze that.
00:31:03
Speaker
you know, installs in operations work as well as they do for something that I, Frankenstein together out of parts so and pieces. So, you know, hats off hats off for that. And, um, but I imagine that's like a, that's a troubleshooting nightmare.
00:31:22
Speaker
Yeah, I know. There's there's definitely lots of lots of work streams thinking about that kind of stuff. like CoPilot kind of fits into that a little bit because like maybe you could ask CoPilot, hey, I'm having a problem with this. Can you look at my screen and help me understand what's going on? and and like there's There's so many cool things. It's just it' just a matter of doing the work on the right one at the right time to get users interested. That's a cool pro tip. Yeah.
00:31:47
Speaker
I didn't think of ah doing that. It's like, Hey, cool. but Why? Yeah, I don't, I don't know. like Troubleshooting other things. I don't know why I didn't think of troubleshooting my system with that. Yeah, I think the cool.
00:32:00
Speaker
aspect of of recall, that as you describe it, is just having that consciousness of what am I looking at. so thats That's got to be an interesting problem to solve. It's like taking pieces of the screen and and and digitizing that in a way and understanding what that was. so then I asked, what was that video thing I was doing with Aaron and James back you know two weeks ago or something like that, that it understands that. How was that problem to solve? how what What was that experience like for you?
00:32:27
Speaker
Yeah, so the the big piece of that is um these things called embeddings or semantic similarity. i'm Just a quick overview of those like the, just as you take a picture or some text and you run a model on it and it generates some coordinates. We could, it's it's much more, but let's say it's an XY coordinate.
00:32:47
Speaker
And so like it's a picture of a fox and we get an XY coordinate for the fox. And then there's a picture of, I don't know what's something related, somewhat related to a fox, a squirrel. It's a picture of a squirrel. And so that has another XY coordinate and it's kind of close to the fox. um But then a picture of a race car has a different XY coordinate that's really far away from both the squirrel and the fox. And so as you like accumulate all these embeddings of these different things, you see which ones are close to each other. And so then but that's how the, you can kind of do some of that really cool searching stuff was like,
00:33:17
Speaker
the one of the examples that you always used was what was the the presentation with the red barn in it. And so that text that says red barn, that has an XY coordinate. And then the picture that is a red barn also has an XY coordinate and those are really close to each other. And so that, that XY coordinate is capturing the the, it's called semantic, but it's capturing the meaning behind that text or behind that image. So even if you said um like the text red barn and the text barn, what's another word for a barn? I don't know.
00:33:46
Speaker
shed house like red barn yeah your red yeah red barn and red shed even though barn and shed are completely different text terms so they wouldn it match text-wise they you're very very close in this semantic space It sounds like we're getting really close to having our computers be able to click a checkbox that says, I am not a robot. yeah I don't know if they can figure out what a crosswalk is. Yeah. we trouble that yeah Or, you know, which pictures have a bus in it? Or a bike. Or a bike, yeah. So you're talking about embeddings. So embeddings, the way I think of embeddings is like it's it's taking something and turning it into a bed.
00:34:26
Speaker
right and and And then those vectors would all live in the same vector space and and each component of the vector term in the vector would would represent the same thing, concept or whatever. um it's Did you find that you had to, for different types of apps, like a ah visual picture, maybe is one embedding embedding mechanism better for pictures versus text or is there one uber embedding algorithm?
AI Innovations in Windows Recall
00:34:53
Speaker
Yeah, I think, um so these are some of the things that I started to investigate and then we also have this whole AI team designing these pipelines of like some stuff goes in here, some stuff goes in there. I think I'm probably getting into stuff that's proprietary at this point, but um but yeah, you can you can imagine there's there's all kinds of cool stuff. Like, I don't know if you've seen part of Recall, it's kind of its own thing now, but if you hold down the Windows key, um the whole screen pops up and it analyzes your screen and like highlights um It highlights the text with OCR. It um finds all the image regions. And so that's that's one of the AI models that we're also using in in the recall pipeline. That's cool. And OCR ah for those playing along at home is what it was an optical character recognition and something we we use and in content spaces a lot for taking a scanned document, being able to recognize what the words are on the page so that it can be digitized.
00:35:50
Speaker
Yeah. So it's, it's really powerful. Like I've, I use it just like, let's say I'm in a, someone's sharing their screen in a, in a team's meeting and I want to go to the same yeah URL that they're in. I can just hold down the windows key and then so highlight the text from the the screen sharing and go right there. wow It's like simple, but so powerful. It almost puts like a, almost like an overlay over your screen of semantic of fine with all the things. And that's cool. I'll have to check that out. I didn't know that existed. That's really neat.
00:36:17
Speaker
Yeah, unfortunately, I think that's only available on the new CoPilot Plus PCs as well, but you should get one. They're pretty cool. Christmas is coming up. So there we go. Shave this plug from our guest. I love it.
00:36:31
Speaker
I think even though like just the like the 10, eight to 10 hour batteries sells itself right there. What other features does it have? What do you want for Christmas? I can't really think of anything. Now I can think of something. Oh my gosh. and We better get a sponsorship check for this one. i That's two. We've got Netflix and Microsoft now. yeah I think they will see a spike in their sales from our viewers. they They're going to see a recognizable spike from this. Absolutely. yeah our two Our two listeners.
00:37:04
Speaker
Is that my parents and your parents, Aaron? No, my parents quit listening, so it may just be your parents now. Clearly not. Come on.
00:37:17
Speaker
Uh, they're like, this is way over my head. I don't, I can't, I love you, but no. All right. So at this point, let's go to ship it or skip it.
00:37:29
Speaker
sheoo skin but ships ski everybody we got to tellbos good So ship it or skip it is where we talk back and forth about a few different topics. And we make a decision on whether it is something we keep in ship it, or it's not really needed or not a good idea. We'll skip that. So with our first one, James, what do we have on the docket for today? Ship it or skip it.
00:37:59
Speaker
a couple of like warm up questions. These should be pretty quick. So, and I, I didn't know how to phrase this. I think it's called just natural scrolling versus reverse scrolling when it comes to like touch interface. What, what do you prefer? Yeah. So I was so, uh, against the reverse scrolling at first, it felt so unnatural. And then as soon as I use a touchscreen for like five minutes, then, then, then the like natural scrolling was the way to go. Like, so I don't guess I don't know which one you're saying. Like the one I like the one where it matches touch. So like, if you, where it feels like as you as you grip the screen and flick in a direction, the screen goes that way. I like that one. Whatever that's called. Yeah, whatever that's called. Yeah. yeah I think that's the natural scrolling, isn't it? That's what I would have called it. I used to have to like switch it every time I got something. It was like, it was, it was the opposite of what I wanted it to seem like. I don't know, but I'm just curious. Yeah. I'm i'm the same. I want my screen to follow the direction of my finger um for that.
00:38:57
Speaker
um However, I think on a mouse, um ah ah no, I'm the same way. And that's one I always have to change on a Mac because Mac is reverse scrolling. um So like when I roll my mouse wheel down, I want it to scroll down. Oh wow, you even change mouse wheel, interesting. Yeah.
00:39:14
Speaker
Well, on a Mac, a Windows does it the way I expect it to do it. So when I scroll when i scroll down on my mouse wheel, my page goes, well, it goes up, actually. So I guess that is reversed. I don't know. it Is it considered? the The scroll box goes down, but the page goes up. So i now I confuse myself. I don't know. I just like what I like. I don't know what it's called. yeah I don't know what it's called. but i This was a bad topic. but i I'm telling you, that's it's like one of those religious like squiggly braces at the end versus the next line kind of thing. Because some people, when I when they see how I have mine set up, like, oh my gosh, how could you do that? It's like, what is wrong with you? It's just what I like. All right, go to the next one, I guess.
00:40:03
Speaker
All right, dark mode everything. oh um I'm not really on board with that. I know I probably should be it supposed to be like save energy or something, but it makes me feel like I'm an evil villain or something.
00:40:17
Speaker
I love that answer. I don't want to be an evil villain. It's bad enough I work for Microsoft. I don't want to be putting that box. Wow. That was our sponsorship. Thanks a lot, Aaron. I had to balance it. I had to balance it. I thought with all that Microsoft money, I was going to be getting a new microphone, maybe some panels up for my wall, some acoustic panels, man. Forget it now. What operating system are you on right now? So let's go on to the next generation.
00:40:46
Speaker
So, um for me, i'm I'm a skip it on dark mode everything. ah I'm looking at my screen right now. One's in dark mode and my other two are in light mode. I like certain things in light mode and certain things in dark mode. It just depends on what I'm doing.
00:41:06
Speaker
I'm sometimes it because especially if you're doing something visual you want the actual and it may be something that you're presenting that is light mode um and having it in dark mode you don't get the you don't get the same kind of wizzy wig feel um in your editor.
00:41:24
Speaker
yeah i mean Skip it for everything. I'm the same as you. I think, depending on what I'm doing, it's i for most everything, I'm dark mode. But like if I'm documents, um just feel it feels more natural to have that lighter background. It feels more like paper to me, I guess. I don't know. That just feels more natural. OK, so 3D or holographic interfaces.
00:41:49
Speaker
Yeah, this one's I think kind of tough. I could, I think the, some of the 3D stuff would be cool if you could have like a virtual large virtual monitor or like, then you could, I don't know, you could sit in your comfy chair and you could have your six monitors right there with you. um But for actually being productive, I don't, I don't think there's a ton else there. So you ah skip it. So I think I'm a skip it. Yeah.
00:42:18
Speaker
I think for me, i've I've always been fascinated by this. I have a hard time doing textual work. it's like you know If I'm doing a bunch of like text documenting or coding, it's it's not a great environment for that. However, in other things that I'm doing, um it would be awesome to have you know those kind of interfaces. but I wouldn't want to have to manipulate with my hands in space. i've I've done some of that before with like the early field recognition stuff and having your hands in a position where they can be read.
00:42:55
Speaker
for a long period of time is very stressful on the shoulders and arms. And so, you know, being able to have like some type of joystick controls for, um, motion and that kind of thing where I can still end up doing typing. And that's the, and that's really the hard thing is the control shifting, um, between the two. So I'm, uh, I guess I'm an in-between.
00:43:21
Speaker
I'm like, yes, for certain things, you know, ship it for certain things, skip it for other things. Like if I was doing 3d modeling in a space, 3d holographic interfaces would be awesome. yes I could then, you know, I could literally move around my model and then, you know, touch and pull and, and sculpt like, you know, um the way that I would want things to move. Um, that would be really cool. Uh, but for, um,
00:43:48
Speaker
coding, you know, a backend server function. No, I don't need that to be so your're mixed. Okay. I'm I am ship it.
00:44:02
Speaker
Because just because like I think when I think of like holographic interface, I think of like Jarvis from Iron Man, right? So, you know, he's got all this stuff floating up in the air, and he's interacting with it and everything. and um And of course, you guys can't see what I'm doing, but I'm mimicking that. But I like just for the fact that like with that type of interface, Iron Man was able or Tony Stark was able to like,
00:44:24
Speaker
Solve time travel after he was like, you know, washing the dishes and these like 20 minutes with Jarvis and figure out time travel. I want to figure out time travel. I got some stuff I want to do. You know, so that's why I would would say shipping if I can solve time travel with holographic interfaces in 20 minutes. Absolutely. I'm going to have to start calling you Hollywood. I think that's a good point. That's probably why we haven't figured out time travel yet. It is going to be the key. Yeah. Yeah.
00:44:53
Speaker
oh I just, I think I lost IQ points with that answer. You're welcome. May God have mercy on your soul.
00:45:04
Speaker
we are all now um There you go. Um, AI as an input mechanism. So like conversational AI. Yeah, I'm, I'm definitely, definitely ship that. I, um,
00:45:22
Speaker
Not just with your computer, like I think it's going to be really cool to talk to your computer and have it have it really know what you're saying in your old language, but all these other things in our life that like I feel like they're they were blocked by not having a good input mechanism, like a smart fridge.
00:45:35
Speaker
like this is This is my secret startup idea for all your listeners. like ah for it like Put a couple cameras in a fridge, connect it to chat GBT, and you can tell it when you're putting stuff in, and it knows what it is, and then it keeps track of it for you, and it tells you what to eat. This is not our biggest fight, but we always my wife always complains that I'm not eating the oldest leftover. like well i just If I could ask chat GBT, what am I supposed to eat today? And the fridge just tells me. like That would solve a lot of problems. Or what's in the back of the fridge? like have those Have those rear cameras. so it's like What's back there? I can't see back there. What exactly what's lingering back there or, you know, some kind of mold alert, let it have it. They have the recognition to say, Hey, uh, this is, this one's going bad. This is going to smell exactly yeah that spaghetti sauce that I forgot about back there. Moldy. Yeah. yeah
00:46:29
Speaker
James, how about you? I like it. Again, I think of Jarvis, you know just being able to talk to the computer and have it do things. I absolutely like like that. Now, i don't as a general computing, that's the thing I guess maybe that in the spirit of this question could have been. I think you know for assistants and those sort of things, yes, it's cool. But I don't know about it like and an operating system level, my day-to-day computing, could I just sit here and talk to my computer and have it, that's my only mechanism for input.
00:46:58
Speaker
Probably not, but for for assistance and those sort of things, for sure. Well, i I am a ship. it I don't ever use a single input um to my system. ah my My desktop is littered with different input devices. um And having one of those being conversational would be amazing. I i have played with voice recognition for my system since probably the late 90s to early 2000s, you know, like when, ah when PC talk, you know, I think, I think that's what it was called first came out or PC voice or something like that. I'm like, this is awesome. You talk into a headset, try and navigate your OS and stuff like that. and But it was never very good. So if we could have AI recognize
00:47:52
Speaker
uh conversational terms and and do what we want and provide suggestions and uh you know be able to modify prompts and things like that that would be just phenomenal so yes please please ship it please ship it i want it very cool how about brain computer interfaces like i'm thinking of like neural link or something like that what do you what do you think about this Oh yeah, I'm i'm definitely a ship of those. I, when I was, I don't know, maybe in high school or something, I wrote like a list of things that I wanted to work on. And that was, that's the only one I haven't accomplished yet. So I could, I can imagine going to work on that at some point. But, but yeah, I think that's just, it's going to be so cool to be able to think and interact with your machine. And I, there's, it just unlocks so many things that the safety aspect is what scares me. And like, I'm glad people are working on it and that it's not ready yet. But yeah, there's, there's a definite future there.
00:48:49
Speaker
Yeah, i'm I'm a hard pass on this one. um i I don't trust my brain enough to ah be able to run a computer, so I don't want it directly linked. I think that would cause all kind of havoc.
00:49:06
Speaker
Especially, you know, if you start getting into that space where I'm, you know, I'm, I'm thinking about 10 different things. I don't want 10 different windows popping up on my screen. That's just going to make multitasking that much harder. So yeah, I'm ah um a hard pass on that one. I'm with you on that. I don't, I don't think I want my, my computer to be like basically rendering what I'm thinking for sure. I think nobody wants that. Um, however.
00:49:34
Speaker
For brain computer interfaces, I think it's interesting from a, I would think like prosthetics. I think that's kind of one of the coolest uses of that where you're able to like, I can think, Hey, I'd like for my, my hand to clench into a fist and it actually does it. And but because my brain didn't, it's a prosthetic. I think that's a really cool use case for, for something like neural link or, or any of the other, are there any other companies really?
00:49:59
Speaker
doing that doesn't what I've ever heard of is neural link right now. I think there's a few others that are way less in your face about it. I gotcha. Yeah. But yeah, I think like you mentioned, I have my wife, I don't know if anyone have ever heard this, my wife says she always sends so many mental texts like in her head, she thinks about a text you want to send someone with and never has a chance to do it. So like even just that ability to be able to think, hey, text this person this and have it send, that would be so cool. Texting exactly what I'm thinking. Yeah.
00:50:26
Speaker
maybe if you have like an activation trigger it's like hey brain yeah absolutely yeah hit brain send you know don't yeah don't text exactly what i'm thinking immediately that's probably not a good idea but i but i like one thing that's cool to use like gtv getting things done in the whole idea of like capturing something and getting it out of your brain quickly if you could use and like a neural link to like jot something down very quickly and get it out like so you don't have to think anymore Ooh, now we're talking, man. Yeah, that would be awesome. That would be really cool. Yeah, the neural link thing is, it's interesting you mentioned that I initially wanted to go into robotics to do exactly that. Um, is to, cause to me, I was always, I always thought of the human body as like a giant an electrical system. And it may be one of my biology teacher's faults. You know, I thought, well, all you need to do is rewire it.
00:51:18
Speaker
And also, you know, Luke Skywalker's hand was really cool. So I wanted to do that. I'm like, why isn't that possible? That looks, it looks like you've got your Tony Stark. I got my Luke Skywalker. So that's fair. That's fair. All right. That's all the time we've got for ship it or skip it today. So we now we move into the most important part of the podcast. You know what you love it. The lightning round.
00:52:06
Speaker
So the lightning round works like this. James and I are going to quick fire questions at you left and right and you can't dodge these. They're going to hit you square in the face and you've got to answer each one. We don't need pontification or long explanation on your answers. Just whatever you feel is the right answer. You may or may not be judged on your answers just to give you fair warning and.
00:52:33
Speaker
With that, are you nat ready for the lightning round? Let's do it. Awesome. James, why don't you start us off because I don't have my question make open. i This one's a doozy. I've never used this one before. This is a good one. Why can't we tickle ourselves?
00:52:53
Speaker
Our brains just don't let it happen. Fair enough. It's not a yeah not a surprise. so So I noticed from your intro, you have a dog, big dogs or small dogs. Big dogs. Nice. What's the fastest speed you've ever driven in a car? Allegedly. Of course. Allegedly 95.
00:53:19
Speaker
Respectful. If everyone in the world had to get married when they reached a certain age, what age would that be? Ooh, let's say 40.
00:53:31
Speaker
It's so funny you answered that because I was just watching a TV show last night. It was an old episode of Reba and she asked her daughter, it's like, when is it appropriate for you know a girl to be active? And she said 40. She's like, did I ever tell you my favorite?
00:53:57
Speaker
um It's just for reminding me. Reba. that Okay. We're going to have to talk about that later. Okay. Um, from on a scale of one to 10, how hot do you like your showers? Oh, this is really depends on how I'm feeling that day. Um, in the summer, like sometimes even a one, uh, in the winter 10. Okay. There. yeah So there's an inverse relationship there. Um,
00:54:28
Speaker
ah Name a word that starts with the letter Q. Quiet. Name a word that starts with the letter Q. Well, no, not okay. Okay, ah do you know the definition of Triska Decaphobia? Triska Decaphobia. I don't know, something about ten-sided It's by an animal, by bugs, maybe 10 legged bugs. I don't know. That's my guess. I have no idea what it is. So is that the fear of the number 13? Is that trachydekaphobia? I don't know. I just, I just saw the question. I'll look it up later, I guess. and Now that's going to bug me. You should have had the answer ready for that one. Listeners, we'll have to come back next week. he try to We're going to test how many episodes you listen to.
00:55:24
Speaker
What sound does a seal make?
00:55:31
Speaker
Sorry, that is the incorrect answer. It's art, art, art, as we all know. art art art If there's a spider in your house, do you kill it or set it free? Neither. I just let it be. Okay, we would have also accepted that answer.
00:55:51
Speaker
Name something you could eat for a week straight. Eggs. So many things you could do with eggs. I like that. Exactly. yeah Our producer has shared with us that Triska Decafovia is an intense, international fear of the number 13.
00:56:09
Speaker
Who's the big brain in the room? That's right. Your Latin classes in high school paid off, I guess. It was more that part of my psychology class I really liked was manias and phobias. Okay. Because there's some weird ones. I don't know how many people suffer from that. I guess 13 is kind of a It's more of a superstition, but like apparently there're so there's a, there's a real like physical fear of it. Like they won't, oh they don't put 13 floors on buildings. Yeah. They don't put a floor. Well, they do now, but they used to not do it. But I guess if you have like this phobia, you won't walk through a door that has a number 13 in its number. So, or go to a house that has an address that has a 13 in it. Um, wow it could be.
00:57:03
Speaker
It can be really strange, but I mean, that's what, that's why it's a phobia. It's an irrational fear. Right. So is, is the technical definition. Anyways, I digress. Um, I think that's it. I think i think that's Yeah. The sponsors will only pay for 10 questions. Our, our, uh, our plethora of sponsors.
00:57:30
Speaker
So that concludes this episode of the Forward Slash, where we lean into the future of IT. Make sure you subscribe to this podcast for future episodes. I'd like to thank our guest, Nat Powlett, for talking with us about development at Microsoft and for being a gracious guest as in and supplying this with lots of funding for our podcast. I'm just kidding, there was no funding.
00:57:56
Speaker
I'd also like to thank my beautiful co-host, James, and all of our production staff that make this podcast possible. I'm Aaron Chesney. Thank you and stay curious.