Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
74: Lights, Camera, Action! (3D Computer Graphics: Part I)
781 Plays2 years ago
The world around us is a four-dimensional world; there are three spatial dimensions, and one temporal dimension. Many of these objects emit an almost unfathomable number of photons. As we developed as creatures on this planet, we gathered the ability to sense the world around us; and given the amount of information represented as photons, it is no surprise that we developed an organ for sensing photons. But because of the amount of photons that are involved, and our relatively limited computational resources, it is necessary to develop shortcuts if we want to simulate an environment in silico. So what is raytracing? How is that different from what happens in games? And what does Ptolemy have to do with 3D graphics? All of this and more on this episode of Breaking Math.
Recommended
Transcript
Introduction to Four-Dimensional Space
00:00:00
Speaker
The world around us is a four-dimensional world. There are three spatial dimensions and one temporal dimension. Many of these objects emit an almost unfathomable number of photons. As we developed as creatures on this planet, we gathered the ability to sense the world around us.
00:00:15
Speaker
And given the amount of information represented as photons, it is no surprise that we developed an organ for sensing photons.
Simulating 3D Environments: Challenges and Ray Tracing
00:00:21
Speaker
But because of the amount of photons that are involved in our relatively limited computational resources, it is necessary to develop shortcuts if you want to simulate a 3D environment in silico.
00:00:32
Speaker
So what is ray tracing? How is that different than what happens in games? And what does Ptolemy have to do with 3D graphics?
Exploration of 3D Graphics vs Gaming Graphics
00:00:38
Speaker
All of this and more on this episode of Breaking Math.
Gabriel's New Job in 3D Graphics
00:00:42
Speaker
Episode 74, lights, camera, action! I'm Sophia. And I'm Gabriel. And this is Breaking Math. We're going to do an episode on 3D graphics, so why this episode Gabriel?
00:00:57
Speaker
Because recently I landed a dream job of creating 3d graphics still an engineering job But I create models of trucks and planes and vehicles and I also create them in a destroyed state as though they've been in an accident and I'm learning all about 3d modeling and video game graphics and simulations and it's so exciting and
00:01:19
Speaker
And so it does sound pretty exciting because I know that emerging art and engineering is definitely one of your things.
3D Graphics Software: Tools and Favorites
00:01:27
Speaker
What cool things have you learned recently? Because you've been doing this for what, a couple months? Sure. Yeah. So actually, for our listeners' sake, what's interesting about this episode is I have been in computer graphics now. I landed this job last November, actually.
00:01:37
Speaker
And just so you know what software I use, there's some fairly well-known software that I use like Autodesk Maya, as well as Adobe Substance Painter and all of the other Adobe products like Photoshop and Illustrator. But I also heavily use a free software that you can literally download right now on just about any computer called Blender. Blender is actually at this present time my favorite.
00:02:01
Speaker
So I've done a lot of practical applications of 3D modeling. It's amazed me how it works. Like, you know, if something's supposed to be, say, you know, a round cylinder, I can make like a six polygon hexagon, but then I'll just right click it and I'll say shade smooth. And suddenly the computer knows to make the light bounce off of it, although as though it's perfectly smooth, even though it's still a six sided hexagon, if that makes any sense.
Sophia's Journey with 3D Graphics and Math
00:02:30
Speaker
Oh, yeah, totally. Yeah. It's one of the tricks you learn early on, I think, in at least you used to be in a light wave. Lightwave is this old dinosaur computer program that nobody uses anymore. I've technically done 3D graphics longer than Gabriel. However, it's been so long since I've done it that a lot of my knowledge is outdated. However, a lot of the math is something that I learned at the university.
Impact of CGI in Movies: A Look at Cameron's Work
00:02:58
Speaker
Yeah, so there's math underlying all of this, and this is the exciting part. Now, I want to just geek out about computer graphics real quick. I think I first got introduced to computer graphics when I was probably in second grade around the time Terminator 2 came out.
00:03:15
Speaker
and they just began to use computers heavily in graphics in movies. And I think that's a James Cameron movie, the same guy that did Avatar and Titanic. He did Terminator 2. He did one movie before that with some heavy computer graphics. I think it was called The Abyss. Did you ever see The Abyss?
00:03:31
Speaker
Uh, no, I did not. Okay, check it out. So in The Abyss, it's a movie kind of like E.T. or Close Encounters of the Third Kind where there's an alien species and it's far underwater. And this alien species has like a telekinetic ability where it can move water in any shape. And I've always been very curious, there's this scene in The Abyss where this water like lifts out of a container and like moves around like a tentacle.
00:03:58
Speaker
that's all done using CGI. And I always wondered if the ability to do that in computers came before the idea for the movie. And I think James Cameron did that. And he thought, wait a minute, I could do some really cool stuff in Terminator with this. He then made a he made the T 1000 Terminator, which is pure liquid metal and is able to turn into any shape. All done with computers, of course.
00:04:23
Speaker
Oh, yeah. And there was a lot of, from what I recall, a lot of advancements made during the movie, actually, on computer graphics. Specifically, I know that the way that they handled reflection with background maps had been done, but getting it to work in a composited environment was a challenge. Yeah. So we're going to talk about all that kind of stuff. But I guess we're going to also talk about, we're going to answer one question, which is, what is the best CGI that
Gabriel's Favorite CGI Scenes from Terminator 2
00:04:50
Speaker
we've seen?
00:04:51
Speaker
Yeah, oh, for sure, for sure, absolutely. And actually, I'll go and answer that now, if that's all right. I already said this before, but in the movie Terminator, back in my single days, that was one of my go-to conversations on dates, because I was always blown away by so many scenes in Terminator 2 for our listeners who haven't seen it yet.
00:05:08
Speaker
I highly suggest just go to YouTube and type in best scenes from Terminator 2 or just watch the entire movie. In this movie, you've got this villain that can turn into liquid metal. There's a part where he is in the floor, like a tile floor. He melts himself down into a tile floor. Then at the right time, he just rises out of the floor. And all of that was done with early CGI, but it was done so spectacularly well that it was just mind blowing.
00:05:36
Speaker
There's also a scene where he jumps onto the outside of a helicopter that's still in the air and he breaks the glass on the helicopter with his helmet. He then melts himself and as a blob melts into the helicopter and then reassembles himself.
00:05:53
Speaker
that is some of the most spectacular CGI even to this day and we're you know 30 some years later it was done so well. Check out Terminator 2 and pay careful attention to the graphics and how they do it. So for me that movie was the most impactful. How about yourself Sophia?
Pixar's Soul: A Blend of 2D and 3D Graphics
00:06:11
Speaker
Well, I think I'll just choose a recent one. Pixar made a movie called Soul, which on the surface doesn't seem like that much of a graphical achievement until you kind of really start watching it. And the kind of merging of 2D and 3D graphics that they needed for that movie plus some of the obviously new shaders, there's a lot to think about with that.
00:06:30
Speaker
What a fun episode because I feel like this episode can fit both with breaking math as well as nerd forensics where we're talking about both the mathematics that underlies something as well as the art and the emotional effect it has on us. So I just think that's really cool. That's really exciting.
Potential Guest Appearance from Nerd Forensics
00:06:45
Speaker
So maybe we'll have Millie from Nerd Forensics, another podcast, although more explicit than this one. We'll have her on to talk about, you know, maybe some moments in 3D movies and talk about the algorithms used to achieve
Mathematics Behind CGI: Movies vs Video Games
00:07:00
Speaker
them and stuff. But for this episode, we're just going to kind of give an overview of the mathematics behind both like movie CGI and video game CGI, which actually is different algorithms and approaches.
00:07:10
Speaker
Yeah, yeah, absolutely. Yeah. And I've got so much to say, especially what I've learned. Again, I'm only maybe, you know, what, six, five or six months into this. And I'm learning stuff every single day about how this works. So and also for our listeners, anyone who wants to try this, I said earlier in this podcast, you can download Blender for free. And there are countless tutorials on YouTube that you can
Listener Engagement and Contact Information
00:07:32
Speaker
do right now.
00:07:32
Speaker
nice oh and uh before we go on i'm going to just plug a couple of things you can email us at breakingmathpodcast at gmail.com you can follow us on twitter at breakingmathpod um you could donate to our patreon at patreon.com slash breakingmathpodcast and we do have a poster still for sale um we're uh if uh if in the past uh you've bought a poster but then had to cancel your order i um
00:07:55
Speaker
recommend buying now again we got our whole process together so yeah just if you had any issues with posters just email us and we'll see what we can get done for you but yeah it's a poster on tensors which are used in the general theory of relativity and it's just kind of a fun poster about how they work
00:08:16
Speaker
All right, so before in the intro we talked about how many photons there are and just kind of to talk about like how many photons there are If you have a 60 watt light bulb right of white, you know, it's just an average white kind of incandescent bulb There's eight point two one times ten to the 18th photons per second leaving that so if you think about that many photons per second if you think about them kind of like grazing your eye and
00:08:39
Speaker
It doesn't actually take that many photons for something to be visible. If it's really, really dark, sometimes only hundreds of photons are entering your eye if you're in the middle of just starlight in the middle of a dense forest or something. And so it really shows how few photons compared to the outside get into your eye. And it also shows how you can't really compute this, right? You can't just scatter a bunch of light and then have it bounce around a virtual scene with all these photons.
00:09:06
Speaker
Because even if you reduce that amount of photon into like the billions, you're probably going to get a low resolution image, right? Yep. And so how did the eye kind of evolve? I think we'll kind of might give us some clues on how to simulate these algorithms. Does that sound good? Yeah. Yeah. In fact, I think there's a really good spot to start here before we get into the math of computer graphics, actually talking about how the eye works and therefore it gives some context to what a computer is trying to do or what we're trying to do with a computer and making the graphics.
00:09:35
Speaker
Oh, yeah, definitely. So let's talk about light sensing animals. Euglena is a genus of single celled flagellate eukaryotes. It's just a single celled little animal. So it has a light sensing organ, but it doesn't it doesn't give any information except for brightness, right? Yeah. Yeah. So it's like, am I in a bright area or am I not? It's basically a one pixel animal. Yeah. Yeah. That's actually so it's awareness of its surroundings, as you said earlier. Yeah. It's just is it brighter or is this darker? That's that's all it's got.
00:10:05
Speaker
Yeah, and that kind of gives us a clue to some of the first stuff we have to simulate. So if you think about a point light source, meaning kind of like a light bulb, an idealized light bulb in every direction, if we're just simulating 3D graphics for Euglena, all we have to do is calculate the distance from the light, square it, invert that, multiply it by how bright the light is, and get an approximation for how many photons hit this animal. Yep. So that gives us a clue that brightness is already part of the equation.
00:10:34
Speaker
So then we're going to take a look at cup eyes, which are used by planarians. Planarians are like these little, um, kind of like flat, like teeny tiny little worm creatures. Um, a lot of drug research is done on them. The bald scientist podcast, we interviewed him on touring rabbit holes. Uh, he works with planarians all the time. That's his favorite, um, worm, I believe. Oh yeah. I would love to have him on this show too. Yeah. Yeah. I'm sure he'd love to be on.
00:10:57
Speaker
And yeah, so the planarian, the reason why it has cup eyes is because if you put your hand in front of your face right now and you rotate it around, you're going to see that the place where everything is dark kind of changes. And if you take something kind of a brightly colored and put it near your hand, you'll see that your hand kind of turns into that color. So that's kind of how cup eyes work. They give you a general sense of direction, but pretty much nothing else. No clarity, nothing like that. Very, very little clarity.
00:11:24
Speaker
What's got so brightness and direction, correct?
00:11:27
Speaker
Brightness and general direction. So it's like very low resolution. So what this tells you is that we're going to have to talk about the direction that the light is coming from, right? Of course, now instead of having just a minimum of one cell to light, now we have a whole patch of cells, right? And that evolves to become a pinhole eye. What happens is it becomes enclosed like a cavity within the animal. An animal that has this kind of eye is called a nautilus.
00:11:57
Speaker
And the way that this works is by simulating a pinhole camera. So think about this. If you have a red object to your left in front of you and a blue object to your right in front of you, the light from the red object is going to the back right part of your eye, right? And the light from the blue object is going to the back left part of your eye. Yes.
00:12:16
Speaker
And let's now let's say we increase our pupils until they're the size of our entire eyes. So we have some freaky open eyes as we increase this more like like at first only the light from the red from the blue pupil can get to the spot in your eye through basically one path, right? Yes.
00:12:33
Speaker
But as you increase that size, the amount of paths increases. Think about it kind of like taking a bundle of spaghetti and kind of like letting it kind of go loose in every direction. So it's kind of like two cones on each side. So that's why things that's why things get blurry when you have larger pupils, for example.
00:12:49
Speaker
And so that's kind of like what you're simulating with a pinhole eye. And then finally, we have lenses. And first, we're just a protective measure later filled up with fluid, differentiated normal evolution stuff. And what that did is it changed the field of what we could see, right? So for example, if light were going to hit a spot next to our pupil, it might hit the lens instead and then reflect back into our eye, giving us a greater range of vision.
00:13:17
Speaker
Yeah, that and the pinhole eyes tell us that we're going to will the pinhole eyes tell us that we're going to need to know where an object hits on the hits on the screen and the lenses tell us that we're going to something is going to be something that the lenses point towards us simulating the rays of light, right? Yes.
00:13:33
Speaker
So without further ado, I think we should go into ray tracing and video game graphics. Sure. Now, obviously, there's a whole lot with respect to our vision that we didn't even get into as well as the vision of other animals, things like rods and cones and differentiating the color palette that we're used to.
00:13:52
Speaker
Oh, yeah, actually, you know, it might be a good idea to talk very briefly about cones. Sure. So red, blue and green cones are used to sense the world around us. But so if you think about it, a computer screen when it's showing yellow, it's showing red and green.
00:14:07
Speaker
because yellow light triggers red and green receptors about the same amount. So if you project red and green light into your eye, your eye is going to see it as yellow. Which means that if you have one of those LED bulbs and you set it to yellow, if you have a red object in that room and you have a green object in that room, the red object is going to look red and the green object is going to look green.
00:14:25
Speaker
However, if you have a light from a sodium lamp, a specific type of sodium lamp, I think it's a sodium lamp called monochromatic light, or if you take a prism and you isolate the yellow and then use a lens to scatter that around the room, the green object and the red object will now appear yellow. And the reason why is because it's not fake yellow light anymore, it's real yellow light. And this actually does come up in some more intense computer graphics stuff that we might talk about later.
00:15:05
Speaker
Rasterized graphics. Yeah, and I mean rasterization really is kind of like the the last step in video game graphics But it kind of gives a clue to how it's used. Yeah. Yeah, so ray tracing is a term It's actually an older term in the history of graphics and it's made a comeback lately There's a couple of graphics cards by one of those by Nvidia and one of them is by AMD. Yeah, very good. I
00:15:28
Speaker
So the term ray tracing is actually a very old term. However, it's made a comeback recently with some very advanced graphics cards by companies like Nvidia, their RTX series was AMD, their higher end series that are capable of modern ray tracing. We're going to talk a little bit about that, but essentially it's getting a computer to simulate what we experience as light and reflection and therefore makes video games look way more realistic.
00:15:55
Speaker
Oh yeah, for sure. So yeah, speaking of that, we're going to talk about how a wrong theory in physics led to a good formula for ray tracing. I love this one. We've actually done entire episodes of this podcast on wrong theories and physics, but one of them is making a comeback because it's so used. I mean, I shouldn't say the theory itself is useful for describing the world, but it literally describes how computers simulate the world. So yeah, it's, it's quite interesting.
00:16:19
Speaker
Yeah. And I wouldn't necessarily call it a comeback. I mean, it's been here for years since the second century. When did the term ray tracing first come out? What's your familiarity with that term ray tracing?
00:16:28
Speaker
Well, my familiarity with it is that in rendered computer graphics, right, like where you're not doing it live when you want more detail and control, ray tracing is what you're going to use, basically. And the reason why is because as you'll see, computer graphics makes even more a broad kind of generalization. So for example, area lights are something that is relatively easy to do in ray tracing. So think of fluorescent lights under a diffuse panel, for example, and offices, for example.
00:16:57
Speaker
That's an area light. So, uh, but rendering an area light, um, in ray tracing is we'll get into exactly what ray tracing is. And we'll talk about why that is. Yeah. And so, like I said, uh, second century, right. Uh, Ptolemy was, uh, creating a book on optics. And I'm sure, uh, Ptolemy noticed that if you, if you put an object, right, let's say a coin behind another coin, if they're both in line with your eye, one is going to be behind the other, right?
00:17:22
Speaker
Yes. So it makes sense that there would be some kind of mechanism for every direction. So the top part of your eye might have a different mechanism than the bottom part of your eye. It's not an illogical theory, right? Correct. Yeah, I'd give them that. Give them some credit. It's not illogical.
00:17:39
Speaker
And so what this theory was was that the eye has a bunch of these little rays within it that go out from the eye and bounce around the world and then report back to the brain. That's amazing. Yeah. Yeah. And I thought that was amazing. What's also amazing is I wonder if there's any relation to seeing light up eyes, you know, to that being embedded in our subconscious or something. My guess for that actually is that it comes from a predatory cats because the fovea of a cat will reflect in light.
00:18:09
Speaker
OK, OK, yeah, because as soon as you see red eyes light up, you know, uh-oh, the bad guy is going to get us. I'm talking about any sci-fi movie or cartoon or whatever.
00:18:18
Speaker
Oh, yeah, I mean the the like just yeah, like Titans the Titans and that one attack on Titan. No, no, not attack on Titan. The the one with Hercules Hercules. Okay. Sorry. The one with Hercules. It's Hercules though. But um, but yeah, so this this was actually refuted by Hassan ibn al-Haytham in a seven part book of optics called Kitab al-Manazir
00:18:40
Speaker
And it was like the first refutation, it was during a scientific sort of philosophical revolution in the Islamic world that happened in the, I believe this is like 10th or 11th century. So before it was debunked, people, or rather this guy thought that eyeballs were like little flashlights. They literally sent out rays of light from each eyeball.
00:19:01
Speaker
Well, not even raise a light because raise a light implies that the raise like go into the eye. It was the idea that light exists separately from eye rays and then that the eye, that the eye being a sensing organ sends out these things into the world, which report back. And I mean, if you think about it, it's not the most like, uh, like ridiculous thing, right? Because I mean, like if you look at, if you look at like, you know, a tree growing over time versus like a blink, it's easy to think, okay, maybe there's these natural phenomenon that are extremely quick that have to do with something as
00:19:30
Speaker
you know, uh, nebulous is sensing, right? Yeah. Yeah. I guess you're right. All right. Again, again, I shouldn't talk too much smack about what people did or didn't know back in the day, because you know, right now, what do we know or what don't we know that we'll be made fun of, you know, in future years. So, Oh yeah. And that's cool about this, this method of ray tracing though, is that it totally works for simulating the real world around us, right? Yes.
00:19:50
Speaker
So, like, let's say you look at a mirror, right? Yes. The rays bounce off of the mirror onto the scene around you. And then it's just like those rays in Newton's optics, where it just talks about, like, you know, apparent size and all that. If you want to know what I'm talking about, just go look up optics on Wikipedia and look at apparent size.
00:20:09
Speaker
But yeah, so the cool thing about this is basically for each pixel you send out a well, I mean the simplest incarnation you send out one ray for each pixel and So let's let's talk about how this works. But before we talk about how this works, we're gonna have to talk about the intersection of rays and primitives.
00:20:26
Speaker
All right. So, um, Gabriel, um, walk us through a raise and what they are in vectors. Oh, for sure. For sure. Absolutely. Okay. So, so raise. Okay. Okay. So a Ray is like, uh, it is, if you go to the mathematical definition from sixth grade, it is an infinite line that has a starting point, or you could also think of a Ray gun like, you know, uh, it's something that has a start and it has a direction and no end necessarily.
00:20:52
Speaker
Yeah, and the start and direction are both represented by triples, right? I mean, one is technically a point and the other is a vector, but they're represented pretty much the same way in the computer, right? Yes, yes they are. And so the rays that we do for a ray tracing thing is, so the way that this is done is you pick, you draw kind of like a square in space, kind of like where you're going to look out from.
00:21:12
Speaker
You could think of this as like kind of like if you if you take a pic if you've ever like taken a picture But before you kind of put your fingers and L's and put them at opposite corners of a box It's sort of like that you set up a bunch of points in a grid like that Yes, and then you have all the rays start at some point behind here So behind the square that all go out to the different points on the square So that's how the rays are defined. Does that make sense? Yes
00:21:37
Speaker
And so we send all these rays out. But how do we know what it intersects with, for example? So obviously, with respect to what we see and how our eyes work and what our eyes detect, that's one thing. But then how computer works, that's a whole other thing, which we'll get to an exciting point here in a little while.
00:21:53
Speaker
Oh yeah. And let's, let's just think mathematically, right? So let's talk about some primitives. Um, there's a couple, there's a few types of primitives. A common one though is a sphere, uh, because you could technically render, um, a scene with a perfect sphere in it. So a sphere is, um, so how do you define a sphere, Gabriel? Oh gosh. Um, a 3d object that of a given length where every point on it, it is the same distance from the origin.
00:22:16
Speaker
Yeah, you got a center, right? And you got a radius. Yes. And so you can define the points of a sphere as all P that are in S such that P minus C, the length of that is equal to R. So now let's derive a ray sphere intersection, right? So Gabriel, if we have if we know that the that that array is
00:22:35
Speaker
The ray is a plus BT right for some or some starting point a yes in some direction B and T is greater than zero, right? Yes So those are all the points that the race weaves out. Okay, so now if we know that array intersects with a sphere, right? It means that it's going to be it's gonna satisfy the formula for a point on a sphere, right? Yes
00:22:56
Speaker
So what we do is we set p to a plus bt. All right, so from here, what we do is you take the point. So you have this p minus c is equal to r, which means that a plus bt minus c is equal to r. And that's the point. So now we need to solve for t, right? Yes. So that's a minus c plus bt is equal to r.
00:23:23
Speaker
We're putting A minus C in parentheses. With me so far? Yes. Oh, absolutely. Yeah. Yeah. I mean, algebraically, just BT equals R minus in parentheses, or rather plus, because you've got to subtract it from both sides. Just algebraically isolate T.
00:23:40
Speaker
All right, so we got magnitude is equal to radius, right? Yes. So if we square both sides, we get A minus, so we got the whole A minus C quantity plus BT squared is equal to R squared. Or minus R squared is equal to zero.
00:23:58
Speaker
So now we're going to set D to A minus C. So we got D plus BT just for E, so we don't have to keep writing A minus C. So D plus BT squared minus R squared is equal to zero. So all we do is we plug that into Pythagorean formula. So what is our A term there? OK, so the A term would be the coefficient of D squared. B squared actually in this case. Oh. Because we're solving for D for the right dimension.
00:24:26
Speaker
Oh, of course, of course. Okay. Yeah. So, so the coefficient of, um, so for the Pythagorean formula, we've got your, your AB and Z, right? Check it out. It's D squared plus two dBT plus B squared T squared minus R squared equals zero. So the Pythagorean formula, our little A is equal to B squared, right? Uh, let's see here. Okay.
00:24:45
Speaker
So that would mean our B would be equal to? Two dB, okay, right. Two dB, and our C term would be equal to D squared minus R squared. Yes. And yeah, so you just plug that into the Pythagorean formula, so you get negative two dB plus or minus the square root of four D squared B squared minus four B squared times quantity D squared minus R squared all over two B squared. And we're not gonna talk about what that is, but basically, you get the two points that intersect the sphere, right? Yes. And do you know why two points?
00:25:15
Speaker
Uh, because if it's a Ray, well, if it goes all the way through it, it's going to be two points. Yeah. It's exactly tangential, which would be technically one point, but yeah. So if you're shooting, like, uh, if you're shooting a, uh, um, uh, basketball with like your Ray gun that you were talking about earlier, it'll hit it in two places. Correct.
00:25:31
Speaker
So yeah, so we got that data. Any idea of how we would get the normal vector where it hits? That's a really good question, actually. So the normal vector obviously is going to be the perpendicular point to any point on a sphere. So every point on a sphere, by definition, will have its own normal sticking straight up. So if you're going from the collision, I'd think that if it was a straight on collision, you wouldn't detect anything because your collision point is the same as the normal. But if it's any other point,
00:26:00
Speaker
Oh, you were very close. But you said was true if you're looking at the point that goes directly from the ray to the center of the sphere. Oh, yeah. But if you do it on the side, all you got to do is take the point of intersection, subtract the center of the sphere from that and normalize that. And that's your normal. OK, normal. Yeah, just normalize it. And then that's your normal as in the mathematical normalizing it. What do you think is too many? Because it's a normal thing to do.
00:26:26
Speaker
It is pretty normal, you know, I mean I did it you did it I'm sure everybody can know everybody's gonna normalize using these normal vectors normalize using normal vectors. Can we tweet that? Yeah, we could do that. I think actually I did that I think so. I think that might have been one of the dorky tweets from last month. Yeah at breaking math pod
00:26:43
Speaker
Yes. So what can we do with the data? We use a normal for reflection, and we use the point of the intersection and the rate from there. We actually use the normal for refraction, too. So Gabriel, what's refraction versus, what is refraction versus refraction? Physics 3 from college, my goodness. I blazed through that class. I'm sorry. So it was reflection versus refraction? Yep. OK. So refraction, think about it this way.
00:27:10
Speaker
If you are going through water and you see that a light ray shifts all of a sudden, that is refraction. Now, that's what you see, but that's not the technical definition. And I want to say that it has to do with the angle that it hits. Yeah, I think it's like tangent of theta 1. Oh, here it is. I looked it up. So the incident index, so that's like, for example, for air, it's like very close to 1, 1.001. For water, it's like 1.6.
00:27:38
Speaker
Oh, that, oh, that brings me back. I believe 1.6 is right on the head. Yep. Yeah. So if you, so if you wanted to see, um, like what, uh, how light would change when it goes into a lens, you take the incident index, which is basically one times sign of how it, uh, the incident angle and the incident angle is just the normal, I mean the, the angle from the normal. And, uh, do you recall how to, how to calculate the nor the angle from the normal? Oh gosh. I'll give you a hint, a dot product.
00:28:03
Speaker
Okay. I don't, I don't, it's been too long since it says, sorry guys. I'm still podcasting for five years out of grad school. And it's been a little while. Yeah, there's no worries. Uh, there's a, the basically, well, the coast side of the angle between the vectors times the length of the, of the vectors is equal to, um, the coast line. Yeah. You can calculate the coastline with it. So if you have two, two vectors pointing in the same direction, their dot product is going to be the same thing.
00:28:25
Speaker
Which is like a cosine of zero, which is like one. If they're pointing 90 degrees to one another, their dot product is going to be zero, which means that their cosine is going to be 90 degrees. So you can actually use that to calculate this using Snell's law, which is n1 sine theta 1 equals n2 sine theta 2. And if you notice, light will kind of like, it'll kind of straighten out with respect to the normal, right? Yes. As it goes through, but when it leaves the light, it does the opposite.
00:28:51
Speaker
And it turns out that that doesn't only work for light, but these imaginary light rays for the eye. It works perfectly, too. You can simulate a lens perfectly with this. So going back to that example with Minecraft, we said that original Minecraft had two maps. One of them, I think, was for the opacity and one of them for color. I don't even think they had a map for displacement that would cause a light refraction if it went into the water. I don't recall that had that or not.
00:29:19
Speaker
Oh, no, I honestly doubt they do because, um, so, so basically Ray, and then the, for a reason, I'll explain real, uh, real quick. So just like finish up Ray tracing, um, you, if a light hits an object, you look at how far it is from the closest light, um, or actually all the lights in the scene and, uh, kind of like add together the inverse square law. Um, and, uh, that's your new, um, light amount.
00:29:41
Speaker
Yes. And also, I want to talk real quick about blurry reflection. The way that that's done is when a light hits an object, you scatter tons of new rays out. So one ray becomes like 1,000 or whatever. We'll put not that, but like hundreds of rays or dozens, right? Yes. And that's how you get things like really kind of like smooth mapping, like more photorealistic kind of stuff. It's increasing the amount of bounces that something goes around the scene.
00:30:11
Speaker
Yes, yeah. And this is contrasted with real time graphics. Well, in the way that that works is that objects are represented by vertices and polygons. And these are kind of objects in like, you know, you create a scene with these polygons and stuff in it, which we'll talk about how to do in a second. But basically what you do is you map them into the map, you map them into the camera's field of view by using a matrix.
00:30:35
Speaker
So you take everything you squish it down so like so it would so you know Like when you're a little kid you might think that the mountains are kind of small you do that one with math basically makes far away things small and Close things up big and you just kind of smoosh everything down together So suffice it to say it's harder to do things like refraction with that kind of thing because then you have to set up new virtual cameras Which is how it's done in modern computers because virtual cameras don't really cost much to spin up but it required a lot of like complicated like
00:31:05
Speaker
A lot of innovative algorithms to do the same thing with real-time graphics that you do in ray tracing Because ray tracing will always come out a little bit ahead as far as that kind of stuff goes but yeah, you're talking about a Minecraft and how and the differences between real-time and raster graphics exactly
00:31:21
Speaker
So this is an exciting thing. And again, this is where it'd be helpful to have a YouTube video for a podcast. For our listeners, if you get a chance, just check out Ray Tracing in Minecraft. In fact, for all you Minecraft players out there who happen to have one of the newer graphics cards that can handle Ray Tracing, so something like a
00:31:39
Speaker
Nvidia RTX 2060 or 2070 or 3060 or above any of those things or even the AMD equivalent if you go to if you open up Minecraft and just search for Nvidia there are some worlds that are made with ray tracing enabled and they have not only the two maps I mentioned earlier I think they have a total of I want to say six maps and
00:32:02
Speaker
where the other six all tell the light what to do. And when you play the game, it's just spectacular because as you walk by a metal object, the light bounces off of it and it looks absolutely shiny. You see your real reflections in things.
00:32:18
Speaker
when you walk by water at night, you'll see the light beams from the moon and it reflects on the water and as the water moves, so does the reflections. So it's an eerie, I almost want to call it an uncanny valley effect, but it is like it's almost hypnotic to look at something with such low polygon counts, such blocky graphics.
00:32:41
Speaker
while at the same time having textures that are convincingly hyper-realistic. It's really magnificent. Sounds good. And the reason why modern GPUs can do that, by the way, is because if you think about ray tracing, we could send out each ray at the same time, right? Yes. And you'd get the same scene, because the scene is essentially static, basically.
00:33:04
Speaker
And so that means that if you have these new graphics cards that both go quickly and have a lot of the way the graphics card works, it has like thousands of little computers inside of it. So if you can all do if you do all these thousands of calculations at the same time, that's how that works, basically.
00:33:19
Speaker
Yes. Yeah, exactly. And any, any of these worlds, basically, as we've said thus far on the podcast, uh, the, the, the computer calculates where the source of the light is from. So if you're outdoors, your main source is going to be the sun or the moon. Uh, if it's, um, nighttime, and then from that it calculates both the, you know, the, the strength, uh, you know, the, the intensity and the direction of the shadows and also the reflection. So if you've got wood, you're not going to have a reflection, but again,
00:33:47
Speaker
as you walk by any kind of metal material, you can have reflection. Also, Nvidia has a website where they literally encourage users to create their own ray traced and labeled textures. I was talking to my daughters about making some lightsabers that emit light in a really cool environment where you see the reflection of your own lightsaber in the environment around you. We just thought that would be the coolest thing in Minecraft.
00:34:11
Speaker
Oh, yeah, for sure. Yeah, definitely. And that's a cool thing as graphics gets more and more kind of like advanced. We're going to be seeing these we're going to be seeing like, you know, these these developments of stuff looking more realistic and then perhaps later looking less realistic kind of like you saw with computer graphics from the 80s to the where they become stylistic as well. Yes. And so we talked a little bit about how real time graphics works, but let's kind of break that down a little bit more.
00:34:38
Speaker
So the way that it happens, you have a graphics pipeline. So first you kind of take your objects and you morph them, do whatever you're doing to them. So for example, if you're doing like a game where somebody's walking, you deform what's called the T pose, which is arms out to the sides facing forward. And you deform that using a rigged model, which we'll talk about that a little bit more next time, but you deform it and you have this new set of points, right?
00:35:01
Speaker
Let's say we have a camera, right? And we're looking at this object and this object is moving forward. How can we, um, the way that we represent it moving forward is actually with a matrix. So this person, this person, uh, this character would be walking kind of like the way that they see it is in place. Yes. However, they're going to position matrix is moved, um, as they walk, which causes the illusion of walking. And then you, when you multiply that by the camera plane, um, you get a projection onto a screen coordinates. Yes.
00:35:30
Speaker
So you know where all the polygons are. And then finally you do clipping. And what clipping is, is you take all these triangles and you see if they intersect one another. And if they do, you kind of clip them, you kind of trim them to size, right? Yes. And you also clip things really close to the camera and really far away from the camera to minimize what you do.
00:35:50
Speaker
If you ever want to see a really great illustration of all the concepts that you're talking about, just do a search on really bad CGI or beginner CGI. Since I've been modeling myself, I've joined a number of Facebook groups all about CGI, and people will often, as a joke, put their first ever animated rig. And it is so poorly done, but it's obvious that the steps that the character is taking don't at all line up with their movement through the world. So it's quite interesting. Oh, yeah, for sure.
00:36:21
Speaker
So let's say we have a bunch of objects in the scene right in the video game. How do we know which polygons are in front of? Well, actually, let's just say we're doing a ray tracing scene. We're ray tracing a scene with let's say a million spheres. And let's say we're doing my first ray tracer program. How would you check which sphere a ray intersects with?
00:36:39
Speaker
Okay. So with, uh, again, I'm assuming, and again, I hope I'm, I'm understanding you, you correctly, but all the information in terms of placement of the objects and the size of the object would be, it would be put in by the original programmer. Right. Oh yeah. Yeah. We're imagining that we have this, we have like a million objects stored in, in memory and they're all within the view of the camera. Um, it's in certain, somewhere in front of others, et cetera. How would you tell, uh, which object, um, array, um, hits first?
00:37:04
Speaker
Okay. So, uh, probably just from the, from the player standpoint, probably the intensity. Well, I'm just talking about, um, we're talking about a rendered scene here, actually. We're talking about like, you know, send, you send the Ray out. How do we know which object to intersect with? Cause it might intersect with more than one, right? Okay. Yes. Yes. So what would you do if you were a Ray Tracer?
00:37:25
Speaker
Okay. So you mean if I was the one who, okay. If you were a ray tracing program. Okay. Okay. So, so based on the instructions given by the programmer who made the scene, you know, there, there, there's going to be a database or a matrix that's going to say, you know, exactly like with a, it'll have a numerical value with how far apart things should be. And like, that's going to inform how the ray tracing is done and is.
00:37:47
Speaker
Yeah, that's part of it, but if you think about it too, like when you're writing this program, you're going to have to check every object, right? Yes. Because you're going to have to check, does it intersect this object? Because the computer doesn't know that off the hand, right? Yep. And that can take a long, long time because you got to check not only which objects intersects, which you got to test every object for, but which is the closest. So you have to save all the intersections and get the one that's nearest to the camera, right? Yes.
00:38:11
Speaker
So the way that this is handled is with something called bounding volume hierarchy sometimes. I did not know that. That's a new one. And it's a really cool algorithm. So what happens is, let's say you have these million objects, right? Yes. About half of them are on one side, about half of them are on the other, right? Yep. So we could do two boxes and calculate which objects are within this box, which objects are within the other box.
00:38:35
Speaker
and which objects are perhaps within both boxes. And then we divide it again, but in a different axis. And we divide each box so it contains exactly half the objects. And we keep doing that until we have this tree in memory. So now, if we wanted to see which object array intersects with, we send it out, and it hits the bounding volume hierarchy. So yes, we know at least one object is going to be made, right? So if I'm understanding you correctly, the bounding box hierarchy is a computer's way of just kind of eyeballing it.
00:39:02
Speaker
Yeah, pretty much. Oh, I get it now. But it totally is a way of eyeballing it, because that was good. Because it's kind of what you do naturally, right? If somebody told you to find something, you could be like, okay, is in this cider at that site. So you send out this ray now, and now it has a check. Okay, does it fit in the left half million or the right half million? And then there, it's like the front 250,000 or the back 250,000.
00:39:29
Speaker
Then so basically all you have to do there is check about 20 20 pairs of objects before Seeing what it intersects with this piece of light fascinating Wow Wow, okay cool. That's that's yeah, that's that's exactly a Computational eyeball on it. Oh, I know. Yeah, that's what I really liked about computer that we really like about computer science It's the science of being lazy
00:39:55
Speaker
We've discussed the algorithms that are used for the two primary forms of computer graphics, one type for realism and the other for speed. Both of these algorithms rely on simplifying the way that light actually interacts with our eye, both over space and time. This was one of the dreams of Ada Loveless, one of the first computational mathematicians. On our next episode, we talk about more specific algorithms for computer vision, and sometimes that films and video games have driven these advancements.
00:40:20
Speaker
I'm Sophia. And I'm Gabriel. And this has been Breaking Math. Email us at breakingmathpodcast.gmail.com. Support us at patreon.com slash breakingmathpod. Tweet at us at breakingmathpod. And also Gabriel, you had a few recommendations for resources. If anybody feels gung ho after this episode and I have a couple too.
00:40:40
Speaker
Oh, my gosh. Yes, yes, I do. I do. So first of all, I'm working with a lot of my nephews, as well as one of my high school friends, 14 year old son. Everybody is interested in computer graphics right now, mostly because of me. Blender is free. It's freeware and it's fantastic. So I would say it's industry. It's industry grade.
00:41:00
Speaker
Yes, it is. And not only that, but there's so many websites that will offer one free course and then they'll offer a lot of paid courses if you like the free stuff. I did the entire free course available at cgboost.com. I don't mind promoting them because it's free, you know? And I did the course where we make a 3D model of a bunch of apples and a bucket and I showed that to you last time you were at my house.
00:41:24
Speaker
Oh, yeah, for sure. Yeah, it's amazing. And you literally like like you take actual photos of apples and you map them onto a 3D Apple that you've created. So it actually looks photo realistic. Then you add a bump map to it so that like the Apple peel is like uneven and bumpy like an actual Apple. It's amazing.
00:41:43
Speaker
Does that sound cool? Yeah, yeah. So just I would download if you're looking for a hobby this summer, you know, just download Blender and go to YouTube and type in type either sculpting in Blender or make something in Blender. I saw somebody do an Iron Man mask in Blender. It was really then 3D print from Blender. You can do that.
00:42:03
Speaker
Oh yeah, and if you want to look up some of the ways that people have advanced the state of the art, I would always check out SIGGRAPH papers. There's websites on the high seas where you can find it. You could also email the authors and they could take their time, but they usually will send you.
00:42:21
Speaker
paper and a lot of times if you go to their website too which there's often these computer graphics people have websites you'll find sometimes models sometimes programs sometimes C code you know whatever to kind of interact with this stuff and there's whole communities of blender plugins, shaders, models, etc. All right sounds good oh and also for next time we will have a discord set up for breaking math fans so until next time lights camera action or something I don't know how to end this episode