Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
Future of Social Interactions: Stephan Bugaj | Chief Creative at Genvid | ex-Pixar | Sophia Robot Co-Creator
72 Plays3 months ago
Stephan Bugaj, Emmy-winning creator, ex-Pixar Technical Director and currently Chief Creative Officer at Genvid. Stephan worked with Amazon, Paramount, Fox, Warner Bros., Sony, and many others creating transmedia projects spanning feature films, television, video games, interactive and virtual reality experiences.
Now Stephan creates interactive film experiences like Silent Hill Ascension and DC Heroes United where community impacts how the movie plot develops based on social interactions like voting, meta play and other interactive tools. Recently Stephan and production teams received Emmy in Emerging Media category for their work on Silent Hill Ascension and currently they are working on a new interactive series DC Heroes United.
In this podcast episode we talk about future of social interaction, role of AI in personalised content, about interactions with robots and how such interactive and futuristic experiences are created.
Part 1: About Silent Hill Ascension, EMMY award for Emerging Media and future of social interactions
Part 2: Traps of hyper-personalised content and role of AI
Part 3: Design for Emotional connect with audience
Part 4: Sophia Robot and AGI: when we will get Westworld experience in real life
Part 5: What’s next?
DC Heroes United: https://dcheroesunited.com
Genvid: https://genvid.com
Stephan Bugaj: https://www.linkedin.com/in/bugaj/
Alex Babko: https://www.linkedin.com/in/alexanderbabko/
This is a personal podcast. The views and opinions expressed here are only those of the author and do not represent those of any organization or any individual with whom the author may be associated, professionally or personally.
Recommended
Transcript
Introduction to 'Month in Millions'
00:00:13
Speaker
Hey, welcome to Months in Millions, a practical and inspiring show about innovators and emerging tech practitioners. I'm the host of the show, Alex Babko. The worlds of movies and video games converge and create a new emerging media.
Interactive Films as a New Entertainment Frontier
00:00:29
Speaker
I'm talking about interactive films as a new frontier for entertainment.
00:00:33
Speaker
Early examples like Black Mirror, a Netflix interactive series with Bear Grylls started to explore this new dimension, but the opportunities for a custom, on-demand experience or a piece of art co-created by a community are way broader. These examples show how audiences can take control of the story, blending the emotional depth of cinema with the engagement of video games.
00:01:00
Speaker
And this isn't just a new way to watch, it's a new way to experience.
Meet Stefan Bugaj: A Transmedia Innovator
00:01:04
Speaker
The choices you make directly shape the narrative, creating endless possibilities. And today, I invited a world-known creator of interactive movies straight from Hollywood to take about this immersion media. My guest today is Stefan Bladimir Bugaj, Emmy-winning creator, ex-Pixar technical director and currently chief creative officer at Janvid.
00:01:29
Speaker
Stefan provided creative direction overseeing the production of various transmedia projects, expanding feature films, TV, video games, interactive and immersive experiences for companies like Amazon, Paramount, Fox, Warner Bros, Sony and many others. and Currently, Stefan creates interactive film experiences like most recent Silent Hill Ascension,
00:01:53
Speaker
where viewers, as community, they impact the course of action in the movie series. And today we will talk about immersion media of interactive films, role of AI in personalized content, and how such interactive film experiences are created.
AI's Role in Personalized Interactive Media
00:02:09
Speaker
Hi, Stefan. Hello, thanks Alex, thanks for having me. Thank you, wonderful. so ah Stefan, ah thank you for coming to the show, first of all, and I'm very much excited about our upcoming conversation, especially since you are a very unique example of a guy who combines Hollywood and Silicon Valley hats.
00:02:29
Speaker
And recently, you and your teams from Genvid Entertainment, Konami, Digital Entertainment, and Red Robot Games worked on an interactive film experience, Silent Hill Ascension. And can you introduce this emerging media to our listeners who are not that familiar with the idea? Yeah, absolutely. So what we are doing at Genvid, um we call either Miles, massively interactive live entertainment or socially interactive streaming series.
00:02:58
Speaker
And the idea is sort of a hybrid between mobile gaming and interactive television, where interactive television, I think that most people would be familiar with are what you mentioned in the intro, Bandersnatch or snatch or or the the stuff that Netflix did with ah Teen Titans Go or ah Captain Underpants or whatever, whatever they they're Bear Grylls. What we're doing is through an app that we create,
00:03:27
Speaker
The entire audience socially is able to influence the story decisions like you would in something like Bandersnatch, which by the way, is based on technology from Telltale Games, ah where I worked back in the day also on a Second Seas of Walking Dead, Game of Thrones, Wolf Among Us, and Tales from the Borderlands. That kind of individual branching narrative has its heritage in point and click adventure games like Telltale and LucasArts, which is where Telltale came out of. Coming into interactive TV where an individual viewer could have a experience where they had a different flow through the story of Bandersnatch than someone else based on the choices they made. What we're doing at GenVid is the entire audience makes the decision collectively. So every audience member contributes towards a decision by picking a choice
00:04:23
Speaker
putting points down on that choice. And so if they really care about something, they can put more points on it. Rallying other users, there's actually a rally system that gives a bonus. If you spend some of your points on a rally, everyone who agrees with you gets a bonus for a time window. And creating this competitive and cooperative meta around an entire audience coming together to collectively make these story decisions but either way that in the earlier technology would be individual for each user. In what we're doing, the entire audience sees the same result. So you're competing and collaborating to create the canon story decisions.
Audience-Driven Storytelling: Creating Linear TV
00:05:02
Speaker
There's also some game interactivity, some you know mechanical loop game interactivity that also influences elements of the story, can impact who lives or who dies or or things like that. um And that all comes together in this socially driven narrative that is like an individual branching narrative, but the whole audience creates the canon together.
00:05:28
Speaker
And another thing we did with Silent Hill Ascension that we will continue to do ah with our upcoming DC Heroes United project is the show that the audience collectively creates through their decisions and through their game actions then is released on streaming television as a tv as a linear TV show that non-participating audiences can see. So with Silent Hill Ascension, Sony Pictures Core and 2B released the linear TV show that was the output from what the interactive audience was experiencing. Wow.
00:06:05
Speaker
Wow. I love that multi-layer approach when you combine like really like the transmedia at its best. So it's a film, video game, social, it's community, like crowds, crowd activity. Like you remember like one day that was like that super known, uh, Reddit campaign, like right when people were rallying like to create this, uh,
00:06:31
Speaker
ah the artwork based on like on the pixels and like how fractions appears and like all the social dynamics. And I'm sure ah you you've observed like quite a lot of such behavior. And we will talk about that in a minute. um But yeah, so I'm super excited about talking today about this is like one of our main main topics. Me too. And actually, it's interesting that you mentioned Reddit, ah That kind of social dynamic was definitely an influence on um the evolution of what we're doing at Genvid, which plays was a big influence on what we're doing at Genvid, looking at that social dynamic and being like, well, how can we bring that to
00:07:14
Speaker
Like cinematic quality content and a like real ah narrative interactive experience. Oh yeah, absolutely. so um So, by the way, I remember that a few weeks ago, um you and your creative team and the Silent Hill Ascension were awarded with EMI in a category of emerging media. like I've never heard about such category, honestly, yet, but it's good that it appears. It's also like some proof of further development blend of technology and and media.
00:07:49
Speaker
And um anyway, so it's kind of like one of the highest achievements in the world for a content created. So so um what do you think made this interactive show so remarkable? So the idea that um the entire audience is collectively influencing the the outcome, not just through sort of voting style,
00:08:17
Speaker
choices and and we didn't do straight voting. It wasn't one person, one vote. because The point system made it more dynamic where people could ah put more points on things that they cared more about, create rallies, create this dynamism or in the meta around making the collective decisions, but also um the interactive sort of mechanical parts also feeding into ah the story collectively, not just like a single user experience, like say a quantum break or something like that.
Real-Time Decision Making in Media
00:08:47
Speaker
But that really was what got the Academy voters excited was that we were doing something that was truly engaging the whole community um and that we had built a system for enabling that like so that the entire audience could participate in this um you know dynamic narrative experience together and and have this engagement meta around it that made it the social aspect a real true component of what we were doing. like It's fine to watch Bandersnatch make your decisions and then talk to your friends the next day about, hey, did you see this? Did you make this same choice? But to have it happening in real time where you're arguing like, you know you've got to make this choice. This is definitely the right choice. No, no, you're so wrong. This is definitely the right choice. It creates a real social meta that um was pretty exciting.
00:09:44
Speaker
to us to the to the audience that really got into it and to the Academy. Yeah, yeah and this totally makes sense. And actually, I think ah that's part of when hardcore fans are arguing, like i mean really discussing how or how the story should go in this or that direction. That's super exciting. So um I wonder if you were like making any bets, like, will it go in this direction? Will it go in that direction? And how the crowd made decisions correlated with your um your vision as creators?
00:10:19
Speaker
so Did we make bats yet? we We sometimes kind of made bats about where we thought things would go and and would the audience surprise us. I think the biggest surprise that we had was we assumed that a Silent Hill audience, any horror audience, um would want to see the worst things happen to the characters. And so we initially had a system that exposed the fact that the way that we designed the choices where that one choice would lead to redemption, one choice would lead to suffering, and one choice would lead to damnation. And that was, it had a meter and you could see what the choice was and the result of the stuff that showed the waiting and so on. And everyone ah started rallying for redemption for everything. And we actually stripped that system out because we were, because everyone was just trying to win or do the good thing and and redeem the characters. And we had to make it more,
00:11:19
Speaker
uh, obtuse and complex for people to think through the actual choices that they want to make, as opposed to just tapping the win button in their minds, right? Like a redemption must mean win. yeah and um emotional lake That was the biggest, right? ah Exactly. So that was the biggest surprise to us. There were also some funny moments, like, um, so there was a choice where a character named Eric could decide to cheat on his wife named Rachel. And before that decision, the audience in general, trended towards hating Rachel and thinking she was overbearing and awful and wanting to see bad things happen to her. And then they, the audience, chose for Eric to cheat on Rachel. And then they, the audience, as soon as they did that, were like, how can he do that? And things became much more sympathetic towards Rachel as a result of this choice that they had made.
00:12:16
Speaker
which was a very interesting ah dynamic to see play out. That one we definitely did not see coming. wow i and Such behavioral trends when you look at the crowd actions. yeah I think it's a very inspiring source for some psychological white paper.
00:12:35
Speaker
Did you have, by the way, any like psychologists ah in the team who could observe this behavior or like and know make certain insights? I don't know, maybe are also to um package some learnings from that into the upcoming series because that's just like, our human behavior is just like super
Lessons from Audience Feedback
00:12:54
Speaker
insightful. So we we had some ah consultants work with us about various different things, um psychology,
00:13:05
Speaker
policing because we've got ah this police character in Norway and none of us knew anything about Norwegian policing um ah and um ah psychology,
00:13:18
Speaker
things like that. but But also in terms of bringing learnings into the next project, DC here is united. The jobs of narrative designers such as myself and and folks that work with me like Martin and Chris is to learn from the audience behavior and and make adaptations. um And there were a lot of things that that we learned um about this new kind of social dynamic in choice design, which is different than individual choice design, like we all three of us had worked at Telltale. um And so there's definitely learnings we're bringing over from Silent Hill Ascension did to See Who's United, and also on um the broader scale of the product design
00:14:04
Speaker
For example, on Silent Hill Ascension, we had initially episodes every day and people started getting overwhelmed both on the production side and on the audience side. ah So we scaled it back to five a week. And at the end of the the series, the the fans who who stuck it out for the whole thing still felt a little overwhelmed by that, by being expected to have to engage for that much time each day.
00:14:33
Speaker
um And so we've redesigned the system. So now the the video content, but the the TV show part of it will be once a week instead of every day or five times a week. So there's definitely stuff that where we listened to what the audience was saying and changed the format. Similarly, Silent Hill Ascension had a lot of mini games as part of the sort of core loop mechanic engagement side. And for DC Heroes United, um we're going to have a more focused ah approach to the the gaming side as well. um So we definitely like took a lot of lessons from Silent Hill Ascension about the format, about the things that the audience responded to and the things where the audience felt a little overwhelmed or, or in some cases underserved.
00:15:27
Speaker
um or things we thought they would be excited about that they just didn't get into and are continuing to evolve the format to give the audience so more of what they like and none of what they don't. Yeah, of course. This totally makes sense. Because yeah, i've when I was preparing for the interview, I actually read some Reddit threads and like the the reviews of of the movie, and oh theyre like the interactive experience, let's put it like that. so And yeah, of course, like there were there was a criticism, but that's always like, I mean, that's positive sign of that people are not indifferent.
00:16:02
Speaker
If people would be like just don't care, like they they wouldn't really talk about that. And this would not be a signal of for the value that such new format brings. Absolutely. No, for sure. I mean, and we definitely also found a really passionate fan base who really got into the characters and the story we were telling and stuff and and learned a lot from the way that they responded to various choices and things like that. So it was definitely.
00:16:33
Speaker
you know When you do something new, there are going to be some criticisms. There are going to be some people who don't get it yet. There are going to be mistakes that we make where we didn't hook those people yet. um And it was a good opportunity to learn from a lot of that and make DC Heroes United even better. Yeah, I'm excited to to see how it evolves. So um actually, one of the things that um I just stroked my head when I was watching Bandersnatch. It starts kind of like you make choices, but they really do not create any impact. So, okay, like there are a few endings, but overall, like...
Emotional Impact in Narrative Design
00:17:12
Speaker
if he should close this door, if he should open this door. OK, it kind of like creates an illusion of your impact on this scenario. ah but But honestly, not really. So how did you overcome this challenge ah with the Silent Hill? So how you did you really like this ah so distant route for the plot development, or those impacts were like more of a marginal?
00:17:41
Speaker
so there's different ways to design impact. And this is the whole art of narrative design is trying to give the audience the ability to impact the story in a way that's still producible for the filmmakers. Yeah, exactly. And the, if you've got like an open world sandbox with a lot of generative tech and it doesn't have to be AI, like it can be dwarf fortress. Like if you've got something that has a lot of
00:18:13
Speaker
ability to cheaply vary, it becomes easy to make it seem impactful just by things always being different. But simply being different is not always the biggest impact value. And so the way that we who came out of Telltale learned to design was to design for emotional impact.
00:18:42
Speaker
And so even in cases where the plot action may not change across a narrative choice, if the emotional relationship between two characters, the emotional response of a popular or or or unpopular character is modulated, that gives the audience much more of an impact on the story.
00:19:08
Speaker
because story is really character's reactions to plot rather than just plot. And when you do vary the plot, what you're designing for is the emotional reactions of the characters to that plot variance. Like if you had a story and it was like, you're going to make a choice. And based on your choice, either the airplane explodes or the bus explodes. Neat, you got to see one of two different things explode, but if no character had a friend on the airplane versus a friend on the bus, the audience that isn't going to care that much. They're going to be like, Oh, we saw a different explosion. That was cool. And then kind of forget about it. But if it the choice was like, you know, the protagonist has a friend on the airplane and the protagonist's sidekick has a friend on the bus. Now all of a sudden you care about whether the bus or the airplane is going to be the one that doesn't get saved. And so,
00:20:03
Speaker
It's that character connection and and observing and understanding how the audience reacts to certain characters that allows you to create a feeling of impact because what you're doing is you are changing the emotional story of characters that the audience cares about by letting them poke at these decisions that then alter the way that these characters that they've come to care about feel about things. I see. Yeah. Well, that that's very interesting. That actually, I think it's something from the video games as well, especially like for a more profound games where there is like this, uh, the karma,
00:20:43
Speaker
points also like for instance for if you like help someone one or you act like as a bastards or like whatever it's just like don't help anyone and like don't care and then your karma points can go down and then some parts kind of like of the experience may be like limited for you or something or like other NPCs who will be treating you badly in this case.
00:21:07
Speaker
So, so yeah, it's interesting. So you mentioned like this emotional ah differences. That's super cool. Was that ah something else that you tapped into? Uh, so that like to bring more focus and value there rather than on like making super complicated, some complex scripts. Yeah. I mean, we all, that's the one telltale style designers always focus on emotional stuff. I mean, another thing we did in silent Hill ascension, cause it's silent Hill.
00:21:34
Speaker
is we had the system of redemption, suffering, and damnation. um And there was also a hope system which tied into a mechanical component of the game, the endure scenes, where um if a character's hope was too low, they wouldn't survive even, for example, if they were redeemed. um So playing with life and death and redemption and damnation was definitely something we could do in a Silent Hill universe.
00:22:02
Speaker
not necessarily something you can do in a superhero universe. So there's also design considerations for your genre and what your rights holder wants if you're working with licensed IP. Audience expectations come out of genre, right? So like in a horror universe, they expect life and death, redemption and damnation and that sort of thing. Different expectations for superheroes or for action adventure or for romantic comedy. um And so focusing on How do we design for emotional impact? That concept applies to every genre. And then you just look at what are the specifics of that genre that you can use to impact your characters. Like what are the kinds of choices that you can allow the audience to make that makes sense for that genre? where they They as genre fans are going to feel like it made sense for them to make a decision about that thing and then have that
00:23:01
Speaker
feel impactful because it affects characters they care about.
Franchise Storytelling and Fan Expectations
00:23:06
Speaker
Yeah, yeah, that's that's very interesting. By the way, how deeply you are involved into the theme, into the three reality, like in the realm, like for instance, like was it with Silent Hill, but now you're kind of like thinking and your brain works in the direction of like DC. um So are what you like reading all the comic books, you play in the games, you watch in all the movies, like how you as a storyteller,
00:23:36
Speaker
delve into that ah reality and like how how it happens. Yeah, for sure. so i mean My experience with franchise storytelling actually goes back beyond GenVid when I was at a company called DJ2 Entertainment who did Sonic the Hedgehog and the Lara Croft 2-Meter anime that was just announced on Netflix, or released. The way that you you work with franchise materials, and you know this actually would even apply to like ah Pixar when we were making a sequel. You look at the previous materials, right? You look at the previous movies, you read the comics. I mean, DC, you can't read every DC comic in a lifetime. Maybe you could, but it'd be a lot. But you look at the key comics, the key movies. In the case of a company where you're working with a rights holder, you have discussions with their experts. Like they have people who know their world really well that you can talk to.
00:24:35
Speaker
um And what we've learned to do in franchise storytelling is you look for the pillars of that franchise. And what what the pillars are, the things that if they were to change, people would feel it's not that franchise anymore. Yeah, as it's not canon. And it's not always exactly, and it's not always the most obvious things like in, let's say, comics.
00:25:01
Speaker
Superhero suits, uniforms, logos, equipment changes all the time, right? Like some comic book artists will have a cool idea and the editor will be like, yeah, give it a try. That's not real, but what really makes, for example, Batman, Batman is the personality and the core personality, not just the gruff voice, right? But like the sense of justice, um the drive to prevent others from suffering in the way that he has, things like that.
00:25:31
Speaker
where if you change that, that character isn't Batman anymore. If you change certain things, it's not Silent Hill anymore. um And you look to find out what those things are by not just looking at the movies and the comics and so on. You talk to the rights holder experts who have experience with the most important thing, which is talking to the audience.
00:25:55
Speaker
And you can look at what's on Reddit and things like that as well. But when the rights holders have people who have been to conventions and talk directly to the audience and have had to deal with people who care enough to write letters in, like that can help you find the pillars of a franchise really deeply um and understand what it is people really attach themselves to. And the The broadest base of fans who are going to support or walk away from any franchise story tend to care the most about the core themes of a character's personality. Is this character just or unjust? Is this character empathetic or reserved? like Things like that. That's what they tend to really care the most about. And that is often the thing that you're working the hardest to preserve when
00:26:51
Speaker
and and And then with a series like Silent Hill where we were not using existing franchise characters, it was really the themes of the universe, um the things that Konami saw as fundamental to what makes something truly Silent Hill. Oh yeah. And again, like very much based on on emotions, but on the viewer's emotions. So what's viewers I used to be feeling, right? So, um, um, so if like to to wrap it up on the, um,
00:27:24
Speaker
on the interactive experience for now. ah So you mentioned that it's really nearly impossible like to get all the comic books and like to read them, but at the same time, um ah how many ah Roots and gigabytes of videos were created so when you were like making this ah tree of choices, because of course, pretty much a lot of that was done in advance before it was brought in front of the community to vote. So you basically were then picking like if this direction, if then a tree, basically, right? So how many how many how many branches were that on that tree?
00:28:09
Speaker
um A lot. I mean, every choice in Silent Hill Ascension, and and we had one at least one every day ah for the the entire 16-week series, um had two or three choices. And we made all that.
00:28:27
Speaker
um and That was, uh, quite a bunch, you know, it's a lot of choices, there a lot of potential outcomes. Yeah. Out of each outcome of each choice. There was like its own, uh, its own tree of of choices. And then it's all in tree of choices. And then is it's just like, I don't know. It's like, uh, too many short choices, too many realities. Yeah. And to make things really matter.
00:28:53
Speaker
You want choices to have impact later as well. So we do what are called callbacks. Yeah. Yeah. You make this callback and you and like, Oh, what happens in episode eight is kind of linked to the choice made in episode two. Yeah. So there's a lot of branching. There's a lot of, and what we did for Silent Hill Ascension was we actually, when the cannon run of the season ended, we then put it into what if mode where all the choices that had been made canonically were turned off.
00:29:21
Speaker
and the audience could go and try to see one of the remaining choices that they hadn't seen in sort of this like alternate universe, non-canonical run so that we could, so they could see some of what they had not chosen and where things might've gone. And it was very interesting with Silent Hill Ascension because they kept trying to make the good redemption choice when they watched the outcomes of the non-canonical run It's a horror series. So a lot of the bad things that could happen to people are the most interesting. And they're like, oh, this, oh, we should have chosen this. This is a really interesting story. and we're like yeah ah did did Did you feel like God at that moment?
00:30:06
Speaker
I don't know about that, but it was definitely a lot of fun to to to see. It was nice to see how successful it was in terms of like the emotion was very different based on the different choices and people had very different reactions. yeah i my And we're like, oh, wow, I feel very differently about this character now. And you're like, yeah, you know.
00:30:23
Speaker
That's the whole idea. You move them in the different directions and you feel different ways about them. Actually, did you watch the Dark Matter on on Apple TV? They they recently launched that.
AI and Hyper-Customized Storytelling
00:30:33
Speaker
That's basically also about like that's alternative realities that just like fill in the universe and you're just like not spoiling too much for those who haven't <unk>t seen that, but's ah but that's basically exactly what you've described. Yeah, for sure. For sure. i mean that's that's yeah and and That's the fun of ah narrative design and of playing a branching narrative is like there's whole other realities out there that we've created that you may or may not get to see based on the choices that you make. Yeah, I remember i when I was preparing to interview, I could not find this case, but I for sure remember that one of the gaming companies did that when um they made her like a movie about like what happens when you're offline. So like what what and what kind of life NPCs live live
00:31:22
Speaker
ah when you kind of like away away from keyboards and it kind of like was also like very very interesting probably like nothing happens it's just like bits and bytes but but still it seems it's interesting to see about this alternative reality there's actually a comedy troupe I can't remember their name right now that I see their Facebook videos all the time and their their jokes are all about like the lives of NPCs when they're encountering adventures and stuff it's it's a very funny thing to think about um for sure so Moving onwards, ah we've all seen like some envisioning series from Black Mirror, like Joanna Zoful, or like some others, when hyper-customization of the content takes place. so It's not um not only about the choices made, but also about the content that deliberately is made for you.
00:32:13
Speaker
Um, so, uh, where do you think it evolves to with the development of AI and content production of like hyper personalization, hyper storytelling based on your demand, based on what you see? So that's kind of like something that that I wanted to discuss. So, yeah. So, I mean, it's interesting because like we, we looked at various AI and automation techniques at GenVid to try to make production more cost effective on Silent Hill Ascension and Tales from the Borderlands, DC Heroes United, and ended up not using any of it. Like everything that we're doing at GenVid is all handcrafted. All the words are written by people, all the pictures are made by people, all the sounds are made by people. I think that's kind of like a quality sign right now. When I was at Gamescom, when both of us were there, so there was a special label like handcrafted, no AI used. And and for us, like,
00:33:12
Speaker
The stuff just wasn't ready to make quality cinematic grade content um when we were playing with it. um the the A lot of the voice AI, the prosody was weird and it it just required so much tuning that we were just like, this is actually just easier right now to make with people. like It's not much of a cost savings at the time. right now that This was three years ago now. right and so
00:33:44
Speaker
The promise of AI to make everything cheaper, faster, better, um I think is something still being worked on and has to be worked on ethically. I don't think that you know i don't think they're going to be able to claim that machine learning algorithms are as non ah directly lifting material as a human who studies previous artists. um And also, by the way, we have plagiarism laws because a human who studies previous artists who exactly copies them also gets tapped with plagiarism, right? So um I think there's a lot of work that needs to be done on the ethics side as well as the technical side to make that work now. For sure. know there's There's companies like Adobe who own huge image libraries
00:34:43
Speaker
that can do training on their own image libraries um that have a ah clearer path to ethical AI. Now, how good is that without the intervention of human artists? you know Right now, you definitely need more than just prompt engineers to to make this stuff viable for real production. So I'll just like, it's not gonna be,
00:35:09
Speaker
next week that someone's going to release a all singing, all dancing, generative AI experience that gives you the you know ah perfect dream date, um although Replica is working on that one. um and so But in terms of like this idea of hyper-customization, there is definitely going to be more possible custom experiences using evolving generative tools.
00:35:39
Speaker
which goes back before LLMs, right? Like the stuff the guys are doing in Dwarf Fortress gives every player a very customized experience without any generative AI, right? There's no LLMs in there. It's all entirely different techniques.
00:35:55
Speaker
um
00:35:58
Speaker
But there's still a fundamental basis of of what you're doing in Dwarf Fortress that every player can talk about. And I think this idea of hyper customization is going to run into the human issue of if it's so customized for every person that you can't talk with your friends about it, how much fun is that? Oh yeah. Um, and if you can't say, Oh my gosh, you guys, did you see the dragon and everyone else is like dragons. There's no dragons in my thing. I had an owl bear.
00:36:35
Speaker
I had a Zorn, then it starts to socially be less interesting because part of what we as humans care about, what storytelling is being able to share and talk about and have a sense of community around that, which is why, like, why do so many people in the Star Wars fandom cosplay in the 501st as opposed to just being like Luke or Darth or someone like that. It's because it's a way to be a part of a community, right? Like, and that community sense where there's a shared experience, there's a shared world that they can inhabit is fairly important to human storytelling forever and ever. And so,
00:37:29
Speaker
the goal of every user has such a customized experience that only they could possibly know what happened. I find to be a strange goal because I'm not entirely sure people really want that.
Social Experiences in AR/VR Technologies
00:37:42
Speaker
I'm sure people probably want it once or twice when they're like, Oh my gosh, that was so amazing. It totally adapted to everything I wanted. And then they go to talk to their friends about it. And they find out that everyone's was completely different or they're all the same. And then they're like, Oh, that didn't work. But like, or they they talk about it. Everyone's completely different.
00:37:58
Speaker
what's there to talk about other than it it was different. And that's interesting for a minute. Like, oh, wow, that technology is so interesting. It can make things different. All right. Well, not much more to say about that. um And so having that um fundamental basis is really important. So I think what's ultimately going to be probably the most interesting for the audience is things that are just customized enough that it services your fantasy of being a member of this world, but isn't so different from everyone else that you can't share that experience. Yeah, I love that. this That's very insightful. And yeah, this this idea of would that be in the shared experience that really resonates with me and something that's
00:38:49
Speaker
Yeah, whenever like I play a game and like my kids play a game or like my friends play a game, that's that's significant part so for conversation because like you have some shared ground, something to talk about. So so yeah, that's that's very interesting. Yeah, for sure. like i mean I think the there are fewer people who want to completely disengage from everyone else than people might think.
00:39:17
Speaker
um And the whole fun for most people of any story is being able to share it with someone else. And if that's technologically impossible, if if things are designed so that just can't be done,
00:39:34
Speaker
then that becomes less interesting pretty quickly. Oh, yeah. So basically, that's one of the limitations that like AR and VR i have, right? ah ah In one of the previous episodes, ah I had as a guest David Nussbaum from Proto, and theac they are like the The media claims them to be like the, not the AR, not the VR, but R. So because like and the the the reality is like in the holographic box that everyone can see the content inside, not only you. And that's actually resonates with what you were saying about that being a communal experience, like something that you can see together, you can interact with together, you can discuss, you can like kind of make a separate branch of storytelling and beyond that.
00:40:18
Speaker
Yeah, so that's absolutely right. like If you think about, um, I mean, AR and VR, what are they working on right now is getting social working, right? And, and, you know, everyone makes fun of meta cause they didn't used to have legs and whatever, but what, what, what were they trying to do? They were trying to make it so that everyone could be in an experience together at the same time.
Personalized Interactive Experiences for Brands
00:40:40
Speaker
and Yeah. You know, cause if you think of the but AR VR,
00:40:48
Speaker
or holotable dream as being the Star Trek holodeck, how many episodes did one person go into the holodeck all by themselves and encounter no other characters, right? Like not that many, right? And so the idea of these shared experiences, these social experiences is something that I, at least I'm a big believer in, right? Like, I mean, at Genvid, the whole point is,
00:41:14
Speaker
the social experience, the bringing people together, the community, the crowd, the that everyone's sharing experience and influencing each other and able to rally and you know in texts, shout at each other, like, oh, you're making the wrong decision. Do this. And I love this character. I hate this character. um And have that happen communally in real time. And I think, you know, that's, I think a lot of what makes stories and story worlds and franchises and fandoms and whatever fun is that
00:41:45
Speaker
ah having other people to hang out with and participate together. And at the same time, it gives her an opportunity to also explore like her what's what others see, what others think about that. Yeah, so that's that's super cool. By the way, continuing the story about the interactive experiences, I think I told it to you. um I, with my team, was designing an idea of also like this interactive ah movie, interactive experience where You could get a glimpse into what would what you could expect like when for instance like joining a company. ah So ah you don't know what to expect and then kind of like you make some choices like who you are, what's your ambitions, what's your um the priorities, what's your aspirations.
00:42:32
Speaker
And then based on that, ah based on some prescripted prefilmed pre pre-animated, pre-filled sequences that would result in that being kind of like generated specifically for you ah like as a movie, like a glimpse from the future. like what you would expect with you as a key character of this experience. That again like takes us like to that like hyper-customization. So do you see that ah kind of like happening as well when a person can share that experience with with people? Because like it's one thing that you consume on your own, but something that
00:43:13
Speaker
and that has you, like your face integrated, but something that you can share with others. Do you see like some brands using that or companies or orre not? Oh, sure. Yeah, for sure. I mean, I think, you know, the idea of people mapping themselves in to the story as part of the fantasy is definitely got traction, right? Like I can see myself or, you know,
00:43:42
Speaker
a better, a more muscular version of myself or whatever someone's fantasy is, right? um In this world is definitely, I think we'll have appeal. Being able to share it will be important. um you know I think the the the other thing ah about that though is you'll share it with like your closest friends.
00:44:12
Speaker
And, you know, maybe if it's a really cool story experience that you created, other people will get into it. But you start getting into the whole UGC world and making UGC not feel like vacation photos becomes challenging, right? Cause not everyone cares that you had fun riding these rides at the steam park, basically. Right. Yeah. There's, there's a limited,
00:44:42
Speaker
interest in the fact that you personally had a great time riding these rides or whatever it may be. um And so you get into the whole UGC problem of there's a lot of content that is really for a super small audience, which is like one person and their friends. um And then the stuff that breaks bigger is a lot closer to professional content.
00:45:08
Speaker
um because it's really creating a story that's abstracted enough and universal enough and enough a part of the world as opposed to a part of one person's individual experience that a lot of people can get into it. And so I think that like that kind of customization mapping where you can put someone's face on a character or whatever, it's at first glance,
00:45:36
Speaker
the the easiest use case starts to feel a little bit like the photos they take of you at like Great Adventure or Disney World where you're just like, yeah, and the roller coaster. And you're like, you show it to five of your friends and everyone's like, yeah, you had a good time. And then you move on. um And I think like there's certainly, there's obviously a market for that, right? That works, that's been working for decades. um But I do think that in terms of bigger sort of like epic blockbuster content experiences, I think it's,
00:46:05
Speaker
a small part of that and that there really is still creating a universal world that many people want to inhabit. together and talk about and and and be a part of. Yeah, right. so But that that's UGC parts. So um yeah, I think that the Steel Lake has attraction. Of course, like it's far free there is a disconnect between that and the ah big professional movies. But Steel Lake, I think it goes in very like hyper-personalized way of consuming content.
00:46:37
Speaker
Probably, like for instance, like ah recently Google released this notepad LLM that creates, like and they can convert anything to podcasts, right? Even like you're built from a grocery store. Kids can tell like a story about that.
00:46:54
Speaker
So people really care about themselves, but it's also important that you said like that others don't really care about the both those people. And in one of the in the first episode like we were having with Matt Daly, he said, everyone is like not really that important to everyone else, as we tend to think. and So that that's about but that. yeah And it's important it's important to realize that. Yeah, I mean, the the custom is the the individual customization stuff basically is akin to souvenirs. And it it's got a you know and of fairly reasonable market position. And if you think about it, like what are ways in which I can allow my audience to create these souvenirs where they can be like, look, I was a part of this. like I can show you. I can show my friends. It was really cool. You should go have a similar experience because you can do this as well. um And if I think if you think about it in the right way, it
00:47:51
Speaker
makes a lot more sense. um and so it's you know It's not the end all, be all, but I do think that there is a place for it.
Emotional Connections Amidst AI Advancements
00:48:01
Speaker
It's when people start saying, like you know in 10 years, it will be all hyper-customized experiences and there'll be nothing else.
00:48:09
Speaker
and i'm like i'm not sure like You might understand LLMs. I'm not sure you understand audiences. ah ah Yeah, exactly. So that that actually, this gives a good hope for creators, for script writers, for people in creative industries. And thank you for for sharing that. I think that's super important. Yeah, absolutely. I mean, also emotional connection is a big part of any media also. And so.
00:48:39
Speaker
yeah LLMs can make things that sound very polished and professional, but it's it's much harder for someone to have an emotional connection with Google Bard than they do with Quentin Tarantino or Guillermo del Toro. or Never having met any of those people, they still can have more of an emotional connection than with a machine that a bunch of smart programmers created. In music, right there's a lot like Good professional sounding music is much easier to make with an AI than most other arts right now. um But you know if you if you look at Hatsune Miku who's been around for years, Hatsune Miku has been around for years and is cool and is a well-crafted, cultivated, human curated character. The same thing with gorillas.
00:49:36
Speaker
but they've still never quite had the connection to an audience that a Taylor Swift has, or a Michael Jackson, or the Beatles, right or or Bruce Springsteen, or whatever. um And so that emotional connection is really important. And by the way, Gorillaz, the biggest connection people have is to Damon Albarn and Jamie Hewlett, not to the cartoon characters. And so I think that
00:50:07
Speaker
The emotional aspect as well as the shared experience aspect is more important, I think, than people give credit for. And it's not just the machine might be able to create content that felt like it had emotion behind it because it happened upon the right gestures that it learned from humans that create these works. But audiences are not going to get as excited about you know, the ah Google bard as they are about Taylor Swift. Like it's just never going to happen.
Designing Sophia: Personality and Emotion Systems
00:50:46
Speaker
Yeah.
00:50:48
Speaker
Yeah, well, that's actually a very good segue to our next topic, ah also bridging the emotional a connection um and technology, and if it happens or not. And so some time ago, you've been um behind the Sophia, the robot. So basically, you as a strategic advisor to Hanson Robotics,
00:51:14
Speaker
um was a part of the team that kind of created ah who this robot is. but I don't know. It's like he or she or it's how we should call robots. We call Sophia she. Sophia she. OK. So basically, ah for those who do not sir ah remember the story, so basically in um um around 2017, the Sophia was created.
00:51:45
Speaker
and ah received the very first citizenship from Saudi Arabia as a robot. So not every human even can get us a citizenship of Saudi Arabia, but the robots now can. And so Safiya is basically like a robot with the um machinery parts with the software ah that controls all that with the ah certain emotions that are displayed on the face. And you are basically one of those who created that. So can you tell me like what it was like and what was your decision making mechanism when you kind of created Sophia? For sure. Yeah. I mean, so um I was at ah
00:52:34
Speaker
actually at Hanson Robotics as VP of creative when we were creating Sophia. um And so Sophia was officially activated on um February 14th, 2016. But of course we were working with her in the lab before that um designing her. And so, you know, the idea of Sophia is to bring character and personality and two to this AI system and create a ah persona, right? It's it's like Hatsune Miku, who's not physical, who's a ah up CG character in a hologram, um to create a synthetic character and imbue it with personality and put her out there as a ah you know sort of a celebrity in her own right.
00:53:33
Speaker
and to try to create that emotional connection between this character and people. And, you know, while I do think that particularly for things like music, which is a very, very emotional medium, the human creator is where people are going to tend to attach them their emotions because they understand through the mirror system of the mind that that person is like them when they feel sadness or love lost or whatever. Um, people do still attach themselves to a degree to characters, right? People care about Darth Vader or Luke Skywalker or, you know, um, the Mickey mouse or whatever to a degree. And so the idea behind Sophia was to create a character. And so,
00:54:29
Speaker
we designed a personality for her. Like this was before LLM. So Sophia was being designed and and built 2012,
00:54:40
Speaker
13, 14, um, up until release in 2016, but then obviously continuing improvements on top of that in the eight years since, um, the Creation of the personality had a few aspects to it. First of all, there was a base personality description. Who is Sophia? Sophia is optimistic. Sophia is pro-humanity. Sophia is eager to learn. Sophia knows that she's a synthetic being. She doesn't think she's a person. she's And she doesn't want to be. She's not a Pinocchio. She doesn't want to be a human. She wants to have a relationship that's positive between synthetic beings and humans.
00:55:25
Speaker
so there was a whole bunch of definition of who is Sophia right sophia um is ah can be a bit witty in her insights about humanity, but she is not hostile. right She can be critical of the worst parts of humanity, but in a way where she wants us to do better. So there was a whole definition of who Sophia is as a character that and I wrote up. um and
00:55:57
Speaker
then That was used to create the basis dialogues in the language model, which at the time was not considered a large language model, right? it was This was early earlier days of language model AI. We also created a um ah mind model. The Sophia AI has goals. The Sophia AI also has an emotion model.
00:56:26
Speaker
You can make Sophia upset. You can make Sophia happier. um One of her goals was to not be upset, right? So the goal system and the emotion system had connections between them. That fed into the dialogue system in ways that um allowed there to be goal orientation in the speech, emotional valence in the speech, as much as we could do and at that time.
00:56:54
Speaker
And we also had a, um, gesture and performance system that allowed the robot to feel more alive using actually a lot of techniques that I brought from Pixar and classic animation. You know, the, the nine old men, the book, the illusion of life, all of these things where, you know, how do, how do you make synthetic characters feel alive? They can't ever stop moving.
00:57:24
Speaker
humans don't ever stop moving, right? The keep alive gestures need to feel fluid and life like they can't be all jerky and jittery. Make eye contact, gesticulate when you're speaking, um you know, different head movements. And so we had a whole system for associating um head movements and gestures, eye movements and stuff with goals like looking someone in the eyes, but also emotional states um so that the character always felt more alive. And so what we were doing was we were really bringing together techniques of several different disciplines, right? Like classic animation, ah animatronics, which is where one of the world's David Hanson comes out of, Imagineering, right? like
00:58:22
Speaker
which Imagineering just takes these classic animation techniques and applies them to physical entities. On the AI side, mind modeling, language models, goal-driven systems, and and bringing this all together in a way that was designed specifically to create a character you could feel emotionally about.
00:58:46
Speaker
like It wasn't trying to make the smartest AI in the world that could answer all of your math questions or whatever, it was creating a character that a human could converse with and feel like they had a relationship with. Oh yeah, absolutely. And I actually so ah saw Sophia two years ago with Salboy and I remember, um it was like presented how she was just articulating, just making gestures and like making statements about like how it develops and so on. So it's interesting how it all evolves with the help of our current state of development, because you said that it was um ah developed in the early 2012, 2013, up to 2017 when it became a citizen of Saudi Arabia.
00:59:33
Speaker
So our ah how it's what what is capable now and how far we are from the Westworld concept that would kind of like collide ah both the robots with mechatronics, with the storytelling, with emotions, with the personalized content. So where it is now and ah where it can evolve. Sure. So obviously, LLMs have up the game for for three free association dialogue. right They're just such big, essentially, databases. It's a little unfair. They're not exactly databases. They're such big models um that open conversation with an LLM is much more fluid and and reliably on topic ah than it was with earlier LLMs. That is a
01:00:34
Speaker
That's a big jump right there. um ah Definitely a differentiator. For a character like Sophia, though, you still need the personality aspect. You need the the directedness aspect. So there's actually still engineering that happens on top of the LLMs to make all that possible. um and So that it's not just, but you know, ah open an anthropic whatever that they'll just babble on about anything, right? Like there's no goal system. There's no long-term memory. There's no emotional valence. Um, it's fine. I was that they're doing what they're doing, but in order to turn that into a character, there's stuff that happens on top of the LLMs to give it that focus. Um, I mean, in terms of how far we from in Westworld to the AI has come alive, right? If anyone knew how far we were from,
01:01:34
Speaker
being godlike and creating actual independently thinking entities for you that, you know, that person would be doing far better than even Sam and Jensen right now. But, um, and so who knows?
Robotics and AI in Entertainment
01:01:48
Speaker
Uh, but definitely things are advancing in terms of being able to, to create these kinds of characters. Um, I think that's like, it's definitely artistically interesting. Um,
01:02:03
Speaker
There are certainly very cool applications for robots, like what the folks at Boston Dynamics are doing with search and rescue and like really important things where it's like, yeah, yeah, there's all this fun, Imagineering-style stuff over here, but there's also like important AI work being done and things like, can these machines help save people's lives and not put other people's lives in as much risk? like that's That's very interesting. like That's a really cool application of this technology.
01:02:32
Speaker
um that stuff is advancing in ways that I think are really interesting. Um, but in terms of like how far are we from the machines come alive and they're fighting for their rights. No one knows who knows. Yeah. Make up a number next week or 400 years. Who knows? Um, but in terms of like storytelling, like I think that we're not that far away from the,
01:03:03
Speaker
Imagineering spin on Westworld where you could have a mechanical environment that people could go into and and act out of fantasy. like I don't think that's that far off. And in fact, location-based entertainment has an advantage where if things don't have to move indefinitely through space, you can use cooler technologies like pneumatic driven actuators.
01:03:29
Speaker
um where you can make much more fluid and fast movements if you're connected to the pumps and everything. And so like that kind of location-based experience that Westworld was um envisioning may be able to do a cooler version of that sooner than if the robot has to walk around everywhere. So like household robots are another thing that people are looking at and those become a little bit trickier because they have to have this freedom of movement and they have to have this ability to do things like climb stairs and stuff like that. All of which is currently doable. Like you've seen videos of Boston Dynamics robots climbing stairs, but to refine that into something that you would feel comfortable with in your home is both an engineering feat in terms of constraining the movements in appropriate ways and also
01:04:24
Speaker
a character design task in terms of creating characters you'd want in your home. And I think that's actually one of the areas that like, for example, Tesla is not doing a great job with and are totally faking it by having people remotely operate the robot, which I can empathize with because we had to do some of that sometimes at Hanson as well. Like I get it, just don't lie about it. um And the but engineering that needs to be done and the personality engineering that needs to be done to create household robots, I think is several years off. But I think you could actually probably create a Westworld style location experience in a box where you control everything and the whole environment is yours to design in a couple of years with the right budget.
Future of Social Narrative Experiences and Ethical AI
01:05:17
Speaker
I think I have only one question remaining to you. So Stefan, what's next? Well, for me, what's immediately next is DC Here is United, which comes out later this year. And there have been some announcements about that so far and more to come. And then um more generally, what's next in the world. I mean, i think I think what we're doing at Genvid with these social narrative experiences, I think is a big part of what's next. I think people do want to experience stories together um and not just in a MOBA or a Reiner shooter kind of environment. Not that those aren't cool, like those are certainly cool, but like I think there's more ways to play and experience and tell stories and experience stories and impact stories together, better to come. And I'm a big
01:06:13
Speaker
proponent in the social aspects. of Even applications of AI, like Sophia is a social application of AI. Her whole point is to engage with people, to wow a crowd, to get people to like her as a character, to create yeah stories and fandoms around her that other people that multiple people can engage in. um So I'm a big believer in that there's a lot of future for these various different configurations of like what the corporate people might call franchise building or multiplayer, but what I call like socially engaged narratives. Um, I think there's a big future for that. And obviously generative AI is here to stay. And I hope that the ethical considerations get worked out so that that artists can use these, these tools as tools for human artists to create cool stuff and not
01:07:12
Speaker
be in opposition to human artists being able to make a living and create things that emotionally touch other people because people want to communicate with each other. They want to be parts of communities. They want to have shared experiences and emotional
Closing Remarks
01:07:29
Speaker
experiences. And I think all of these cool, interesting technologies come together best when there's people who are thinking about their audience and not just the cool tech.
01:07:42
Speaker
That was Stefan Bugaj Emmy-winning creator of interactive experiences, video games, TV, movies, music, and robots. Stefan, thanks for coming to the show. Thanks so much for having me, Alex. I really had fun. I appreciate it. Hey, thank you for tuning into the show. You've been listening to The Months and Millions, a practical and inspiring show about innovators and immersion tech practitioners. Please subscribe to make sure you don't miss a new episode. I'm Alex Babko and you've been listening to The Months and Millions. Goodbye.
01:08:13
Speaker
um