Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
5. Level Up
532 Plays9 months ago
For the final episode of this series, Andrew is joined by Dr. Jacquelyn Schneider, the Director of the Hoover War Gaming and Crisis Simulation Initiative. The episode focuses on the value and variety of wargaming today. What does the field of wargaming look like now? What does it need going forward? And how can you play some of these games with us?
Key Points
- Wargames are a useful tool for problem sets that lack good data, either because they happen rarely, like nuclear use, or because they’ve never happened, like a nuclear war.
- As wargaming methods experience a renaissance, the traditional defense wargaming community is increasingly interacting with social scientists in the academy. Each group can learn from the other, and ideally those interactions will lead to engaging games that follow rigorous methodologies to answer analytical questions.
- Emerging technologies, and novel use cases of existing technologies, are a particularly interesting area for wargaming. Much like nuclear use, there are many outstanding questions about the role of artificial intelligence, cybersecurity, hypersonic missiles, and other technologies in war, with limited to no real world data. Wargames offer a chance to explore these technologies in a synthetic environment.
- Wargames can also be a valuable tool in exploring the perspectives of other countries. Jacquelyn discusses her experience wargaming cyber vulnerabilities and their impact on nuclear stability in the U.S. - China context. In those games, keeping the gameplay abstract (ie. using fictional countries rather than real ones) allowed individuals to speak more freely than they would have during a real world scenario.
Jackie’s Reading Recommendations
- Erik Lin-Greenberg, Reid B.C. Pauly, and Jacquelyn Schneider’s “Wargaming for International Relations Research” for European Journal of International Relations
- Andrew Wilson’s The Bomb and the Computer The History of Professional Wargaming 1780-1968
- Thomas B. Allen’s War Games: The Secret World of the Creators, Players, and Policy Makers Rehearsing World War III Today
A reading list to accompany this podcast series can be found at: https://brsl.berkeley.edu/podcasts/
Recommended
Transcript
Wargaming vs Gaming: Real-life Simulation
00:00:03
Speaker
I discovered what makes wargaming different than gaming is that there's this buy-in to the game that you are not playing the game to win a game, but that the interaction that you have inside this kind of stimulated environment represents how you would behave in real life.
Introduction to Risk Calculus Podcast
00:00:25
Speaker
Hello and welcome to the Risk Calculus, a podcast from the Berkeley Risk and Security Lab. I'm Andrew Reddy, the lab's founder and your host for this series on wargaming. Over the last four episodes, we've approached wargaming through various lenses. We've talked about the history of this method from antiquity through the Cold War, how it's been applied to different problems from nuclear use to cybersecurity, and how these historical games have influenced how we think about risk and strategy.
Current and Future Landscape of Wargaming
00:00:49
Speaker
We've also looked at game design, what's in a game, and why these choices matter.
00:00:53
Speaker
In this final episode, I want to focus on the value and variety of Wargaming today. Specifically, we asked three questions. What does the field of Wargaming look like today? What does it need going forward? And how can you play some of these games with us?
Dr. Jackie Schneider: Transition from Defense to Academia
00:01:14
Speaker
Today, we're delighted to be joined by Dr. Jackie Schneider, a Hoover Fellow at the Hoover Institution, the Director of the Hoover Wargaming and Crisis Simulation Initiative, and an affiliate with Stanford Center for International Security and Cooperation. Jackie, of course, your experience is particularly unique given that you started Wargaming in the defense community and have now been at the forefront of bringing the method to the academy. I would just like us to start by kind of talking through how you think about using Wargaming in your work.
00:01:39
Speaker
Yeah, so the benefit of working at a place like the Naval War College is you learn, I would say, probably the best Department of Defense ways to run a war game. You get very immersed in the intricacies of building scenarios, building out players, the program management part of war gaming.
00:01:59
Speaker
But the program management part of Wargaming, the places like the Naval War College do so well, sometimes means that it can be difficult to do the social science part of Wargaming. That the ability to control the way you build scenarios and pick players and execute the game, to try and find, I'm going to say kind of a nasty word in political science,
00:02:22
Speaker
causal relationships or get close to causal relationships. When I left the Naval War College and came to Hoover and Stanford, it allowed me the ability to try and take what was a pretty amazing learning experience at the Naval War College and try and apply social scientific methods.
Merging Defense and Social Science Methodologies
00:02:44
Speaker
I generally ask questions where we don't have a lot of good data,
00:02:49
Speaker
because it either hasn't happened or because they're horrific scenarios like nuclear war. And so we're glad we don't have the data. And so I turned to war games to try and answer some of the questions within that kind of novel scenario space.
00:03:05
Speaker
And what I've been trying to do is to introduce these two disparate, I'm going to say fields, but they're not really fields, these disparate communities to each other, which is on one hand, defense war gaming, which comes with probably a century of knowledge about what works and what doesn't in terms of games, and then introduce that to social science methodologies.
00:03:26
Speaker
leaning on the work of experimentalists, as well as people who think through process tracing and qualitative methods and try and marry the two so that we run rich, evocative, interesting games, but we're able to do it in a way that controls as much as we can for bias.
00:03:44
Speaker
Yeah, I really like that you've noted both the kind of experimental approaches, but also the more case studies, right? Like board games do provide us that really rich case data that you could go and use for telling stories about how introducing variation in one particular game changes a particular result.
Wargaming Beyond War: Emerging Tech and Human Behavior
00:03:58
Speaker
And then of course, couldn't agree with you more in terms of where this method is useful is where we don't have a large amount of data already to go and use.
00:04:06
Speaker
So you've kind of already started addressing this, but other research problems that you think were games are particularly well suited to you mentioned a nuclear use case or their geographical context where you think they're particularly interesting or other types of technological problems.
00:04:18
Speaker
So I think the logical subject matters are things that have to do with the emerging technology and conflict, the kind of the futures world, or for technology that we really don't see used a lot in conflict. Nuclear weapons being a really great example. These are things like artificial intelligence, cybersecurity,
00:04:39
Speaker
hypersonic missiles. We're trying to understand how systems that we haven't seen fielded a lot might influence combat or might influence the chance for war. Because the chance for war is not something that we can stimulate very nicely in a survey experiment. War games are probably a much better way of generating data. But I think that's like just the first logical place I'm actually really interested
00:05:04
Speaker
and how games can help us understand human behaviors in a lot of different types of situations. So not just conflict or war, but thinking about human behaviors and how they affect democratic movements, how they affect decisions about economics. I think actually there's a great extension of work that can be done using war games to understand core questions about human behavior and group dynamics in all sorts of different social processes.
00:05:32
Speaker
Yeah, I like that. I mean, do you feel as if almost wargaming is kind of a pejorative label that's been given to the field and it doesn't actually need to be wargaming, it can just be kind of gaming methods writ large?
00:05:41
Speaker
Gaming is a better way to describe it than wargaming because I think wargaming takes it down a particular pathway. And our initiative is wargaming in crisis simulation because our director said, hey, I love this, but generally a lot of people think about wargaming as only worse. And if we extend it to crisis simulations, then you're thinking about something beyond just conflict.
00:06:04
Speaker
But if you call it just gaming, now you have to differentiate between so many other different types of games.
Evolution and Design of Wargames
00:06:12
Speaker
And I think at its core, what makes wargaming different than gaming is that there's this buy-in to the game that you are not playing the game to win a game, but that the interaction that you have inside this simulated environment represents how you would behave in real life.
00:06:30
Speaker
And that's the difference between a game for fun and this war gaming thing, which is about understanding behaviors in a way that stimulates real life. I think there's an interesting story. Like if you go back to Kriegsville, one of the first war games.
00:06:49
Speaker
Preachable actually evolved from a series of kind of social games. The people were playing these games as a kind of parlor game. It was the upper middle class elite were playing games. But then they realized, oh, what if I actually make these rules somewhat similar to how things happen actually in combat? Yeah, geography. Yeah.
00:07:12
Speaker
Yeah, and they brought in these people who are military experts and they got them to build these really complicated rule sets that were basically these set of beliefs about how war would actually play out. And then they got it in front of these senior oppression leaders. And this is when the senior oppression leader says, my God, this is not just a game, this is war.
00:07:34
Speaker
But it's because war games, they introduce this level of vividness and reality that invest their players in a way that make it more like real life and less like you're playing a game.
00:07:49
Speaker
Perfect. So I'm wondering if we can move the conversation to something that's a little bit more concrete in terms of how you've actually built war games to study a series of really important challenges, cyber escalation, for example, with your international crisis war game. How do you think about the design process for the ICWG or another war game that might be one of your favorites?
00:08:10
Speaker
So I start with war game design like any social science research design. What is my core question? What am I trying to understand? And based on that research question, I'm going to build out my game. Now, this first game series, the International Crisis War Game Series, we had a research question. We were trying to understand how cyber operations affected nuclear stability.
00:08:36
Speaker
specifically how cyber capabilities to exploit nuclear command control and communications would affect decisions to use nuclear weapons at all or earlier or to take different counterforce campaigns. So in order to design that, we needed to build a scenario that allowed us to control a little bit about
00:08:58
Speaker
are variables. And so once I've determined what my variables are, then I go into designing a scenario and a game design that allows me to look at those specific variables. I know a lot of people when they start a word game, they want to start with scenario. And I would say that's where you really get messed up because you start building all these details and they end up making it really fun to play, but may make it difficult for you to pull out those variables and look at those relationships afterwards. Don't
00:09:27
Speaker
So we actually built it kind of in a quasi-experimental way. So we had controls. We introduced different treatments based on the cyber capabilities and cyber exploits. But in order to do that, we also made a lot of things really simple. We didn't have adjudication in between moves. We made it a hypothetical scenario.
00:09:48
Speaker
We had groups, but they were relatively
Insights from Wargaming: Beyond Quantitative Data
00:09:50
Speaker
small. So we made traces that gave up some level of interaction in order to be able to control because our goal with the game was to run it as many times as possible to try and understand how iteration and heterogeneity in our population could help us move towards generalized ability or at least being able to say, this is not just the behavior in one game, but that this is a behavior
00:10:16
Speaker
that we see over a lot of different games, over a long period of time, and a lot of different characteristics of the popular. I would love to pick up on that point around your design being abstract. It's something that we chatted with Ellie Bartels at Rand about as well. What drove that choice to be abstract from your perspective?
00:10:33
Speaker
We did this for a few reasons. One is we were talking about cyber and nuclear, and we find that those are two subjects where people are generally, if they've had some sort of experience working in these fields, they might be less likely to open up in a real world scenario because they don't want to reveal classified information that they may know. Also, the first iteration of this game was in a track two between two nations who have had a series of nuclear crises between each other.
00:11:02
Speaker
we felt like if we have an abstract or hypothetical scenario, then we're likely to speak freely.
00:11:08
Speaker
than if we give them a real world scenario. We also wanted to generalize beyond a specific incident. So our research question was not about how does cyber affect nuclear stability in a US China scenario. It was how do these variables in general, separate of the context in which they're introduced, affect the nuclear dynamic. And so because of that, we thought abstraction was the right choice.
00:11:32
Speaker
Now, there's two critiques. One is, if you pick an abstraction that you think is abstract, but it still leads people to think of a specific scenario, then you're not getting the goodness of abstract, right? You're not abstract enough. The second potential problem with abstraction is that players don't connect with the game in a way in which they are acting the way they would in real life, i.e., we are capturing
00:11:56
Speaker
general reactions to the scenario, but those general reactions are not specific enough to help us understand how players would react in a US-China scenario or in a Russia scenario. And I think that's a problem that anybody who uses abstract scenarios is going to have. And so you just have to be clear about that when you're talking about how your findings indicate what the limits are about whether they are predicting future behaviors.
00:12:25
Speaker
And what did you end up finding? Did the cyber capabilities end up impacting nuclear stability? Yes, but in a counterintuitive way. So we actually went into this game, we had a series of hypotheses, but the hypothesis that I thought was going to play out was that cyber vulnerabilities would make players feel insecure and lead them to using their nuclear weapons earlier.
00:12:46
Speaker
That's not what happened. In fact, what we found was that players tend to underestimate their cyber vulnerabilities and overestimate their cyber capabilities. We found that what led to more nuclear instability was actually the introduction of cyber exploits, so telling a team that they have a capability to attack the opposite team's nuclear command control communication.
00:13:11
Speaker
that would lead states to take more aggressive counterforce campaigns, preemptively put their own forces on nuclear alert, and then states that have the vulnerability but not the capability, what we found is they were more likely to move to automation or this use of dead hand or lower levels of demand to actually make nuclear decisions.
00:13:34
Speaker
The implication was that it was overconfidence in cyber that could lead to nuclear instability and accident. I really appreciate you taking us through kind of one of your example game designs and demonstrating how it's contributing to kind of an active debate is happening inside the scholarly literature. I think it's one of the best examples that we have of linking the Wargaming Method to the way that we think about doing this work as scholars.
00:14:09
Speaker
So Jackie, what explains from your perspective why we're really focused on this method today? There's a few things that were happening that led to where we are in war games. One is not happening in the academic realm. One was what happened in the Department of Defense, which was a focus under Deputy Secretary of Defense Robert Work to reinvigorate war gaming. And one of the initiatives that he was pushing for was the use of meta-analysis of games
00:14:37
Speaker
He was trying to build a wargaming collection that would allow the Department of Defense to look at all the games that are occurring across the Defense Department and try and make generalizations across games. He also invested in different methodologies, different ways of thinking about using games, and I think that provided an impetus inside the Department of Defense, but it also actually trickled down a little bit into the think tank community and also into academia.
00:15:06
Speaker
at the same time that that was occurring, which is kind of at the same time that the new, he had a strategy, the third offset. So it's right around this time that all these things are occurring. And I think at the same time, academia is honing its experimental method. And so you have a rise in understanding about how to use experiments
00:15:25
Speaker
within international relations. But at the same time, the experiments are becoming a little bit less satisfying. You're having problems with thinking about the population of people or the sample of people that are doing their surveys. There's a lot of critique that's occurring about like, do these survey experiments actually represent how defense decision makers think? Can they say anything about international relations? And I think those two come together also at a time
00:15:53
Speaker
give a little credit to bridging the gap and people like Jim Goldgeier, who were actively building communities of scholars that were interested in doing work that bridged this gap between policy and academia. And I think you have young scholars coming up like you, Andrew, Eric Lynn Greenberg, the archival work of Reed Folly, and that led to a series of games that were coming out around the same time.
00:16:18
Speaker
that we're all trying to use this war gaming method as a method, leaning on both experimental work and qualitative work and saying that war games were going to present data in a way that can help us answer questions that we had trouble generating data. So I think these two worlds fed upon each other and there wasn't like one particular moment
00:16:39
Speaker
It wasn't one particular catalyst, but it was the infusion of interest, scholarship, new thinkers, and all of those coming together. Yeah, perfect. Well, I think it's been almost a decade, I think, since Robert Work's memo outlined why we needed to kind of have a renaissance in word giving methods. As you kind of reflect on that decade, are there other lessons that you think that we've learned as a field or are still learning?
00:17:02
Speaker
Well, I want to say that I think there's a big difference in what the Department of Defense is learning or thinks it's learning in war games and academics. Academics in general, the community is always moving forward with thinking about how they use data. And so you see that thinking about using war games as data, I feel like that's maturing at perhaps a
00:17:24
Speaker
quicker rate than what's happening at the participants, probably because we don't have a lot of people that have high stakes in the game. So you're building a brand new, potentially set of communities. The bigger trouble in academia is figuring out how this method is different or the same or better or not as good as existing data methods.
00:17:44
Speaker
I think we're learning a lot in that way. So I will say, when I started working in this particular War Game series, I thought that my value add was going to be quantitative. But in running the games many, many times, I'd be able to actually say something quantitatively that would help us move War Gaming to the next level.
00:18:03
Speaker
When I started writing the paper, I realized that even though I had run this game with 580 players over three years, I had teams of four to six. I only really had an N of about 115. I had four treatment groups.
00:18:19
Speaker
So now I'm looking at basically groups of 20 to 30 that I'm comparing. There's actually not a really great quantitative thing here. Basically, I would describe my work as a bit of qualitative plus because what I realized was I had these numbers which was helpful for framing and understanding the outcomes, but they didn't really explain why we got to the outcomes.
00:18:43
Speaker
the qualitative, the how people filled out their response plans, all the extra information they gave in surveys, that actually was the value add of the war game. And so I'm coming around more and more to
00:18:58
Speaker
how games provide that, not just how we can use games to provide more ecologically or externally valid quantitative measures, but also how the richness of games and that like putting people in a room together for, you know, three hours and then seeing what they say, how that actually can give the really amazing insight into human behaviors.
00:19:23
Speaker
I really like that. I mean, obviously correlations are great, but you do need to get the deep conversation to get at the why. You know, I think that's a really important distinction that you're drawing there and we'll kind of see how the field moves forward from there as well. Also, as you kind of reflect on your experience in the field as one of the academic pioneers in this space, are there any kind of notes of caution around either how we're using it inside the Ivory Tower or how are you seeing war gaming methods being used in the defense community? Well, we can save the defense community for the second part because that could take a while.
00:19:52
Speaker
So I think on the academic side, there's some core questions about what makes a good game that we still haven't answered. And so I mean, one game that I reviewed, they didn't even send any of the game material. That's not okay, right? So that's progression. We need to think more about how we look at the entirety of your game materials. I think that games are expensive.
00:20:17
Speaker
and take a lot of time. And that can be a big limitation. And so we need to find ways to be able to decrease that cost so that you have more gaming. There is no magic number of iterations or players. We don't know when we've run the game enough times to stop. I think we also don't know when games are interactive enough.
00:20:41
Speaker
We have not found a way to measure or convey how decision making in games is different than decision making in surveys, for example. I wouldn't say they're words of caution, but I would say that these are things that we as academics need to better understand.
00:20:57
Speaker
How do you measure buy-in from players? How do you understand how the group functions? What are the dynamics of the group and how you introduce them into the game? These are all four questions about war game design that we really don't have answers for.
Academic vs Defense Approaches in Wargaming
00:21:11
Speaker
So I wouldn't say that's a question I would say, I hope we work on that. And I would say the academics have to work on it because the DOD will not.
00:21:19
Speaker
What the DoD is kind of designed for is to build games that make the person who are paying for the game generally happy. How do you make someone generally happy about a game? Well, two things, you get to an outcome that the sponsor would prefer, and two, you do it in a way that's generally enjoyable, right? So the primary way that you're evaluating whether it's a good game for a Department of Defense game is how do the players feel?
00:21:44
Speaker
And I think that's coming back to player buy-in, but there's not like time within that DoD board gaming planning process to ask these questions about, okay, we need player buy-in, but at what level? And how does that affect bias? And so a lot of the core questions about what makes games good are just never going to be answered in the Department of Defense, who is also the primary user of board games. And so I think academia can look at those type of questions and that will actually end up changing.
00:22:11
Speaker
probably the way games are evaluated by the Department of Defense. That's a really nice segue. I'm interested in what you think the field needs to move forward. I mean, obviously, all the technologies that you were mentioning before, which are good fodder for substantive areas of research to kind of apply war gaming methods, some of those same technologies are also offering new ways to actually perform war gaming as well. So I'm just kind of wondering, what's your sense of where the field might move in the next five, 10 years?
00:22:37
Speaker
So, I mean, the first thing is, as you highlighted, I really feel like we only ask almost the most expected questions out of war games. And I hope that they move from something that is entirely looking at war and look at larger social processes. So that's where I hope the field moves. I also see a lot of goodness in bringing in younger academics, graduate students, assistant professors, people who are kind of coming up in their career, but
00:23:04
Speaker
The problem is running games is expensive and difficult to do. And so we need resources for young scholars. I mean, if you think running a survey experiment is expensive, games are extraordinarily difficult to do and to do well. I think the biggest difficulty in games and the hardest thing for young scholars is getting the sample because you either have to pay the right sample to participate in either sitting in a room for a long period of time that, you know, that's an honorarium.
00:23:34
Speaker
or that's paying to get them there or that's paying for their food and entertainment. You have to incentivize and sample becomes extremely expensive. If we can find ways to help younger scholars be able to generate their sample and to give them some of the administrative support to be able to run the party planning part of games, I think we'll find that they're very innovative when it comes to game design.
00:23:59
Speaker
if we can build incentives for scholars to do methodological work, so not just running games, but also looking across games at what works best, doing the same kind of work that you would see in the survey world or the experiment world where
00:24:14
Speaker
We have done enough analysis across surveys or across experiments to understand what works better and what leads to substantial bias or limits generalizability. But in order to do that, you have to have a place where scholars can publish and potentially to have fellowships or some sort of space so that scholars can take time away from just running games and actually look at the methodology.
Future Directions and Resources for Wargaming
00:24:39
Speaker
And of course, you're running one of the initiatives that I think will make that type of meta analysis easier and also contribute to building a library of where folks that might be interested can actually pick up and start to play these games. So can you give us a little sense of your vision for the archive that you're putting together that will hopefully serve as a repository for games across the country and across the globe?
00:24:58
Speaker
Yeah, we're really excited. We're putting together a collection of board games and board gaming materials inside the Hoover Library and Archives. And we're approaching this in a little bit of a different way than some of the existing collections. So actually, Andrew, it is shocking how much board gaming stuff is out there. And there's a lot of individually curated websites
00:25:24
Speaker
that all have links to different types of games. But what you realize when you start looking at all these games is that nothing's super curated. It's very, very difficult to look across games. So we're building our collection with a focus on data first. So I think of it as a database, not just the collection. Building out the metadata and the technological infrastructure to be able to look across games extensively, methodologically,
00:25:51
Speaker
to be able to download games and replicate games. But this is actually a bigger leap forward than I realized because there's actually a pretty strong divide or stovepipe between library and archives and the big data world. And so we're trying to bridge those worlds and building out this collection.
00:26:13
Speaker
The other thing that we're doing, and this is our focus in the first three years, is trying to bring in as many historical games as possible. That way we can look across games to understand how games have historically influenced big choices that have been made in foreign policy and domestic policy for the last few centuries.
00:26:31
Speaker
So bringing in all these materials so that people can run games, people can analyze trends, people can look at specific games to understand how those games influence historical decisions and ultimately hopefully to make games better.
00:26:46
Speaker
Yeah, and you've got big supporters over here across the bay in that endeavor as well. So beyond folks following the development of that archive and database at Hoover, just a last question from us. If you were to give advice to somebody just starting out in the field, where would you send folks to kind of learn more, whether it's a particular article or a book that you found particularly inspirational as you kind of launch your journey in Wargaming?
00:27:08
Speaker
Well, this is horrible. I'll promote myself, but I'm going to promote my co-authors more. So I've been really inspired by working with Eric Lynn Greenberg at MIT and read Polly at Brown. We have a piece in the European Journal of International Relations that talks about how to use war games in terms of designing war games to generate data, and also how to think about war gaming data as a piece of novel data.
00:27:34
Speaker
I think it's useful, you know, Eric Reed and I came up with a bit of a how to say, okay, you want to run a war game. Here are some steps and questions that you need to ask when thinking about the types of rules, the types of players, the way you collect data. So I'm going to put a little plug in for myself in that paper, because I think it can be helpful for people thinking about how to use data and how to design games.
00:27:58
Speaker
And then lately I've been doing a lot of work thinking about how games influence foreign policy and their role in history. And I just want to recommend an oldie, but a giddy, the bomb and the computer by Andrew Wilson, and then a new one. I just bought war games, the secret world of the creators and policymakers rehearsing world war three today. And this is by a guy named Thomas Allen, and it has a lot of more contemporary war games that have been run in the department of defense.
00:28:27
Speaker
Perfect. That sounds great. Jackie, thanks so much for joining us. I really appreciate your
Conclusion and Preview of Future Topics
00:28:32
Speaker
time. We'll include your suggested literature in the show notes, and we'll also post a list on BRSL's website. Although we've reached the end of our first miniseries, this is only the beginning of the risk calculus. In future seasons, we'll be digging into other aspects of our work here at BRSL, from industrial policy to AI regulation. And for those of you that have made it the full way through the season, you'll know that we've left plenty of meat on the Wargaming Bow, and so I'm sure we'll have a second series on Wargaming as well. I hope you'll join us.
00:28:57
Speaker
for that. Thank you, Dr. Jackie Schneider, for joining us. Thanks to Andre Anderson and Citrus, the recording studio hosts. And special thanks to our amazing producer, Jane Darby-Menson. And finally, to all of you for tuning in. Until next time, I'm Andrew Ready. You've been listening to The Risk Calculus. Obviously, the tree guys are here. No worries.
00:29:26
Speaker
Uh, and I don't know how to make my dog stop barking because he is legitimately breaking at people at our house. It's okay. Should we give him like a second? Let's come back and try this again.