Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
#15 - Arash Azhand - Trusting the Machine: A Deep Dive into AI Safety and Assurance
58 Plays3 months ago
How can we ensure AI systems are innovative, safe, and trustworthy?
In this episode, I sit down with Arash Azhand, an expert in AI safety and assurance, to explore the challenges and strategies for building reliable AI systems. From safety in medical AI to continuous assurance practices, we dive into the processes shaping the future of trustworthy AI.
Key highlights include:
- Addressing regulatory gaps in AI for critical applications.
- Tools and frameworks for making AI decisions more transparent.
- Generative AI’s role in both cybersecurity risks and solutions.
Curious about the future of AI and how it’s staying accountable? Catch the full episode to learn more!
Recommended
Transcript
Introduction and Trust in AI
00:00:00
Speaker
Hey, what's up, everyone? This is Ajmal Savari, and welcome back to another podcast episode. Let me ask you something simple. Can you trust AI making life-changing decisions? In this episode, I'm joined by my friend Arash, a brilliant scientist and AI expert to uncover how we make AI not just smart, but truly trustworthy. We dive into the hidden world of AI safety, how it's tested, certified, and made ready for critical tasks like saving lives in health care or keeping us safe on the roads.
00:00:30
Speaker
It's a conversation that pulls back the curtain on the magic and the challenges of building responsible AI. Enjoy. Before we get started, I want to introduce you to MI4People, a tech nonprofit that's close to my heart. Their mission is to leverage innovations from AI and machine learning for the global good and make a real difference in people's lives.
00:00:50
Speaker
They already created an AI-driven radiology assistant that helps doctors in underserved areas to diagnose patients faster. They also run a marine litter project where they leverage satellite images to detect trash in our oceans to combat pollution. Sounds cool, right? Because it is. The only issue is that MI for people can't do it alone.
00:01:11
Speaker
If you're someone with a cool project idea or an existing project that can benefit from support, just reach out to them. You can also support them by volunteering your tech skills, team up as a partner organization, or support them with a donation. Visit mi4people.org. That is M-I-4, as in the number four, P-E-O-P-L-E dot O-R-G. Mi4people.org. Get involved and help keep the ball rolling.
AI in Nonprofits and Critical Fields
00:01:54
Speaker
Hey everyone, and welcome to another podcast episode. If you're new here, my name's Ajmal. I'm a neuroscientist and entrepreneur. On this podcast, we explore the links between science, technology, business, and the impact they have on all of us. Today, we talk to Arash. Arash is a scientist and researcher with a strong background in artificial intelligence, data science, and theoretical physics.
00:02:17
Speaker
Currently, he focuses on AI safety assurance at the Konyong Geembeha, where he designs frameworks to ensure the safety and reliability of AI systems in critical fields like autonomous driving and healthcare. care Previously, he developed advanced computer vision algorithms that supported medical applications for senior care, co-authored study papers, and even supervised the development of a pet.
00:02:40
Speaker
His career is all about using AI to solve meaningful challenges and build trustworthy solutions. The list goes on and on. All right, enough background. Let's get into it, shall we?
00:02:53
Speaker
Hey, Arash, it's ah great to have you. So first off, um it's been a long time in the making. We have met each other during our time at Lendera. Unfortunately, back then, my first week was your last week. yes But luckily, we managed to stay in touch and well exchange ideas over many, many beers together, which I appreciate it a lot ah because your perspective from from theoretical physics has always been very enriching to me. and well During the last time ah we met each other and talked about things, you actually invited me to come to the Applied Data Summit, which is hosted by Deconium, where you're working at the moment. and well I must say, i I didn't really know what to expect. I just saw the website and I thought, okay, this looks
00:03:46
Speaker
This looks like a crazy day and it was a crazy day.
AI Safety in Complex Systems
00:03:49
Speaker
It was it was a lot of fun. um Many, many topics discussed. And one of the topics that day were where that was talked about was AI safety assurance and trustworthy AI, which is exactly what I want to talk to you about today.
00:04:10
Speaker
Yeah. And I wanted to ask you, you know, to get all the listeners on board. Can you tell us a bit about what AI safety assurance and trustworthy AI are and why they are so important, especially nowadays? Yeah, sure. Yeah. First off, thank you very much for inviting me. So also, as you said, along in the making.
00:04:33
Speaker
ah Of course, big fan of your podcast. but Thank you, thank you. Always watch it. Yes, so the question, why actually, why the heck? So why doing AI safety assurance? um at the The issue is, ah ah the answer why it's important is um because there are several domains um where AI can be applied, where we need higher safety assurance. So just an example to um get it a bit more lively. ah Just imagine ah domains like medical devices are automotive. and These days we all hear about um driving assistance.
00:05:31
Speaker
that should come into the car. And yet probably one time in the future, ah we expect to have fully automated driving assistance. And i kind that that will get also very complex because you will probably
AI Applications and Trust Building
00:05:50
Speaker
not have a situation where ah you only have automated or robot cars alone.
00:05:59
Speaker
ah You will have probably situations where you have humans st driving alone completely combined with several different types of cars who are maybe fully automated or ah partly automated. And this is a highly complex, a very nonlinear complex system ah where you will have a lot of feedbacks. But even apart from it, just um specific, um ah easy to imagine assistance systems are highly um um complex and also very safety relevant. What does it mean, safety relevant? That means that um ah you ah need to ensure that the system is working ah such that it's not endangering, for example, people on um or other
00:06:59
Speaker
um people who are on the streets. And on a medical area, you can imagine we both worked in a medical area. We were at Linderra. That was actually previously before I came to the economy. My experience was um application of AI in a safety relevant area. And in a medical like Linderra, we were um a medical device.
00:07:28
Speaker
um Even for non-AI parts like software, we needed to ins ensure that the system is safe. It's producing or it's working on a safe regime. If it's um assisting or predicting for ah medical situations, it has to be checked and tested before it is going to production. And for AI, it's the same. so ah When AI is coming into play and tries to ah do predictions for them for a specific medical ah question,
00:08:08
Speaker
then you need to ensure that the system is working according to expectations. And um so this is actually ah to summarize why.
00:08:24
Speaker
so ah On the other side, if you just use AI for fun products like, for example, ah enhancing your images on a smartphone with AI, that's not an issue or detecting cats or dogs or so. But when it's coming to kind of detecting cancer, for example, on X-rays, I imagine you have done an AI and then you, ah let's say you you have an AI, you 100% want to rely on it. You say it's okay, I don't ask any questions. ah The product is detecting cancer on x-ray and then say, I rely, I don't question it. And then it's ah maybe um um is predicting um that the cancer image is normal. And the the patient is that doesn't get the care it needs.
00:09:22
Speaker
um that's That's the issue, so pra yeah image-wise, to put it. And that's why it's important to you have to have ah specific processes to ensure safety of these AI systems.
00:09:40
Speaker
Right, right. I mean, i can I can definitely see it, especially as you already said, we're from the medical perspective, right? I mean, drugs are being tested as well, right? Before they are getting to market, they're also tested for the risk and benefit ratio. And ah the same goes for also the regular software where AI is not a part of it, you know, when it comes to um just any type of a potential ah ah digital diagnostics um that don't even involve AI, um you can not even diagnostics. i mean If you think about, for example, specific devices that um
00:10:24
Speaker
that have a type of automation for, for example, insulin delivery, yeah or for diabetic patients, right? What would happen if the device malfunctions and does not provide the insulin that is required in a specific moment, or it gives too much insulin at specific moments where it might not even be required, right? The downside ah is immense, right? it's ah it's a life It can be a life and death situation, right?
00:10:52
Speaker
yeah Yeah, definitely. That's true. And of course, and ah you can have several um different levels of complexity even also. So so safety relevant the use cases not equal safety relevant use case. So you have um the most, I think one of the easiest is the AI enhanced assistance systems where you, ah for example, assist.
00:11:21
Speaker
in a case for medical personnel. And the system is not um doing stuff, but it just does recommendations. And then the decision is still on the side of the human. um That's ah kind of, that's the way we started off um in this area, because others, what you, for example, gave us an example is a very, and um sophisticated, where then even the decision would be on the side of the device. And then it's it's getting even more complex. But ah starting easy and then gradually ah applying our learnings on that to the others ah will help, I think. For sure.
00:12:12
Speaker
And how do you how did you relate the AI safety assurance to now um the discussion about trustworthy AI? and How couple do you think are those things? I think um it's um ah both are kind of um helping each other or influencing each other. So the thing is that um ah We are humans that all kind of are very um driven, I would say, to ah to trust. Or if um you you don't use when you don't trust it. and so right um that's So trust is kind of ah For us, it was it's a kind of very catchy work and it's kind of what we call in German plakatif or I don't know how to... Yeah, I know what you mean. Yeah, ah it's important. um So um you need, especially in these domains that I talked about, safety relevant, trust is a kind of
00:13:39
Speaker
um important resource also for the companies who are in this area um because you it's easy to lose trust but it's difficult to gain it and you have um you you had several examples of the negative parts of, for example, companies who ah in regard to data, for example, we had several um examples where data breaches happened and then it was um very difficult afterwards for the companies to gain trust again. um It took time and similar as this was AI. so if you
00:14:27
Speaker
ah so Ensuring our safety assurance is a lever, I would say, to gain trust. And then um if you and then on the other side, you have trust. you As a company, of course, you then get more resources from this trust to develop further and bring your products to the to the people.
Continuous AI Assurance Vision
00:15:00
Speaker
ah So I would say it's just both influencing each other. ah So it might just be, I mean, I never really thought about it. Sure, I have thought about it, but not in terms of thinking of trust as ah as a resource or maybe even a currency, you know, because yeah sure a good fight if i if I don't If I don't trust something, right? Even if it might, you know, there's this yeah on the marketing level, it all sounds so great. It will benefit me. But if I don't trust what they say about this product, yeah and depending on how how big the downside could be to me, I will stay away from this, right? That's the thing. And also the thing, um one came just to my mind, one important but funny thing is,
00:15:49
Speaker
um that we talk on the one hand one side about assurance and on the other side trust. um And for me, I think for you also, for regarding your background, for me as a physicist, um it's um the measuring, so the measuring stuff is the only stuff that is mattering.
00:16:17
Speaker
so You have assurance, you have quantitativeness, you measure stuff. um And on the other hand, you have trust and trust is the opposite. So trust is something qualitative. You cannot measure trust. um And that's, I find very funny because you have these two sides of a medal for me and to one is yeah measurable and the other one is kind of very It's also human-wise, feeling matters, and and that's why it's also funny to think about this. Yeah, that's true. i mean You're right. I'm also a lot about numbers. as yeah if something If something is so and ah based on the numbers, let's say 95% accurate,
00:17:10
Speaker
I have ah my bias to trust it is of course higher. i um I mean, I would say, of course, but I see over and over again, a lot of people are like, yeah i don't I don't really care what it says. This number says, or based on the statistics, I heard this one person had a bad experience with it and therefore it's out of the question. yeah and's it's It's interesting. you know And in the institute where I used to work,
00:17:37
Speaker
There was also a lot of research on neuroeconomics and how people make decisions. how and and One area that was studied also there, which I never really thought about was um um research on trust. How and why do people trust other people? yeah Not only how do they trust other people, how does their trust behavior change if they think they are interacting with a machine? That I thought was very, very interesting and you could clearly see differences in their behavior just by them
00:18:15
Speaker
thinking they are interacting with the machine, even though they have been interacting with the machine all along. you know yeah yeah but It's like, okay, this is, as you said, you know this there it has this qualitative
00:18:28
Speaker
There's some qualitative parameters in there that it they can be quantified, but just in, you have to kind of go around the corner, right? It's not yeah ask yeah directly what you feel. I have to do a lot of other tests to actually know what you feel.
00:18:45
Speaker
Yeah, yeah it sounds very complex so it's very You studied because cognitive science and it's important. And of course, ah you know, in the end, um I think with the technological revolution, right? That's basically happening right now. Or i feel I shouldn't say right now, it has been continuously going on. This is just another big step in the evolution. It's now a accelerate keeps just keeps on accelerating. I think it's really important on many, many different levels, that people are able to, to trust this new technology going, as you said, I think from very simple systems that
00:19:30
Speaker
that don't require such high safety assurance because yes there is no downside. If I you know if i swap my face with so ah in a picture with somebody else, one of the very early ones, we we we did this all the time. It was so much fun, you know just for a giggle. Well, it's it's fine. But if I'm supposed to use an app that is that is supposed to diagnose, let's say, arterial fibrillation based on my smartwatch. That's a different story. Yeah, yeah that's true. And um and of course, ah for me, um ah was also kind of um this important point to ah ah to continuously kind of learn and develop this.
00:20:21
Speaker
And as um actually my, um the idea where why I came to this was also from the other side, actually not the trustworthy. So to extend on this kind of this duality a bit, ah this,
00:20:42
Speaker
notion of trustworthy AI came actually afterwards, after we for several kind of couple of years worked on this topic. What I actually, and together with some colleagues, we termed this um kind of AI assurance or con continuous AI assurance. And then afterwards, um the trustworthy AI notion came. And I would suggest or I would see this, the trustworthy AI as a kind of an outcome, an output or outcome of a a continuous AI assurance process in the end. um yeah and so my And so the the reason ah I came to this and then I had this vision actually was to
00:21:38
Speaker
Uh, because as you said, uh, worked at Linda for a couple of years and then working in a medical startup was, let's say at, at, at large, we were 30, 40 people, uh, was a great product and also very important product, but it, um, ah it was quite, uh, difficult for us as a startup to go through this kind of all this, uh, uh,
00:22:08
Speaker
medical device regulation, getting it. I'm not even talking about AI, just the cluster the other stuff, the software part was also difficult and you we had to ah consulting companies who came actually from ah domain from different domain from actually from a classical medical hardware devices and yeah yeah ah pills. and that's easier i would say yeah And then also products that are ah taking maybe years from yeah kind of building and then coming to product ah software-based software software AI based product that you have these faster cycles, update cycles, and then
00:22:55
Speaker
It was quite difficult and then consulting companies told, gave us ah not good recommendations, which, recommendations, which were actually fitting to, to these classic old hardware devices. And, and then, um, to my mind, then I went after Lindira, I joined Deconium and then.
00:23:18
Speaker
um It was kind of a coincidence that I came to a project where I worked in automotive ah and and then for Volkswagen Cariad, and then the question was there to um develop an end-to-end process for building AI, ah which at the same time is agile and also adheres to safety and all these um kind of rules and regulations. rules and regulations um And that's kind of these duality, again, and that's duality, agile and regulations.
00:24:05
Speaker
yes And um so then I came to this idea, okay, um what about having a system, a process there, ah where which at the heart of it,
00:24:18
Speaker
has um technically um developed a method to um really um evaluate the AI development while it is developed automatically and continuously.
00:24:35
Speaker
um And then producing all the needed kind of documentations, metrics, and evaluations without the developer and data scientists and so on taking too much care about these documentation and regulation stuff. Yeah. Actually, um, actually like you would imagine, uh, in a classical software systems was the continuous tests and integration CI CD, uh, systems.
Evolution of Safety Standards
00:25:04
Speaker
Actually, actually I envisioned a similar type of process like CI CD, but now for AI.
00:25:12
Speaker
But then again, because the problem or the the really the challenge was ah really to map from this classical software CICD to the AI, because as you also know, um kind of um and AI is not similarly testable like ah classical software.
00:25:39
Speaker
And right there there everything began. So that was the kind of the birth of this continuous AI assurance. Yeah. No, you're right. I remember still the audits we had to go through. So for example, when the TUF, for the international listeners, um it's ah and You can think of it like the DMV, but more for overarching technical things, it doesn't really have to be about cars. So the technology is fine. They are also checking you know your quality management system for medical devices, give you a stamp of approval. So you send them all your documentation, they come in, they grill you on things.
00:26:27
Speaker
And you're you're right, it's already hard enough for just being regular, plain old software, but not even considering AI to be part of it. But of course, for us, AI was part of the product. And I still remember this guy grilling me on it. And he just asked this, this you know this he went through his checklist. And he asked me, OK.
00:26:53
Speaker
um So what are your reference measures? And I looked at him and I said, what do you mean? What do you mean reference measures? Yeah, your reference measurement devices. He said, you mean like a ruler? He's like, yes, a ruler. and so I was just staring at him blankly and I said, what?
00:27:14
Speaker
What are you talking about? we we We don't have a ruler. This is done in the camera from the space on the software, which is based on the training dataset, which has been validated. So I just kept giving him all the information, at least I thought. to And then when I was finished, he was just staring at me blankly, simply saying, so where is the ruler?
00:27:36
Speaker
I was like, okay, I guess you didn't understand a single word I just said, yeah and which also means that, um and I think that's also very important, is that a lot of the rules and regulations are not based on the latest technology. yes that's and and And even if they were to have the thinking about timeframes or your your product life cycle, which is also incorporated into these rules and regulations, you know life cycles for pharma products
00:28:16
Speaker
are are very different ah than for software. And it's the same for, for let's say, traditional hardware medical devices. it's You maybe make a new hardware product let's say every four years, at least yeah you're not expected to do it faster. So a lot of the rules and regulations are taken from there and pushed onto the software medical devices. And as you said, in agile, well, two weeks, four weeks, you ship an updated version and yeah the regulators are just like, wait, you're not so supposed to change the product.
00:28:58
Speaker
Yeah, that's kind of, yeah, that's, that's difficult. And also, I think, um ah through my kind of work in this area, um ah it took time. But after a while, I understood also from historical perspective, that it always was the case. So you brought the yeah very nicely this, the tooth into the arena.
00:29:25
Speaker
And I also learned ah the history of TIFF, for example, according to my work. So you said it's technical evaluation um consult yeah group. And actually, they got also 150 years old. So now, um I think last year, last year, and they were actually also were founded in the Kaiser Reich, so in the old German, German speaking domain, and they were all actually called Dampf über Prefungsverein, so steam, steam evaluation, because, um as I learned, um they were back then these steam engines that were founded, and then they
00:30:22
Speaker
industrialization began afterwards. But the issue was that these steam machines engines were prone to explode. And then when they exploded, that was very, very dangerous. And then I saw some pictures of large factories that were just burned down completely because of an explosion of a steam engine. And then the Kaiser,
00:30:50
Speaker
in essence, sold them, the industry, either you take care of this and to make your steam engines secure or I will do it for you. And then the industry just founded this downfield approach was fine. And then from that, so then afterwards really, ah they developed also these standards and these kind of um how to test and how to make them secure. And back then, of course, you could imagine um they at that time didn't have the ruler for the steam engines. But after a while, methods and standards were developed evolutionary, I would say. And then um that was so successful ah after a while that they then
00:31:47
Speaker
um later started to evaluate also other industrial machines, not only steams and then later cars came. And we can, we, um, the modern, uh, humans, no tooth in Germany or German speaking countries because of cars, because you have to go with your car, but it was actually steam engines. But now it, there are others like medical products came and then software is coming now AI and so on.
00:32:17
Speaker
And um that brings me to actually to what I said, that I learned a bit. And so, okay, we need to convince the people. And the tooth became, ah the tooth now, when I intact with them, with several of these tooths, they are much more pragmatic than at the back of the time when we, what do you said.
00:32:44
Speaker
We will start it there. They have a lot of technical people, data scientists, and ah for example, they found it also recently, the TIFF AI ah kind of department, yes, section, who are kind of researching the standards of the future for several domains.
00:33:13
Speaker
And um so that's very interesting. but So then I understood, okay, um as you very nicely said, the ruler, so in many areas, we're trying to, in an evolutionary sense, develop these rulers. So ah just as an example, when we, um um in inside kind of these measurable assurance,
00:33:44
Speaker
ah We ah maybe ah want to have one area where it's about um robustness of an AI system. So if I imagine ah the question is, if I have a medical AI who is, so let's say, um um segmenting an x-ray and finds the parts in the x-ray where there is a lung pneumonia,
00:34:11
Speaker
um yeah Then you want to know, okay, how robust is the system when I disturb the input image? yeah Many people know this ex example adversarial attack examples where you do put attacks or kind of random noise onto the image and then the system suddenly is not able to detect the pneumonia.
00:34:40
Speaker
um And so this is the robustness, so how much can I disturb it? And then we developed kind of methods to measure this robustness yeah quantity and for a specific use case. And then the question which comes then afterwards, okay, what is the standard there? Your TIFF guy, for example, asked, what is the reference?
00:35:10
Speaker
And yeah you can now define ah basic references for use case, um ah which then so finally get standards. And then you have these reference systems for robustness for an image detection system, for example. So right that's kind of closing the circle, actually. So we started very ah like,
00:35:38
Speaker
um one of our farmer well former Chancellor Merkel thought hashtag Noiland for internet. So it was kind of hashtag Noiland also in this kind of AI assurance area because we did not know what to do and which methods to use but gradually step by step ah you um ah you learn and you develop new methods and and also new standards.
00:36:09
Speaker
Right, right. you know i think I mean, it's definitely Neuland, the new frontier, yeah a new frontier that needs to be well discovered, investigated, of course, a lot of work. and I'm glad you already answered a question that I had in mind, which was,
00:36:31
Speaker
um I wanted to ask you if you would think that the the ah AI safety and assurance is just a static thing, you know just this does done once, you get your stamp of approval and off you go. And of course, it's not. I mean, you obviously already said continuous, that's a critical word here, yeah um specifically when it comes to and the the fast development times timelines and yeah cycles. You cannot just ah you know sit back, you got your one product done and ah look at it four years later. The software develops so fast, the environments change, you need to keep up with the maintenance
00:37:14
Speaker
And with the maintenance changes there all affect your product in the end. So of course it needs to be done on a continuous basis. um I wanted to ask you ah specifically when it comes to the safety assurance and also the trustworthiness.
00:37:31
Speaker
um I mean, you're very aware of this. AI is often described as ah as a black box. You put something in, some magic happens. Even heard ah people really describe it as magic or you need when they say, oh yeah, we need to add some fairy dust to our products. So just include some AI. I was thinking you lost your mind, but fine. It makes no sense. But okay, you put something in, some magic happens and you get something out.
00:38:02
Speaker
I was wondering if you think if that is a problem when it comes to safety and trustworthiness of AI products, this terminology of this magical black box. Yes. Yeah. um Yeah, it's absolutely an issue. And I think, I don't know how this, it's fitting to that what you asked. I heard this famous saying of someone It was very smart. um but Actually, it went like this. Any sufficiently complex technology looks like magic. Yeah, yeah exactly. Human is not understanding it. And that's actually and the other part of ah one important part of making AI trustworthy in the human domain is to also educate, in some sense, the people
00:39:00
Speaker
ah Because um one issue that is adverse to trust building is this magical thinking. um ah Because people say, okay, I don't understand it. Of course, um I also don't understand many how a car, in essence, or a plane functions. But there is a kind of um historically, we both trust because we know, okay, ah many important people understand how it functions. And that's okay for me. um And that must be also the case for AI at some ah time in the future, that people think, okay, ah I at least understand from a broader perspective, from a higher high level perspective, um that it's a technology. It's
00:39:58
Speaker
In the end, also AI is just software. It's a different type of software, but it's software. um And um it is still, even if it's complex, it is still manageable. We need to build processes and methods to manage it.
Explainability and Certification
00:40:17
Speaker
And and then um one way, so one technical way to um to um enable ah trust in this area and that um ah that's why I find the question very very good ah is the domain of explainability, which is also very important within this process of continuous AI assurance. And um it's also very complex um because ah you um you can have different levels of explainability.
00:40:57
Speaker
um And it's also depending on the domain you're working. Just as an example, if you're in finance, for example, ah let's say these use cases where you um ah give credits to people or no credit to people, ah then you need the granular ah explainability and the banks and the insurances need to ah report to the customer directly why they didn't get the credit in a specific case. And then for that, ah that's the reason why um until today probably um ah in this domain, ah highly sophisticated AIs like this neural networks are um very reluctantly used or even not used. Mostly it's these classical
00:41:55
Speaker
linear and logistic regressions, because you can just do feature explanation. And then at the end, you say you give real explanations. So this is the granular explainability on the individual level. And on the other side, on other use cases, um like in, let's say, um in industry,
00:42:21
Speaker
um ah business to business, like in medical and automotive, for example, um you don't need this granular up until the end. You don't need to explain to each individual why it has happened, but you maybe want to explain it to medicals, to doctors, ah why a system that detect this as a cancer. And then, for example, there are very good methods like this. You get an image, it detects cancer, and then you get kind of heat maps, you paint onto the image the regions um which decided why it's cancer. And that's also in, this is in itself also a useful product
00:43:14
Speaker
because you then can show to the doctor, okay, this is the reason and if it's wrong, you can get it as a feedback from the doctor and then improve the system in the next round for sure. and yet And then I think what for me is actually done the most just because I'm physicist, the most ah intriguing or what I like ah is then the mathematical looking into the systems. My dream or my vision at the end is to really um and not on the kind of, ah because currently in AI, what we do with explainability is kind of this empirical. ah We just, we assess ah individual models and then explain, okay, this this specific model,
00:44:11
Speaker
ah this decision A and B because of that. But um what I would like to do at at some time in the future is to build ah meta explanations for for all neural networks, for example. For example, to say if you train a neural network like that, that is the result which you will get at the end.
00:44:38
Speaker
And this is, ah um in essence, ah theoretical understanding, ah ah basic understanding why a neural network functions the way it does. Why is this specific training method functioning and another training method not? And and at at the end, it will also then um improve overall ah the whole process of training because you you will be more efficient and will not ah because currently you also lose a lot of energy because you're empirically trained stuff which is not yeah
00:45:19
Speaker
functioning. Yeah. Yeah. Yeah. I mean, some people might not be aware. There's not just one type of model architecture that you can use. There's not just a single type of approach you can use for training AI. You know, there's not one AI. There are many, many different ways of how you can go about it. You can think about it for, let's say, the the the normal people, but that the the regular people, right? there's there There are many different types of cars, right? You can have a Ferrari, you can have a Volkswagen, a Golf, you can have a BMW, you can have a Lamborghini for that matter, right? They all have their pros and cons. And I think very importantly, their price tag is very, very different. The big difference to what you said is, I think that's very important.
00:46:14
Speaker
At the moment, the evaluation is done after the fact. right yeah Sure, we have a few indicators. right i mean If I want to train a large language model, yeah, I will most likely not use a boosting algorithm. Well, that's fair, right? We have some yeah we have some rough yeah ideas, right? yes But not as granular as you said, and that would be very, very beneficial. If we could already have some predictions about the, ah let's say,
00:46:52
Speaker
outcome of the training ah based on well how the architecture is and the type of input we give it, rather than just ah you know a gut feeling because that is definitely quite expensive in terms of energy and also just i mean manpower.
00:47:13
Speaker
right is this Sure, the training happens on you know systems and stuff, but labeling the data, collecting everything, you know putting in the pipelines, checking if it works, that's still a lot of manual work.
00:47:28
Speaker
yeah A lot that a lot of people forget. But we get to the to the tech a bit more technical details in a second. You know you mentioned the um to get it maybe a bit more, I mean, I love to talk about the super technical and theoretical aspects, yeah um but to to reel more people in so they actually get a grasp of what we mean, yeah um you mentioned um When we talked about robustness, when you mentioned robustness, yes you talked about ah system that was you know for example detecting pnemonia yeah you were involved right in a system that in an AI system for pneumonia detection. Yeah, ah that's that's true. Actually, um and this was the project that I initiated together with a team at Deconium Data. Great, that's awesome.
00:48:27
Speaker
Uh, because we, uh, so the reason was not to, um, uh, that would, that is nice. Uh, uh, product would be, of course. So, but, um, the idea was not to build, um, pneumonia detector as a product for the corneum. Uh, but the reason was to, um, uh, to build, uh, uh, continuous AI assurance. Process. Uh, and then.
00:48:56
Speaker
um because you cannot certify a process because a certain process is too broad. TIFF would never certify it and then say, okay, your process for every use case is check mark. No, that would do that and no ah you need a use case as a representation. I see. We thought, or actually,
00:49:25
Speaker
and that ah I did a research because we, and the decision that we would do medical ah was clear ah because before we worked, I worked in a project for Cariad automotive. So we automotive, we did already for Cariad and it's also um kind of a USB of Cariad Volkswagen. We are also not allowed to talk ah publicly about the details so we said okay we ah we want to build for Deconium as a general process not only for automotive but medical finance industry um a process which is functioning in essence for all of it
00:50:16
Speaker
But we said, okay, we start with medical ah because firstly, I worked before at Lendera in a medical domain. I knew the challenges there. And then ah we wanted to have an image-based use case because image just images are nice. um I worked before in computer vision. I knew computer vision and we had several people in our team who are also ah pro ah very versed in it. And when we came to pneumonia detection as a representation, ah because um there is a Kegel, I think six years ago, five, six years ago, um and there was this Kegel challenge ah for pneumonia detection, ah which was brought there ah by
00:51:13
Speaker
ah the American Society of Radiologists, and um the data set that they provided is one of the best ah medical ah public medical data sets where you have millions of this kind of, r i will I don't want to say something wrong, but very, very huge data set was x-ray images, which were um uh, extensively also labeled by, uh, many, many, uh, kind of radiologists who were also double checking each other with the labels. Oh, that's very important. Right. Uh, so that was the labels were good. And then we took this data set and then we said, uh, we want to, um, um, to build a process. Uh, and how we want to certify or bring a product
00:52:13
Speaker
a medical product, AI product to certification. And then I have to maybe get one step back to explain to everyone who are listening ah the reason why. So there is a reason well we came to this before, namely ah that TIFF Austria, so the Austrian section of TIFF, were um creating a catalog as the certification catalog together with ah the Technical University of Linz. And this catalog is, um I think, around 200 or so requirements for Trusted AI. They called it ah the Trusted AI Catalog. And it's a kind of an end-to-end catalog. So it is starting with the business case definition.
00:53:12
Speaker
goes over to the data part. It has even a chapter about ethical ah requirements. And then it's the it has a chapter about the model ah modeling requirements and then deployment. um And then we said, okay, um we had contact with TIFF Austria ah even before for that project the project for Cariad.
00:53:38
Speaker
And then we did the certification for CARIAT for that use case back then. And then we said, OK, as the econium data, we want to do it also for us, for the econium data. um And then we took this planar ammonia and said, OK, we tried to um to build this process and on the basis of the planar ammonia and then go to TIF Austria and let it certify as a And that's called a conditional certificate. So conditional means it's a certificate for that for ammonia use case. And then it's um the reason is to show how the process works. So ah you then say, OK, if you go with our process for your use case, ah the probability that you will be able to certify also
00:54:36
Speaker
under this catalog, it will be high. And so we wanted to learn. So the the reason was also to um to learn how to ah to bring medical product with AI up to certification. And that was very, very interesting, very also fun because then we um ah we um tried to build this evaluation systems, how to do explainability, how to show the robustness and uncertainty stuff. It's another topic there. And then at the end, Tif Austria, who has have a dedicated technical team with data scientists, did an audit of that primordial detection.
00:55:26
Speaker
with us okay and we created a report and we've we made fun and we and the team we thought we called this report that we generated our master thesis because it's kind of 90 pages report that we did. 90 pages. Yeah. And then we also kind of built a kind of a system. we cook It's not fully automatic, but we tried several ways to kind of For example,
00:55:58
Speaker
um um create automatically some part of this re report when, for example, kind of evaluations of one part is coming out, it's generating automatically some PDF, which is then extended to that report and so on. And then we sent we sent that report to TIFF Austria and then Uh, we had, I think six to eight, uh, calls each kind of two hours. And then the, as you called it back then grilled us in a yeah nicely manner. right And then they are always very respectful, I should say. Yeah. And then, um, and then, um, yeah, told us really also in the first, um, um, first phase.
00:56:50
Speaker
after the first audits really said, okay, this chapter, you were really good. That was a pass. This chapter was also good. And then this chapter, we have recommendation, for example, um ah about the ah qualitative model evaluation, where we did kind of how the model is performing in several the domains.
00:57:17
Speaker
um I said, okay, here we have some recommendations to do something more. Like like imagine you would, in essence, ah submit a paper to a journal to get reviews and so on. And then you just extend it. And then in some other parts, they said, okay, this is not enough. You have to do this more and more. And and and then we had the second phase ah where we then in the final audit,
00:57:46
Speaker
ah addressed all these issues and then afterwards they ah granted us the conditional certificate for that primonia detection use case. That's really, really cool. ah just Just so for me to to fully understand, I don't think I really got what you meant with the conditional certification. I mean, yeah it it was a pass in the theoretical sense, or i mean because you didn't really make a product to get the staff, yeah right? But they they would have given it to you. does Is that what it means? No, it is actually. so
00:58:22
Speaker
ah Yeah, so um yeah, it's difficult to explain you, right? It is actually, if we would go for this product part, we would also, we we would have the certification. So in that sense, it's not theoretical. okay um ah Conditional means just that um ah The certificate is only valid for that for that pneumonia use case. Right. Oh, that's what you mean. Yeah. Yeah. It's not so you cannot take the certificate. Let's let's imagine. Let's just ah um just as a kind of ah theory, let's say I would build a detector for cancer and x-ray.
00:59:11
Speaker
And then I would bring it to market and then use my, my certificate for that. Right. Okay. That would not be valid because it's just only, uh, it's only valid for that pneumonia. But, uh, nevertheless, uh, the, uh, the idea for us was not the, to bring this to market, but to say, okay, we, uh, we want to extract, um, um,
00:59:41
Speaker
usable standard methods ah for other use cases in that image-based use case, at least to say, okay, we've done it for pneumonia, but our ah best practices that we built for that are also valid for for the cancer case or for any other case, which is in the image domain. And then our idea was to um also in the future, extend that to, for example, to language systems. ah yeah Of course, we have to adapt and because just as an example, that the listeners understand that really directly.
Understanding AI Processes
01:00:28
Speaker
um If we speak about robustness, does that as an example?
01:00:35
Speaker
There are tools to measure robustness of often AI, which is ah for images. um ah Some of these methods may be functioning also for voice or image, for voice or text-based AI, but many not. So you cannot just take this and then apply it for that. So we need then other methods and tools for this other use case.
01:01:03
Speaker
ah So that's what it make challenging. And also um ah the endeavor to build cont continuous AI assurance for any use case is quite challenging. And so my idea, so we we can talk about this a bit more in detail, but just as an interesting fact. So also one of my um early ideas back then was to ah built a technical framework which is automatically like CI CD running and then you adapt it for another use case ah with the help of a configuration file like ah maybe some of ah you or some of the listeners know
01:01:53
Speaker
ah YAML files as in as an idea, so you have this XML or YAML paste, and then you have a file which is then defining on a configuration level, let's say use case polynomonia, and then under it in the text, in this YAML file, you write, um ah tool one, ah robustness equal XYZ.
01:02:21
Speaker
and so on and so on. And then if you have a voice yeah ah use case, you take another um configuration file and write another tool. And then the framework just builds itself on top of this configuration file for that other use case. yeah So that was the idea back then, but it's still kind of an evolution because the yeah as you said, you have ah thousands of AIs.
01:02:50
Speaker
And then ah you have done for each of these thousand different flavors. Yes. and But it could still happen because, I mean, just as you described it, and ah you talked by now about YAML files or, you know, the XML files. Yeah. you could
01:03:10
Speaker
I think you're for me to make this very graspable, at least for the other developers or not so much developers out there, it's like I always think of a Dockerfile. You have a yeah yeah five five lines of code. If you click, let's go, and all of a sudden, this these five lines of configuration create an entire finished system architecture with all the necessary components taking care of everything. um And there's also, of course, a massive catalog out there of what other people have built that you can just use, you know, there's not, yeah which is great, right? And it just, based on what you said, it reminded me a little bit of this, it's just like the does the architecture.
01:03:57
Speaker
Yeah, and that's a great image and that's also kind of a very good um ah good picture was the Docker system, Docker file, and something in a similar way I heard, for example, also I had contact with people, for example, from this mission Cai, or mission Cai,
01:04:21
Speaker
from the Fakias Ministry of Traffic traffic gets on and Transport. They built this mission and there is ah one group which who are developing so-called smart machine readable standards.
01:04:44
Speaker
And this is similar, so they also talk about this kind of XML, YAML-based files, and then the where you define standards, but in a machine-readable way, so that then at the end, the machine system will read in the standard and then test your AI system against these standards. the standard yeah yeah yeah That's really, really cool. I think a lot a lot there can happen. i A lot of this is going to happen. yeah What I really like about how you went about it with the Pneumonia, it's
01:05:22
Speaker
It's literally a bit reverse engineering, right? You you yeah yeah take something and you only take it to understand how you could develop a process to make it work. yeah right It's like ah literally the the reverse engineering way. not Not to reverse engineer the product, but to reverse engineer the process. Yes, absolutely. Do you directly talk with the regulators?
01:05:46
Speaker
yeah who then, which I also think is very important to mention, right they are not that they're portrayed as your enemy very often yeah from a business perspective because they're just someone you have to go through.
01:06:00
Speaker
um
01:06:02
Speaker
but they are all willing to hear you out, right? I mean, similar to the ones that were in the audit, they were very respectful. And it wasn't just a, you didn't answer my question. You didn't get this point. It wasn't like that at all. It was, okay, let's talk this out because it seems like there's some communication barrier here. yeah And after one hour of conversation, it was clear that, oh, okay, that is what you mean. It was just a vocabulary issue.
01:06:32
Speaker
And it was all fine. No, it was a pass in the end. Similar to what you said with, okay, so this is a pass. This is a pass here. You need to adjust this a little bit and then it's going to be fine. and Literally like a paper review. So yeah, I would say more stressful, but still more stressful. And I think it's really cool that you went about it in this reverse engineering way and that the turf was.
01:06:58
Speaker
was actually willing to go through it with you. I think that's what's what's really, really cool and that's what we often do not hear um about the TUF is they are, and and I also must say they were also present at the the applied data summit, yeah they they are now everywhere. When it comes to AI stuff, they go to every single conference because they understood at some point, it's like, okay, we need to catch up, yeah so let's do it. yeah Which is something that doesn't often happen when it comes to regulatory, um governmental regulatory instances. They are often just saying like, we're the final boss and that's that.
01:07:45
Speaker
I mean, they have kind of pragmatic, um, perspective in the end. i think that's And also, um, kind of, as I said before, so we have this, um, uh, the TIFF, TIFF AI lab, which was built, uh, I think kind of, uh, originally TIFF note, uh, section. Yeah. And then, but the TIFF AI lab is tasked, uh, with building, uh, these new standards, which then at the end,
01:08:14
Speaker
ah should serve all the TIFs. And then ah ah you mean, for example, at Applied Data Summit, there were our colleagues here from TIF IT. So the IT shares also from TIF node section. And there, for example, ah for many years, um have expertise in classical cybersecurity and come from that direction, and which is also very interesting.
01:08:43
Speaker
ah because there is, um to my view, a very yeah interesting and important link between the cybersecurity domain and the AI robustness. um Also in the future will be more. And that was kind of the good contact that we got with our colleagues from TIF IT.
01:09:10
Speaker
um because they're also ah what is quite fun and I like much ah in a collaboration with this technical people is this sense of kind of this curiosity to build new systems and processes and I like it because I just had it was so fun when I started nothing was there and no one knew how to do continuous assurance or any assurance of AI. And then by kind of, as you said, nicely, the reverse engineering, the process from ah from a use case, it has a very scientific ah feel to it. Like, yeah for example, in many areas ah in science, in biology, you have these your toy systems like the drosophila we um make fun of. it
01:10:05
Speaker
the toy system of the biologics. And then in physics, we we often said the harmonic oscillator is this. And then ah we need for in AI section also domain these kind of, let's call it toy system or just um learning systems, yeah where you learn. And then you only can build the systems if you go and build it.
01:10:33
Speaker
So um you cannot, you can, ah this is what I always try to ah to say to ah to people and um who are kind of want to know how to to really build safe AI. um I say you need to be pragmatic. You cannot build trusted or safe AI in theory.
01:11:02
Speaker
So you cannot think about building it. You need to build it. You need to build also AI. And also, you must be prepared that it's failing. Because when it fails, you learn yeah how to mitigate this failure. as a Because ah you cannot, in theory, think about what is failing and then mitigating before. So this kind of also fitting with the notion of risk.
01:11:32
Speaker
for example, because often ah we tend to be risk averse instead of risk managing. And for risk management, you need to build the systems and you need to really ah take the risks head on and don't try to um mitigate them beforehand.
01:11:56
Speaker
yeah Yeah, exactly. It reminds me of ah what my professor used to say. He said, your job as a scientist is not to find data that supports your theory. You're supposed to find data that breaks your theory because then you are actually expanding on your understanding. The rest is just, it's just well, how how would he phrase it?
01:12:22
Speaker
It's like a a wall is painted white and you're just throwing more white paint at it. You're not looking for anything else. yeah you're You're not learning anything. There is no contrast, no additional information. right yeah that that's So your job is to try to break your best idea so you yeah get to a better one. yeah which Well, it was definitely good advice from a scientific perspective and with the AI systems, it's the same, right? When it comes to risk, you have to break it to then get to a point to then be challenged to actually think about, okay, yeah it broke. It broke under these circumstances. So now what are we going to do about it? Yeah. And then you develop these methods, you develop kind of systems around it.
01:13:11
Speaker
ah to exactly manage this risk and so yeah. and yeah Let's get a bit more deeper. yeah and it's I think it's really good that we talk about these frameworks and I think from for a lot of people understand now a bit better what the AI assurance is, specifically, I would say the ones that are not the developers, but now thinking about a developer perspective um or researcher perspective.
01:13:38
Speaker
you know We talked at the beginning about you know accuracy and assurance, you know something quantitative. and yeah That's not supposed to be qualitative. yeah um What tools can you recommend or what tools are out there that developers or AI researchers use or could use, maybe that that you use on a day-to-day basis,
01:14:01
Speaker
yeah that that that help you in in understanding the model better to do actually the quality assurance. yeah um Yeah, there are a lot, but I give a kind of sample of those, which are kind of ah one of these, um which I would suggest highly.
01:14:27
Speaker
and recommend is the tool called ah Weight Watcher. So it has also funny names, so it has nothing to do with the Weight Watchers. ah But it's it's called Weight Watcher because it literally watches the weights of the neural network. So um yeah ah it was developed by actually also bio theoretical physicists who are also very active in further developing and he's currently working also on a paper that was in contact with him the last months, working on a general theory of neural networks. And oh, that's cool. Actually, it's actually based on a specific it's ah based on a
01:15:17
Speaker
um theory ah the tool actually is based also on a theory that is actually coming off ah from statistical physics ah area that's why I liked it because I'm also coming from that area I understood it quite well and what the tool is why the tool is so good to use first off of course important for developers it's ah um It's a Python package, which is available very easy with PIP. So you can just do PIP install Weight Watcher. It has a great, ah very very nice documentation page with tons of examples ah ready to to apply on, ah let's say you can take take it to a Jupyter Notebook and immediately start working on it.
01:16:14
Speaker
and um And the second great advantage of this tool is um that it only needs the model and no test data. So it just looks on the quality of the model from the perspective for each layer, in essence. So it' the input of the the tool will be the layer matrices, the weight matrices.
01:16:44
Speaker
And then it just derives a quality metric for each of these layers. And without going to detail of this kind of complex the theory behind it, there is a kind of, ah i it's called alpha metric. And the alpha metric, according to the theory, has to be between two and six. So and the numbers two and six,
01:17:12
Speaker
are a good or well-trained model. And that brings me kind of to the summarizing advantage of the tool that you you can take this this nice one-number metric in order to kind of also track the evolution of your model while you're training. You can just, ah for example, together with this what the the data scientists always do with the accuracy tracking and the loss of the training tracking. You can also plot the alpha metric of the Weight Watcher tool and then see, okay, whenever the the alpha is going, maybe ah tending to go below two, you stop your training because you then go into over-training.
01:18:10
Speaker
so the the essentially the model says so that the theory behind the model, the stool says, um when you're above six, you're in the under training regime. And when do you you when you go below two, you're over training. And, hu and ah what is also great, um you, ah you can put to the stool, not only the
01:18:41
Speaker
this classic, now classic um AI in neural networks, like um let's say convolution networks and RNNs and so on. But you can also assess the quality of LLMs. So oh yeah was it? Oh, that's amazing. And you can also do put ah output qualitative kind of plots, which also give kind of a metric of yeah the quality of the model.
01:19:11
Speaker
you can compare different model versions with each other and then also say, okay, how well was version two compared to version one? um So that's looking probably one of the, ah which is, um if I would look, if I would be searching for a quantitative tool, I think that is coming very much near to to it. And I'm curious,
01:19:42
Speaker
um ah so based on this theory, I mean, just to to so people know, it's a theory of heavy-tail self-regularization, right? Yeah. I'm wondering, so people people can, of course, look that up because that's ah yeah it's a-hour podcast just in itself, just to scratch the surface.
01:20:06
Speaker
um I'm curious, i mean you said ah at the moment we have to assess a model, we look at accuracy, we look at loss. you know yes has there been I didn't look too much into into the paper for Weight Watcher.
01:20:25
Speaker
um When you do train a model and you know you plot the the loss and the accuracy, where of course the accuracy should go up and the loss should go down, yeah is there then also a graph that shows that you actually do end up between you know when you look at this alpha value along those two?
01:20:49
Speaker
um, other metrics that it does actually end up between two and six. And when you try to do the, the validation or the cross validation of the model that with a parameter of under two, it performs very poorly. If you do a test data set. Yeah. um Yeah. has that been Yeah. There is, there has been, and also, uh, kind of, um, we was the primonia that's a good, uh,
01:21:15
Speaker
a good question, actually, which brings me to also to to our Neumannian case because there, for example, we explicitly showed that there will be a naive, ah very well-performing neural network. yeah We trained the model on the system, which was very highly accurate, well above 90% accuracy. Wow.
01:21:42
Speaker
ah but it actually did not look at the regions of the paranoia at all for its prediction. So we used another tool, this this kind of, ah the tool is called, this would be the second, which I maybe recommend, it's called SHAP, Shapley additive explanations. um And then we used this tool to to track down for for all tests sets for the whole test set to track which regions of the primonia actually decide ah for that this this image is primonia or not primonia. And then um the challenge is, of course, you could do it quite qualitatively, but it's very cumbersome to look at thousands of images and decide, OK? But then we developed a process, a technical ah process, where
01:22:42
Speaker
ah where we did run SHAP on whole batch of test sets and then extracted the mean SHAP values across all test sets. Then you get a kind of an averaged. For each class you get an average and then you compare it to um and our naive model was actually the ah The heat map was completely distributed along the image. And not actually looking at these regions where the primone is. And then we um we found later out what actually decided this. And and that was not actually the the region, but there was a hidden variable or hidden, I would say hidden feature
01:23:40
Speaker
in the data set that you're not seeing at first sight, namely um the question, if the people were measured via the x-ray at the patients, were they actually standing when they were measured or were they lying in bed? ah The people who were lying in bed were very, very sick.
01:24:09
Speaker
and the The system actually did ah had a classifier. At the end, it was a classifier for being lying in bed or standing. And the the only thing is that the correlation the correlation of lying in bed with the pneumonia, with sickness, was much higher. yeah Because the people who were standing most probably were not sick.
01:24:36
Speaker
yeah and Yeah. And that was actually the correlation, which was not indirectly in the image, but it was hidden behind. In the data of and it's crazy how did you find that? How the did you find that? and said We just did in a team kind of the, um ah we did kind of analysis actually. And then what we find out, we did this Why we found it is that this first off that the system is not functioning very well ah is via this shape and the image thing. And then we had some discussions and then we actually tracked it down by a kind of logic discussions. And so that's I think we did also some analysis, but what I wanted to say, because your question was regarding Weight Watcher, we did also an analysis of the system was the Weight Watcher.
01:25:31
Speaker
And essentially we saw that ah the overall, the specific layers, but also the the model ah was ah kind of over trained on the training set. And so the alpha value was way down.
01:25:50
Speaker
Wow, that's the two. And you can track it down. And then we just what we did, just as an interest, because maybe interesting for you and also for the listeners, we had then to to mitigate this issue of this wrong correlation. We ah built a two-stage system, AI system, where we ah partitioned each X-ray image into
01:26:21
Speaker
ah kind of smaller sized images. And then we um first um kind of classified with a model these small images. And then we had another system which only operates on the set of the sub. And then we tried to just ah by that the um we found that the ah kind of the system then overall looks more into the regime where the primonia actually is. Right. That's crazy. I mean, that's it's it's
01:27:03
Speaker
A beautiful example that shows that the type of training data and the the balancing of it is just so, so, so, so important yeah because, ah and you cannot, you cannot just rely on this um accuracy score. You just count. Yeah. Yeah. That's, that's essence actually. So we cannot, and also um ah these days kind of the,
01:27:33
Speaker
accuracy and precision and so on are ah kind of the classic metrics. But this is ah ah you will not come through certification with accuracy and precision, in essence. So you have to really show um directly into the model what yeah actually what is but by what is it deciding. So what are the deciding factors? Yeah.
01:28:02
Speaker
yeah yeah Wow. Yeah, okay. So ah just so that as a quick summary, it's the SHAP tool, which is also a Python package, right? Yeah, SHAP is also, and the good thing about SHAP is also um why I like it. It's um um it's applicable ah both on for image-based use cases. It's also, you can use it for language systems.
01:28:29
Speaker
And then it makes them the heat mapping on the text. So it says which words and so on, decided. And voice, I think you can do also. um And what I found, there is another tool, but it's kind of, it's similar to SHAP, but it's developed by Fraunhofer.
01:28:52
Speaker
ah I think in Berlin, the and hats yeah phone of find hats ah This one is called LRP. It's also kind of Python package, ah but I think it has another kind of license. It's for scientific license.
01:29:15
Speaker
okay ah But ah the the good thing they found, they developed also a version of this LRP. It stands for Layer-wise Relevance Propagation.
01:29:31
Speaker
um It's also they developed a version for LLMs where they just track also what's really, I i heard it last year when I was at TIFF AI conference, um the professor of that group, Professor Wojciech Samek um ah had a talk about this and then I found really interesting what he showed about um and was a tool. They were able to also um track down ah concepts that the model learned. So ah concept classes and so on. and Using this approach, this package. me And then showed also ah what happened. So ah they could really track down which kind of neuron in which layer of the LLM.
01:30:30
Speaker
Uh, if you is responsible for which concept and then if you block it, then it's doesn't find the concept.
01:30:41
Speaker
and
01:30:44
Speaker
Huh. That sounds very interesting. Um.
01:30:52
Speaker
Wait, I'm a little baffled. That's pretty cool, actually. I didn't expect that. Okay. So, Weight Watchers, Shapp, and ah LRP, and with LRP for the commercial ones, check the license first before you yeah yeah before you just yeah use it. but you can come I think the good thing is regarding the tools, and I'm also very um interested in, us on a daily basis, also researching. and trying to find new tools because the tool area domain is evolving very much and also it's an interesting area and I would also ah recommend people to um yeah not only go into ah AI, the basic AI, but also what is interesting is development of meta tools for AI.
01:31:46
Speaker
Uh, in the future. So this becoming very important, but very important. And also kind of, uh, uh, what I've found very important in that regard is to, we need to really, uh, take care of these tools. And I would say 99.9% of these tools are open source. Yeah. Uh, but, um, not many people, uh, in invest.
01:32:17
Speaker
resources in development in these tools. But these tools will be important in future when we want to build um agile assurance systems ah for our AI. And so we need to create standards for these tools. and Because one of our my colleagues, you you you met him also at the Apply our conference, Applied Data, Alexander Akka, who had also talked about LLMC, taught me one day, um yeah, if we use a specific tool for robustness evaluation, for example, ah who and by which tool ah is actually measuring yeah the robustness tool itself?
01:33:15
Speaker
So if, yeah so the robustness tool is essentially, then the circle is with what you said, the that robustness tool is the ruler. Yeah. So exactly how will you measure this? that The ruler is functioning correctly. And then also he said, ah maybe company A was a similar use case is using for their testing.
01:33:43
Speaker
um robustness tool A and company ah B as to using C yeah and you have to well how to compare this to robustness exactly with each other. You know what would be great is maybe it's just a wishful thinking if ah if these type of tools because the
01:34:11
Speaker
It is a continuous process. Documentation is a lot. And the more things can be automated also means the um the less manual labor it will be for the tooth, for example. yeah So ah my gut feeling says that it would be of great benefit um if some already existing open source projects were supported by the TUF and at the same time checked for their integrity so it has the the quote-unquote TUF stamp of approval for your AI assurance and equality assurance so that you as a company also know I can use this tool and as a TUF says it's fine. Yeah. That would
01:35:01
Speaker
I guess that would also close the circle well. Yeah. Yeah. And this is also kind of many people working currently also on to this also the two free eye lab. I know that they also currently um assessing different tools, and different open source tools. And um I think tools like weight, water and shop and so on of course have kind of high probability to get into this domain standard tools. ah But there is room for other tools. And we also kind of regarding uncertainty estimation, which is we didn't talk about is also important. There are also tools. It is. And um yeah, but this I think is just
01:35:57
Speaker
so ah In essence, a field of its own where people can get into and have an impact.
01:36:10
Speaker
I think a massive impact because if you think of the downstream effects, it's it's massive. right research Research can be slow. I mean, it's always how it goes. right Research is slower than the product development cycle based on the research yeah and then also the tools to create the products. It's a cascade um that takes place.
01:36:33
Speaker
yeah So yeah, this is super, super interesting. Arash, I have many more questions for you, but we've already been talking for quite some time. um I leave it up to you.
01:36:46
Speaker
i mean I still, maybe we can just scratch the surface on it. You mentioned the cybersecurity aspect. Yeah, we can talk about this. let's Let's start superficially first, because this is also a very deep hole you can get into. um But I would like for people just to understand what the impact of AI can be on cybersecurity and the other way around as well, and which is also not often talked about.
01:37:17
Speaker
Yeah. um and ah yeah let's let's let's Let's go with that as the last main topic. Yeah, um yeah that's a good, also interesting topic, which I came um the last weeks and months extensively to.
AI and Cybersecurity Intersection
01:37:34
Speaker
um it's Actually, it's to be on a ah high level first, um and also very important and these days was the Gen AI.
01:37:46
Speaker
ah revolution. ah It actually came also um and very much linked to the Gen AI and LLM topic. um One issue, ah let's start first with the classical cybersecurity. What we know is that um you talk about um um the ability to attack classical kind of IT systems.
01:38:16
Speaker
yes by different means and you want the ideas to ah get a hold onto the system and then do whatever you want with the system or hack the system and the cyber security um ah tries to mitigate this so find ways how to make the system secure and- The holy grail is root access. Yeah, and and what we, of course, we need to, um ah with one thing, we have to be clear at the start, namely that there is no 100% security.
01:38:57
Speaker
it um ah We know it in the domain of virus, for example, yeah um It's also not, there will be always a virus that is getting past any antivirus software. And ah similarly with cyber security systems or anti-cyber security frameworks, you get, ah there will be always an attack that will come past the framework. yeah um The idea is to
01:39:35
Speaker
um to guess get the best security you can get. And yeah um ah what now with AI and specifically was the age of Gen AI tools come into it is the to my view in a novel um way or novel door for attacks.
01:40:02
Speaker
ah because you have now the combination of the classical IT systems enriched with AI. And and there is new ways how you can ah kind of um attack the systems via the AI. ah So think about kind of you use chatbot.
01:40:28
Speaker
Everyone can use it from from internet, and then um you try to kind of ah convince the chatbot to do a something you want. ah The early the systems were very easy. ah you You had something, you just, yeah, Dan, for example, do anything now.
01:40:53
Speaker
yeah that but
01:40:56
Speaker
I think this one, this one is closed already. So I had also um recently was on a talk um or a talk series regarding GI and chatbots and then the, the open AI and the other companies who are working in this area already closed many of this easy stuff. But there is very much as a sophisticated ways.
01:41:24
Speaker
ah for example via prompt injection and other stuff you can do. And um now to come to kind of the interesting cybersecurity threats, imagine you have um ah these systems that you interact with like chatbots and so on integrated within, um ah we call this,
01:41:53
Speaker
are um Experts call it cyber-physical systems, cyber-CPS. So CAR, for example, with software, NEI, is a cyber-physical system. So you have a physical system, and then it's cyber-enhanced. And then if your medical device is also kind of a cyber-physical system, um when you enrich the systems with let's say a chatbot sophisticated genii, which is also maybe connected to internet. And then you have interaction, uh, probably you have then a way to attack it. A malicious actor maybe is able to do prompt the injection and ah overtake the car. Let's go that way. Um, yeah um that is one maybe very
01:42:50
Speaker
ah visual visual way to but to get it understandable. And um there, for example, it has a link with a topic we talked before, namely robustness. And that's why we had um with our colleagues from TIFF IT that you mentioned, to ah they were also an applied data.
01:43:12
Speaker
Toga Makat and Vasilios Tanos. They're um coming from cybersecurity and we I worked in robustness topic for for a while. And then we had this connection between these two topics because um um now you can even um argument that um robustness validation is very important to mitigate cyber security threats um because you need to to use robustness evaluation validation in order to close these loopholes where people can attack the system via kind of these prompt injections or injecting changed input
01:44:08
Speaker
and so on. And so that's on a higher level. um ah By introducing, to summarize, to by introducing these kind of elements into these classical hardware systems, you ah you expose a new yeah kind of window. A new way. For people, a new way to attack the overall system. Of course, yeah if the system is just for entertainment, let's say in a car, you have chatbot just doing entertainment stuff.
01:44:39
Speaker
It's maybe, but then again, still you have to, because the overalls a car as a CPS, cyber physical system is very complex. i You need to ensure that um it has not by any means, maybe a connection to a to a system which is say safety relevant. right But then maybe in the future,
01:45:09
Speaker
you want to have a chatbot, it would be be be nice for the driver ah to have a chatbot ah to also control driving.
01:45:21
Speaker
yeah um and Then getting interesting regarding safety. Yeah. So I think it's um from a cybersecurity perspective, right? From a classical perspective, as you said, right? You have these, ah you have login portals, you have communication between servers, you have different systems. And now you add another system, which obviously if connected to
01:45:54
Speaker
the downstream um elements of of, let's say, the back end, or in this case, a car or or a device um can well can be manipulated in one or another way. yeah It came to me now, just now, even another example, maybe that is maybe also easier to to come grasp.
01:46:19
Speaker
Let's just think about um chatbot systems or these AI systems that are integrated in our smartphones and serve as our personal assistants. Yeah. And then maybe we are controlling it by our voice and by our personality in a sense, but what if someone is kind of convincing our assistant that he or she is us yeah and then, um,
01:46:51
Speaker
extracts information about us from my phone yeah via the chatbot over the internet. and You know, it reminds me a little bit of, um, it was, I think mid nineties, I think at the beginning of the 2000s, they already closed this. Uh, it was, do you remember the old analog phones right before, yeah for smartphones, you know, when you had the, there's still the dial phones and you had phone booths. It's a long time ago. Uh, wait, if you rec, and
01:47:25
Speaker
By dialing a number, a tone was transmitted through the system, meaning the tone itself carried information. And if you had a specific combination of tones, you could, for example, fool the system to make um ah to make long distance calls that were for free, because yes, back then you still had to actually pay for individual phone calls. yeah um And there was no WhatsApp, there was no signal, there wasn't any of this. And that they closed that, um and of course that was deterministic. But now, if you were to include a chatbot, you can think of these tones as these individual words or phrases that you just
01:48:10
Speaker
ah w lish ah yeah you You throw it into the system, you kind of already know what's going to happen, of course, in a slightly more fuzzy way. yeah But you just throw it out like a script, like ah like a bunch of commands, in essence. And it doesn't have to work 100% of the time, it just has to work 5% of the time, because now it's a numbers game. Yeah, that's true.
01:48:39
Speaker
yeah Yeah. This is interesting. I never thought about this. This is a ha.
01:48:47
Speaker
ah Okay. Yeah. But this is of course also um from the, I think from safety and security research perspective is quite very, very interesting topic. I even, I had by research found some papers which were really fun.
01:49:08
Speaker
and interesting to read, who even just kind of ah dive into some kind of machine psychology, in essence, because then it's kind of um really this convincing of a machine to do I think is then when the the machine, these LLM's are so complex, then it's kind of ah advanced like and psychology for machines. yeah Yeah, it's basically, well, machine social engineering, except there is no social because it's a machine. But yeah, you would still apply a very similar tactics to see how you can get there. And the next level, I think you had also, I think one of your previous ah podcast episodes, I forgot the name of
01:50:06
Speaker
the person, but you talked a lot about agents, so agentically. Yeah. Yeah, that was with Sven. Yeah. Yeah, Sven. Yeah. And, and the next level is kind of in this area, um which you then get another non-linearity into it, the, when you have several agents of the technology interacting with each other. and Yes. Yeah.
01:50:36
Speaker
who
01:50:38
Speaker
Okay. Yeah, just to to scratch the surface when it comes to cybersecurity, because this, I could talk about this for hours. It's also something I really like to talk about. um Yeah, but this may be an idea for upcoming. For next time. Yeah. For sure. For next time. But just to get back
01:50:59
Speaker
to the main parts when it comes to ah trustworthiness of AI and continuous assurance.
Challenges in AI Assurance
01:51:08
Speaker
um which pressing i mean You've been working on this for quite some time and you've been getting closer and closer to to um you know something that say non non AI people can also understand and use in their daily practice. But which pressing questions and hurdles are right on the horizon that still need to be tackled that you see? Yeah, um I think um ah one of the most pressing um apart from this, I think what is currently I think the most pressing is the
01:51:49
Speaker
assurance of these large geni eyes like LLMs, um which is posing a big challenge. But apart from it, for me personally, I would ah put something which was um already a big challenge before ah the advent of LLMs, and that's the um the challenge of um ah continuously learning or dynamically learning AIs. um Because even today, to this day, ter um there is no standard or there is no process for certification of dynamically learning AI.
01:52:40
Speaker
that's try so So it's all only possible to certify in this domain those freeze our frozen systems. Classically, we trained a system, then it's coming in a production, and then in when it's in production, then it's slow it's frozen. But it would be nice to have at some time ah systems that um also learn while in production. But this is opposing a real big challenge for kind of this continuous certification.
01:53:19
Speaker
ah You need to build kind of ah very smart monitoring systems in the end, which is kind of kind of assuring on the go or kind of that have kind of alarm systems, which maybe tells, okay, the system is learned, but it's going maybe in the wrong direction. You need to take care and then kind of fail safe systems also maybe a way to fall back to the frozen system when it's going into a wrong direction. But it's, ah you see, I have already kind of thinking about ideas, ah but it's ah really challenging. I know from also from um our colleagues from Tif Austria who actually developed this catalog, they're also working on this topic with Theo Linz.
01:54:19
Speaker
um on this challenge. um I personally would like to to see some improvements in that area. For sure. I yeah i mean, I remember when I was at the DMEA last 2025 now, I think there was 22 or three. and I don't remember. um I mean, I gave a talk about you know am AI and health tech and all of this stuff. And one person literally, and i was there was also a person from Tuft there as well, and from the the VDI,
01:54:52
Speaker
so yeah photo d d i v d yeah find deutsche engineer ah on yeah I think it's electrotechnic. Yeah, but they were there as well. And one person from the audience asked, well, you know, wouldn't why, exactly as you said, why aren't there any systems that dynamically learn based on, you know, the doctor says, actually, this diagnostic was wrong. This, ah it should have been this. And why isn't that immediately fed into the system? And, well, I literally said what you just said, it's like, well,
01:55:30
Speaker
We have no way of knowing if this data was any good because it it needs to it needs to be cross validated by someone else as well. you know If we're talking specifically about these high risk systems that is so important what the system is fed and you cannot always tell after the fact what the effect would be.
01:55:53
Speaker
And the TUF person was also like, yeah if we we don't we barely have a certification for it as is. yeah we cannot we cannot ah we we We don't even know how to think about this yet, which I thought was quite
Rapid Development and Causal AI
01:56:08
Speaker
quite interesting. And now it's two one or two years later, yeah technology is developing so fast. and yeah Everybody's still like, okay, how do we do this? Back then, back then, uh, chat GPT wasn't there. So no yeah, that's true. And then this came and then everything. Exploded. Exploded. We're in different world now.
01:56:33
Speaker
You know, I mean, all of these things that you do are super interesting, what you're working on. um For the listeners, are there any books or resources that you can recommend if they want to learn more? And this can be about, you know, anything, even if you think like, Hey guys, you know, ah this book about theoretical physics or this theory would be quite interesting to you, maybe whatever it is. Yeah, I would, um I would think um there is actually, um,
01:57:03
Speaker
one great book that I would recommend it's actually not directly um directly was that topic but it's still what I really liked and I think influenced me a lot last year was the ah the book of why by Judea Pearl which is actually about um But Judea Pearl actually is the farthest godfather of Bayesian networks. Ah, yes, yes. um And the book of Y is about the question of what you call not the correlation, ah causation. So about causel causal AI.
01:57:58
Speaker
So that's really written greatly and interesting. But, yeah, apart from it, what I would also recommend
01:58:16
Speaker
because, yeah, it's more technical, I would just recommend to um to step into, if you if the listener is interested also in kind of ah theory,
01:58:29
Speaker
um oh What I really liked, it's not a book, but mostly kind of a collection of ah papers, the information bottleneck theory. and It was actually developed by um this um ah physicist, also Israeli physicist Naftali Tishbi,
01:58:58
Speaker
Um, there is also actually ah great, um, uh, article about this. Uh, if you ah search on Google, uh, for, uh, after the information bottleneck theory, you find this, uh, article very, um, that's the written article on Quanta, Quanta magazine, which is free available. And they, uh, actually, um, what I found interesting. So the.
01:59:25
Speaker
motto or the title what we could put onto this ah is that the most important thing in learning is forgetting because um ah the theory actually shows or tries to um show how a neural network actually learns and then finds out that um there are two phases in learning.
01:59:55
Speaker
So first the network while training, yeah it extract extracts all information it gets like a vacuum cleaner from the data. And then there is a phase transition. So a certain phase transition, like in physics, we call it like in transition from fluid to gas, ah from so solid to fluid and so on. There's this kind of phase transition ah to a generalization phase. And in this phase, it the system is not extracting information. It throws away information. So it that abstracts away and throws away. And then the theory shows that actually, while yeah after this
02:00:49
Speaker
You just, the information measure of according to data is going down while the information about the label is going up at the same time. So you get this kind of generalization. um That sounds interesting. I will definitely give that a read. Yeah. And I would, yeah, just especially while this year was kind of this, uh,
02:01:16
Speaker
Nobel Prize of Physics got was for two researchers behind our networks. um In essence, sense I think it could be ah trigger a revival for um more physicists and more theoretical physicists stepping into this area. And I see already some who are working on this.
02:01:49
Speaker
That sounds really, really fun. I will definitely give this a read. i gotha You know, after this conversation, I have so many more questions than I had before. There's definitely going to be around too, yeah to also dive deeper into the cybersecurity aspect. Well, ah ah thank you so much for this great conversation and your time. I mean, when people want to reach out to you, you know, to where can they find you?
02:02:18
Speaker
I'm um i'm um ah firstly and mostly ah accessible over LinkedIn. so yeah ah instagram ah and I put all those links in the show notes just so yeah yeah yeah yeah yeah so people don't have to write it down. Yeah, I think mostly I think I would say LinkedIn. LinkedIn. Yeah. Cool. Let's do that.
02:02:46
Speaker
Yeah. And ah you can write to me and contact me over there. I'm very active on LinkedIn. All right. Well, then that's definitely the one I'll put there um for people to easily reach out to you. Well, again, I can only thank you so much. We have been already two hours. um yeah Thank you. much and It's been a pleasure. And well, to everyone listening, you have a great day.
Closing and Call to Action
02:03:17
Speaker
Hey everyone, just one more thing before you go. I hope you enjoyed the show and to stay up to date with future episodes and extra content, you can sign up to the blog and you'll get an email every Friday that provides some fun before you head off for the weekend. Don't worry, it'll be a short email where I share cool things that I have found or what I've been up to. If you want to receive that, just go to achmal.com. A-D-J-M-A-L dot com. And you can sign up right there. I hope you enjoy. it