Become a Creator today!Start creating today - Share your story with the world!
Start for free
00:00:00
00:00:01
AI-Powered Job Skills Inventory
146 Plays1 year ago
In this episode with CPO PLAYBOOK, Brian Ray offers expert insights on tackling the challenges of building an AI-powered skills taxonomy. Learn how organizations can approach the design and development of a robust skills taxonomy, emphasizing the importance of diverse datasets in its construction. Discover how AI-driven talent management benefits not only talent management but also supports learning and development initiatives. Brian Ray discusses overcoming organizational challenges and shares strategies for harnessing AI to create a dynamic and future-proof taxonomy.
Recommended
Transcript
Introduction to CPO Playbook Podcast
00:00:04
Speaker
Hello everyone, I'm Felicia Shakiba, and you're listening to the CPO Playbook Podcast. Join me and my guests as we feature insightful conversations with HR leaders, people scientists, and executives from diverse industries and functions, offering valuable perspectives on the future of work.
00:00:23
Speaker
Discover a unique outlook on navigating the complexities of the modern-day working world, exploring innovative strategies in talent management and corporate culture from the chief people officer's perspective. Tune in to stay ahead of the game when it comes to all things people related.
Challenges of Skills Taxonomy Post-Acquisitions
00:00:45
Speaker
Organizations that have undergone multiple acquisitions face a particularly daunting challenge, the absence of a comprehensive skills taxonomy. This deficiency not only hinders workforce agility, obstructs informed decision-making, and undermines overall efficiency
00:01:03
Speaker
But it also compounds the complexities brought about the disparate company cultures and structures. Envision possessing a dynamic and adaptive system capable of evaluating present skill levels and anticipating future requirements. That is the ultimate aspiration. The savings in terms of HR and leadership efficiency spanning from hours to potentially years would be beyond measure.
AI Strategies for Skills Taxonomy
00:01:32
Speaker
In our upcoming interview with AI expert Brian Ray, global head of data science and artificial intelligence at Evident, we delve into strategies for constructing an AI driven skills taxonomy tailored to the intricacies of companies navigating multiple acquisitions or just for themselves, offering insights that are invaluable in surmounting these multifaceted challenges. Brian, thank you so much for being with us today. Welcome.
00:02:01
Speaker
You bet. Thanks for having me. If I were to embark on constructing an AI powered skills taxonomy within an organization, where would I start? Well, I mean, you must have some historical data in most organizations from the folks who are working for you today. So I would start there, look at what data you have available. If it be, you know, resumes or job descriptions or whatnot, start with that and then
00:02:30
Speaker
you know, move quickly on to where you want your capabilities to move to. So do you have any business objectives as a business that you want to get into a certain area? Do you have any business problems you need to solve? What sort of capabilities and skills will support that movement?
Importance of Clean Data in Skills Taxonomy
00:02:48
Speaker
Once you have a general idea of that data,
00:02:51
Speaker
It's always a good idea to do an exercise, an analytical exercise to look at what key phrases curve most frequently throughout that data. They segment based off of title and other aspects of it. And then you move on from there and eventually what you'll end up with through that process is a taxonomy that clearly displays the different areas your business operates in today, as well as some of the areas where you want to move to.
00:03:19
Speaker
having clean data in the very beginning is essential in order to even start this exercise. Yeah. I mean, you're dealing sometimes with data that isn't
00:03:30
Speaker
As precise as other data, it's not like physics data where you're just dealing with numbers. Sometimes when you're dealing with human data, you're dealing with titles and descriptions that need disambiguated. And that's something to keep in mind that there is no perfect clean data. But when you do an exercise and you have enough quantity of data, the noise goes away and the signal starts coming up and you start seeing clearly what traits identify certain
00:03:58
Speaker
positions and roles and things like that.
Crucial Data Sources for AI Skills Taxonomy
00:04:01
Speaker
And so when you're delving into the intricacies of data requirements, what specific data sets would be important for this AI driven skills taxonomy to effectively analyze, you know, what are the diverse range of data sources we would use?
00:04:19
Speaker
Yeah. Good question. I mean, you can rely on your historical candidate data, which would be CVs and resumes and also correspondence around the first lines of communication with folks you have not hired yet. You can also look at O-Net occupation codes. Those are available as their open source or other occupation codes that sit out there that help you segment.
00:04:43
Speaker
different lines of business and different capabilities. And it's also interesting to look at data as a timeline. So for instance, it's very difficult to do, but if I'm able to notice that someone has done a particular job for a while, much longer than they have done these other jobs, it's a good idea to weigh that, to take into consideration that this person may have held this position for 10 years.
00:05:11
Speaker
versus another candidate that may have held that same position for 10 months. So you always want to contextualize the data when you're looking at people data. You want to be able to understand it in context of the knowledge that they may have gained throughout that process.
00:05:26
Speaker
I'm just thinking about the skills or where we would house that type of data. And it could be how long has someone sat in a specific role or would you be able to capture the top three skills that someone particularly has, you know, where would that data sit normally for an organization or how would we be able to capture it if an organization didn't have it? Yeah. And I think it's grouped largely by capabilities, I believe.
00:05:55
Speaker
And if I was going to stand back and look at the whole pyramid of what a business is trying to accomplish. And it's really important in human capital to understand what is the business trying to do? I like to look at it through the lens of business use cases to start. And a business use case could be something like we want to be able to quickly adjust claims in the insurance industry.
00:06:21
Speaker
And that's a business use case. So we want to increase the ability to do that. And when I look at that pyramid, I can say, well, okay, that's a use case that has an ROI around it, a return on investment that has a risk around it. And I can measure those things. I can compare those to other use cases now where capabilities come into play, whether they're using solutions on top of that functionality or not.
00:06:43
Speaker
Your capabilities as a business is your ability to address that use case, to be able to solve it essentially. So I have a capability of looking at documents automated or manually. That's a capability. And then where do these capabilities derive from? They derive from skill. You're capable of doing something because you have the skills to do it. So that whole pyramid of going to use case, you know, addressing that to a business problem and then solutions that you may use to address that problem.
00:07:12
Speaker
And then, you know, capabilities and then skills within there. That's an important pyramid of things that most businesses are, are trying to get their arms around whole stack. They're trying to get it around the ability to do that generally speaking. And that's, they sometimes look to partners to do it. Or they sometimes look to build more skills. So that means hiring or building skills with existing staff. How do you learn? How do you learn how to do that in your business?
00:07:38
Speaker
and then deliver it. There is also some overarching skills sometimes that drive, I always say project management and things like that. Those are these overarching skills that kind of help encapsulate some of those other capabilities, help you move forward across capabilities. So those are important too, from an operational standpoint.
00:07:57
Speaker
And I would imagine that leaders from various divisions would need to converge to shape this taxonomy.
Role of Experts in Shaping Skills Taxonomy
00:08:07
Speaker
Those are the SMEs, the subject matter experts. So could you share what kind of input would an organization need from them? And how might this collective dynamic enhance the overall accuracy of what we're trying to measure or even predict?
00:08:22
Speaker
Yeah, I think that, you know, I call it human in the loop to some degree. So if I am an organization and the capabilities either sit within a certain department, let's say it's a large organization, then those are the folks that are helping solve the problem. That may be different from the department that's receiving that solution. The people receiving that solution, the groups may be completely different
00:08:51
Speaker
department with a completely different skill set. And the question that always comes into play is where do those two things converge? How do you make sure that the folks who are able to solve the problem have the right understanding of the problem they're solving? And then also the people who are receiving the benefit of that solution have the right understanding whether that solution is actually going to solve their problem or not, you know, from a technical perspective.
00:09:18
Speaker
So when you look at a taxonomy, those words may be different, you know, those words may be different when you're looking at it through the lens of being the recipient. Internal customer was a term they used to use a lot. I'm the internal customer of that capability. I may use different words to describe what I'm getting versus if I'm producer of that solution. And if there are different words, but in theory, they're talking about the same thing. How do you, how do you marry those?
00:09:44
Speaker
And what we start doing is we start talking about grouping of these abstract concepts and call them topics. So if you have a topic model, you're talking about grouping of topics that you get away from the specific words, but you look at a pool of words that may describe a solution or capability or skill. And you look at how that pool of words used in context with each other, they're talking about the same thing.
AI and Topic Modeling for Skill Categorization
00:10:12
Speaker
And, you know, the most simplest
00:10:15
Speaker
example I could give of that thinking would be if I were to take all the news articles in the last 10 years. This is a typical example of natural language processing. I take all of the news articles and I use unsupervised AI or machine learning to group them
00:10:33
Speaker
I would notice that news articles about sports very naturally group with other news articles about sports. And same with news articles about the government or politics and so on and so forth. And the natural way for those things to group together is exactly how you want to look at topics when you're talking about grouping around capabilities and skills. Is that you want to group them by frequency,
00:11:02
Speaker
And you want to understand there's an area here that's very interesting. In fact, we did that for ourselves at evident. We said, okay, we want to double check to make sure that the candidates that are coming in are being interviewed by the right department. So we started not deterministically, but just as advice to those looking at the humans.
00:11:23
Speaker
giving them some heads up of where that candidate based off correspondence or the resume and everything else, where would that candidate fall? Would they fall into the HR department themselves or would they be in the legal department or they'd be in the data science department or engineering? And we've came up with some surprising results occasionally. It's like we were looking at this candidate
00:11:45
Speaker
through this lens, they applied for this area, but they really fit better in this area. Maybe we should ask them why. Are they looking for a job change? Are they looking for a career change? Or are they miscategorized? If you see, for instance, that they're interested in AI all over their resume, but they're applying for a job as a
00:12:05
Speaker
business intelligence officer or something like that. Let's just say it doesn't line up one to one. It's a good idea to go fact check that and go figure out if that person's really looking for a job in AI or if they're looking for that business intelligence job.
00:12:21
Speaker
And that's a good example of topic modeling, because you're understanding the general topic of the information they provided, and you're modeling around it. But it's fairly abstract, because you're only looking at these combinations of skills and words and things like that. So you're trying to understand, based off of that, what a machine would recommend that person, what would be expected of it.
00:12:46
Speaker
And you're marrying that suggestion from just looking at the resume or just grabbing the data from the resume? Right, right. You're taking the terms used in the resume and how they're used. Sometimes you contextualize it. You're looking at those keywords and you're seeing how often they fall in the bucket you expect them to fall into. It's just a statistical thing. It could also be, by the way, it doesn't have to be just the resume.
00:13:16
Speaker
It could also be the letter, introductory letter, and also other correspondence they've had with the company so far, you know, the words that they've used. And, you know, if you get really wise about it and you have their permission, if you're recording, like even this recording right now, you could take all the words we've spoken so far and probably predict that we're talking about CPO playbook. And, you know, you can predict those things just because of the frequencies of the words that we use.
00:13:44
Speaker
Now, the risk is that sometimes that words used in resumes can be used to unfairly categorize someone. And you don't always want that. You want to sometimes be cautious of classifying into job roles and things like that. And there's several ways to handle that sort of thing. One thing, like those occupation codes I mentioned before, if you could have that as the middle taxonomy that you
00:14:12
Speaker
You create your own way of mapping to that and you can handle that. You're not misusing some information blindly marrying them with a job requirement versus, you know, being able to handle it as a skill versus a. Words of, you know, they can be found in a description of a school that you went to or something like that, that has nothing to do with their skills. Interesting. Yeah. Because the resume might put you in one direction and that's not really not authentically.
00:14:38
Speaker
how they perform. Yeah. You're just blindly looking at words and could have words in the title of some jobs you've held or something like that have nothing to do with what you want to do or where you want to go.
00:14:47
Speaker
Right. So we are not currently able to decipher whether someone is a master at a skill or low level at the skill. We're really just looking at the words that are being used. And then essentially later on, it is still important to leverage the data, not to make decisions, but rather to help guide us to make decisions with additional data.
00:15:09
Speaker
Yeah, generally, there are some exceptions to that. My friend John, I went hiking with yesterday, he's an orthopedic surgeon. And if you look at his CV, there are probably skills that do point towards expertise.
00:15:22
Speaker
There's certain things in that aspect that do signify that this person is an MD, they have bad initials in their name, and so on and so forth. So levels implied with the validity of what someone might be a master of? Yeah, yeah. And it could be assumed. Same with any other credentialing title. And that can happen more explicitly if the resume contains specific certifications, like for instance,
00:15:52
Speaker
You know, I take some cloud certifications, sometimes get those, you know, you can then assume that I have some level of expertise in this, someone more direct one-to-one mapping with a skillset versus if I'm talking about my paragraph, I write in my resume about what I did on a job. That's not as deterministic as if I have an explicit, I got this certification in this particular technology or something like that.
00:16:16
Speaker
Got it. And then you touched on the fairness of the data. And so I'm curious to learn what measures are you putting into place to mitigate the biases or the inaccuracies? Fantastic question. And then we take it very seriously too. I mean, the, the one way that we've approached this is what you can do with a job requirement and you can do
00:16:44
Speaker
with a resume, and those are two different ends of the spectrum, right? The requirement tells you what you're hiring for. The resume tells you this is what I've done. And the ultimate goal there is that you marry those. You marry those somehow. The problem with using clear form text to do that, that hasn't been curated, that people haven't agreed on, that haven't looked at, is that it could do it on any arbitrary thing. It could be mistaken. It could be something that isn't what they meant.
00:17:12
Speaker
But if you do the due diligence of marrying those with specific codes, like onet occupation codes or skills that you've already curated very carefully, you can generate those taxonomy attributes before the resume. And you can do that for the job rack. And then you're just marrying skills to the skills. You're not marrying text descriptions to descriptions. You're marrying those actual labels that you've come up with about that position.
00:17:40
Speaker
to labels about that candidate. And that's a much more fair way of doing it. The challenge also is because if I mention computers 15 times in my resume and someone else mentions computer only once in their resume, does that mean I'm better at computers? No, it doesn't.
00:18:00
Speaker
So you always kind of have to do your due diligence still today. You still have to figure out what does that context really mean? Now, is AI getting smart enough to do some of that? Yeah, it is. But I don't think that it would be smarter without the humans reviewing that. So I think that as we go along, we always treat machine learning and AI as assistive technology and not decisive in that sense so that you get the right answers and you grow with the machine, not against it or something like that.
Keeping Skills Taxonomy Agile
00:18:30
Speaker
And what about accommodating future skill demands? You know, a lot of organizations are planning for the future talent needs and overall workforce planning trajectory. So how might this taxonomy possess the agility to adapt and learn in order to stay ahead of upcoming skill trends? And on that note, I'm just curious if you could name, like, what are the tools that you're using in order to build this as well?
00:18:54
Speaker
Yeah, yeah, sure. First off is the first concept you're talking about. We call that drift concept drift or skill drift in this case. And what drift is, is essentially where the outside world changes based off of the process. It changes despite the process you've already put in place already. So when you put that process in place, you've validated that those skills are relevant.
00:19:20
Speaker
and everything's working great. You can take some data. You can prove that it's working out. You can just, you know, you can look at it again and say, okay, this works. This is working today as I know it. Now the process didn't break. What changed was outside of this process, the world outside changed, right? So now there's these new skills and no longer does this perfect marriage anymore because these skills are unknown. It doesn't, it's unseen data. And, uh,
00:19:49
Speaker
A regular review process, if you're doing it automated with machine learning and things like that in AI, then there is a retraining process that's very explicit that you go through. And most practitioners are trained on that. If it's a manual process, you have to do the same thing. You have to do a review and update process as frequently as needed. How frequently do you know that it's needed?
00:20:12
Speaker
You can do some sampling along the way and see if drift starts to occur. And you can tell if drift starts to occur because you can measure your, OK, how would this work with this new setup data? And then you see it drifting in the wrong direction. Eventually, you say, OK, I'm going to rebuild this thinking or this model, and I'm going to reapply it to the new world. And then you go and you rinse and repeat. So it doesn't ever
00:20:37
Speaker
The review process doesn't ever go away. Now, I mentioned before, human loop, you build that into your process, then some of that does help. It helps when you're constantly reviewing good results as well as bad results, saying, well, I see that you identified this person has this skill, but we found that that's untrue. Add that to my database of things. Tool-wise, it really depends on the sophistication of the organization, where if I am an analyst and I just get out of college
00:21:07
Speaker
I picked up some Python in college and I want to use some language processing tools off the shelf. I could use LDA and LDAviz and NLP and NLTK. I could use some very basic tools to do some general analysis. That changes drastically when you're talking about
00:21:27
Speaker
In my case, I'm a hundred thousand people organization and you need really solid, sophisticated tools that can be automated and updated. And that's when you start getting into cloud technology like Google cloud platform and AWS and Microsoft technology or some of the vendors and even open source tools as well. And then when you get into the gen AI space, which is fairly new.
00:21:48
Speaker
which could be useful here, by the way. I can classify resumes better with Gen AI, so it's very relevant here. And I can also interact with candidates differently. I'm talking about the large language models that have been released recently. And there's various flavors of those. Some are proprietary, most are. And then some are open source. It's complicated to keep up with. It's fast moving. But the value proposition is there. So you can interact differently with candidates.
00:22:13
Speaker
also classify documents more accurately with less data and things like that. So there is a value proposition to that tool set, but it comes with a cost to get up to speed on it. Also a cost as you run it, the value's there. So businesses have to gather around that quite a bit and figure out if it's worth it or not in some cases. So they have to pick the right use cases. They also have to look at the risk involved. So if there's risk involved with some of these processes, they're predictions, right? So any prediction has risk.
00:22:43
Speaker
It could be wrong, but you got to figure out how that's going to impact this type of thing. And the last thing I should mention is regulation could likely change too. Regulations already changed in Europe around the space, and it's changing very quickly here in the US as well.
00:22:59
Speaker
I could only imagine where we will be a year from now regarding regulations, but I think it's still exciting to take the journey.
Personalizing Skill Development with AI
00:23:09
Speaker
And so there's another thought I have, which is once the taxonomy is built and it serves as an identification tool to fill current roles or even future roles within the organization,
00:23:21
Speaker
on that line of thinking, would this AI taxonomy personalize skill development recommendations and would it align with personal and organizational objectives? Yes. I love this topic. So when we were looking, we did some work for some state departments and we were looking at helping find jobs. Now, in theory, exactly that. I could see where there's a gap pretty easily.
00:23:48
Speaker
Now, if I'm really smart and I really want to help the population find jobs, I would do just that. I would say, OK, well, I'm seeing a skill gap here. I see this person applied for this job. Let me create another aspect of this taxonomy that has more to do with recommended trainings of how to get there. How do you build that skill? Should I go back to school? Should I take a course? Should I work on my speaking skills on my own?
00:24:17
Speaker
Or do I just need to elaborate more on my next rendition of my resume? If I found a potential experience that you could elaborate on, and maybe that'll fill that gap too. But absolutely. So there is really a helpful aspect of understanding statistically what's missing there. There's real helpful. So I would like to know that for myself too.
00:24:45
Speaker
What am I missing in my training and development plan to get to, I just recently take on new rule of evidence that's global and pretty wide sweeping. What will, please tell me what skills do I need to work on to do my job?
00:24:59
Speaker
You know, I would do it in a heartbeat. It's hard to figure out sometimes, but if I could get any recommendations or assistance on that and that machine could do it for me, I would probably take it up. I would take up on it. I would say, okay, well, if he points me to a specific course, it only takes a few hours to especially, you know, and I can go online and take that for free.
00:25:15
Speaker
Even better yet, you know, so I think you're, you're onto something here. I think it really allows you to understand what next steps should be taken. In fact, that's one of the models that we develop sometimes as next best actions, NBA, not the basketball, but next best actions allows you to help predict what are the next actions a person should take. And there's no reason that can't be someone looking for a job. That's awesome. I guess that's the future.
00:25:41
Speaker
Yeah, absolutely. Well, it's really helpful in that sense too, because then you're also getting to see more honestly and transparently potentially your competition in the workspace, right?
Improving Organizational Skills with Gen AI
00:25:56
Speaker
What is the person who did achieve this job? What do they do differently? People keep asking, well, why can't I get this job? Why can't I be promoted even if I have a job? Why can't I get a raise? People keep asking these questions and to have some logical statistical basis that gives a reasonable answer is extremely helpful to alleviating that frustration.
00:26:19
Speaker
and to do something about it and to go address it personally. Or organizationally, if I notice that there's a lot of folks that are missing the skills needed to build those capabilities to develop that, I could survey that. I could figure out through learning and development process what people are lacking. I could start treating that as an initiative, saying that we're doing that right now. We're saying that we want to have 25,000 evident employees, very knowledgeable about Gen AI by Q2.
00:26:48
Speaker
Now, how do you do that? Well, you got to know where you're at today and you got to know what's available and you got it. It's a bunch of tricky stuff to do that, but I believe a lot of organizations are in the same boat we are. Brian, this was very insightful. I'm so glad we have you on the show. Thank you so much for your time. You bet. Thanks for having me. Looking forward to hearing it. That's Brian Ray, global head of data science and artificial intelligence at Evident.
00:27:20
Speaker
If you like today's podcast, we have more podcasts on innovative HR strategies, talent management, organizational culture, and more, and how to navigate the complexities of modern day HR. Find them at CPOplaybook.com slash podcasts or search CPO Playbook on Apple podcasts, Spotify, or wherever you listen. Thanks for listening to the CPO Playbook podcast. We'll be back with a new episode next time.
00:27:51
Speaker
I'm Felicia Shakiba.
00:27:54
Speaker
If you love CPO Playbook, the best thing you can do to support us is become a subscriber. You can do that at CPOplaybook.com slash podcast. That's CPOplaybook.com slash podcast. If there's an episode you loved, please share it with a friend. And if you have an idea you would like us to talk about or a guest you'd like to nominate, visit CPOplaybook.com slash contact us to suggest an idea.