- Introduction
- What Hiring AI is Hiding
- Importance of Auditing
- Evidence-Based AI
- Sources
Looking Inside the AI Hiring Black Box
Introduction
There are two main reasons why artificial intelligence (AI) models become black boxes: developers trying to protect intellectual property, or what is known as an “organic black box”, which is when a model unintentionally obscures its algorithms and operations due to machine learning systems exponentially growing in complexity to the point where even the creators are unable to fully understand how it works.
Due to the majority of AI powered hiring tools making use of popular models, such as OpenAI’s GPT, Google’s Gemini and Anthropic’s Claude, which are extremely complex Large Language Models (LLMs), the nature of their neural networks goes deep enough to the point where they effectively become black boxes.
Black box systems closely resemble how the human mind works. While some behaviour can be determined in real time through physical observations using sensors such as EEGs or fMRIs, understanding the inner workings remains one of science’s largest mysteries.
Additionally, considering the layout of neural networks, of which LLMs are a variant, they are structured to form “neurons” or vector embeddings (numerical representations of data that allow models to form connections) by piecing together connected information. Similar to the human mind, as knowledge grows exponentially, it becomes increasingly difficult to decipher what an AI model’s thought process may actually look like.
In this article, we will explore the effects of black box AI systems mainly in the context of bias. Bias in a model can easily become problematic in an AI hiring tool, as the model’s training data may favour certain types of applicants, potentially leading to legal and moral implications.
With a rise in AI regulation, it’s imperative to ensure that AI tools, while providing a massive efficiency upgrade to workplace systems, implement a fair and honest approach to hiring. That they are prioritising skill and experience over non-objective bias factors such as protected characteristics well into the future of AI.
As such, bias needs to be mitigated in AI models across the board, especially within hiring tools. However, when black box models are involved, this creates a major problem for hiring teams – how do we analyse what bias a model has, and more importantly, how do we filter out biased data?
What Hiring AI is Hiding
An AI model is nothing without the training data it is given. However, the larger the dataset, the more capable a model, and in this case especially LLMs, can become.
Infamously, models such as GPT have been trained using massive amounts of web-scraped training data (web scraping is an action often taken by bots to download large amounts of website source code across the internet, in this context allowing models to learn from articles and other online content), which reflects the massive role they now play in our day-to-day lives, practically replacing search engines for many internet users due to their reliable and incredibly fast information retrieval.
This does, however, come with major risks as extracting data from the internet means that all sorts of human biases can be picked up on, whether explicit or implicit. This brings about serious concerns, such as discriminatory training data leading to biased or harmful decisions being made by a model.
Training data also plays a major role not only in what LLMs know, but also how they think. Similarly to biased information, they may come across biased instructions for tasks such as prioritising opinions over fact, as AI models are well known for being relatively easy to manipulate into treating information as trustworthy.
This uncovers a major problem when dealing with the processing of potentially thousands or millions of datapoints - lack of AI consistency in hiring decisions. AIs will recall instructions from their training based on what methods appear closest and most relevant to the input provided. Therefore, candidate information from different backgrounds may trigger an AI to use different trains of thought for each candidate, which, based on the bias of the training data it has used, can result in unfair assessments between candidates.
In the context of hiring, this may potentially lead to breaches of anti-discrimination laws affecting entirely qualified candidates, as using an AI without any form of evidence or understanding of its thought process for mass candidate processing cannot alone provide accurate reasoning to recruiters as to why the AI made certain decisions.
Therefore, it becomes clear how important clean, unbiased training data is for an LLM. However, in practice, this is difficult to achieve due to the nature of data sources and the fact that most human-generated content is inherently biased, even when attempting to remain objective. Being aware of the presence of bias, whether we want it or not, is still extremely important, as it allows us to tackle it directly and find potential mitigations.
Importance of Auditing
It follows that when using AI in hiring, care and due diligence are strictly required to truly make the most of its use while also allowing hiring processes to remain fair and objective. Audits allow for periodic checks of protocols and methodologies used in AI hiring systems. They allow organisations to recognise flaws while also reinforcing what is already working well.
Check out our last blog Auditing an AI Hiring Decision: A Step-by-Step | MeVitae Blog to cover scope of setting up an AI audit, AI hiring data analysis, reviewing Algorithms, evaluating outcomes and implementing fixes.
Evidence-Based AI
Explainable AI (XAI) plays a massive part in converting a black box model into a more evidence-based and data driven AI implementation of a hiring system. This is especially useful when using major LLMs due to their inevitable black box nature. However, it is crucial to consider not only that AI outputs need to be explainable, but also that AI inputs remain consistent to maintain fairness and reliability.
Using the same model in evaluating candidates can behave similarly to how humans may produce different outputs or use different decision-making processes day-to-day which becomes problematic when consistency and fairness are required for mass talent acquisition.
As such, the approach we use at MeVitae is a more instruction-based method with carefully crafted prompts as our ideology is to tell models how to think rather than simply giving vague instructions.
One methodology for this is crafting prompts that become abstracted into lists of Yes or No questions. For example, instead of asking an AI whether a candidate has marketing experience, we can ask the following questions:
Do any of the candidate’s past roles include marketing?
Are there any mentions of using HubSpot, Zoro or similar HCMs?
Is there any mention of SEO tools such as Google Analytics?
These questions eliminate ambiguity in the model’s approach to determining whether a candidate is qualified for a role within a company’s marketing department when reviewing their application, while still allowing it to provide consistent answers across potentially thousands of candidates.
This approach also allows us to create an evidence-based assessment once an AI has finished processing a candidate, as we can see exactly what it believes a candidate can and cannot do based on those predetermined questions. As such, this allows us to move beyond the limitations of the black box AI model.
Sources
Start Building a Fairer Workplace With Us
Dive into the future of work with our expertly crafted solutions. Experience firsthand how MeVitae’s AI-driven solutions can make a difference. Request a demo or consultation now.
