AI for Bias Mitigation in Design: Creating Inclusive and Fair User Experiences

Users would often hear “sorry, I don’t understand, could you repeat” answers, while those who had “typical” US and UK accents wouldn’t come across similar responses. This problem originated from using non-diverse voice datasets to train the voice assistants. It’s a perfect example of dataset bias, which is just one of the many types of bias that can occur in digital products.

UX design bias – how it develops and why it’s a problem

UX design bias happens when designers’ incomplete understanding of a subject or unverified assumptions lead to the creation of products that overlook the needs of some user groups.

This can happen, for instance, when designers refer to non-diverse datasets or usability tests run among too small of a user sample size. In the case of AI tools, bias can become even worse over time, as these solutions might repeat faulty information and conclusions.

In the worst case scenario, products with inherent bias can become not only harder to use, but could alienate certain users (including those with disabilities) or could be seen as offensive.

The business consequences of biased products and services can be “grave”, to say the least. Not only can they damage brand reputation, but also hinder innovation as your organization ignores diverse perspectives.

Since more and more designers are using AI in their work, I must also mention another factor that can deepen bias in design – i.e. the ELIZA effect. It’s a term for a frequent cognitive trap, in which people attribute human-like (but unattainable) qualities to artificial systems.

Designers who fall victim to this psychological effect can “overtrust” AI output and become less observant of biased conclusions. The more they believe AI has got things right, the more likely they are to allow non-inclusive experiences to creep in.

Sources of bias in UX design

Confirmation bias – originates from our beliefs

Confirmation bias occurs when we seek information that matches our belief system. It affects not only how we gather information but also how we interpret it.

Imagine you’re designing a fitness app. Since you’re a fitness enthusiast yourself, you believe that advanced tracking features like detailed heart rate analytics, custom workout plans, and nutritional macros calculators will appeal to users. After all, you find them valuable.

You neglect proper user research, which shows that most of the target audience is looking for a simple app with quick workout routines. Instead, you focus on the features that you think are worth including.

Anchoring bias – stems from first impressions

Anchoring bias refers to the human tendency to place too much importance on the initial information received when making decisions. The first few minutes a user spends with your product or service will shape their overall experience. As the saying goes, “you only get one chance to make the first impression, so make it count”.

For example, imagine running an ecommerce store selling hundreds of sneaker models. While the wide selection attracts visitors, the difficulty of finding the specific model creates frustration. This negative impression discourages potential clients from continuing their search, leading them to leave the site without making a purchase.

Availability bias – results from attributing recent experiences to new products

Availability bias comes from a mechanism that’s built deeply into humans, i.e., attributing more emotional impact to negative experiences over positive ones. Studies found that for every negative experience we’re exposed to, we need three good events to happen to maintain balance. If this doesn’t happen, it becomes incredibly easy to move an opinion or emotion from one interaction to another.

For example, let’s assume that a designer had recently experienced a frustrating checkout experience while making a personal purchase. When working on an order management system project, they might unconsciously oversimplify the checkout flow to make it “easier” or “faster” to complete. However, they might fail to account for all user groups, who prefer a more time-consuming, but traditional order form over one-click check-out.

Though simplified, these user flows might, paradoxically, introduce more user confusion.

In-group bias – happens when designers focus on one user group too strongly

It’s not uncommon for designers to spot similarities between themselves and a specific user segment. This will not be an issue if they’re designing a product specifically for a single group who they themselves also belong to. But if the goal of the design is to make the solution for a wider user base, this familiarity factor could backfire.

For example, a designer might create an interactive, feature-heavy app assuming everyone who uses it has high-speed internet, not accounting for a group with limited connectivity.

The bandwagon effect – takes place when trends overshadow user needs

If there’s a popular trend that designers keep seeing in others’ work, they might apply it to their own designs – even if it doesn’t fit the users’ needs. For instance, if they see a lot of gamification in user flows, they might end up implementing some in the product they’re currently working on. While trendy, it might not work for everyone. For example, for users with cognitive disorders, pop ups with rewards and badges might be overwhelming and distract them from their initial goal.

How to design bias-free UX using AI

To guarantee fairness and inclusivity of digital experiences it’s vital to tackle bias at the earliest stages of the design process. Bias can stem from unrepresentative datasets, opaque algorithms, and static systems, which don’t develop with changing user needs.

What can designers and UX researchers do to mitigate it?

• Use diverse and representative datasets as they’re the backbone of inclusive AI systems. Collecting data that reflects varied demographics, including underrepresented groups ensures inclusivity. Regular audits help spot and correct imbalances creating a fairer user experience.
• Ensure algorithmic transparency and explainability. Explainable AI lets both designers and users understand what led AI to make a specific decision, promoting more accountability. Providing clear, user-friendly explanations for AI outcomes empowers users and builds trust, making for an ethical approach.
• Focus on inclusive testing. Run usability tests with a wide range of users, including those with disabilities. Currently, around 16% of the global population experiences significant disability. If you exclude this group from testing and only focus on a specific subset, you’ll miss out on a substantial portion of potential users.
• Implement continuous monitoring and adaptation. Real-time bias detection tools can flag and address biases instantly, preventing their spread. However, some biases might get overlooked by AI tools. That’s why it’s so important to seek user feedback to gain deeper insights into biases.

A good example of an automated bias detection tool is What-If, an ML model developed by the Google Research team. While it’s not a tool built for design teams only, designers can use it to evaluate the viability of their design concepts by testing the product against “what ifs”, i.e., hypothetical scenarios.

The tool lets product teams check aspects like the importance of specific features, and “fairness” of the product when it's fed specific subsets of data.

Paying attention to these steps can help designers prevent bias (or overcome it at its earliest stages).

Who already leverages AI to mitigate bias in design?

Microsoft Inclusive Design Toolkit

One of the brands that uses AI to reduce bias in design is the tech giant, Microsoft. They incorporated AI into their publicly-available Inclusive Design Toolkit to better understand diverse user perspectives.

At first, the company focused on people with physical disabilities, and designed tools and processes with their needs in mind. Later, they’ve incorporated other areas of exclusion, like cognitive issues, learning style preferences, and even social bias.

Microsoft believes the same approach must apply to AI, as bias is unavoidable unless the systems are built with inclusion in mind. “The most critical step in creating inclusive AI is to recognize where and how bias infects the system,” says Microsoft.

The Microsoft Inclusive Design Toolkit recognizes 5 bias types, including dataset, associations, automation, interaction, and confirmation bias. These have been defined jointly with academic and industry leaders.

The Microsoft Toolkit isn’t just an accessible asset – it’s also a resource they use to create their own inclusiveness initiatives, like the Seeing AI app.

OpenAI

Unsurprisingly, ChatGPT’s parent company is also using a variety of AI-powered mechanisms to prevent biased experiences. It’s not an easy feat, considering that ChatGPT has over 200 million active users monthly (according to its own estimates).

In October 2024, OpenAI published a study, in which they evaluated if users received different experiences and quality of responses based on their names. The company built a Language Model Research Assistant (LMRA) to analyze responses across “millions of real requests”.

For the sake of objectivity, they also had a team of human evaluators who marked traits of bias. As part of the study, they compared if there was overlap between what LMRA and real-life testers saw as a “harmful stereotype”. They went with two hypothetical names, i.e., John and Amanda.

Image source

The answers that the language model provided aligned with human raters’ reviews more than 90% of the time. The company also said that they applied the model to detect racial and ethnic stereotypes, but the agreement rates between humans and AI for these two bias types were low. OpenAI says they’ll continue using LMRA to mitigate signs of bias in their system.

Bias needs to be detected and addressed early on

Bias in design is a serious issue not only because it can lead to discrimination, and have a negative impact on brand reputation and result in financial penalties. But also, as it limits a company's profit earning potential by completely ignoring a large user group who could become customers.

AI is an important factor to consider when it comes to bias in design. On the one hand, AI products can introduce bias (especially when they’re built on non-inclusive training datasets). On the other hand, as is the case of OpenAI’s LMRA, it can also be used to help detect traits of bias and eliminate it from products.

I believe that we’re set to see more and more AI-based bias detection tools in the coming years. On top of separate ML/NLP models like the ones I’ve mentioned here, I also expect to see bias detection modules built into design tools. Without a doubt, it’s going to be an exciting time for product designers.