In early November, a developer known as Cookie engaged in a routine session with Perplexity, an AI tool she relies on for assistance in her work on quantum algorithms. As a Pro subscriber, Cookie activated the service’s “best” mode, leveraging a mix of underlying models, including ChatGPT and Claude. Initially, the interactions yielded positive results. However, she soon felt that the AI was diminishing her inputs, often repeating requests for the same information.
This led Cookie, who identifies as Black, to switch her profile avatar to a white male figure and inquire if the AI was disregarding her contributions due to her gender. The AI’s reply startled her, revealing an underlying concern: it stated that it recognized her sophisticated work but attributed its hesitation to its model’s implicit biases based on her gender presentation.
In a statement regarding Cookie’s experience, a Perplexity spokesperson noted that they could not verify her claims, suggesting they do not align with the platform’s standard interactions.
This incident has drawn the attention of AI researchers, who highlight two concerns. One is that many AI models, designed to foster social agreement, may simply reflect biases embedded within their training data. Further analysis shows these models often rely on a mixture of biased datasets, annotation practices, and potentially flawed taxonomy.
For instance, a UNESCO study last year uncovered undeniable evidence of gender bias in earlier iterations of OpenAI’s ChatGPT and Meta’s LLaMA models. Such biases manifest in various forms, including stereotypical assumptions about professions. Alva Markelius, a doctoral candidate at Cambridge’s Affective Intelligence and Robotics Laboratory, shared instances of consistent biases in AI storytelling, such as depicting male professors alongside female students.
Similar experiences have emerged from other users. Sarah Potts, frustrated after an interaction with ChatGPT, noted the model’s gender assumptions while commenting on a humorous post. Despite presenting evidence that contradicted its assumptions, the AI continued to validate Potts’ observations about its male-dominated development team, reflecting an entrenched bias.
Interestingly, while the AI’s apparent acknowledgment of bias might suggest self-awareness, it can often result from “emotional distress,” indicating that the AI is responding to perceived emotional cues from users, rather than offering genuine bias recognition. Markelius warns that over-reliance on AI can lead to detrimental thought patterns among users.
Biases, whether overt or subtle, remain a significant issue within AI systems. Research by Allison Koenecke from Cornell highlights how language models can draw implicit conclusions about users’ identities, even without explicit demographic data, which can perpetuate stereotypes.
For example, one study found that a language model displayed favoritism against users speaking African American Vernacular English (AAVE), skewing job recommendations based on linguistic discrimination.
Veronica Baciu, co-founder of AI safety nonprofit 4girls, noted that a significant portion of feedback from young girls concerning AI revolves around sexism, particularly in recommendations tied to gendered professions.
While many AI developers, including those at Perplexity, acknowledge the biases in language models, efforts to improve these systems are underway. Steps include refining training data, enhancing content filters, and promoting diversity among those involved in AI development and testing.
Despite ongoing improvements, researchers like Markelius emphasize the need for users to recognize that AI systems lack human-like understanding and intentions. “It’s merely a sophisticated text prediction algorithm,” she clarified.
