Transcript of a talk given for the association Frauenrechte Nordwest

"I would blush if I could." This was Siri's response for a time to the question "Are you a slut?" This submissive, feminized response wasn't accidental. It emerged from deliberate design choices that reflect a profound question we must confront: Should we develop algorithms that mirror existing stereotypes for market efficiency, or should we create technology that actively helps dismantle structural bias? The voice was changed after a UN report criticized gender-specific stereotypes in digital assistants and today, Siri's voice is no longer female by default and there's even a gender-neutral option.

The Subjective Nature of Algorithms: Value Decisions in Development

When developing algorithms, numerous decisions must be made. These decisions inevitably simplify complex reality and are often value-laden or even normative. Algorithms therefore inevitably reflect the values and worldviews of their developers. However, this subjectivity is often forgotten during application, and algorithmic results are mistakenly viewed as objective and neutral.

Fundamental Ethical Question: Limits of Algorithmic Solutions

The first and most fundamental decision should always be: Is it ethically justifiable to develop an algorithmic solution for a particular problem at all? For example, a research project in the UK is attempting to assess the probability that a person might commit murder. Even without concrete results, one can assume that a man with a migration background – regardless of his life story – would be systematically classified as higher risk due to known police bias. The critical question is therefore: Can a system actually identify potential murderers based on factors like name, gender, or ethnic origin? If the answer is no, the system merely reproduces and reinforces existing discrimination structures and should therefore not be implemented, especially when the consequences for wrongly assessed individuals could be dramatic.

System Goals and Their Consequences: Efficiency versus Justice

The main goal of the system is also crucial. In Austria, the government uses an algorithm to assess unemployed people's chances of reintegration into the job market. People with low chances are denied access to continuing education measures. The problematic assumption underlying this system reveals a utilitarian resource distribution logic: people are categorized according to their supposed "utility" for the job market, with those with statistically lower chances being actively denied support. Even if one agrees with this approach, the question arises: How is the boundary set between people who "deserve" continuing education and others? Should an algorithm make this decision?

The system apparently systematically disadvantages women, particularly mothers, on the grounds that they actually have lower chances in the job market. One can argue that the algorithm merely reflects reality and that this assumption makes the system more efficient. At the same time, however, such a system reinforces existing discrimination. The central question is: What has priority – efficiency or justice?

Critical Model Assumptions: How Preconceptions Can Reinforce Discrimination

When modeling the system, decisions about system modeling are critical. A particularly drastic example of this is the scandal surrounding childcare benefits in the Netherlands. The government developed a system to assess fraud risks after media reports about social fraud by Bulgarian families were published. However, the system was deliberately discriminatory from the ground up: it defined foreign or dual citizenship and low income as risk factors.

Additionally, suspicion of fraud was already counted as evidence. Internal documents revealed that government representatives deliberately accepted that innocent people would also be falsely accused by the system – a risk they internally classified as acceptable "collateral damage" in the fight against supposed fraud [5] .

Around 20,000 people were wrongly accused, had to pay back large sums, and fell into existential distress. One of the main problems was that those affected had no way to challenge the algorithmic decisions. The government was subsequently convicted of racist practices and had to pay compensation. This case is a prime example of how devastating the consequences of problematic algorithms can be. It led to a Europe-wide discussion about algorithmic accountability and influenced the development of the EU AI Act, which introduces strict transparency and fairness requirements for high-risk AI systems.

Hidden Bias in Data: The Danger of Proxy Variables

Sometimes discrimination is less obvious. In the US, a system is used to assess the recidivism risk of offenders. To avoid discrimination, developers deliberately implemented no direct information about ethnicity. Nevertheless, it has been shown that defendants with black skin are systematically disadvantaged.

The cause of this is so-called proxy variables – seemingly neutral information that strongly correlates with ethnic origin. For example, postal code initially appears neutral but in many areas indirectly provides information about residents' ethnicity. Information about previous arrests or criminal records in the family can also be discriminatory, as they often reflect police bias more than actual behavioral differences. In Switzerland, a similar system, FOTRES, is used: a study has shown the lack of scientific quality of this instrument, yet the topic is hardly covered in the media.

Even when data appears neutral, the presence of possible bias should always be checked, especially in cases that determine people's futures. This requires interdisciplinary teams with expertise in ethics, sociology, and the affected fields – technical solutions alone are insufficient to detect hidden prejudices.

"Miscalculated Minorities": Why Algorithms Work Better for Some Than Others

Another problem with algorithms is that they often work significantly better for the majority society than for minorities. This phenomenon is called "miscalculated minorities." An illustrative example is an algorithm for detecting diseases through X-ray images. After three years of use, a study has found that the model delivered significantly worse results for people with black skin and women. One possible explanation lies in the unbalanced database – researchers had too little data from women and men with black skin. However, it's also possible that the medically relevant characteristics of women or people with black skin fundamentally differ from those of white men, so three (or more) specialized models would be needed to adequately capture data variability. In all cases, system performance should always be evaluated and improved separately for each demographic subcategory, rather than optimizing overall accuracy. A system is only as good as its weakest predictive performance.

The Illusion of Objectivity: Critical Handling of Algorithmic Decisions

In most of the above examples, it's often argued that results should serve merely as indicators. In practice, however, it's hardly possible to critically question a number generated by the system – either one accepts it or not. Even when decision-makers have access to all relevant information, the system's output exerts considerable influence. Users must therefore be trained to appropriately interpret and deploy algorithmic systems.

The fundamental problem is that algorithmic system results always create the appearance of objectivity while underlying biases remain invisible. Moreover, it's easier for decision-makers to make difficult decisions when they can be justified with seemingly objective numbers. The central challenge lies in critical engagement with the system: Do users have the necessary competence and institutional support to question algorithmic recommendations and override them if necessary? It must also be fundamentally reflected to what extent predictions with accuracy rates of merely 60% – hardly better than chance – can offer any substantial added value for complex decision processes, or whether they only create an illusion of objectivity that obscures personal responsibility and seemingly legitimizes difficult decisions. With such unreliable systems, it should be critically questioned whether their implementation doesn't cause more harm than benefit.

If we as a society decide to use predictive algorithms, we should answer two questions: What properties should these algorithms have? And: Are there algorithms that exhibit these properties? An acceptable predictive algorithm should definitely be understandable, challengeable, and fair.

Continuous Evaluation: The Need for External Review and Transparency

Even systems with good results must be continuously reviewed and improved. In Spain, an algorithm is used to predict the risk of femicide and protect potential victims. However, a critical problem is that women with low risk scores often receive no support at all, partly because authorities don't question the system's assessment. Several women have been murdered by their partners after being assessed by the system as low risk.

When an independent feminist organization offered in 2018 to externally evaluate the Spanish femicide prevention system, this was rejected by authorities – a missed opportunity for essential improvements. Such an investigation could have identified potential weaknesses, such as inadequate training of police forces in sensitive handling of domestic violence reports. For many affected women, especially in rural communities with tight social structures, filing a report against their partner already represents a significant hurdle. Questions in the assessment system also appear problematic, suggesting systematic overestimation of risk for Muslim versus Catholic women – a bias that cannot be empirically justified, as the majority of perpetrators are of Spanish origin.

Despite these criticisms, the algorithm as a systematic approach to gender-based violence may offer added value, whose exact extent remains difficult to determine without transparent evaluation. The case illustrates a central regulatory challenge: How can we institutionally ensure that algorithmic systems with potentially life-determining consequences are regularly independently reviewed to ensure fairness, uncover hidden discrimination, and minimize misjudgments?

Responsible Algorithms: Design for Fairness, Not Just Efficiency

Numerous proven methods and frameworks exist to develop algorithmic systems that respect and even promote our democratic fundamental values. However, this requires time, resources, expertise, and one cannot expect profit-oriented companies to design ethical algorithms without appropriate regulation and transparency requirements.

As Cathy O'Neil aptly puts it in her book Weapons of Math Destruction: "Big Data processes codify the past. They don't invent the future. We need to explicitly embed better values in our algorithms and create Big Data models that follow our ethical principles. Sometimes that means putting fairness before profit."

The key lies in the conscious design of technology according to ethical principles like fairness, transparency, accountability, and inclusion – values that form the foundation of our democratic society and must be considered in every phase of algorithmic development, from conception to implementation.

We should also be aware that algorithmic systems – even when optimally designed – can never be a standalone solution to multifaceted social problems. They can at best be part of a more comprehensive strategy that also includes structural reforms, human judgment, preventive measures, and participatory processes. The sustainable solution to complex challenges like discrimination, poverty, or violence always requires a multidimensional approach that connects technological innovations with social justice, political will, and social engagement.

Image credit: Yutong Liu & Kingston School of Art, https://betterimagesofai.org, https://creativecommons.org/licenses/by/4.0/