AI data labelling: a pathway or peril to refugee self-reliance?

Executive summary

Humanitarian budgets are collapsing at the same moment that AI systems are generating major demand for high-quality human-labelled data. Considering flexibility and low entry requirements of AI data labelling, social enterprises have moved quickly to position refugees at this intersection, framing this digital work as a scalable, dignified pathway to economic self-reliance. Drawing on organisational documents, social enterprise impact reports, and first-hand worker testimonies, the policy brief examines whether data labelling constitutes a meaningful livelihood solution for displaced populations or whether it reproduces new forms of digital precarity within global AI supply chains.

The analysis focuses on four interrelated dynamics: the dual identity of social enterprises, which pitch their impact-oriented services to tech clients while selling the same work as empowerment to humanitarian donors; the self-reliance narrative promoted by social enterprises; and the lived realities behind this narrative, including both the positive aspects of giving refugees access to AI work and structural labour harms embedded in this labour.  While social enterprises are providing hope and some insulation for refugees entering the labelling market, they cannot overcome structural harms of the market, leaving refugees to bear the cost of that failure. This paper recommends that UNHCR publish binding operational guidelines for social enterprises, drawing on the Fairwork Framework, to ensure that AI-related digital work can support refugee livelihoods without entrenching the very exploitation it claims to address.

1. Context

Global emergencies are intensifying: the number of state-based armed conflicts reached a 70-year record in 2024 – numbering 61 (PRIO 2025). At the same time, global warming neared a 1.3 anthropogenic increase (World Weather Attribution 2025), and authoritarian repressions keep rising under the flag of a right-wing turn (Human Rights Watch 2024), leading to growing displacement all over the world. Despite this increasing polycrisis, in 2025 UNHCR’s budgets have diminished by 30% year to year and are expected to further fall by 15% in 2026 (Le Poidevin 2026). The greatest impact of these shortfalls has been felt on refugees themselves, whose numbers soared to over 36 million in 2025 (UNHCR n.d.).

By 2023, two-thirds of refugees lived in poverty, and 62% resided in host countries that severely restrict labour market access in practice, (UNHCR 2023, 1) despite the 1951 Refugee Convention’s formal guarantee of the right to work. The International Labour Organization (ILO) thus framed the gig economy as a “stepping stone to jobs and income”, (ILO 2025a, 19) while UNHCR highlights that web-based economic activities afford displaced individuals the vital opportunity to earn a living (Easton-Calabria, Hackl 2023, 7). In 2022 UNHCR and ILO launched an 18-month project in 8 countries called “Promotion, Inclusion and Protection of Refugees in the Gig Economy,” aiming to equip displaced populations with new skills and knowledge connected to digital hygiene and cyber risks (ILO n.d.).

Remote digital work like AI data labelling is thus uniquely attractive because it bypasses physical borders and discriminatory national labour laws, offering low barriers to entry and flexible, piece-rate work (Clark 2021, 770). For displaced populations, it theoretically offers a “virtual entry” into the global workforce, presenting an immediate income path for those routinely blocked from stable, local jobs (Altenried 2024, 1120). For tech companies, this transnational digital matching grants access to an “extremely scalable workforce” on a “pay-as-you-go” basis (De Stefano 2016, 4-5).

2. The hidden infrastructure of AI: what data labelling is

“AI isn’t magic; it’s a pyramid scheme of human labour” – Adio Dinika, a researcher at the Distributed AI Research Institute explained to The Guardian (Bansla 2025). AI market has been rapidly expanding with UNCTAD projecting for it to grow by 25 times – from 189 billion to 4.8 trillion USD between 2023 to 2033 (UNCTAD 2025). Microsoft estimated that one in six people are using generative AI worldwide with Global North leading the way with almost a quarter of the population being GenAI users (Microsoft 2026, 2). Its major branches like large language models (LLMs), developed for chatbots like ChatGPT, and computer vision systems, used to identify objects in self-driving cars, require massive amounts of high-quality data. If AI is trained on raw, unlabelled data, it absorbs inaccurate and different in format information, ruining its output (Jenkins 2024). To prevent a drop in quality, tech giants rely on millions of human data labellers, who are important for teaching AI systems to do tasks that are hard to describe clearly, such as finding emotions or bias in text (CWA Union 2025). These workers draw boxes around objects in images, transcribe audio, tag particular words and mistakes in AI-generated text and review datasets to ensure consistency and accuracy (SuperAnnotate 2025). Labelling is quite easy to complete and only requires language knowledge (often non-English languages on a native level), basic computer literacy and common sense (Kaneti 2025, 611).

3. The social enterprise model and its self-reliance pitch

Social enterprises entered the humanitarian sector in 2010-s, offering humanitarian agencies a mechanism to embrace market logics while maintaining moral legitimacy (Portales 2019, 57). By presenting themselves as hybrid entities that combine the efficiency and innovation of for-profit businesses with the passion and values of non-profits, they now act as a “bridge” between these two worlds, creating a “blended value” that appeals to both corporate clients and humanitarian donors (Battisti 2019, 140).

3.1 Hybrid structure and promise of social enterprises

If anyone visits the website of a social enterprise with AI data labelling programs, they find a platform created to speak to two entirely different worlds simultaneously. To tech giants, mostly based in the Global North, the enterprise pitches seamless, high-quality data labelling services at competitive rates, promising to build the future of AI. To humanitarian donors and foundations the pitch is more about resilience and showing refugees economically and mentally benefiting by entering the global digital economy.[i] To succeed in disseminating both pitches, social enterprises make their whole structure hybrid. The first half is a commercial agency that targets corporate tech contracts, showing they can successfully work in a private setting. The second one is a non-profit foundation that benefits from tax reductions and charity donors that give out grants specifically to NGOs. It is responsible for the social mission part and takes on digital literacy training and providing infrastructure, like laptops, to their refugee workers.[ii]

By utilizing these two faces, the social enterprise positions itself as a cushion between the client AI data labelling platform and refugees. However, the main website page of the corporate agency of EmpowerAI does not make major emphasis on datasets being produced by refugees: the visitor would need to scroll and click to learn about this.[iii] The reason behind it is that the social impact pitch is deemed unattractive to corporate clients that are “accustomed to no compensation floor and thriving on a rampant race to the bottom”. When one enterprise tried to establish stricter rules on fair compensation, they faced so many difficulties in getting new clients that the hired consultants recommended it to “tone it [social impact narrative] down” (Kaneti 2025, 618, 620).

3.2 Selling “self-reliance” to the market and the worker

The narrative of self-reliance has been present in refugee support field, being grounded by UNHCR in their 2005 “Handbook of Self-Reliance”, aiming to empower refugees “to meet essential needs <…> in a sustainable manner and with dignity” (UNHCR 2005, 1) It has later become a running theme in most social enterprise reports under an umbrella narrative of “work instead of aid” (Sama 2022, 12). They set an inspiring tone by making the title “Empowering Change in Conflict-Affected Communities” and featuring “inspiring case studies” that “captured positive changes our initiatives brought to communities in need” (Humans in the Loop 2024, 1). Social enterprise operations are thus driven by a goal of “economic self-sufficiency of disadvantaged populations,” while a longer-term vision is “refugees [being] able to overcome economic & social barriers and become pioneers in the tech sector” (Subul 2024, 10-11). Here social enterprises reframe refugees from “tech user[s] to tech empowered,” (Techfugees n.d.) actively promoting a narrative change of the role refugees can play in digital economy. Considering the ambition, the effects are supposedly going beyond giving short-term solutions, instead aspiring to provide “financial independence and career development”, serving as an opportunity to “re-start their careers” (Humans in the Loop 2024, 3). However, analysed social impact reports provide no longitudinal data on whether participants transition to stable employment. Instead, it highlights “success stories” of workers who leveraged their annotation gigs with the only “ripple effect” being that participating refugees started “investing more into their families, education and communities after securing digital work” (Sama 2022, 24).

4. The realities of AI data labelling programs for refugees

To understand how social enterprises succeed or fail in fulfilling their promise of self-reliance to refugees, it is necessary to consider both the bright and the dark side of AI data labelling programs.

4.1 The bright(-er) side of data labelling

Hope given through market access

As it was mentioned above, in many host countries displaced people routinely confront obstacles, including the denial of work permits, unrecognized educational credentials, language barriers, widespread discrimination and geographic isolation in camps or peripheral urban areas (Altenried 2024, 1120). Hence, AI data labelling can give much-needed accessibility to income that does not necessarily require a 9-5 office presence. It makes it accessible to refugees caring for families, those with physical or mental disabilities or facing religious restrictions on mixed-gender workplaces (CWA Union 2025). Another benefit is transportability of data labelling work as regardless of refugee’s next destination, it will still be available online, which is a ray of stability important for ever-changing livelihood of displaced people (De Stefano 2015, 5). Moreover, from the psychological perspective, earning money independent of social allowance can in itself be empowering with refugees feeling “on top of the world” when they first got it (Berhane Gebrekidan 2024, 7). A job in a prospective field of AI serves as motivation and a point of pride for people who lost their homes and had to restart their lives in a new country.

Social enterprise mediation

Importantly, social enterprise model offers an additional insulation for refugees that would otherwise be thrown into the ocean of data labelling work without any safety nets. Refugees are usually unaware about how to register and engage on digital platforms or do not even possess the needed infrastructure (Kaneti 2025, 616). Most social enterprises are at least trying to negotiate better conditions for their workers, for instance, setting fairer pricing models, arranging clearer task scopes and deadlines, and implementing specific ethical guidelines (Ibid, 615). They are also taking on most of the administrative burden: finding clients and negotiating connection with refugees in spite of their documentary situation. This support is especially vital for refugees who often lack official documents or bank accounts due to host state restrictions and other bureaucratic obstacles. So, social enterprises focus on mediating bank transfers from the clients, which refugees themselves call “very convenient”. While it might take some time: “We sometimes do not receive our work allowances until after two months” (Al-Hammada 2024, 11), this is still much better than not receiving anything at all. In case of office work in host countries, social enterprises can also offer the space and infrastructure (computers, Wi-Fi etc.) needed for data labelling. Finally, as mentioned above, they are conducting trainings that go beyond data labelling, important for those without education, computer or soft skills, which can thus improve refugees’ employability and future work opportunities.[iv]

4.2 The dark side of AI data labelling for refugees

Non-transparency and communication hurdles

When a refugee starts work on the first project, the enterprise gives instructions from the client. They are individual for each project and can be 30-40 pages which workers must learn to avoid mistakes. However, workers have no direct line to the client, so any unclear cases are communicated back through a long chain: enterprise team manager to senior staff to client (Ibid, 9). This layered communication frequently produces misalignment between what the written policy says, what the trainer explained, and what the client actually wanted. The consequences then fall on refugees. When a completed and submitted project fails to meet client expectations, workers are expected to redo the work under revised guidelines, importantly doing it for free. As one Syrian refugee data worker in Lebanon put it: “Why do we have to show flexibility and re-implement the work even though it is not our fault, as the instructions were not clear from the beginning? Is it fair to work twice on a project for one wage?” (Ibid, 10).

Non-disclosure agreements (NDAs) remain an issue for most data labellers as well, as they are forced to sign these documents to formally protect data of the client company. NDAs function as instruments of forced silence and intimidation, making it punishable to form unions to try improving working conditions of labellers. While some social enterprises are trying to take on more advocacy functions and negotiate the unionization of workers, it remains extremely difficult due to clients justifying NDAs as necessary to protect their intellectual property and corporate confidentiality (Equidem 2025, 42).

Algorithmic control and NDAs

The mental strain of meeting targets is serious – in most projects the lowest quality threshold stands at 90-97% (al-Hammada 2024, 7). A score below this could trigger warnings, lost incentives or even placement on a “performance improvement plan,” which workers described less as support than as a formal corridor toward dismissal (Berhane Gebrekidan 2024). To make matters worse, workers might have to face triggering text or pictures, often connected to their war-torn homelands, as they are often hired to label in their native language. While most times social enterprises are trying to avoid such projects, they are often left unaware about concrete tasks and end goals of their clients (Kaneti 2025, 617). “I hope this isn’t for weapons” is a valid concern among labellers, however the non-transparency of end client makes it impossible to be certain about the usage of labelled data. It might indeed be for weapons (Miceli 2024).

The time pressure remains massive as well. If the worker takes too much time on a task, their rating will fall, resulting in less or lower-paid tasks in the future (Hao, Hernandez 2022). It leaves them in a vicious cycle of trying to complete tasks fast but retain the highest quality. One data worker summarized this pressure: “How can you read this essay, grade it accurately, and then provide feedback in five minutes? The time per task was not adding up, and then the timer put on pressure” (CWA Union 2025). This is usually the point that clients of social enterprises are not willing to negotiate on, as famously said: “time is money,” and the labelled datasets are expected by the company on time, regardless of how much work and rework it actually requires (al-Hammada 2024, 7).

Psychological effects

In an attempt to mitigate negative effects on workers, most social enterprises introduce wellness breaks and support sessions with a psychologist. Despite flexibility being one of the appeals of this work, in reality, those working in the office have a standard 8-hour day and highly restricted wellness breaks. Workers in similar facilities reported having just “a 20-minute wellness break twice a week” and noted that they “have to beg for a wellness break on the group chat for it to be granted” (Berhane Gebrekidan 2024, 14). Psychological support is also unprofessional. Instead of receiving help, workers state: “I always go to the counsellors and come out more triggered and more crazy” because some counsellors “think you are faking it or trying to manipulate them so you can get a leave permit when you cry” (Ibid, 32-33).

Economic precarity and dependence in the absence of choice

When there is “only one project every few months <…> Many annotators feel compelled to accept any available work” (al-Hammada 2024, 32-33) The task can also come in the middle of the night or when they are unavailable. Some workers resort to using a “browser extension that sounds an alarm when a new task appears” and keep it “on loud even when [they] sleep, to wake [them] up in the middle of the night” (Hao, Hernandez 2022). However, this additional payment may be fundamental for the whole family, so missing or declining it is not an option. Yet this work is still poorly paid. Labelling objects on 10,000 images over several days is compensated with just 140 euros – an amount that barely covers ten days of food in a country like Lebanon (al-Hammada 2024, 9). In a year the “salary”, already positively negotiated by one data labelling social enterprise, amounted to just a bit above 2000 euro (Kaneti 2025, 615), which is far from enough for survival of even one person.

Non-transferable skills

Finally, the long-term effects of trainings conducted by social enterprises for refugees might not be as useful as promoted. Instead of acquiring high-level coding or web development skills that foster genuine upward mobility, the social enterprise only taps into these spheres on a surface level, primarily focusing on training refugees to carry out routine, low-skill microwork. The trainings range from turning on a computer to English language or soft skills, however they rarely go into developing any of these skills higher than pre-intermediate level (Humans in the Loop 2025, 15). While this is important for some displaced data labellers, many other are not only literate but also have a higher education (al-Hammada 2024, 8). Transferability of these skills thus remains doubtful, trapping refugees in the cage of labelling microtasks for specific clients who have narrow rules and skills needed for usually short-term projects.

5. International policy landscape: Emerging good practices

The regulatory vacuum that historically enabled these harms is beginning to fill. The following instruments constitute a growing toolkit that UNHCR can leverage in its frameworks for refugee platform work. These examples were chosen to represent relevant policies in different regions and are not meant to cover the entire regulatory landscape of platform work and are listed by scale of coverage.

Who introduced Measure / Instrument Relevance for Refugee Contexts
ILO Convention concerning decent work in the platform economy, ILC.114/Convention No. 193: recognizes workers’ rights to decent labour and occupational health and safety; requires human involvement in algorithmic management changes; mandates disclosure of monitoring and decision-making systems; classifies workers based on the terms of their work  (ILO 2026). Provides UNHCR with a multilateral standard to cite in partnership negotiations and advocate for in UN digital governance fora.
EU Platform Work Directive 2024/2831 (EUR-LEX 2024): mandates employment status clarification, algorithmic transparency, human oversight of automated decisions and worker representative consultation. Directly applicable to BPO and platform-mediated data work. Creates enforceable transparency standards UNHCR can require of EU-headquartered partners.
Malaysia Gig Workers Act 2025 (Malaysia Parliament 2025): national legal framework regulating digital platform workers; prohibits unfair wage deductions, mandates payment transparency, ensures algorithmic transparency, introduces occupational safety requirements and creates a grievance tribunal (ILO 2025c). First comprehensive national gig worker law in Southeast Asia. Provides a model for host-country advocacy in regions where UNHCR has operational presence.
Australia Fair Work (Digital Labour Platform Deactivation Code) 2024 (Department of Employment and Workplace Relations 2024): minimum standards for worker deactivations; requires warning notices, opportunity to respond, review processes, and record-keeping. Applies to all “regulated workers” on digital platforms (ILO 2024). Addresses the arbitrary termination dynamic documented in refugee data labelling contexts. Model for NDA-alternative grievance pathways.
India (Bihar) Bihar Platform Based Gig Workers (Registration, Safety and Welfare) Act 2025 (PRS 2025): Welfare Board, Social Security Fund, registration, payment regulation, data protection, algorithmic transparency, grievance redressal (ILO 2025b). Demonstrates that emerging economies can legislate binding digital labour protections. Directly relevant to UNHCR operations in South Asia considering large numbers of gig workers in the region.

Considering the existing legislation, it is also important for UNHCR to take into account special vulnerabilities of refugees when developing own policy for work conditions of social enterprise programmes in data labelling, recommendation for which will be presented in the last part of this paper.

6. Policy recommendations

  1. Prioritize integrated pathways to sustainable livelihoods

To strengthen the long-term impact of digital livelihood programs, UNHCR should move beyond short-term output metrics and tie recommendations to longitudinal evidence that refugee participants progress toward stable, formal employment. Innovation grants can be impactful if directed toward programs that build genuinely transferable skills, such as advanced coding, ensuring that refugees, including those with higher education, are supported toward meaningful economic mobility instead of entry-level microwork.

  1. Adopt minimum labour standards via the Fairwork Framework

Current UN initiatives have made valuable progress on digital literacy and cyber risk awareness. To build on this foundation and ensure that digital livelihood programs deliver on their self-reliance promise, UNHCR should formally endorse the Fairwork Framework (Fairwork n.d.) as a baseline for recommendations to social enterprises across five dimensions:

  • Fair pay

UNHCR should work with social enterprises towards replacing piece-rate structures that produce poverty-level earnings with living wage thresholds adapted to host-country cost of living. Enterprises should be required to publish transparent, disaggregated income data as a condition of humanitarian endorsement, replacing the current practice of curated success stories with longitudinal evidence that the work actually generates economic self-reliance.

  • Fair conditions

Because employer-provided mental health support has been documented as often inadequate and even harmful for workers, UNHCR should mandate that trauma-informed mental health care for refugee data laborers be funded and managed entirely independently of employer control with particular attention to the double exposure risk for refugees labelling content from their own countries of origin.

  • Fair contracts

UNHCR is encouraged to work with social enterprises to review the use of NDAs in refugee data labelling contexts, exploring whether worker protections and client confidentiality can be balanced through alternative grievance mechanisms. Proactive disclosure of client identities and dataset purposes, where feasible, would help workers make informed decisions about the work they undertake.

  • Fair management

UNHCR should call on social enterprises to introduce meaningful human oversight of algorithmic performance systems, including the 90-97% accuracy thresholds enforced under time pressure, and to ensure that no worker faces income reduction or dismissal risk for raising concerns about task content. Task timelines should be set in consultation with workers, instead of dictated by client delivery schedules alone in order to improve both working conditions and output quality.

  • Fair representation

UNHCR should require social enterprises to build worker representation into their governance structures, viewing them as central voices with real influence over pricing negotiations, task standards and client selection. These should include, among others, regular anonymous worker surveys, introduction of Beneficiary Advisory boards which can receive concerns from workers, as well as allowing to form unions.

7. Literature list

  1. Al-Hammada, R. (2024). “If I Had Another Job, I Would Not Accept Data Annotation Tasks.”
  2. Altenried, M. (2024). Mobile workers, contingent labour: Migration, the gig economy and the multiplication of labour. Economy and Space.
  3. Bansla, V. (2025). How thousands of ‘overworked, underpaid’ humans train Google’s AI to seem smart. Internet source. URL: https://www.theguardian.com/technology/2025/sep/11/google-gemini-ai-training-humans
  4. Battisti, S. (2019). Digital Social Entrepreneurs as Bridges in Public-Private Partnerships. Journal of Social Entrepreneurship.
  5. Berhane Gebrekidan, F. (2024). Content moderation: The harrowing, traumatizing job that left many African data workers with mental health issues and drug dependency. Data Workers Inquiry.
  6. Clark, T. (2021). The Gig Is Up: An Analysis of the Gig-Economy and an Outdated Worker Classification System in Need of Reform, 19 Seattle J. Soc. Just.
  7. CWA Union. (2025). Ghost Workers in the AI Machine: U.S. Data Workers Speak Out About Big Tech’s Exploitation. Internet source. URL: https://cwa-union.org/ghost-workers-ai-machine#workers-value
  8. De Stefano, V. (2015). The Rise of the “Just-in-Time Workforce”: On-Demand Work, Crowdwork, and Labor Protection in the “Gig Economy”.
  9. De Stefano, V. (2016). The Rise of the “Just-in-Time Workforce”: On-Demand Work, Crowdwork, and Labor Protection in the “Gig Economy”. Comparative labor law.
  10. Department of Employment and Workplace Relations. (2024). Fair Work (Digital Labour Platform Deactivation Code) Instrument 2024.
  11. Easton-Calabria, E., & Hackl, A. (2023). Refugees in the digital economy: The future of work among the forcibly displaced. Journal of Humanitarian Affairs, 4(3).
  12. Equidem. (2025). Scroll. Click. Suffer. The Hidden Human Cost of Content Moderation and Data Labelling.
  13. EUR-LEX. (2024). Directive (EU) 2024/2831 of the European Parliament and of the Council of 23 October 2024 on improving working conditions in platform work.
  14. Fairwork. Principles. URL: https://fair.work/en/fw/principles/
  15. Hao, K., Hernandez, A. (2022). How the AI industry profits from catastrophe. Internet source. URL: https://www.technologyreview.com/2022/04/20/1050392/ai-industry-appen-scale-data-labels/
  16. Human Rights Watch. (2024). World report 2025. New York.
  17. Humans in the Loop. (2024). Impact Report 2023.
  18. Humans in the Loop. (2025). Impact Report 2024.
  19. ILO. (2024). Fair Work (Digital Labour Platform Deactivation Code) Instrument 2024 under the Fair Work Act. Internet source. URL: https://digitallabour.ilo.org/legislation/fair-work-digital-labour-platform-deactivation-code-instrument-2024-under-fair-work-act
  20. ILO. (2025a). Exploring the gig economy: Challenges and opportunities A self-guided resource.
  21. ILO. (2025b). The Bihar Platform Based Gig Workers (Registration, Safety and Welfare) Act, 2025. Internet source. URL: https://digitallabour.ilo.org/legislation/bihar-platform-based-gig-workers-registration-safety-and-welfare-act-2025.
  22. ILO. (2025c). Gig Workers Act 2025. Internet source. URL: https://digitallabour.ilo.org/legislation/gig-workers-act-2025
  23. ILO (2026). Convention concerning decent work in the platform economy. ILC.114/Convention No. 193. URL: https://www.ilo.org/sites/default/files/2026-06/ILC114-Instrument%20C.193-EN.pdf.
  24. ILO. Promotion, inclusion and protection of refugees and host communities in the digital and gig economy. Internet source. URL: https://www.ilo.org/projects-and-partnerships/projects/promotion-inclusion-and-protection-refugees-and-host-communities-digital.
  25. Jenkins, T. (2024). Garbage in Garbage out. Internet source. URL: https://www.applogicnetworks.com/blog/garbage-in-garbage-out
  26. Kaneti, M. (2025). Social Enterprise Solutions for Migrants, Microwork, and AI Ethics: The Case of HiTL. Human Service Organizations: Management, Leadership & Governance, 49(5), 609-627.
  27. Le Poidevin, O. (2026). Exclusive: Cash-strapped UN refugee agency to cut more jobs, even as crises mount. Internet source. URL: https://www.reuters.com/world/europe/cash-strapped-un-refugee-agency-cut-more-jobs-even-crises-mount-2026-05-18/
  28. Malaysia Parliament. (2025). Rang Undang-undang Pekerja Gig 2025. D.R. 27/2025.
  29. Miceli, M. (2024). “I hope this isn’t for weapons.” How Syrian data workers train AI. Internet source. URL: https://untoldmag.org/i-hope-this-isnt-for-weapons-how-syrian-data-workers-train-ai/
  30. Microsoft. (2026). AI Diffusion Report. Global AI Adoption in 2025—A Widening Digital Divide.
  31. Portales, L. (2019). Basics, Characteristics, and Differences of Social Entrepreneurship.
  32. PRIO (2025). New data shows conflict at historic high as U.S. signals retreat from world stage. Internet source. URL: https://www.prio.org/news/3616
  33. PRS. (2025). The Bihar Platform Based Gig Workers (Registration, Safety and Welfare) Act. No 8.
  34. Sama. (2022). Environmental & Social Impact Report 2022.
  35. Subul. (2024). Annual Impact Report 2024.
  36. SuperAnnotate (2025). What is data labeling? The ultimate guide. Internet source. URL: https://www.superannotate.com/blog/guide-to-data-labelling
  37. Techfugees. From tech user to tech empowered (#Empowerment). Internet source. URL: https://www.notion.so/covidrefugees/2-From-tech-user-to-tech-empowered-Empowerment-11392e4039c8492d8cef375417c01166
  38. UNCTAD. (2025). 2025 Technology and Innovation report. Internet source. URL: https://unctad.org/publication/technology-and-innovation-report-2025
  39. UNHCR. (2005). Handbook for Self-Reliance. Geneva.
  40. UNHCR. (2023). Refugees’ Access to Jobs and Financial Services. Background Guide. Challenge Topic #3.
  41. UNHCR. Refugee Data Finder. Internet source. URL: http://unhcr.org/refugee-statistics.
  42. World Weather Attribution. (2025). Unequal evidence and impacts, limits to adaptation: Extreme weather in 2025. Internet source. URL: https://www.worldweatherattribution.org/unequal-evidence-and-impacts-limits-to-adaptation-extreme-weather-in-2025/

Endnotes

[i] The difference is mostly visible in these two websites of the same social enterprise: The NGO part: https://humansintheloop.org/impact/foundation/, the corporate part: https://humansintheloop.org/

[ii] See, for example, two entities of this social enterprise:  Na’amal. URL: https://naamal.org/, Na’amal Agency. URL: https://agency.naamal.org/.

[iii] See the website: HITL. URL: https://humansintheloop.org/.

[iv] Look, for instance at the following social enterprise sources: Techfugees. Inclusion. URL: https://techfugees.com/inclusion/,  Medium. Na’amal, DOT, and The Conrad N. Hilton Foundation Launch Follow-Up to Successful Digital Skills Programme in Ethiopia. Internet source. URL: https://na3amal.medium.com/naamal-dot-and-the-conrad-n-9c2268fd56b2

Arbitrary cognitive offloading to GenAI: Does the current policy landscape account for the right to quality education of children and youth in the European Union?

Abstract

General-purpose generative AI (GenAI) is increasingly used by young learners in the European Union, with 20% of individuals aged 15–29, approximately 15 million people, relying on it for educational tasks such as information retrieval, feedback, or full task delegation. While cognitive offloading is a natural part of human learning, GenAI’s ease of use risks disrupting efficient learning processes, potentially depriving children and youth of their right to quality education, including foundational skills like literacy, numeracy, and critical thinking.

This paper argues that the unmediated use of GenAI in educational settings undermines the right to quality education by interfering with cognitive development. It examines the role of cognitive load, metacognition, and cognitive offloading in learning, linking these concepts to GenAI’s potential to disrupt memory consolidation, problem-solving, and critical thinking. The analysis evaluates the current European policy landscape, focusing on the EU AI Act, OECD Digital Education Outlook 2026, and the European Parliament CULT Committee’s 2026 briefing to assess whether existing hard and soft policies address the cognitive risks of GenAI. While these policies recognize risks, they primarily target AI-enhanced educational technologies rather than general-purpose GenAI, leaving a regulatory gap.

The paper concludes that arbitrary cognitive offloading via GenAI jeopardizes the normative goals of quality education, as defined by frameworks like UNICEF’s and Bloom’s Revised Taxonomy. It calls for a revision of the EU’s normative approach to general-purpose GenAI, emphasizing the need to protect young learners’ right to cognitive development and autonomy.

1. Introduction

Generative AI (GenAI) is usually defined based on its ability to create (“generate”) new audiovisual and textual content, but it can also be defined by the fact that this ability enables it to serve a broad array of use cases. In other words, GenAI is also general-purpose AI. It is in this latter respect that young learners in the European Union use it. Indeed, they are its heaviest users. In 2025, 20% of them use it in the context of their education. This amounts to a total estimated number of approximately 15 million[1] people between 15 and 29 years that use GenAI to find information, seek feedback for their assignment, or delegate entire tasks.

Given its general-purpose nature, the sheer ease to delegate cognitive tasks has been shown to disrupt the efficiency of learning processes and therefore carries high risks for young people to de deprived of the quality education they are entitled to.  At the same time, offloading cognitive tasks is part of every human experience. A young learner usually is cognitively not yet in a position to learn how to solve a problem and to solve a problem in the same instance. Learning strategies provide means to reduce occurring cognitive load, and to foster an effective learning process. GenAI seems to interfere with the equation of keeping required cognitive effort to produce deep understanding.

In this paper, I am going to argue that GenAI in educational settings[2] deprives children and youth from their human right to quality education, specifically from their right to acquire basic formative skills, including literacy, numeracy and critical thinking. I am going to lay out this argument in three steps:

In the first part of this paper, the background section, I provide context about the human learning process, the influence of cognitive load, and subsequent strategies such as cognitive offloading and metacognition. I establish links to GenAI’s possible interference with learning and human cognitive development. Especially young learners who are building their cognitive architecture are susceptible to irreversible consequences. Such consequences include failed memory consolidation, cognitive atrophy, illiteracy, impaired problem solving, lack of critical thinking, and consequently loss of autonomy and human dignity. I connect these concerns to the human right to education, examining the meaning of quality education and its links to educational goals. In the middle part, I am using a case study to establish a comprehensive understanding of GenAI’s intricate impacts on cognitive and metacognitive dimensions of young learners.

In the third part, I analyse the current remedial standings of European hard and soft policies for cognitive impact dimensions of general-purpose AI. The EU AI Act being the baseline, the OECD Education Outlook 2026 report and the CULT Committee’s briefing 2026 are centre pieces of the analysis. These reports recognize the pressing dangers on cognitive development of GenAI in education and propose risk mitigations. Amending a hard regulatory element of a cognitive criterion in the EU AI Act is a deliberative proposal forward but designed with AI-enhanced education technologies in mind.

Finally, the paper takes the occasion to discuss GenAI’s normalisation and that the current understandings should lead to revise our normative approach to general-purpose AI.

2. Methodology

This paper originally set out to examine the cognitive influences to which young people are exposed through the use of GenAI during their formative years, and how the European policy landscape is currently addressing this. Following a literature review in February 2026, the initial finding was sobering: no significant policy instruments yet existed. However, when the two documents OECD Digital Education Outlook 2026 and CULT Committee Briefing were published in March 2026, it became necessary to revise both the initial research question and the methodology for addressing it.

Conceived as a policy analysis, this paper now examines whether GenAI influences the human learning process to such an extent that the right to a quality education is at risk, and how the latest policy measures address this issue to significantly reduce this risk.

To answer these questions, a qualitative approach was taken, overlaying existing scientific findings with internationally applicable law and political regulatory procedures. Sources were selected accordingly, including scientific studies, background papers, journalistic sources, and a statistical report.

  • Scientific studies: A total of 20 primary scientific studies were selected from the fields of cognitive psychology, cognitive science, computer science, educational science, and neuroscience. These were identified via keyword searches (e.g. cognitive offloading, cognitive overreliance, cognitive atrophy, extended mind, critical thinking, artificial intelligence and the education of children and young people) and through references within the background papers.
  • Foundational papers: Nine foundational documents on general-purpose AI and education in the European context (OECD, European Commission, European Parliament) were selected. These were identified via keyword searches (e.g. cognitive dimensions, AI harms, European policy, European Education). Two of these documents, namely the OECD Digital Education Outlook 2026 and the CULT Committee Briefing 2026, for in-depth policy analysis as they were the only ones addressing the issue at a pan-European level at that time.
  • Journalistic sources: These serve to enrich the paper qualitatively, but do not contribute to the onus of proof.
  • Statistical report: One selected document (“Young people in Europe: A statistical summary 2022”) provides descriptive evidence of stated user figures, but does not analyse the figures themselves.

Based on this data, a comprehensive picture of the state of scientific knowledge in this area can be formed, enabling valid conclusions to be drawn about the comprehensiveness of policy measures. However, the consequences and effects of the measures cannot be predicted based on the available data. This also means that the conclusions of this study are limited in terms of the possible positive and negative consequences of GenAI use on cognitive development. Consequently, any argument relying on this evidence base is euqally limited.

3. Background

This section of the paper introduces concepts and issues required for an analysis of the paper’s leading question on the impact of arbitrary cognitive offloading on the goals of education, and the influence of European policy: cognitive load theory, metacognitive laziness, the goals of education and EU education policy.

Cognitive Load Theory

Learning is a process involving information. When there is more information than the brain’s working memory can handle, or too little, the learning process is hindered.

Cognitive Load Theory is based on the assumption that the working memory has a very limited capacity, and that learning places a cognitive load on it. In contrast, the long-term memory has virtually unlimited capacity. Learning involves taking in information and processing it inside the working memory before subsequently storing it in the long-term memory. This storage, in turn, takes the form of schemata: complex knowledge structures containing multiple types of knowledge, e.g. declarative (knowing-that), or procedural (knowing-how) knowledge. It is important to emphasise that accumulating knowledge alone does not lead to expertise; instead, complex schemata differentiate experts from novice learners.

Managing cognitive load in the working memory influences the efficient construction of a person’s cognitive architecture. According to John Sweller, cognitive load originates from intrinsic, extraneous or germane sources. Intrinsic load arises from efforts to process the content of a task or the complexity of a problem, whereas extraneous load originates from the instructional design and accessibility of the task. Germane load contributes to schema construction, a highly desirable learning outcome. Thus learning conditions should reduce extraneous load, while intrinsic and germane load should be increased. This combination improves learners’ chances to build the foundations for deep learning and, consequently, deep understanding. For more information on this topic, see Sweller et al. (1998).

Metacognitive Laziness

Cognitive offloading is a strategy to regulate efforts of learning. Formally, it is defined as “the use of physical action to alter the information processing requirements of a task so as to reduce cognitive demand” (Risko & Gilbert, 2016, p. 676). Counting with one’s fingers, taking notes or navigating with a GPS, these are all forms of cognitive offloading.

The kind of load that is offloaded through these tasks depends on the intended learning goal: if someone is trying to get to a location in a city quickly perhaps due to an emergency, it would not be detrimental for them to use a GPS, but it would be detrimental for them if they were trying to learn the city’s geography to ease navigation later. In both situations, the intrinsic load (the factual information itself, e.g., the ways around the city) and germane load (the motivation and effort that are required, e.g., to build the schemata of the city) are offloaded. However, in the second scenario, the person would not want to offload these, because they are purposefully trying to build schemata in their long-term memory.

Given that the learning process is an essential part of the cognitive development of children and young people, the function of cognitive offloading should be to make learning objectives achievable, not to effectively do the objectives for them. Take for example using a calculator for arithmetic operation versus counting by hand. Children who use their hands show substantive cognitive resources and are more efficient in arithmetic calculations  (Thevenot et al., 2025). In fact, germane load has been shown to enhance the learning effect, for example determining which new information aligns with existing knowledge (Gerlich, 2025; Risko & Gilbert, 2016, p. 685).

Taking self-regulated learning decisions, such as deciding over single cognitive tasks to perform, or to offload, is a human intellectual capacity called metacognition. Metacognition is thinking about thinking, and supplies an additional form of thinking that supports the learning process. Essentially, the learner can realize what they do not know to perform a task. Through metacognition, thinking has the chance to become critical thinking, e.g. when continuing the search for more reliable sources of information, or adjusting ones strategies of fact-checking (Singh et al., 2025; Winne, 2021). The deliberative effort from metacognition and its attached decision-making creates an additional cognitive load, the metacognitive load. Take for example the act of evaluating the perceived difficulty of a task. Usually such evaluation consists of a multi-stage process where the individual assesses if they can retrieve the specific knowledge from their memory, how easy or hard this retrieval might be, or if the expected utility of, e.g., conducting a Google search right away, would be higher than actually engaging in a recalling from memory process. A metacognitive belief is formed that forms an argumentative ground for the learners final decision.

These metacognitive actions cause an additional cognitive load in the working memory of the learner. Challenges to sustained metacognitive engagement originate in self-perceived capability to solve a problem, low achievement motivation or complexity of a given task. Unlike offloading cognitive tasks for load reduction as discussed earlier, metacognition does not need offloading in that sense. Instead, the learner metacognitively disengages. This behaviour is called metacognitive laziness.

Pedagogical approaches propose metacognitive scaffolding techniques that enable learners to self-regulate and stay engaged with the task at hand. Scaffolding proposes to solve a task by means of distinct engagement phases: planning (“What is it that I must do? Have I done a task like that before? What can I solve independently? What can I not solve independently?), execution (How well do I understand the material? Can I use this source? Should I seek help for this part, and why?) and reflection (Have I managed to solve the task to my satisfaction? What have I learnt about my approach of such a task? What does that mean for next time?). This approach funnels information in a way that they turn into germane load. In that case, what could have been an extraneous load, is transformed into a learning plan, or tasks that are broken down into subtasks and evolve in difficulty.

By the same token, freeing up mental resources from low-cognitive involvement tasks, e.g. grammar checking, can facilitate critical thinking which is considered an activity of high-cognitive involvement. Again, depending on the goal of the learning task, these activities oscillate between different types of cognitive load.

The Goals of Education

Education is a fundamental human right. Article 26(2) of The Universal Declaration of Human Rights (UDHR) sets forth that “Education shall be directed to the full development of the human personality and to the strengthening of respect for human rights and fundamental freedoms. It shall promote understanding, tolerance and friendship among all nations, racial or religious groups, and shall further the activities of the United Nations for the maintenance of peace” (United Nations, 1948).

In this sense, education is not the direct realisation of human dignity, but a necessary consequence arising from the exercise of the right to education. Therefore it stands to reason that access to any form of education alone is not sufficient to achieve this level of impact; quality standards must also be met. To get to the bottom of the question of whether arbitrary cognitive offloading undermines the impact of education, the first step is to examine whether international quality criteria are in place and examine whether the of GenAI violates these criteria.

The minimum criteria of quality in education aren’t prescribed in an international treaty and therefore, quality education cannot benefit from an international standard that nations can be held accountable for[3].
A paper authored by UNICEF (2000) characterizes quality education through the dimensions of

  1. safe and inclusive schooling facilities, well-managed classrooms
  2. participants and their mental health
  3. child-centred teaching processes and assessment 
  4. curricula and content that enables the acquisition of basic skills (literacy, numeracy, and skills for life)
  5. outcomes that encompass knowledge, skills and attitudes, and are linked to national goals for education and positive participation in society.

Under point 4., the paper states that learning should be student-centred[4] and tailored to individual learning differences, and that “in general, curricula should emphasize deep rather than broad coverage of important areas of knowledge, authentic and contextualized problems of study, and problem-solving that stresses skills development as well as knowledge acquisition”. Here, it is also emphasized that content cannot be separated from the process, that is, how children learn to read is a product of what they are reading, and respectively, the literacy they develop. Numeracy being taught in an integrated manner provides foundations to develop logical reasoning and advanced interpretative communication skills. Both literacy, and numeracy, so goes the paper, form foundations to acquire life skills that include topics such as health, hygiene, etiquette, vocational skills. Thus, at the heart of defining the meaning of quality education lies a mediation of life skills imparting from content and process in equal terms. The process should rely on student-centred methods, and practices to elevate skills to competence. 

UNICEF’s definition builds on Robert Marzano’s New Taxonomy. It draws a distinct line between lower-order versus higher-order thinking skills, and the learner’s active engagement in deciding how to transform information (Marzano & Kendall, 2007, pp. 17–19). This one, in return, builds on Bloom’s Revised Taxonomy. The latter provides a framework that has largely informed our understanding of the role of schools, universities and further education institutions prior to UNICEF’s definition project. It assigns a cognitive, procedural and a declarative knowledge dimension to the intrinsic goal of education. Within the boundaries of this framework, education mediates capacities that require the interplay of knowledge acquisition, its immanent updating, and cognitive processes. Therefore, with an increase in complexity, school curricula should entertain ways to foster essential human skills and traits:

  1. memory consolidation and remembering
  2. understanding concepts
  3. applying judgement
  4. analysing
  5. evaluating
  6. creating

On the bottom level of complexity, lower-order thinking skills include storing and remembering facts, as well as understanding concepts. Higher-order thinking skills comprise the analysis, evaluation and application of complex judgement. Critical thinking, problem solving and decision making are compounds of these skills. The process that bears these compounds requires executive functioning[5] and metacognition. UNICEF makes mention of the thinking about thinking-activity in its definition of quality education under student-centredness. In other words, quality in education should account for mediating metacognitive skills because of its impact on forming cognitive abilities.

However, The metacognitive dimension of learning can be an inconsistently explicit aspect of the lived learning process in schools. Young children in early education years configure their workload by strategies such as counting with their fingers. Progressing towards more complex educational goals, older children begin to coordinate their cognitive skills with the task they have been given and develop an understanding about effective learning strategies. The development of these learning strategies can be actively construed, or accidentally, not knowingly, assimilated.

More traditional teacher-centred approaches to education assume a unidirectional transfer of knowledge from the teacher to the student, whereas constructivist approaches emphasise the active involvement of the student. At the heart of constructivism lies the idea that learner’s prior knowledge influences how and what of the new information is assessed and stored. Therefore, learners actively shape their personal learning process through monitoring and planning activities and consequently alter their individual epistemic journey. More importantly, constructivism values the learning process as a source for knowledge in and of itself. This is what Willingham referred to when he wrote that thought processes are intertwined with what is being thought about. According to him, it is impossible to engage in critical thinking when one has nothing to think critically about, e.g. a learner can only think critically about something based on their existing prior knowledge (2019).

In contrast, more traditional approaches operationalise the teacher’s cognitive skill and knowledge as the teaching transaction and assume that critical thinking can be facilitated by what is transmitted from the teacher to the learner. This pedagogical approach bears higher chances to encroach higher-order thinking skills, than a constructivist approach does. This insight matters for the critical assessment of the impacts of GenAI in the classroom.

EU Education Policy

What does current EU policy say about the intersection of AI and policy? To answer this question, we will use the EU AI Act as the baseline, as well as the OECD Education Outlook 2026 report and the CULT Committee’s 2026 briefing. In brief, the EU AI Act introduces a a trade-off between the acquisition of digital skills versus the acquisition of basic learning skills. The OECD report recommends a more effective use-approach to GenAI and thus contains no notable references to regulating cognitive influences. In contrast, the CULT Briefing considers amending the AI Act to include an assessment of precisely this cognitive impact. However, regulatory recommendations seem to be have been designed to address AI-enhanced learning technologies, and not GenAI. Therefore, further analysis is provided to propose additional mitigation strategies.

The EU AI Act

Recital 56, the EU AI act explicitly states that the deployment of AI systems is a realisation to the right for high-quality education, naming the acquisition of digital skills and competences, including critical thinking skills.

It reads “The deployment of AI systems in education is important to promote high-quality digital education and training and to allow all learners and teachers to acquire and share the necessary digital skills and competences, including media literacy, and critical thinking, to take an active part in the economy, society, and in democratic processes. (…)”.

At the same time, it classifies the same AI systems as high-risk in cases where they assess, monitor or evaluate individuals, and consequently the trajectory of their lives. This reasoning displays a profound gap regarding the perils of GenAI in the classroom – and outside of it. Fundamentally, the right to quality education encompasses the right to form digital literacy and critical thinking skills. In the case of GenAI, this seems deeply conflictual. The general-purpose nature of GenAI provokes arbitrary cognitive offloading when metacognitive capacities are not or only little developed and can imply cognitive risks for millions of young people. In other words, teaching digital skills using GenAI entails the risk of undermining other fundamental learning processes and skills. Given the concept of quality education as outlined here, it is reasonable to conclude that the consequences of such an undermining would cause greater harm than the teaching of digital skills could justify. The scientific groundwork for this hypothesis is presented in the case study of section 4.

This lays the ground for a suspicion that the EU AI Act’s specification to account for the right to quality education disables itself. Given the current evidence regarding GenAI in the classroom, it appears contradictory to ensure that digital skills are taught (e.g. prompting, understanding capabilities of an LLM) while also guaranteeing the development of higher-order thinking skills (e.g. discerning truth from falsehood, drawing conclusions from prior analysis). This contradiction is evident in the fact that GenAI allows for arbitrary cognitive offloading, resulting in children and young people receiving a lower standard of education than they would otherwise. Consequently, children and young people are unable to enjoy their full right to education.

The following parts provide an analysis of recent educational policy publication as well as a case study of a learner’s behavior when solving a learning task involving GenAI. The case study shall provide context for the analysis at stake: does European education policy recognize and address the cognitive dimensions of GenAI and their problematic impact on children and youth as beneficiaries of the right to education?

OECD Digital Education Outlook 2026

The OECD Digital Education Outlook 2026 (2026) has been chosen for review inside this project for two reasons. First, in 2023 the OECD published a background paper with the title “Generative AI in the classroom: From hype to reality?” which provided first insights of major issues with GenAI in classrooms (2023). The paper’s tone reflected on potentials for more dynamic classrooms and individualized learning. Second, it is authored by a long-standing international organisation, combining the represention of economic interests with a transnational analytical rigour. Findings and recommendations of OECD reports are of non-binding nature, collecting scientific evidence for informed public and political discussion.

The analysis focuses on the categories of learners, teachers and educational institutions[6] and sets out recommendations for all these target groups aimed at using GenAI to improve learning, teaching and institutional management. The report is framed around outcomes these groups create, are responsible for or are involved with.

The rise in content generated by GenAI constantly challenges the limits of our knowledge in everyday life. Misinformation and deception pose a challenge to society as a whole, the scale of which is already socially and politically evident, for example through media attacks on democratic systems or infringements of personal rights. Right at the outset, the report emphasises that the utility of higher-order thinking skills,  namely critical thinking, problem-solving and decision-making, as well as metacognitive abilities, will increase.

The report collected statistics from OECD member states to draw a more precise picture on age groups and level of education where GenAI is used and for which purposes. Globally, teenagers (age 12-17) make for the group among all age groups with highest usage, e.g. 90% of Estonian school students report making use of GenAI, followed by higher education students, e.g. 85% of French students reported having used a GenAI tool in 2025 at least once, where numbers were up to 94% for German students, with 65% of them reporting to use it daily or weekly. Looking at the most popular uses of GenAI within these groups, the report refers to a study conducted in seven European Countries by Vodafone Stiftung (Stiftung Vodafone, 2025):

Type of cognitive engagementType of learning taskSelf-reported use in %
Low-cognitive involvement tasking[7]finding information56
receiving explanatory support for terms and concepts45
High-cognitive involvement tasking[8]using complete solutions to given tasks    31
Meta-cognitive taskingmake use of interactive content to guide learning experience29
structuring and planning a personalised learning path20

The report indicates significant variances between countries which may be due to national differences in access, proliferation, dissemination, and cultural perception of GenAI tools, but differences in research methodology and assessment criteria might explain them equally well. Therefore, interpreting these numbers should be approached with caution.

CULT Briefing “Artificial Intelligence in the Classroom”

In early 2026, the Committee on Culture and Education (CULT) of the European Parliament (EP) received the briefing “Artificial Intelligence in the Classroom”, the first one centered around this question. Committees of the EP examine proposals prior to bringing them before the whole assembly. The CULT Committee works on european educational policy that promotes engaged and resilient European citizens through programs such as Erasmus+. Given that education is a matter of national sovereignty, CULT’s responsibility consists of vision building, convening, and deliberating educational opportunities, risks and challenges in the European Union. Findings and recommendations of CULT briefings are of non-binding nature that collect scientific evidence for deliberation. Inspite of their soft policy nature, it is said that these committee briefings project a baseline understanding of contemporary issues among stakeholders of European policy making. This marks the reasoning why this briefing has been chosen for review inside this project.

The report differentiates between GenAI and AI-enhanced educational technologies, and clarifies that most of the reported risks are associated with GenAI in an unmediated education context. In comparison, AI-enhanced educational technologies are product optimised for learning purposes. Therefore, there needs to be a bifurcated approach for governance and responsible use.

The CULT Briefing refers to identical usage figures already introduced in the OECD report discussed above. It adds that two thirds (63.8%) of young people aged 16–24 in the EU used GenAI in 2025 which makes for almost twice of the adult population (32.7%). Among this age group, 39.3% stated they were likely to use AI tools for formal education (Chounta, 2026, p. 2).

That means that around 20%[9] of young people in the EU use AI tools for formal education. Despite grounding its argumentation in the distinction between the substantive risks imposed by GenAI versus AI-enhanced educational technologies, the report does not issue usage numbers of the latter.

4. Case Study

To illustrate the challenges brought on by GenAI’s introduction into education, let us suppose a learner named “D” in early high school is given the task to compare two historical texts A and B from two authors about the same event and articulate the authors’ divergent viewpoints of the event. For example, do the authors reference different facts, what are their general tones, do they provide valid reasoning for their differing viewpoints, do they employ manipulation strategies and if so, what might be their motivations in doing so?

Learners are free to decide how they approach the task, e.g. reading text A first and taking notes, then text B and taking notes, or skimming both and underlining similarities and differences afterwards, or not making any marks at all. The aim of the task is not to resolve which of the authors is “right”, but to train the learner in at least four cognitive skills: working memory (keeping one text in mind while reading a second one), reading comprehension (decoding text, extracting meaning, constructing a representation of the text), sustained attention (identifying different or identical facts, deviations in wording), and critical thinking (analysing line of reasoning and manipulation strategies).

D trains their metacognitive skills when they decide which approach is more promising for them to achieve the task, when they monitor how well they have understood the text, if they need a second pass, and if they have sufficiently engaged with the critical questions. Last, the task imparts knowledge-that (declarative) of the event, and knowledge-how-to (procedural) about the approach of such a task. The latter fosters the development of mental schema, and repeating similar tasks will reinforce their availability, and differentiates unexperienced learners from experts.

Cognitive processes

Since there is evidence of positive learning impacts from usage modes where a chatbot takes on the role of a deliberative interlocuteur, a replication of the Socratic Method[10], it is possible to argue in favour of D making use of a modified GenAI (Monzon & Hays, 2025; Singh et al., 2025; Xu et al., 2025). However, the initial reading (reading comprehension skill) and note-taking (sustained attention) should not be skipped by D: The problematic use case we are concerned with here is if D uses GenAI to complete the entire task, e.g., prompting it to ingest and compare the two texts, their tone, the presence of manipulative tactics, and so on. In which case D neither works on the information contained in the task, nor gains the skills required to complete it.

Over-reliance

This skipping-through-the-learning-process qua the aid of GenAI has been shown to bring about a few direct and indirect consequences for the learner. For example, studies by Gerlich (2025), Kosmyna et al. (2025) and Zhai (2024) have shown that students who arbitrarily offload whole tasks to GenAI, such as D does, indeed hamper their cognitive development: They eperience distorted memory consolidation, aren’t able to speak about the topic later on, or cannot make cross connections to other learning contents. Overreliance captures the process of handing more tasks over to AI than the learner should have handed over according Cognitive Load theory. For example, D could ask for background information to the two texts D should compare, why they are meaningful, or what their public perception might me. In this case, D would involve GenAI for a lower-engagament portion of the task, and keep the rest of it for himself to process, thus he would rely on GenAI, but not overrely.

The suggestion that offloading low-cognitive-involvement-tasks facilitates critical thinking provides an important provocation for the critical assessment of GenAI in an educative setting: can we save critical thinking by enumerating the low-cognitive involvement tasks that children and youth can offload, and the critical thinking (along with other high-cognitive-involvement-tasks such as problem-solving and decision-making) will build and develop naturally?

Several clues should lead to the safe assumption that this is not the case:

In the case of D, the initial reading and taking notes, is typically the low-engagement portion of the task they should not offload if their learning goal is to improve critical thinking abilities through the appropriation of text. Through the act of reading, D’s metacognitive capabilities will do their part to assess how well they understand the text, if they possess information about such the problem of comparing two texts and how to solve them, and if they want to seek help.

By retrieving and deciding about the information to retrieve, D is already with the problem in a critical way (Willingham, 2019). What is more, the metacognitive decision process that underpins the analytical process just described would be massively hampered, or disrupted, if there was a generic answer to the offloading question. Hence, offloading a cognitive task for reasons of ascribed cognitive engagement status does not solve the problem of cognitive development, even if it does solve the problem of arbitrary offloading. This is an important insight to come back to when evaluating possible solutions to the problem.

Atrophy

Cognitive atrophy is a consequence of cognitive offloading of single tasks, and to overreliance, particularly when heavily repeated. A well known example of cognitive atrophy is a form of functional illiteracy: the degration of once acquired reading and writing skills. It occurs when these skills are not regularly used, for example when D consistently watches images or videos instead of reading texts, or prefers digital text-to-speech applications over D’s own writing, or simply lets GenAI do the reading. When the cognitive pathways that are involved in reading and writing process are not demanded, they atrophy. In fact, the last OECD’s Survey of Adult Skills (2024) revealed that adult functional illiteracy is rising globally while it had been steadily reduced in the past century[11].

Metacognitive processes

Still in the young internet days, Nicholas Carr described how using intellectual technology, whereby referring to the internet, had changed his ability to absorb a long read, and to memorize what he had read. He asked if losing the ability for deep reading eroded the ability for deep thinking (Carr, 2008). This question kicked of a series of Google Effect-studies that primarily focused on individual memory allocation in tension with an externalised memory (the internet), and others provided the first links between offloading strategies best described as metacognitive laziness and illusions of competence, also called false mastery. For example, one study showed the augmentation of people’s self-esteem, and overrated feeling of having-knowledge-in-their-head when they had completed a quiz successfully with Google’s help. They would also overestimate how well they would perform in the next test of that kind without an external aid (Fisher et al., 2015). In a similar setting, people should solve a quiz and were divided in two groups that either was allowed to use, or not to use google. Before answering the quiz, they were asked whether they would know the answer. Interestingly, the group that was allowed to google the answer made significantly less effort in thinking about whether they would know (retrieve) the answer by themselves (A. F. Ward, 2013; D. M. W. Ward Adrian F., 2013).

Laziness and the illusion of competence

One of the many outcomes teachers have observed, and research evidence confirms, concerns the dissociation between improved performance on assigned tasks and longterm skill or knowledge acquisition: these studies show that writing an essay with GenAI improves essay scores (the outcome) but does not increase knowledge gains. Furthermore, in another study, students were asked to assess their own pen and paper writing for improvement: the setting allowed one student group to seek help from human tutors, and another one from of GenAI. The results indicate that the GenAI-group would skip a pre-assessment all together, and ask GenAI directly for improvement suggestions, whereas the human tutor group followed the model of the “help seeking” theory (Chen et al., 2025; Fan et al., 2025).

The latest studies examining the impact of GenAI on metacognition confirm correlations between metacognitive laziness and illusions of competence. The randomized study by Fan et al. (2025) found that the convenience of AI can undermine learners’ engagement in the learning processes (planning, monitoring, and revision). The learner hands over their metacognition to the tool.

Another study confirm that students may solve more tasks in less time when they ask for the solution but show lack of genuine understanding. What is more, learners’  self-reported confidence of learning gains is systematically overrated (Singh et al., 2025, p. 2). For novice learners who are still building schema and have less knowledge about what they do not know, this often invokes the Dunning-Kruger effect, where people tend to vastly overestimate their level of knowledge, particularly in new, unfamiliar domains (Kruger & Dunning 1999).

Overerstimations of gained competence are not exclusive to GenAI but span across digital learning where ease of use is confused with depth of learning. GenAI’s design, its availability and accessibility, or its generalness, suggest an ease of use for any given task and provides ground for metacognitive laziness through which arbitrary offloading is catalyzed. The category mistake of users to confuse ease of use with cognitive involvement impacts the development or training of critical thinking abilities, and is, consequentially, detrimental for developing expertise (Lodge & Loble, 2026, p. 23). These findings are evidence for GenAI promoting metacognitive laziness.

How do these findings translate to D’s behavior? During the initial reading, D feels a certain cognitive and metacognitive burden synthesizing in their head, e.g. the difficulty of the vocabulary, the length of the text, grasping the overall tone and meaning of the text. Overwhelmed, D decides to prompt GenAI to generate the required output. D sees the output, looks at it briefly with satisfaction, thinking: “That does not look so difficult, looks like I could do that myself”, and pastes it into D’s homework document. D having formed a belief about a newly gained competence in solving such a task, 2 weeks later, a class exam contains a similar assignment. D reads the instructions, and then both texts. Soon, D metacognitively reflects that the task appears to be at least as difficult as it did two weeks ago, and that D beliefs not being up to the task. Nervous and cognitively overwhelmed, D takes notes while struggling to remember the first text during the lecture of the second. D then then turns to the main part of the task to analyze divergent viewpoints (e.g. by presented facts, general tone etc.). D tries to recall the ouptut the GenAI had produced to guide the process but fails to do so. Even more nervous, D undertakes the analysis nonetheless in an ad-hoc, unsystematic fashion.
If the condition was not the exam setting, but comparable situation from the initial confrontation with such a task, it is legitimate to assume that D would once more rely on GenAI and arbitrarily offload D’s cognitive involvement to solve this task, and the vicious cycle between over-reliance, illusion of competence and metacognitive laziness would continue.

Metacognitive laziness and social costs, transaction costs

Learning strategies often involve seeking help when facing challenges. In that case, the learner metacognitively assesses if they involve teachers, peers or GenAI to solve their learning tasks. Studies show that seeking help from GenAI alters the help-seeking process. Students would seek help earlier in their process from GenAI than they would from a human, and skip to evaluate the generated answer they received. The study reasons this finding with the involved costs of help-seeking:  if the perceived knowledge-gain is lower than the cost of the social interaction with the teacher, e.g. because of possible reputation loss, then they either avoid seeking help, or involve GenAI instead. This finding confirms the risk for metacognitive laziness in front of GenAI, and it social costs precisely to inhibit this laziness (Chen et al., 2025).

Another study could show that students who ask for explanations, instead of solutions, increase their understanding of the topic, but not the volume of solved tasks. The same study reveals that simply disabling the copy and paste function invokes higher transaction costs, and prevents a decrease in required learning efforts (Lehmann & Cornelius, 2025, p. 24).

Reviewed studies confirm deep disturbances with learning efforts that shape the development of cognitive abilities and skills of novice learners like children and youth. The cognitive consequences include long-term memorization issues, and distorted schema building. Compound higher-order thinking skills depend on the quality of metacognitive strategies, that is, they either do not form or atrophy when effort is replaced by laziness. There is a critical subsequent dynamic at play that brings about overconfident learners who overestimate their acquired skills while they miss opportunities to develop long-term expertise. To summarize, associated risks of GenAI for learning purposes could outweigh its anticipated benefits (Yan et al., 2024).

Other issues

The impacts of arbitrary (meta-)cognitive offloading span from failing to store and recall information in the long-term memory, making it difficult for learners to refer and discuss what they thought they had learned, to cognitive atrophy and metacognitive laziness, which imply the loss of literacy, numeracy, critical thinking, and adaptive problem solving. Beyond these effects, they distort judgements of the self as a knower, the trustworthiness of the source, and the epistemic value of the learning process. In the grand scheme of education, numerous consequences for young people, and society at large, arise.

Among many, the impact on already weakened democratic decision-making will increase. If young people cannot rely on learnt schema for critical thinking, their ability to discern truth from deception will cause even greater luring into epistemic bubbles, and a loss of common social concerns related to work, environment, health (Rose, 2026).

Furthermore, If learners use GenAI to request the entire result of what would have only been the last stage of the learning process they would have gone through, what does that mean for the value of the process of education, a process that forms a human from a very young age to young adulthood? Does that mean that only the final result matters, the grade, the credential? While workplaces are becoming more and more enshrined with GenAI, for many it seems like schools are the only places where humans can still cultivate thinking. If they are deprived of the right to earn the fruits of the laborious process of education, how can they become the autonomous, self-reliant person entitled to dignity qua their status of being a human (Pollmann, 2026; Rose, 2026)?

By the same token, what does that mean for the value of a genuine human creation, e.g. a piece of writing, versus one that has been entirely simulated to be such? With the ever more sophisticated GenAI models, detecting human-made versus GenAI-made content is more and more difficult discern, and raises the same pressing questions surrounding the status of truth, and human dignity.

5. Potential Remedies

This section surveys some of the potential remedies to the problems discussed above.

OECD Recommendations

One key finding the report highlights is a study showing student’s inclination to increase the use of GenAI with an augmented workload, for any given task. Concurrently, higher usage was correlated with procrastination, self-reported memory loss, and finally overall academic performance. Worth mentioning here is that an arbitrary delegation of cognitive work to GenAI increases student’s propensity for naïve overreliance on these systems. As a result, students struggle to assess trustworthiness and quality of GenAI’s outputs, and submit to this discrepancy, leading to lost opportunities to develop critical thinking, or related skills (Abbas et al., 2024; Zhai et al., 2024). After reviewing 106 experimental studies on human-AI collaboration, a meta-analysis found that task performance was worst in the collaborative setting compared to AI or a human solving a task independently (Vaccaro et al., 2024). This is yet another confirmation towards overreliance and the illusion of cognitive mastery. These findings demonstrate GenAI’s negative potential on cognitive development.

The OECD report proposes that learners and teachers adopt an effective-use approach to GenAI. Recommendations for effective uses list behavioral and procedural interventions for learners and teachers to prevent large scale negative impacts on societies and economies, such as an increase in illiteracy. While these recommendations align with latest scientific findings, it remains questionable if they provide actionable measures for students and teachers. In the face of it, learning to learn with GenAI sounds like a reasonable recommendation. However, which of these findings provides us with evidence to infer meanings about effective uses of GenAI for all learners at all stages? What constitutes use cases that respect the learner’s right to quality education in a way that encompasses GenAI without compromising on all indicated dimensions? In addition, the effective use-recommendation raises concerns regarding the student-centredness: if school curricula shall be designed to adopt to student needs, it seems contradictory to require the learner to adapt to how the tool functions. Shouldn’t rather the tool be designed in a way that adapts to the learner by default? Here, the CULT Briefing’s recommendations offer a range of approaches to adequately address the complexity and impact of socio-technical systems within education.

However, the OECD’s recommendations shift the conversation from a frame of positive potential by GenAI on to a frame of adaptation of learning to GenAI. By outlining these findings, the report challenges the idea of GenAI’s positive potential to promote high-quality education and underscores the need for purpuseful adaptation in the classroom but does not provide solid grounds for continued delivery of quality education.

CULT Recommendations

Unlike the OECD report, which puts its entire remedy belief into effective use pedagogies, the CULT Briefings’ suggestions cover a mix of pedagogical interventions, new regulatory framing, and cognitive impact assessment.

Pedagogical interventions

The educational interventions incorporate the ‘effective use’ recommendations from the 2026 OECD report[12] which are intended to create the conditions necessary to promote “cognitively safe and productive use of AI” in schools. Interestingly, the term ‘cognitively safe’ appears here without being further explained, and interestingly the OECD report makes no use of this term at all. [13].

Firstly, the efficient use of AI systems is reiterated in the CULT Briefing as a pedagogy-first recommendation, in line with the evidence-based proposal of the OECD report. AI systems should only be used if they serve clearly defined learning objectives, and not be “deployed as a general-purpose convenience”(Chounta, 2026, p. 6). Secondly, this is followed by the recommendation to use AI systems that are specifically designed for teaching purposes; therefore, general-purpose AI should not be used at all. Thirdly, metacognitive scaffolding (planning, reflection and evaluation) and a mastery orientation (augmenting knowledge through exploration, questioning and connection) are preferred approaches to effective use in all cases (Xu et al., 2025). However, the third point in particular raises the question of whether this refers to general-purpose AI or an AI system designed specifically for education. Fourthly, age and the associated learning objectives must be taken into account: “The cognitive and developmental differences between a seven-year-old and a sixteen-year-old demand genuinely differentiated governance frameworks, not a single age-neutral approach”. (Chounta, 2026, p. 7)

Regulatory frame

Legally binding frameworks such as the General Data Protection Regulation and the Council of Europe Framework Convention on AI and Human Rights, Democracy and the Rule of Law do not address the risks as they are set in the CULT briefing. The EU AI Act (Annex III) on the other hand classifies applications as high-risk which decide over individuals’ learning trajectories through conformity assessments, admissions, evaluation and student progress monitoring, in short the trajectory of someone’s biography (European Data Protection Supervisor, 2025). The CULT briefing suggests the revision of some areas that create ambiguity, e.g. the report asks if adaptive learning systems and intelligent tutoring systems (which generate personalised learning pathways and may implicitly steer students’ learning process) would constitute high-risk systems, given that they influence learning trajectories (Chounta, 2026, p. 7).

The report critically remarks, that “none of the existing regulatory instruments requires the systematic assessment of AI systems for their cognitive impact on child users before deployment in school settings. An AI system that increases short-term task performance while undermining long-term skill development would, under current frameworks, likely pass regulatory review” (Chounta, 2026, p. 8).

Consequently, the report conceptualizes a new cognitive outcome criterion for AI systems before their deployment in school settings. This represents a novelty in the European AI-related jurisdiction.

The briefing suggest the following measures for AI Systems:

  • mandating a cognitive impact assessment as an education-specific layer, within or alongside  the fundamental rights impact assessment (FRIA) of AI Act (Article 27) for high-risk AI systems, to be conducted before any AI system is deployed in compulsory school settings; it could require the assessment of cognitive load, metacognitive development, the learner’s developmental stage, and equity impact.
  • establishing minimum evidence standards for pedagogical efficacy claims as a condition for  AI systems used in education, e.g. providers must provide evidence for their claims, and standardisation labels could lever a signal to deploying institutions.
  • introducing a mandatory pedagogical evaluation (including cognitive dimensions of  learning) for AI systems acquired by public educational institutions before procurement as a condition of award, analogous to accessibility requirements embedded in public procurement for digital tools.

Cognitive Impact Assessment

The briefings’s analysis of the impacts of GenAI on the cognitive dimensions of education, elaborates on the consequences for cognitive development, including fundamental and  metacogntive skills, as well as autonomy, over-reliance and dependency. Thus, it elaborates on all aspects of the human right to quality in education, including “curricula and content that enables the acquisition of basic skills (literacy, numeracy, and skills for life)”.

Criticisms of the OECD Recommendations have raised concerns about scenarios in which learners’ rights to a quality education are violated when young students are required to learn how to learn with GenAI.Here is where the CULT Briefing emphasizes the importance to favor AI-enhanced learning technologies that are adapted to student’s needs over GenAI that cannot meet this need because of its general-purpose nature.

Interestingly, the report identifies the use of AI systems tailored to the learning context as the most important educational measure. It then recommends, in line with the OECD’s recommendations, that procedural and self-learning measures, such as adapting learning content to focus on learning how to learn, be given greater emphasis in the curriculum. Ultimately, however, neither this recommendation nor the one to use AI systems in an age-appropriate manner offers any new insight or a convincing solution to the problem of cognitive development in the face of GenAI.

The introduction of a new regulatory layer for cognitive impact assessment in the EU AI Act for high-risk applications is a novel regulatory approach that merits further consideration. The suggested remedies acknowledge the far-reaching consequences for young citizens when both binding and non-binding measures are distributed among several pedagogical stakeholders, caretakers as well as EdTech and GenAI manufacturers. While it is essential to incorporate an assessment of cognitive impacts in educational context in the AI Act, the suggested regulations should, in principle, be straightforward for AI-enhanced learning technology to comply with: these tools are already designed for educational purposes and built in an education-specific way. It will therefore be feasible for most providers to comply with the suggested standards and for deployers to perform the assessment.

Unexpectedly, however, there is no bifurcated approach to regulating GenAI versus AI-enhanced learning technologies. The briefing does not propose any formal or binding remedies to protect the right to a quality education despite GenAI. Instead, it creates the impression that the suggested amendments have been designed through the prism of AI tools already adapted to educational purposes in mind. For example, ‘establishing minimum evidence standards for pedagogical efficacy claims as a condition for AI systems used in education’ can only apply to producers who advertise their products as being designed for educational use.

Since both educational and regulatory interventions focus more on AI-enhanced  learning technologies than on GenAI, the question arises as to whether the measures that have been ‘left out’ suggest that GenAI is too general to be equipped with such guardrails?

This leads to the conclusion that the suggested mitigations do not suffice to protect children’s right to quality in education and that the problem of quality education in light of GenAI persits. The infringement has been partially addressed by the CULT Committee’s interventions.

6. Conclusion

Arbitrary cognitive offloading is detrimental to the human learning process, as much as a surplus of cognitive load is. An efficient learning process favors a balance between beneficial intrinsic and germane load, after the detrimental extrinsic load have been reduced. In the face of a learning problem to be solved, the beneficial load can be composed of low- and high-cognitive engagegement tasks, just as when D reads two literary texts with sustained attention, memorizes the content and then analyses and applies judgement. Beneficial load facilitates learning, and is highly dependent on the goal of the given learning task. When learners prompt GenAI to solve whole tasks for them, they offload cognition in an arbitrary manner. In that case, cognitive overreliance on GenAI reduces or even erases the possibility for a beneficial load, and the hampered learning process results in distorted memory consolidation, difficulties in retrieving facts, refering to and speaking about the supposedly learnt content, not even to speak of crossreferencing it. In the long term, cognitive development is impacted and basic cognitive skills might not form.

Learning is an active process in which learners learn how learning works, and that it is beneficial to offload irrelevant information to the external environment. However, the ease of use of GenAI can lead learners to reduce their metacognitive involvement. Metacognition constitutes a highly dynamic process that provides the learner with insight about their own thinking. Commonly described as thinking about thinking, it constitutes a part of cognitive load itself, and that often causes students to bypass this activity. Metacognitive laziness manifests in this bypassing and results in behaviors such as leaving generated output unverified, illusion of competence, limited capacities to develop expertise, and ultimately crippled critical thinking skills.

Against this backdrop, studies by Fan et al. (2025), Singh (2025) and Xu (2025) recommend that learning curricula should focus on teaching metacognitive skills. This would better equip learners to use GenAI efficiently in their learning process. For example, GenAI could be used to provide explanations rather than complete solutions, or suggest learning strategies and guide learners step by step through the learning process. These studies have shown that this approach improves learners’ critical thinking skills, deepens their understanding, and enhances their ability to learn independently.

The right to quality in education enshrines foundational learning objectives. Although the preamble to the Universal Declaration of Human Rights (UDHR) does not provide specific guidance on this matter, a UNICEF report from 2000 offers concrete normative guidance. Accordingly, quality in education is characterised by a focus on the learner and their mental health, an appropriate learning process and a curriculum that covers basic literacy, numeracy and life skills. The outcomes of quality in education contribute significantly to learners’ socialisation by shaping the development of knowledge, skills, and values. A closer examination of these qualitative learning objectives reveals their reference to Bloom’s revised taxonomy. Consequently, the normative understanding of the goals of quality education is underpinned by low- and high-order thinking skills, metacognition, and executive function. Preserving and promoting the process of achieving these goals is fundamental to the right to a good education, bridging the gap from cognitive development to human autonomy and dignity.

Recital 56 of the EU AI Act explicitly states that deploying AI systems is an expression of the right to a high-quality education, including digital and critical thinking skills. The OECD’s Digital Education Outlook report and the CULT Briefing largely concur in their analyses of the impact of GenAI on cognitive overreliance and metacognitive laziness. This is why the non-regulatory measures largely coincide and draw on similar evidence-based references. Recommendations include teaching metacognitive learning strategies, using GenAI for metacognitive scaffolding and taking a more differentiated approach according to age group, particularly for younger learners. However, particular emphasis should be placed on the CULT Briefing’s recommendation to use learning-specific AI-enhanced technologies (i.e. not GenAI) in education, as these could yield positive cognitive gains in the long term. Further regulatory recommendations are also based on this premise, including novel cognitive impact assessments, evidence standards for educational AI claims and mandatory pedagogical evaluations prior to school procurement. Unexpectedly, however, there is no bifurcated approach to regulating GenAI versus AI-enhanced learning technologies. The briefing does not propose binding or formal mitigations to protect the right to quality in education in the light of GenAI, and instead creates the impression that the suggested amendments have been designed  for education-specific AI. From here it is not implausible to infer that GenAI is too general to be equipped with such guardrails.

Thus, instead of arguing to attain the right to education qua the use and deployment of GenAI, evidence suggests to argue for the opposite: the right to education is not saved, instead it is infringed. This dichotomy should alarm policy makers, teachers, and social institutions to steer the narrative away from benefits to perils. The normal development of (meta-) cognitive skills is perturbated in a way that the goal of high-quality in education can no longer be achieved with this tool, but is instead jeopardised by it.

The EU AI Act’s recommendation to use GenAI in education to develop digital literacy and critical thinking skills is contradictory. Rather than supporting the right to a quality education, it is being undermined. Furthermore, the two mitigation strategies presented do not adequately minimise the risk of arbitrary cognitive offloading when using GenAI.

The CULT Briefing provides crucial insights when it argues that distinguishing between GenAI and AI-enhanced learning technologies is of fundamental importance. It proposes a regulatory approach that introduces a novel criterion to assess the ‘positive cognitive impact’ of an AI system. AI systems used in the classroom should therefore be restricted to those specifically designed as educational digital tools and marketed and sold as such. GenAI cannot be included in this category. It is precisely the generalness of GenAI that can lead to cognitive and societal harm. In contrast, AI-enhanced learning technologies are specific; for instance, learning management systems provide learners with tailored motivation based on their behaviour and progress.
The briefing highlights the impossibility of reconciling general-purpose AI systems with student-centred values, such as contextualised problem-setting and problem-solving that emphasise cognitive development, and knowledge transmission.

This is merely the tip of the iceberg, and it entails further unconsidered pitfalls. To address the conflictual proposal of the EU AI Act the more directly, means to adjust recital 56. This change should reflect the insight that only an adapted AI system can be in a position to not infringe on the right to quality education.

In other words, an amendment of recital 56 seems necessary.

The use of AI Systems is considered fruitful for high-quality digital education and training when all learners and teachers are enabled to acquire and share the necessary digital skills and competences, including media literacy, and critical thinking, to take an active part in the economy, society, and in democratic processes.(…). General purpose AI-systems do not fulfill this objective.”

The shift from the question of how GenAI might be handled within the context of high-quality education to the realisation that general AI is entirely unsuitable for the educational context gives rise to a new normative framework. So if GenAI were not used – and consequently the digital skills that can only be acquired through GenAI were not taught – would this constitute a violation of the right to education, or is a new understanding of digital skills required instead? Does “to prompt engineer” constitute a fundamental digital skills?

Declaring GenAI an AI system with limited use for specific domains where generalness is more beneficial than harmful could cause GenAI to lose its naturalised status. This would pave the way for education to make use of AI-enhanced learning technologies, through which digital skills could be developed, and the problem of arbitrary cognitive offloading could be solved.

It seems that the generalness of GenAI acts as a catalyst for arbitrary cognitive offloading, thereby rendering it unsuitable for educational use. Excluding it from school curricula, in turn, would prevent the teaching of digital skills associated with its use. This raises the question: which digital skills would be considered as such if GenAI was not a generally accessible AI system, but was instead replaced by a multitude of specialised AI-enhanced purpose-built technologies? Would the skills acquired in this way not be particularly valuable precisely if GenAI was no longer the standard AI system? What if a new norm for specialised, specific-purpose- applications was established? Whether it be resilient microservice software architectures, biodiversity approaches or diversified supply chain management: diverse student-centred AI learning systems promise to advance critical thinking and problem-solving whilst significantly reducing the risks of single-system dependency, a cognitive, infrastructural and political terms alike.

Bibliography

Carr, N. (2008). Is Google Making Us Stupid? Atlantic Monthly, 302(56–62 July/August). https://www.theatlantic.com/magazine/archive/2008/07/is-google-making-us-stupid/306868/

Chen, A., Xiang, M., Zhou, J., Jia, J., Shang, J., Li, X., Gašević, D., & Fan, Y. (2025). Unpacking help-seeking process through multimodal learning analytics: A comparative study of ChatGPT vs Human expert. Computers & Education, 226, 105198. https://doi.org/10.1016/j.compedu.2024.105198

Chounta, D. I.-A. (2026). Artificial Intelligence in Classrooms: Cognitive Dimensions. EPRS: European Parliamentary Research Service. https://coilink.org/20.500.12592/3cf1fxv

European Commission. Statistical Office of the European Union. (2022). Young people in Europe: A statistical summary 2022. Publications Office. https://doi.org/10.2785/684958

European Data Protection Supervisor. (2025). AI Act Regulation (EU) 2024/1689 – Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence and amending Regulations (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Artificial Intelligence Act) (Text with EEA relevance). Publications Office of the European Union. https://doi.org/10.2804/4225375

Fan, Y., Tang, L., Le, H., Shen, K., Tan, S., Zhao, Y., Shen, Y., Li, X., & Gašević, D. (2025). Beware of metacognitive laziness: Effects of generative artificial intelligence on learning motivation, processes, and performance. British Journal of Educational Technology, 56(2), 489–530. https://doi.org/10.1111/bjet.13544

Fisher, M., Goddu, M. K., & Keil, F. C. (2015). Searching for explanations: How the Internet inflates estimates of internal knowledge. Journal of Experimental Psychology: General, 144(3), 674–687. https://doi.org/10.1037/xge0000070

Gerlich, M. (2025). AI Tools in Society: Impacts on Cognitive Offloading and the Future of Critical Thinking (SSRN Scholarly Paper No. 5082524). Social Science Research Network. https://doi.org/10.2139/ssrn.5082524

Holmes, W. (2023). The Unintended Consequences of Artificial Intelligence and Education. Education International.

Kosmyna, N., Hauptmann, E., Yuan, Y. T., Situ, J., Liao, X.-H., Beresnitzky, A. V., Braunstein, I., & Maes, P. (2025). Your Brain on ChatGPT: Accumulation of Cognitive Debt when Using an AI Assistant for Essay Writing Task (arXiv:2506.08872). arXiv. https://doi.org/10.48550/arXiv.2506.08872

Lehmann, M., & Cornelius, P. B. (2025). AI Meets the Classroom: When Do Large Language Models Harm Learning? https://arxiv.org/abs/2409.09047

Lodge, J. M., & Loble, L. (2026). Artificial intelligence, cognitive offloading and implications for education (p. 1487662 Bytes). University of Technology Sydney. https://doi.org/10.71741/4PYXMBNJAQ.31302475

Marzano, R. J., & Kendall, J. S. (2007). The New Taxonomy of Educational Objectives (2nd edn). Corwin Press.

Monzon, N., & Hays, F. A. (2025). Leveraging Generative Artificial Intelligence to Improve Motivation and Retrieval in Higher Education Learners. JMIR Medical Education, 11(1), e59210. https://doi.org/10.2196/59210

OECD. (2023). Generative AI in the classroom: From hype to reality? EDU/EDPC 2023-11.

OECD. (2024). Reader’s guide: Do Adults Have the Skills They Need to Thrive in a Changing World? (OECD Skills Studies). OECD. https://www.oecd.org/en/publications/do-adults-have-the-skills-they-need-to-thrive-in-a-changing-world_b263dc5d-en/full-report/reader-s-guide_aee63130.html

OECD. (2026). OECD Digital Education Outlook 2026: Exploring Effective Uses of Generative AI in Education. OECD Digital Education Outlook, 2026. https://doi.org/10.1787/062a7394-en

Pollmann, A. (Host). (2026, May 31). KI in der Wissenschaft: Das Ende der Bildung? [Broadcast]. Deutschlandradio. https://www.deutschlandfunkkultur.de/ki-in-der-wissenschaft-das-ende-der-bildung-100.html

Risko, E. F., & Gilbert, S. J. (2016). Cognitive Offloading. Trends in Cognitive Sciences, 20(9), 676–688. https://doi.org/10.1016/j.tics.2016.07.002

Roebers, C. M. (2017). Executive function and metacognition: Towards a unifying framework of cognitive self-regulation. Developmental Review, 45, 31–51. https://doi.org/10.1016/j.dr.2017.04.001

Rose, J. (2026, April 30). The more young people use AI, the more they hate it. The Verge. https://www.theverge.com/ai-artificial-intelligence/920401/gen-z-ai

Singh, A., Taneja, K., Guan, Z., & Ghosh, A. (2025). Protecting Human Cognition in the Age of AI (arXiv:2502.12447). arXiv. https://doi.org/10.48550/arXiv.2502.12447

Stiftung Vodafone. (2025). AI in European Schools. Vodafone Foundation. https://www.vodafone-stiftung.de/wp-content/uploads/2025/01/AI_in_European_schools_A_European_report_comparing_seven_countries_IPSOS_Vodafone_Foundation_EN.pdf

Sweller, J., van Merrienboer, J. J. G., & Paas, F. G. W. C. (1998). Cognitive Architecture and Instructional Design. Educational Psychology Review, 10(3), 251–296. https://doi.org/10.1023/A:1022193728205

Thevenot, C., Krenger, M., & Poletti, C. (2025). Finger counting as a key tool for the development of children’s numerical skills. Journal of Experimental Child Psychology, 252, 106156. https://doi.org/10.1016/j.jecp.2024.106156

UNICEF. (2000). Defining quality in education: A paper presented by UNICEF at a meeting of the International Working Group on Education. UNICEF. Education Section. Programme Division. https://unesdoc.unesco.org/ark:/48223/pf0000199104

United Nations. (1948). Universal declaration of human rights. United Nations. https://www.un.org/en/about-us/universal-declaration-of-human-rights

Ward, A. F. (2013). Supernormal: How the Internet Is Changing Our Memories and Our Minds. Psychological Inquiry, 24(4), 341–348. https://doi.org/10.1080/1047840X.2013.850148

Ward, D. M. W., Adrian F. (2013, December 1). The Internet Has Become the External Hard Drive for Our Memories. Scientific American. https://www.scientificamerican.com/article/the-internet-has-become-the-external-hard-drive-for-our-memories/

Willingham, D. T. (2019). How to Teach Critical Thinking. https://education.nsw.gov.au/content/dam/main-education/teaching-and-learning/education-for-a-changing-world/media/documents/How-to-teach-critical-thinking-Willingham.pdf

Winne, P. H. (2021). Cognition, Metacognition, and Self-Regulated Learning. In K. Hytten (Ed.), Oxford Research Encyclopedia of Education (p. 0). Oxford University Press. https://doi.org/10.1093/acrefore/9780190264093.013.1528

Xu, X., Qiao, L., Cheng, N., Liu, H., & Zhao, W. (2025). Enhancing self-regulated learning and learning experience in generative AI environments: The critical role of metacognitive support. British Journal of Educational Technology, 56(5), 1842–1863. https://doi.org/10.1111/bjet.13599

Yan, L., Greiff, S., Teuber, Z., & Gašević, D. (2024). Promises and challenges of generative artificial intelligence for human learning. Nature Human Behaviour, 8(10), 1839–1850. https://doi.org/10.1038/s41562-024-02004-5

Zainuddin, Z., Churiyah, M., Xiaoyu, W., & Halili, S. H. (2026). From AI literacy to cognitive AI safety: Rethinking safe AI use in education.

Zhai, C., Wibowo, S., & Li, L. D. (2024). The effects of over-reliance on AI dialogue systems on students’ cognitive abilities: A systematic review. Smart Learning Environments, 11(1), 28. https://doi.org/10.1186/s40561-024-00316-7


[1] According to the newest statistics cited in the report “Young people in Europe 2022”, 73 million young people between 15-29 live in the EU in 2021 (European Commission. Statistical Office of the European Union., 2022, p. 5). No later census data is available. 

[2] Explicitely refering to two educational spaces: 1) the classroom or the university as on premise-space where teachers and peers are present, and 2) the educational non-formal setting for homework or project work, alone or with peers, parents etc.

[3] Thre are good reasons why this is the case, e.g. sovereignty of nations; geographical, social and cultural differences. Appointed by the United Nations, the transnational agency UNESCO is responsible to establish conditions that supports the national provision of quality education.

[4] The term student-centred here is used to expressed the opposite of teacher-centred that illustrates a shift away from a unidimensional transfer of knowledge from the teacher to the student towards a constructivist approach.

[5] Akin to metacognitive skill that enable learners to monitor and plan their task engagement, executive functions, as their name suggests, allow them to reguate their behavior and thinking, e.g. maintaining focus, continuing the work even when motivation declines, to control frustration or impulsive behaviors. For further details about the effect of executive functions and metacognition for cognitive development, please see Roebers (2017).

[6] This categorisation reflects active stakeholders in the education sector and was also made, for example, in the report “The Unintended Consequences of Artificial Intelligence and Education” by Education International (Holmes, 2023).

[7] Categories of low-, high- and meta-cognitive tasking have been chosen according to Bloom’s Revised Taxonomy.

[8] Task with highest arbitrary offloading potential. The more kinds of cognitive involvement are required, the higher the potential for arbitrariness.

[9] 63.8% of all young people use AI Tools and thereof 32.7% use AI tools for formal education, that means approximately 20% of young people in the EU use AI tools for formal education.

[10] The Socratic Method is an argumentative dialogue informing a structured way to learn and practice critical thinking. Among others, Monzon and Hays (2025) propose using GenAI to create desirable difficulties, instead of bypassing effort.

[11] Exact numbers of functional illiteracy are particularly difficult to estimate. An overall estimate of illiteracy worldwide claims that there are around 750 Million people who cannot read and write sufficiently but it is unclear if this metric accounts for functionally illiterate populations.

[12] Furthermore, the European Commission’s Ethical Guidelines on the Use of AI and Data in Teaching and Learning for Educators (2022) are cited as the source for the following recommendations.

[13] AI Safety includes research, governance and application of AI Systems that prevent harmful use or mitigates harmful effects of AI Systems. However, the phrasing “cognitively safe AI use” has no established meaning at the time of the Briefing’s publication date. One study from May 2026 by Zainuddin et al. (2026) tries to introduce the concept of “cognitively safe AI use, but this is only a first attempt.

Question Zero: Why Responsible AI Begins Before AI Adoption

Abstract

The Question Zero (Q0) Self-Assessment Tool for Responsible AI, developed by the AI Policy Lab at Umeå University, supports organisations in posing foundational questions before adopting AI. Grounded in the concept of Question Zero: “Under what conditions should an AI system be adopted, if at all?”, the tool offers support for cross-functional team discussions covering motivation, stakeholder mapping, system type, adoption process and infrastructure. It is designed for public institutions, civil society organisations, policymakers and other actors looking for responsible AI decision-making approaches. With the help of the tool, we argue that purposeful, problem-led assessment must precede any procurement and deployment decisions.

Keywords: Question Zero, Q0, Responsible AI, AI Self-Assessment, AI Governance, EU AI First Strategy, AI Policy, AI Adoption Motivation, Stakeholder Mapping, Explainability, AI Policy Lab, Umeå University

 

The Problem with Starting from the Answer

Public and private organisations face growing pressure to adopt artificial intelligence (AI). Governments across the EU and beyond are committing vast public resources to AI acceleration. Organisations, large and small, feel urged to integrate AI into their operations for fear of being perceived as falling behind. The European Commission’s Apply AI Strategy (European Commission, 2025) has formalised this pressure into policy direction, framing AI as the default first response to organisational and societal challenges. The same trend can be seen at a national level, as well as across sectors and organisations.

However, this urgency leads to a fundamental disruption of sound decision-making. Rather than beginning with a problem and asking what solutions are available, an “AI First” logic begins with the technology and asks where it can be applied. As we have argued elsewhere, the result is that the question of whether AI is appropriate in a given context is hardly ever properly posed (Dignum et al., 2025). The consequences of skipping this step are not abstract.

The Dutch childcare benefit scandal, in which an algorithmic system wrongly flagged tens of thousands of parents as committing fraud, disproportionately affecting ethnic minority and low-income families (Amnesty International, 2021), is one of the examples of what might happen when AI is deployed without adequate prior deliberation about necessity, appropriateness and who might be harmed. Similarly, the UK’s A-level grading algorithm reinforced existing inequalities, putting students from smaller schools and lower socio-economic backgrounds at a disadvantage by relying on historical data to determine results (Kolkman, 2020). In the United States, ProPublica’s investigation into the COMPAS recidivism algorithm revealed significant racial biases, with Black defendants nearly twice as likely as their white counterparts to be wrongly flagged as high risk (Larson et al., 2016). Australia’s Robodebt scandal exposed the dangers of deploying flawed automated decision-making systems at scale, showing how vulnerable citizens can be unjustly penalised without adequate oversight, transparency and safeguards (Royal Commission into the Robodebt Scheme, 2023).

Together, these cases illustrate how without robust AI governance, such systems can deepen discrimination against marginalised groups and undermine public trust in key institutions. While these cases differ in context and technical form, they share an important feature: technology was introduced without adequately addressing Question Zero (Q0), meaning the prior scrutiny of necessity, appropriateness, accountability and social impact was weak or absent.

The question Q0 (“Under what conditions should an AI system be adopted, if at all?”) we pose, is a call for rigour and strategic foresight. It asks organisations to define the problem before considering technological solutions, to assess alternatives, to map who benefits and who bears risks and to ensure that any decision on how to proceed is genuinely justified. Only sometimes, not always, is AI the right answer. Thus, the purpose of the Q0 Self-Assessment Tool is to facilitate a structured pause to allow for better decision making about responsible AI.

What is the Q0 Self-Assessment Tool?

The Q0 Self-Assessment Tool is a free assessment tool designed not as a compliance checklist or technical audit, but as a structured self-reflection instrument for cross-functional groups within organisations. Its primary goal is to support decision-makers in working through the foundational questions of responsible AI governance before procurement or deployment decisions are finalised (Titareva et al., 2026).

The tool is organised into five sections: Why, Who, What, How and Where, each targeting a distinct dimension of responsible AI governance. These sections are not a linear sequence; organisations are encouraged to move between them iteratively, as answers in one area frequently raise additional questions in others.

Section A: Why (Motivation) asks organisations’ decision-makers and employees to articulate the problem they are attempting to solve, the reasons they are considering AI, the alternatives they have considered (including human and non-AI technical solutions) and why and whether AI is the best solution for the existing problem. This is the core of Q0: encouraging explicit justification before any further investment of time or resources into AI procurement and deployment.

Section B: Who (Stakeholder Mapping) asks organisations to identify the stakeholders who may be directly or indirectly affected by the adoption of an AI system. This includes colleagues, employees, customers, and members of marginalised or underrepresented groups who are often overlooked in initial assessments (as well as at later stages). The section also prompts organisations to consider who currently performs the tasks the AI system is intended to undertake, who is likely to benefit, who could be harmed and whether meaningful opt-out options exist. By mapping the full stakeholder landscape, organisations can assess how different groups may experience the adoption of AI and ensure that no group is disproportionately burdened, excluded or deprived of meaningful choice.

Section C: What (Type of AI System) asks organisations to define and describe the specific AI system, method or tool under consideration (e.g., predictive, generative, categorising or hybrid), as well as whether it will be procured off the shelf, developed in-house or built for specific needs of an organisation. This section encourages organisations to ensure that the chosen system fits the problem they want to solve and that the system’s complexity, development approach and flexibility match its intended use.

Section D: How (Adoption Process) asks organisations to consider how the AI system would be deployed, how existing workflows would change, how outputs and performance would be monitored, how data security would be ensured and how affected users would be able to understand the basis of the system’s conclusions. It also prompts organisations to define how complaints and errors would be handled and who would be accountable for oversight throughout the system’s lifecycle. This section emphasises that accountability must be embedded from the beginning rather than added retrospectively, and that explainability should be treated as an ongoing organisational practice rather than merely a compliance requirement.

Section E: Where (Infrastructure and Control) asks organisations to consider where the AI system will operate, where data will be stored, where the provider is based and where the training data originates. These questions address issues of digital sovereignty, data jurisdiction and the extent to which organisations can maintain meaningful control over the AI systems they adopt. This section emphasises that effective governance and accountability depend on understanding where the system operates, who controls it and how data is collected, stored and managed. It therefore encourages organisations to ensure transparency regarding infrastructure, models and training data in order to meet legal, security and ethical responsibilities and requirements.

Each section concludes with a confidence rating on a five-point scale, allowing organisations to reflect on how confidently they can answer the questions within that dimension. Rather than measuring compliance or assigning a level of AI maturity, the ratings are intended to stimulate discussion, identify areas of uncertainty and highlight where further information, stakeholder engagement or organisational preparation may be needed. The aggregated scores are visualised in a radar diagram, providing an overview of confidence across all five dimensions. This visualisation enables organisations to compare results across teams, over time or between different AI use cases, thus supporting internal learning, reflection and continuous improvement. The ratings are not intended for external accreditation, benchmarking or audit, but as a self-assessment tool that helps organisations recognise strengths, identify gaps and guide responsible AI governance.

A final section of concluding questions invites decision-makers and employees of organisations to reflect on the insights gained throughout the self-assessment process by evaluating any pilot experiences, considering the expected benefits of the proposed AI system alongside its possible risks and harms and identifying appropriate next steps. Rather than prescribing decisions or producing a pass-fail outcome, the tool is designed to support informed, context-sensitive judgement, recognising that responsible AI adoption depends on an organisation’s objectives, values, legal obligations and operational context. As a decision-support instrument, the Q0 tool helps structure deliberation, surface assumptions and strengthen transparency, but responsibility for interpreting the findings and deciding how to act upon them remains with the organisation and its employees as part of its internal AI governance.

Figure 1.Question Zero Conceptual Operationalisation
                          Figure 1. Question Zero Conceptual Operationalisation

Positioning the Q0 Self-Assessment Tool Among Existing AI Assessment Frameworks

The Q0 self-assessment tool builds on a growing ecosystem of AI governance and assessment instruments. At the same time, it addresses a different stage of the AI lifecycle. Existing frameworks such as the European Commission’s Assessment List for Trustworthy Artificial Intelligence (ALTAI), UNESCO’s Ethical Impact Assessment (EIA), the UNESCO Readiness Assessment Methodology (RAM), the UNDP Human Rights Impact Assessment Toolkit and similar international and national frameworks support responsible AI through structured reflection on legal, ethical, technical and organisational issues. Although these tools have different purposes and target different audiences, they share a common goal: They help governments or organisations identify, manage and govern the risks associated with AI systems and promote transparency, accountability, human rights and public trust.

The Q0 tool shares many of these principles. Like ALTAI, it addresses human oversight, transparency, accountability, fairness and data governance. Similar to UNESCO’s Ethical Impact Assessment, it encourages multidisciplinary participation and considers the wider social impacts of AI alongside technical issues. The UNDP Human Rights Impact Assessment Toolkit also emphasises the importance of identifying rights holders and affected communities. UNESCO’s Readiness Assessment Methodology takes a broader perspective by examining institutional capacity, governance, infrastructure and organisational readiness. Q0 tool follows a similar approach by treating responsible AI governance as an institutional challenge rather than only a technical one.

The main difference is not the questions themselves, but when they are asked. The assessment tools mentioned above often assume that an organisation has already decided to develop, procure or deploy an AI system. Their purpose is to assess whether that system is trustworthy, legally compliant or ethically governed. Some frameworks do include questions about necessity or proportionality during the pre-adoption stage. However, these questions usually form one part of a broader impact or compliance assessment when the AI system’s adoption project has already been initiated.

The Q0 tool starts one step earlier. It asks whether AI should be adopted at all, and if yes, under what conditions. This is the central purpose of Question Zero. The tool encourages organisations to define the problem first, consider non-AI alternatives and examine governance, stakeholder impacts and organisational readiness before resources are committed or procurement begins. It also places greater emphasis on collective deliberation and participation. It recommends cross-functional participation, the inclusion of affected stakeholders where possible, iterative use of the tool, documentation of disagreements and integration with existing governance processes. The goal is not a single assessment but an ongoing process of organisational learning and reflection.

A second difference is the intended use of the tool. Many existing assessment instruments support regulatory compliance, formal impact assessment or risk management. The Q0 tool has a different purpose. It is a self-assessment instrument. It does not generate a compliance score or certify that an AI system or case is trustworthy. The confidence ratings are intended to stimulate discussion, reveal uncertainty and identify gaps in knowledge. The radar diagram allows organisations to compare discussions across teams, projects or points in time. This helps identify differences in understanding and areas that require further attention.

The Q0 Self-Assessment Tool is intended to complement, not replace, existing governance frameworks. Organisations that decide AI is appropriate can then carry out more detailed assessments, such as ethical impact assessments, human rights impact assessments, data protection impact assessments or assessments required under the national or regional legislation (e.g., the EU AI Act and others). Q0 tool therefore supports an earlier stage of decision-making. It aims to improve the quality of the decision that determines whether these later assessments will be necessary.

The Q0 tool also has limitations. It is a self-assessment tool. Its value depends on organisations’ employees engaging with the questions and including a diversity of perspectives. It cannot remove organisational bias or guarantee that all relevant stakeholders are represented. It cannot ensure that recommendations will be implemented. It is also not a substitute for legal, technical or human rights assessments where these are required. Finally, no questionnaire can capture the full complexity of every organisational context. The Q0 tool should therefore be seen as a governance aid rather than a decision-making mechanism. It supports better judgement, but responsibility for the final decision always remains with the organisations’ leadership and employees.

Example of the Q0 Self-Assessment Tool in Practice

A university department is considering introducing a generative AI assistant to answer students’ questions about courses, deadlines and administrative procedures. Instead of immediately assuming that an AI chatbot is the right solution, the department first uses the Q0 Self-Assessment Tool. Addressing the questions in the section Why”, university employees from diverse departments and functional areas define the problem, for example, long response times during busy periods. They also consider other options, such as improving online guidance, extending office hours or employing more student assistants. AI is treated as one possible solution, not the starting point.

The remaining Q0 tool’s sections help to broaden the discussion. Under the section Who, the participants identify as many categories of stakeholders as possible, who may be affected, including students, lecturers, administrative staff, IT services and others. It also considers students with disabilities, international students and others who may be disadvantaged by digital-only support. Under What, participants compare different AI systems and ask whether a simpler non-AI or rule-based solution could meet the need. Under the sections Howand Where, they discuss governance, accountability, transparency, data security, hosting arrangements and whether the university keeps appropriate control over its data.

At the end of the process, the group gives a confidence rating for each section. The radar diagram shows where participants feel confident and where more work is needed. In this example, confidence could be high for the motivation behind the project but lower for infrastructure, data governance and long-term accountability. These results do not determine whether AI should be adopted. Instead, they help the department identify where further discussion, expert advice or pilot testing is needed before making a final decision.

Conclusion

AI is not inevitable. Its adoption is not inherently beneficial. Responsible AI governance begins before procurement, development or deployment. It begins by asking whether AI is the best solution for your current problem, if so, under what conditions. This is the purpose of Question Zero.

The Q0 Self-Assessment Tool provides organisations with a practical way to support this early reflection. It does not replace existing governance frameworks or assessments. Instead, it complements them by addressing a stage of decision-making that is often overlooked. It focuses on motivations, stakeholders, system choice, adoption, infrastructure and organisational control.

The value of the Q0 Self-Assessment Tool lies not in providing answers, but in improving the quality of the questions organisations ask before adopting AI systems. It creates space for discussion, makes assumptions explicit and helps identify issues that require further attention. In the end, the responsibility for deciding whether to adopt an AI system remains with the organisations’ employees. The Q0 process helps ensure that this decision is more informed, transparent and accountable.

Access the Tool

The Q0 Self-Assessment Tool (Version 3, April 2026) is available: HERE 

 

References

Ala-Pietilä, P., Bonnet, Y., Bergmann, U., Bielikova, M., Bonefeld-Dahl, C., Bauer, W., … & Van Wynsberghe, A. (2020). The assessment list for trustworthy artificial intelligence (ALTAI). European Commission.

Amnesty International (2021, October 2025). Xenophobic machines: Discrimination through unregulated use of algorithms in the Dutch childcare benefits scandalhttps://www.amnesty.org/en/documents/eur35/4686/2021/en/

Dignum, V., Carli, R., Ericson, P., Titareva, T., & Tucker, J. (2025). ‘AI First’ to ‘Purpose First’: Rethinking Europe’s AI Strategy. AI Policy Exchange Forum (AIPEX)https://doi.org/10.63439/LPOU6506

Dignum, V., Carli, R., Dahlgren Lindström, A., Ericson, P., Titareva, T., & Tucker, J. (2026, June). Question Zero for Explainability and Vice Versa: The Case of the EU’s AI First Strategy. In International Conference on Human-Computer Interaction (pp. 17-31). Cham: Springer Nature Switzerland. https://doi.org/10.1007/978-3-032-29456-2_2 

European Commission (2026, March 27). Apply AI Strategy. Shaping Europe’s Digital Future. https://digital-strategy.ec.europa.eu/en/policies/apply-ai

Kolkman, D. (2020, August 26). “f**k the algorithm?”: What the world can learn from the UK’s A-level grading fiasco. The London School of Economics and Political Science (LSE) Impact.  https://blogs.lse.ac.uk/impactofsocialsciences/2020/08/26/fk-the-algorithm-what-the-world-can-learn-from-the-uks-a-level-grading-fiasco/#:~:text=%E2%80%9CF**k%20the%20algorithm%E2%80%9D?:%20What%20the%20world%20can,address%20through%20making%20their%20algorithms%20more%20explainable.

Larson, J., Mattu, S., Kirchner, L., & Angwin, J. (2016, May 23). How we analyzed the COMPAS recidivism algorithm. ProPublica. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm

Royal Commission into the Robodebt Scheme (2023, July 7). The Report of the Royal Commission into the Robodebt Scheme to the Governor-General, His Excellency, General the Honourable David Hurley AC DSC (Retd). https://robodebt.royalcommission.gov.au/publications/report

Titareva, T., Carli, R., Dahlgren Lindström, A., Dignum, V., Fabris, B., Ericson, P., & Tucker, J. (2026). Q0 Self-Assessment Tool with Guiding Questions for Responsible AI Approach by AI Policy Lab (AIPL). Test version 3 from April 2026. Umeå University.

UNESCO. (2023). Ethical Impact Assessment: A tool of the Recommendation on the Ethics of Artificial Intelligence. https://doi.org/10.54678/YTSA7796

UNESCO. (2023). Readiness Assessment Methodology: a Tool of the Recommendation on the Ethics of Artificial Intelligence. https://doi.org/10.54678/YHAA4429

United Nations Development Programme. (2025). Human Rights Impact of AI Assessment Tool.  https://www.undp.org/eurasia/publications/human-rights-impact-ai-assessment-toolkit

Is Simulated Evidence Still Evidence? A Warrant-Based Policy for Governing Synthetic Data

Executive Summary

Generative AI can produce data that looks like a record of the world without being one: a scan with no patient behind it, a measurement of no event. Such synthetic data is often benign and valuable, but as its fidelity rises, so does the ease of passing a fabrication off as a genuine trace of reality. Existing regulatory rules do not directly address it. They govern synthetic data as a matter of privacy, fairness, and disclosure, and they attend to its accuracy and to the integrity of the AI models trained on it, but they do not frame the challenge in terms of warrant, which is to say, they do not approach synthetic data from the perspective of its representational value, or of whether it can be relied on as evidence of something real. In response to this gap, this white paper proposes a dedicated, government-sponsored system of detection and provenance, coupled with an ambitious moonshot, a research program to measure representational value itself.

1. Policy’s Metaphysical Reckoning

Data is supposed to be a promise. Because the instrument that produced it had to be pointed at something, the datum testified that the something had actually been there. Generative AI has disrupted if not cancelled that promise, since it can produce the scan without the object, the measurement without the event, the record without anything to record. Philosophy offers a useful pair of terms here. A simulation stands in for something real and can be checked against it, while a simulacrum is a copy that has come loose from any original, a likeness with nothing behind it [1, 2]. Synthetic data – artificial data that purports to reproduce the features of real data [3] – can be either.

The issue is that not only is our capacity to create synthetic data rapidly increasing, but also our ability to make such data more plausible. I am purposefully invoking plausibility here instead of fidelity because the two concepts are subtly different from each other, and the impulse is to assume that simulacrality is a function of the latter. Fidelity is a relation between a synthetic artifact and the real data it was built to imitate, a measure of how well the copy reproduces the original’s features. Plausibility, meanwhile, is a relation between the artifact and whoever receives it, a measure of how readily it will be believed. These two relations naturally go together, and indeed, the pursuit of better and better synthetic data presumes that they do. However, they can also be separated, and the danger of simulacrality arises when they do. An artifact answerable to nothing real can still be plausible, because what makes it convincing is not a tie to any original but its fit with what an interpreter expects to see. This means that as our capacity for both generation and plausibility climbs, so too does the ease with which we can create simulacra that pass themselves off as faithful simulations, or even as the real thing [2].

The threat posed here is to the very notion of evidence, which is not a property a thing carries on its own but a standing it is granted. A datum (a fingerprint, a scan, a sensor reading) counts as evidence only when some interpretive stakeholder (a court, a hospital, a scientific community) accepts it as genuinely representing the part of reality that system is charged with judging. That act of acceptance is warrant, and it rests on a wager about representational value, the artifact’s actual tie to the reality it depicts. Because warrant is conferred rather than contained, it can be granted to something that does not deserve it, and a sufficiently plausible simulacrum is built precisely to be accepted. The danger, then, is not only that a machine can fake a datum, but that our institutions can be brought to vouch for the fake as though it were real.

Policy must reckon with metaphysics, although it need not resolve the unresolvable to make meaningful headway. The right reframing can both put existing tools to work and prompt the development of new ones. The way through is to notice that two different questions are actually in play, and that they differ in kind. Whether a datum is synthetic is a matter of how it came into being, a fact that holds regardless of who encounters it. Whether it counts as evidence is a matter of how it is interpreted and used, a fact that holds only relative to the interpretive stakeholder that relies on it. The underlying distinction here is also from philosophy, namely John Searle’s distinction between facts that are observer-independent and facts that are observer-dependent [4].

For policy, determining the first fact (is it synthetic?) has a single answer everywhere, while determining the second (is it evidence?) has a different answer in every sector, and necessarily so. A workable policy would therefore govern the two facts differently. For the first it needs a universal layer, a way of establishing whether an artifact is synthetic and where it came from, through detection and provenance, administered by a measurement-and-standards body that sets methods and keeps records rather than a regulator that rules on anyone’s claims. For the second it needs restraint, leaving interpretive stakeholders – academic domains, economic sectors, civil society – free to decide what its data warrants.

The metaphysics will still linger, though. Detection and provenance only make it possible for stakeholders to grant or withhold warrant responsibly. There is a path for policy to attempt to wrangle whether an artifact has genuine representational value, any real tie to what it depicts, and that is to undertake a moonshot research project. And it would be a moonshot because ultimately it is probably not achievable, but failure could nevertheless yield insights we cannot currently anticipate.

This white paper will proceed as follows. The next section develops the concepts the argument rests on, evidence and warrant, simulation and simulacrum, and makes the danger concrete through worked examples [§2]. I then survey the rules that currently govern synthetic data and show where they fall short [§3]. Following this, I set out the measurement-and-standards body and the service through which it would work [§4]. I close with the moonshot – the attempted measurement of representational value in-itself – which such a body could lead but only with the assistance of the world’s academic and corporate institutions [§5].

2. Synthetic Data and Simulacra

This section is concerned with grounding the key metaphysical considerations from above. I will first make precise the pairs of ideas that my argument rests on, namely, warrant and evidence, simulation and simulacrum, and fidelity and plausibility. I will then show that what matters in the threat of simulacral synthetic data is how plausibility and fidelity can disconnect.

2.1. Evidence as warranted data

Classically, knowledge was defined as justified true belief, until Edmund Gettier showed that one can hold a belief that is justified and true, but true only by luck, thereby undermining whether it is actually knowledge [5]. Alvin Plantinga’s diagnosis was that justification itself – understood as doing one’s epistemic duty, weighing the evidence responsibly – was the wrong thing to add to true belief, since a conscientious believer whose cognitive faculties are malfunctioning can fulfill every duty and still fail to know. What knowledge additionally requires is warrant, the property a true belief has when it is produced by properly functioning faculties working as they are meant to [6, 7]. Warrant, in short, is what turns a true belief into knowledge rather than a lucky guess.

A belief does not carry that property on its own. It is warranted only relative to some system of standards entitled to confer it, and those standards differ from one interpretive stakeholder to the next [31], the institutions and communities each charged with deciding what may count as an adequate representation of some part of reality.1 A court decides what may stand as proof that a defendant did something; a scientific community decides what may count as a finding about the natural world; a hospital decides what may be read as a sign of injury or disease; an intelligence service decides what may be treated as a fact about a threat; a public administration decides what may be accepted as a record of obligation or entitlement. Each grants standing to some of the data that reaches it and withholds standing from the rest, by criteria of its own.2

Evidence, then, names a status rather than a kind of thing. To call a datum evidence is to say that some interpretive stakeholder has granted it standing, taking it to genuinely represent the specific aspect of reality the system is charged with judging. The ascription is not, in principle, arbitrary, which is why the same datum can be evidence in one system and merely data, or nothing at all, in another. A fingerprint on a windowsill is not evidence until a court uses it to warrant the claim that a defendant was at the scene, otherwise it is simply a dirty smudge. Similarly, a timestamped GPS record is evidence that a suspect was near the scene when a court admits it as such. Yet, that very same record, given to an epidemiologist studying movement patterns, is merely one data point among millions, warranting nothing about any specific individual. In this case, the datum does not change; what changes is whether a system has meaningfully staked its judgment on it.

2.2. Simulacra in the pipeline


Figure 1: Radiograph of author’s fractured distal phalanx obtained from a clinical imaging procedure.


Figure 2: Synthetic radiograph in which the fractured distal phalanx has been replaced with a finger bone.


Figure 3: Synthetic radiograph containing a duplicated instance of the fracture elsewhere in the foot.

When an interpretive stakeholder grants warrant, it is basically making a wager that the given datum has representational value, a real tie to the part of the world it purports to depict. A simulacrum is a wager lost. The interpretive stakeholder unwittingly grants it warrant – it confers the status of evidence – because it looks “right.”

Here I must return to philosophical distinctions raised earlier between simulation and simulacrum on the one hand, and fidelity and plausibility on the other. A simulation has representational value because it stands in for something real and can in principle be checked against it; a simulacrum has none, its likeness answering to nothing, however convincing it looks. What separates them is not how faithfully each reproduces its source, since both can be high in fidelity, but whether anything real stands behind the likeness at all. This is easier to see once fidelity and plausibility are conceptually pulled apart. Fidelity measures the artifact against the data it was built from; plausibility measures it against the expectations of whoever receives it. The two measures are independent, and that independence is the whole problem.

What can be perplexing is that every simulacrum is, at heart, a simulation, or at least a purported one, and plausibility seems to depend on a high degree of fidelity. Imagine a radiograph of a foot. What appears in your mind’s eye is a simulation: a grey-tone image, the bones in stark white against a dimmer background, the broad cuneiforms and metatarsals at the core, the smaller phalanges of the toes, perhaps a corner of metadata noting a patient and a timestamp. The image is plausible precisely because you perceive in it enough fidelity to actual radiographs you have seen. Now add a sixth toe, anywhere you like. Nothing yet has gone wrong: a deliberately fanciful image, offered as fanciful, is an honest simulation of a foot that happens not to exist, no more deceptive than a painting.

Now suppose that fanciful image instead exists as a digital file on your computer, and then you post it online. That fantastical radiograph is not yet a simulacrum, but now you have published it online, shared it with friends who have shared it their friends. What turns it simulacral is that somewhere along the way, someone stakes a claim about reality on its basis, e.g., “This is a human foot with six toes” (as opposed to, “This is an imagining of a foot with six toes”). The moment that six-toed image is put forward as the radiograph of a real patient somewhere, it asserts a tie to a body it does not have, and its undiminished fidelity is exactly what enables the false claim to persuade. Simulacrality, then, is not a property of how an artifact was made but of the gap between what it claims to represent and what it actually answers to.

So, fidelity and plausibility can part, and the gap is where the danger lives, since an artifact tied to nothing real can still be plausible if it fits what an interpreter expects to see. That is to say, warrant is kept or lost on plausibility, not fidelity, and a simulacrum needs only the former.

Simulacrality also admits of degrees, meaning that not all simulacra are equally dangerous. During the preparation of this paper, my distal phalanx (the top bone of my big toe) suffered a fracture – hence the inspiration for the fantastical radiograph. Now, the authentic radiograph of my injury counted as medical evidence to the orthopedic specialists tending to it because my real body met a real X-ray machine at a real moment in a real hospital ward. The image inherited its authority from that causal tie to the world, not from its visual content alone. Or, more precisely, the visual content was also supposed to supply the credentials for rightly believing it to be tied to the world. The point is, that tie was its warrant: the fracture in the image corresponded to an actual fracture in my toe [Fig. 1].

Now consider two synthetic radiographs derived from that original: one in which my toe bone has been replaced with a finger bone [Fig. 2], and one that depicts a “Fomenko toe” [Fig. 3]. These demonstrate differing degrees of simulacrality. The first reads as “an injured bone” to an untrained eye or a coarse audit, but it answers to no real anatomy (a finger bone does not belong in a foot), and so it can be caught. The second is subtler. The genuine fracture has been copied and a duplicate inserted elsewhere in the foot. Because the duplicate is a copy of real pathology, it carries the exact statistical signature of a genuine injury, and it will pass the audits while corresponding to an injury that never happened. Its fidelity is real but pointed at the wrong thing, faithful to the source pathology and answerable to no actual patient, and that is exactly what makes it plausible. In both cases the image counterfeits warrant, keeping the look of a datum tied to the world while the tie itself is gone.

The Fomenko toe also shows in miniature a general asymmetry: what a simulacrum must do to succeed depends on whom – or what – it must fool. To deceive a person, it must be sensorily plausible, i.e., the anatomy on display and the metadata wrapped around the file must look as a human viewer would expect, if more demandingly for a radiologist than for a layperson. To deceive a machine, none of that is required; the artifact need only fall within the range of variation the system treats as normal, close enough on the features the system attends to not to be flagged as an outlier [8]. The danger therefore sharpens as the audience shifts from human to machine, because deceiving a machine collapses the distance between fidelity and plausibility: statistical conformity is both the easier thing to manufacture and, to a system with no referent of its own, sufficient on its own. Plausibility is bought without any tie to the real, which is the simulacrum’s whole advantage.3

Notes

  1. Here I should clarify that I am borrowing warrant from Plantinga for its essential idea rather than his specific understanding of it. For Plantinga, warrant is externalist and individual: a true belief is warranted when the believer’s cognitive faculties are functioning properly. The way I use warrant here is institutional and conferred: an interpretive stakeholder grants a datum evidential standing by an act of judgment. What carries over from Plantinga to me is only the underlying move, namely, that a true representation must earn something beyond its surface features to count as knowledge. The account of what that something is, is my own.
  2. Those criteria are tailored to the system’s particular adjudicative task. How they are arrived at, and what role historical conditions and biases play in their formation, lie beyond the scope of this white paper.
  3. I should note that a simulacrum need not be intended or malicious to be simulacral. An honest synthetic image, e.g., a couple’s picture of the child they hope to conceive, can become a simulacrum if it later surfaces in a background check and is read as proof of a real dependent. In that case, the artifact’s setting changed, not the artifact itself. A lab augmenting a dataset of rare plant samples can generate a subtly malformed one that accidentally slips through quality control, thereby also becoming a simulacrum. The adversarial cases are simply the most alarming version of the same thing, e.g., a doctored receipt claiming a larger reimbursement, or an astronomical pipeline fed night-sky scans with a reconnaissance satellite edited out or an unidentified aerial phenomenon edited in.

3. Existing Policy Responses

The simulacrum was a philosopher’s puzzle long before it was a policy problem. For Plato it was the degraded copy, the image too many removes from anything real to be trusted [1]; for Jean Baudrillard, more radically, it was the copy with no original at all, a representation of the world that had secretly detached from the world entirely [2]. Every regulator who has written a rule about synthetic data has been legislating about simulacra, just without knowing it.4

This section examines that unwitting body of regulation. I will begin by sorting out the vocabulary, which is still evolving and has an important lacuna. I will then survey what is on the books in the European Union, United States of America, and People’s Republic of China. The section will close by showing what they collectively leave untouched.

3.1. Synthetic data, media, and evidence

As the vocabulary around this phenomenon is still evolving, it will help to fix terms before outlining the existing rules. The bare term “synthetic data” appears often, but the emphasis falls on the noun’s modifier rather than the noun: what the rules care about is that the data is synthetic – artificially generated rather than collected – and they regulate that fact about its origin. What they do not ask is what the data is taken to show: whether anyone relies on it as standing for something real, and whether it is entitled to. That is, again, the question of warrant, which makes it a question about evidence, not data. In practice, then, regulators have been governing synthetic data while leaving synthetic evidence untouched.

This is not to say they have either overlooked or ignored the harms. Where synthetic data is used maliciously, regulators have attended mainly to synthetic media. Popularly known as “deepfakes,” this is synthetic data made to be seen, disseminated in open communication systems and aimed at persuading human audiences (hence, “media”). The case they have largely overlooked is synthetic evidence, which is synthetic data taken up by closed analytical systems whose primary purpose is interpretation, where a datum is relied on as warranting a claim about something real (hence, “evidence”).5 From a prima facie reading of the rules, it is not clear whether synthetic media and synthetic evidence are actually understood to be kin, two uses of one underlying artifact rather than unrelated problems.

3.2. Current rules

Turning to the rules, the European Union runs the most layered regime for synthetic data, along tracks of purpose limitation, privacy, fairness, and labeling, with detection and provenance emerging as a fifth. The purpose-limitation track works indirectly, at the source. Article 5 of the General Data Protection Regulation (GDPR) requires that personal data be collected for a specified, legitimate purpose and not reused in ways incompatible with it [12]. Although synthetic data is not explicitly targeted, the Regulation nevertheless reaches it through the real records from which that data is generated: those records may be fed to a generator only insofar as synthesis is compatible with the purpose for which they were first gathered, which keeps the resulting artifact loosely tethered to the licit reason it was made.

The privacy track treats synthetic data as a tool for anonymization, and can be understood as a partial extension of the same logic. Because data that no longer relates to an identifiable natural person falls outside the GDPR entirely (per Recital 26) [12], synthetic data is prized from the Regulation’s standpoint precisely because it can carry the statistical shape of sensitive records while pointing to no real individual [13].6 Nor is this favor confined to privacy, for the AI Act also prizes synthetic data as an instrument of fairness and as a means of correcting against discrimination [13, 15]. In these respects, European law actively values synthetic data for what it protects, and never asks whether it answers to anything real.

The labeling track, meanwhile, treats synthetic data as a disclosure problem. The AI Act’s Article 50 requires generative-AI providers to mark generated audio, image, video, and text in a machine-readable form, and deployers to disclose deepfakes to those who encounter them. Per Article 99, the obligation is broad, binding any provider operating in the EU market regardless of size, and it is backed by substantial fines [13].

The United States has no federal statute comparable to the GDPR or the AI Act. Federal agencies have some remit that is relevant to the problem, which I will discuss in a moment [§3.3]. Overall, though, the country leans on the voluntary guidance of the National Institute of Standards and Technology (NIST), whose recommendations on synthetic-content detection and provenance carry no force of law [16], and on a patchwork of state laws. Within that patchwork California stands out. Its AI Transparency Act obliges large providers, those above a million monthly users, to embed a latent, metadata-level mark in what their models generate and to offer the public a free detection tool, and it requires large online platforms, those exceeding two million monthly users, to detect and surface that provenance metadata as content circulates [17]. A companion statute separately compels generative-AI developers to publish summaries of the data their models were trained on [18].

The Western labeling regimes are softer than they look. The European marking duty is binding, but the means of satisfying it are not yet fixed, as no current technique meets the AI Act’s own criteria of robustness and reliability, the harmonized standards are still in development [19], and the obligation is qualified to hold only “as far as technically feasible” [13]. California’s law, for its part, has many loopholes. The size threshold exempts everyone smaller; the visible, human-readable disclosure is left to the user’s discretion rather than mandated; and the latent mark relies on the same strippable, Coalition for Content Provenance and Authenticity (C2PA)-style metadata [20].

The People’s Republic of China has a far more aggressive labeling regime. Its Cyberspace Administration’s Measures for Labeling AI-Generated Synthetic Content set no size threshold at all. Instead, every provider of AI-generated content and every ordinary user must attach both an explicit, human-visible label and an implicit label embedded in the file’s metadata, in the precise form dictated by a mandatory national standard. Distribution platforms must flag content they algorithmically suspect of being synthetic, and noncompliance can bring business suspension, revoked permits, even criminal liability [21, 22].7

3.3. An embryonic fifth track

Running beneath all three jurisdictions is an embryonic fifth track of detection and provenance. NIST’s guidance catalogues detection methods and provenance standards; California requires its largest providers to field tools for detecting their own AI-generated content, and from 2027 will require large online platforms to surface provenance metadata as content circulates; Chinese platforms must flag content they algorithmically suspect of being synthetic. The connective tissue across the West is the C2PA Content Credentials standard [20], the main cross-industry effort to bind a tamper-evident record of origin to a file, toward which both NIST’s guidance and California’s provenance rules point. However, none of these efforts represent settled regimes, to say nothing of robust ones. The Californian regime binds only the largest providers and platforms, the Chinese regime serves labeling rather than any independent test of reliability, and the C2PA mark, being metadata, can be stripped or spoofed.

These efforts are not only regulatory. Governments are also funding the underlying research directly, and some of it is aimed squarely at the pipeline rather than at media. In the United States, the National Science Foundation (NSF)’s Cybersecurity Innovation for Cyberinfrastructure program runs a dedicated funding area, Integrity, Provenance, and Authenticity for AI-Ready Data (IPAAI), whose stated purpose is to improve the integrity, provenance, and authenticity of the scientific datasets that AI systems consume [24]. In all but name, this is public investment in the detection and provenance of synthetic evidence, and it is far from the only example.

European law, for its part, comes closer to the simulacrum problem along a different dimension, that of data quality and system integrity. Article 5 of the GDPR requires that personal data be accurate [13], while Article 10 of the AI Act requires that the datasets training high-risk AI systems be relevant and sufficiently representative [15]. The AI Act further requires that such systems be made resilient against data poisoning, the deliberate corruption of a training set by injected data [15], a threat that NIST has begun to catalogue in detail [25]. These duties touch synthetic data, but obliquely and to other ends.

Accuracy asks whether a record faithfully reflects the person it describes; representativeness and poisoning-resilience ask whether a dataset is fit to train a reliable model. Those are not nothing, especially representativeness and poisoning-resilience, but they look past the question that matters here. Representativeness is a statistical property of a dataset in aggregate, whether the collection as a whole has the right shape to train a model that generalizes; it does not fully address whether any single datum within it answers to a real thing. Poisoning-resilience treats corrupted data as a threat to a model’s performance, something to be absorbed or filtered in bulk, not as a false claim about the world to be adjudicated one artifact at a time. Warrant runs the other way: it asks of a particular artifact whether it is entitled to be treated as a genuine stand-in for the specific real thing it depicts, whether, so to speak, it is authentically simulative of an authentic thing. A dataset can be impeccably representative in the statistical sense, and a model impeccably poison-resilient, while every synthetic record in play is a flawless simulacrum.

The United States’s federal remit I mentioned earlier tells a parallel story of aiming its arrow at a target it does not seem to fully realize is there. The Federal Trade Commission (FTC) can pursue AI-enabled deception under its standing power over unfair or deceptive practices [26], and the Federal Communications Commission (FCC) has declared AI-generated voices in robocalls illegal under the Telephone Consumer Protection Act [28]. In practice, both have trained that authority on synthetic media, the cloned voice and the fabricated endorsement aimed at a human audience, rather than on synthetic evidence entering an analytical pipeline. The reach exists; it has simply not been pointed at the pipeline.

All of which leads to the gap the next section sets out to close. These regimes differ in important ways – Europe’s mandate is broad if still undetermined, California’s label is narrow and partly optional, and China’s marking is universal and criminally enforced – yet they converge on a single omission, or, more precisely, on an implicit and undeveloped idea. Each governs how a synthetic artifact is made, anonymized, disclosed, or kept from poisoning a model; none governs whether the artifact is warrantable, whether it is entitled to be believed as evidence of something real. Where the regimes brush against that question, through an accuracy duty or an unaimed federal power, they engage it only implicitly and leave it undeveloped. However, one near-miss is different. Detection and provenance, the embryonic fifth track, is already pointed at the right target; it has simply never been built out or tied to the question of warrant.

  1. To be sure, the metaphysics is not always unconscious. Noteworthily, England’s national synthetic cancer dataset, generated for health research, is named The Simulacrum [9].
  2. This cybersecurity use of the term “synthetic evidence” is an extension of the term’s use in legal scholarship. There, “synthetic evidence” denotes AI-generated documentary or audiovisual material entering court proceedings [10, 11]. The legal usage is concerned with courtroom admissibility and the doctrinal challenge of authenticating artifacts whose probative claim is built from probabilistic inference over training data rather than from a causal relationship to events.
  3. Unfortunately, this has also proven to be a faulty assumption, as studies have shown real people can be reconstructed from synthetic data [14].
  4. The Russian Federation is omitted here. For present purposes, it is better understood through the lens of adversarial use than of regulatory modeling [23].

4. Government-Supported Detection and Provenance

Mitigation, like the problem, is at root philosophical, which is that society must ensure that synthetic data is simulative and not simulacral. The work centers around three tasks, all of which a single governmental entity can either do itself or spearhead. They are detection, provenance, and representational value, and they ascend in both difficulty and ambition. Detection asks only whether an artifact is synthetic. Provenance records where it came from and carries that record forward to whoever later relies on it. Representational value, the hardest of the three, asks whether the artifact answers to anything real at all. The first two can be accomplished today with tools that already exist, while the third remains a research program rather than a capability, one I take up in the final section below [§5].

This section sets out how such an entity might work. I will begin with the kind of institution it should be, namely, a standards body rather than a regulator. I will then describe the mechanism I propose for detection and provenance, a service I call integrity gating, and a second provenance strategy, a hardware root of trust, that could run alongside it. A final subsection takes up the limits of both.

4.1. A measurement-and-standards body

The entity itself should be a measurement-and-standards body rather than a regulator per se, an institution that sets methods and keeps records without ruling on the truth of anyone’s claims. Such bodies already exist in the form of national metrology institutes. In the United States the closest fit, if not the actual candidate, is the National Institute of Standards and Technology (NIST). China likewise has one in all but name, since its National Information Security Standardization Technical Committee, known as TC260 and working under the Cyberspace Administration, authored the labeling standard surveyed above.

The European Union is the harder case. The natural candidate, its Agency for Cybersecurity (ENISA), does not fit, since the agency’s remit is advisory and concerned with the security of networks and information rather than with setting measurement standards or keeping technical registries. The role described here is currently distributed across Brussels’s harmonized-standards bodies, such as the European Committee for Standardization (CEN) and the European Committee for Electrotechnical Standardization (CENELEC). While these entities drafted the technical specifications underpinning the AI Act, no single institution owns that function as NIST and TC260 own theirs, so the Union would have to assign it.

All that being said, whatever precise shape the proposed agency takes in these contexts, what I am proposing is a toolmaker and a registrar rather than a tribunal of the Real, and that distinction is the whole point of the design. A body empowered to rule on whether a synthetic artifact truly answers to the world would be deciding, from a single seat, what counts as genuine evidence in medicine, in law, in journalism, in intelligence, and elsewhere, all at once. That is nothing less than the authority to declare what is real, and no free society should vest that power in a single institution. The architecture proposed here depends on keeping that authority where it has always lived, with the sectors that exercise it, while tasking the central body with the narrower work of establishing what an artifact is and where it came from. The entity thus supplies the means by which each stakeholder can settle warrant for itself.

4.2. Integrity gating


Figure 4: An overview of integrity gating in the specific case of academic science.


Figure 5: A mock-up of the Integrity Gating service dashboard.

Detection would be the heart of the entity, because it is the precondition for everything that follows. One cannot record the origin of an artifact that no one has yet recognized as synthetic, so the work has to begin by catching synthetic material that arrives undeclared, at the moment it enters a workflow, before anyone mistakes it for a genuine record. Provenance comes afterward, taking the form of a durable and checkable record of origin that travels with the artifact to everyone who may later encounter it and come to rely on it.

Obviously, neither capability is new. Detection tools exist across several research communities, and provenance already has maturing standards in C2PA Content Credentials [20], in NIST’s synthetic-content guidance [16], and, for scholarship, in the decades-old digital object identifier (DOI) [27]. The idea here, then, is not to reinvent the wheel, but to organize existing tools and future research around the threat they are already addressing from different angles and without coordination.

The institutional form I propose for that coordination is integrity gating, which is, in plain terms, a detection-and-provenance service that researchers and laboratories can use in the ordinary course of their work [Figs. 4-5]. The term “gate” signifies a procedural checkpoint “placed” wherever undisclosed synthetic material would do the most lasting damage.

Identifying the best locations for these checkpoints will vary by interpretive stakeholder, meaning that another task of this entity will be to conduct attack-chain analyses with, or on behalf of, stakeholders. For example, in academic science, which is often dependent on open source datasets hosted on public repositories like Hugging Face, Zenodo, Kaggle, figshare, and GitHub, there are at least two key moments where integrity-gating would have the strongest impact. The first is pre-training, when data enters the training set of a model, and the second is pre-publication, when results stand to enter the published record [Fig. 4]. In pre-training, the threat is that undisclosed synthetic artifacts – accidentally incorporated into the dataset, or smuggled there by a malicious actor – become consumed by the model, while in pre-publication, the threat is that it is the researcher themselves who fail to properly disclose their use of synthetic content, thereby introducing it into the scientific ecosystem.

Two practical questions now follow: how would the integrity-gating service work, and how could the proposed body manage it? The answer to the first question is straightforward, as it essentially entails making integrity-gating a best practice among stakeholders. To use academic science as the example again, before training a model, a researcher would submit the dataset they intend to train on to the first gate, which runs it through the detection regime and returns a report on any potential undisclosed synthetic material within it. When the work is written up, the manuscript and its related materials are likewise run through the detection regime, which issues a disclosure record that travels with the manuscript, basically like today’s Digital Object Identifier (DOI) [27]. It is this record that would enable reviewers and readers to directly assess the artifact’s provenance instead of having to painstakingly reconstruct it for themselves. And to be clear on a key point: the aim throughout is not to punish researchers, but to surface and disclose, to log a synthetic artifact’s origin before it hardens into someone else’s ground truth [Fig. 5].

As for how the standards body could run this service, that question has several workable answers, differing mainly in how much the body centralizes. For example, it could host the detection regime on its own cyber-infrastructure and process every submission itself. This would be the most concentrated arrangement, which I would anticipate Beijing to favor. Alternatively, it could fund and equip established stakeholder institutions such as universities to operate the gates on its behalf, a strategy that suits the European habit of working through designated bodies. Or it could push the capability outward almost entirely, seeding it through grants and letting a broad community of adopters build and run their own gates, as the United States research system and its funding agencies tend to do [§3.3]. In every version, however, the entity keeps the standard and the registry. From a cybersecurity perspective, the more distributed the arrangement, the less the entity becomes the single bottleneck and monoculture that an adversary would most want to attack. Again, it is meant to be the rails on which the many gates run, not the one gate through which everything must pass.

4.3. Hardware root-of-trust

Another provenance strategy that can be pursued at the same time is a hardware mandate. The clearest proposal of this kind comes from ventures such as TripleID, which would build the guarantee into the chip itself rather than supply it via a software intervention. In their approach, each processor is given a unique cryptographic identity at the point of manufacture, one meant to be impossible to clone or forge. A recorder built into the device then signs every output the AI produces with that hardware-rooted key and links the signatures into a tamper-evident chain, while a central authority registers each chip’s identity and keeps the standing power to revoke it.8

How such a mandate might come about would differ sharply by jurisdiction. Beijing is the best positioned to impose one outright, since it already compels labeling and holds direct leverage over the chip makers operating within its borders, and could route device registration through the same machinery that administers its synthetic-content rules. Brussels would more likely arrive at it through product law, writing a hardware-identity requirement into conformity assessment and the CE mark, or into a future revision of the AI Act, with its standards bodies specifying the technical form. Washington, DC is the least likely to legislate a mandate directly, but it holds a different kind of leverage, since the most capable AI chips are designed by American firms, so the same end could be approached through procurement rules, export controls, or NIST standards that become binding in practice if not in law.

4.4. Limitations

Both strategies have limitations. For integrity gating, the biggest limitation is that detection is never finished. The detector is always a step behind the generators it must catch, and the most dangerous artifacts are precisely the ones built to pass, such as the Fomenko toe [Fig. 3]. Gating lowers the volume of undisclosed synthetic material moving through a pipeline, yes, but it cannot promise to catch all of it.

Another limit is that a gate only works where a pipeline passes through it, which is to say, if it is not enforced with the weight of law, it requires voluntary stakeholder adoption (the best-practice strategy I mentioned earlier). If so, then like current labeling regimes, gating could end up binding good-faith actors.

As for a hardware root-of-trust, its very strength is its weakness, for it secures the act of creation rather than the act of entry. It can certify the synthetic artifacts that a registered machine produces, but only that registered machine; it does not tell us who the creator was, and chips can move around. It also does nothing about an adversary who simply generates on an unregistered chip. Indeed, even if this strategy was implemented tomorrow, chips without this adaptation will be in circulation for a long time to come. That is to say nothing of the fact that the root-of-trust itself could likely be spoofed by a determined enough state adversary.

Worse, a chip that signs everything an AI does, and that an authority can switch off, is also an instrument of surveillance and a remote kill switch. That is a cost no rights-respecting society should accept without extremely careful consideration.

Now step back, and a deeper limit comes into view, one the two strategies share, and one that no improvement to either could remove. Integrity gating guards the point of use and a hardware root-of-trust guards the point of creation, so together they watch more of an artifact’s life than either could alone. However, notice what they establish even when they work perfectly: that an artifact is synthetic, and where it came from. That is a fact about its history, not its truth. A synthetic artifact can be correctly flagged and honestly provenanced, its every credential confirmed, and still depict something that never happened. This is not a failure of the system I am proposing, but a boundary of it. Detection and provenance were never instruments for measuring whether an artifact answers to the world; they secure its history so that each interpretive stakeholder can then answer that question for itself.

What this system cannot do is confirm the artifact’s representational value. That is the question that matters most, and the hardest to resolve, but as I will propose next, we can at least try.

  1. TripleID is pre-product and operating in stealth as of late May 2026, and has not publicly disclosed the cryptographic primitives, attestation protocol, or registry underlying its design. The account here reflects the company’s stated architecture, conveyed in personal communications, rather than independently verified or deployed technology.

5. A Representational Value Benchmark

If detection asks whether an artifact is synthetic and provenance asks where it came from, representational value asks the question that actually matters: whether the artifact answers to anything real. A benchmark for it would measure how faithfully a synthetic datum stands in for the part of the world it claims to represent, turning the difference between a simulation and a simulacrum into something quantitative. This section sketches what such a program could look like and then confronts why it may never fully succeed.

5.1. Building from what exists

The evaluation of synthetic data is already a mature field, but it has concentrated on three properties other than representational value: fidelity, how closely an artifact resembles the source data it was built from; privacy, how well it shields the real individuals behind that source; and utility, how useful it proves for a downstream task [29, 30]. Representational value is a fourth and orthogonal axis, and the distance between it and the other three is exactly the danger this paper has tried to articulate. Indeed, it is telling that existing toolkits already report that an artifact’s statistical fidelity and its downstream utility need not move together [30], which is a kindred divergence to the divergence between fidelity and plausibility that a simulacrum exploits.

Measuring this fourth axis would not, however, mean starting from scratch. The mechanisms I propose here each already exist in some corner of the literature, but they sit apart, divided by modality and by purpose. The core idea here is to unite them, treating representational value as a single property that can be approached across images, text, and structured data alike.

What might such a program do at the outset? The honest first move is to give up on a single universal measure and begin where the problem is most tractable, with falsification rather than verification. It is far easier to show that an artifact answers to nothing real than to show that it does, because the faker must get every detail right while the detector need find only one thing wrong. Impossibility leaves traces, e.g., the finger bone that does not belong in a foot, the duplicated fracture, the timestamp inconsistent with how the record was supposedly made [Figs. 2-3].

In fact, this is the most developed of the three approaches. Current benchmarks test whether generated images and video violate physical and anatomical possibility [32, 33], and a long forensic tradition catches manipulated media by their internal inconsistencies. A first benchmark in the representational-value project could therefore measure how reliably a system catches artifacts that could not correspond to any real referent, climbing the difficulty gradient from the obvious finger bone toward the subtle Fomenko toe. The less-charted frontier may then be to carry this same impossibility-testing beyond unstructured audiovisual data into structured tabular data, catching the logically impossible record or the transaction trail no real process could have produced [8].

A benchmark of this kind would also take an already proven shape: a reference collection of artifacts with known provenance, some authentic and some simulacral of graded subtlety [e.g., Figs. 2-3], paired with a public challenge inviting laboratories to tell them apart. This is how detection has long been advanced, from media-forensics challenges to the Collaborative Research Cycle for synthetic data [29].

Two further measures could be built outward from there. A grounding benchmark could measure how much of an artifact’s content is traceable to a real source rather than invented, generalizing the attribution and faithfulness tests already used against hallucination in language models. That generalization is itself the hard part, since such tests live almost entirely in text and would have to be carried into images and structured data. A decision-equivalence benchmark could ask whether substituting a synthetic artifact for real data changes the judgment a given task would reach. Here the groundwork is firmest, laid by the train-on-synthetic-test-on-real paradigm and, most directly, by Synthetic Ranking Agreement, which measures whether synthetic and real data rank competing models the same way. What a representational-value benchmark would add is a weighting-by-stakes, since not every decision a simulacrum distorts matters equally: a synthetic artifact that flips a trivial classification should count for little, while one that flips a diagnosis, a verdict, or a threat assessment should count for much. This has the further merit of keeping the benchmark a tool each sector calibrates rather than a verdict imposed upon it.

Taken together, these three benchmarks would share a deliberate modesty, as none would measure truth directly. They would measure what a simulacrum cannot do, the impossibilities it cannot avoid, the grounding it cannot fake, the decisions it cannot preserve. That is less than the moonshot promises but far more than nothing, which is where a hard program rightly begins.

5.2. Why it may never fully succeed

The proposed standards body, whatever precise form it takes, would be best suited to lead this program, since the endeavor would require a long horizon to either achieve or conclusively fail. Even a highly centralized version of this institution could not pursue the research on its own, however, for the intellectual and technical resources required would be too great. The same academic and corporate laboratories that build our AI systems could and should be enlisted into it.

That said, a benchmark for representational value as such is probably not achievable, and it is worth being honest about why. One obstacle is technical: whether a synthetic datum represents reality well enough depends on the use to which it is put, so a single universal measure may be incoherent, and a forest of interpretive stakeholder-specific ones may be the most anyone can build.

Another obstacle is political. A body that could pronounce on representational value in general would be deciding what counts as a faithful picture of the world, which is the very authority this brief has argued must remain with stakeholders. The benchmark must be built as a tool that interpretive stakeholders can use on their own terms, rather than a verdict imposed upon them.

Yet, perhaps the deepest obstacle is philosophical. Representational value is not a property inside the artifact, the way a watermark or a file signature is, but a relation between the artifact and that part of the Real the artifact is standing in for. The feasibility of such a benchmark thus rehearses the moment depicted in Raphael’s School of Athens, with Plato pointing skyward, Aristotle earthward. Is the Real, if I may put it this way, its own independent reality, or is it only ever found here, with us, with the subjects and objects of experience themselves?

That such a project may never conclusively succeed is not a reason to abandon the architecture this brief proposes. It is the reason for it. If representational value cannot be inherently read off an artifact, then any governance built on inspecting artifacts was never going to reach the thing we most care about, and a regime built instead on detection, provenance, and the standing of each interpretive stakeholder to judge its own evidence is not a placeholder awaiting the real solution. It is the right response to a property that was never going to sit inside the file. The unmeasurability is not a void beneath the design, but the ground the design stands on. There is, then, something fitting in a standards body funding research into the one thing it may never be able to standardize, regulating the disclosure of synthetic data in the present while underwriting the longer effort to understand what synthetic data is worth. Whether the bind is peculiar to synthetic data, or whether synthetic data has only exposed something always true of the metaphysics of evidence, namely that warrant is conferred and never simply measured, is a question I leave open here.

What is not an open question are the stakes. Synthetic artifacts are already entering the analytical record, some in bad faith and many in good faith but merely undisclosed. If they harden into the ground truth on which the next AI system trains, the drift will become very hard to reverse.

Competing Interests

The integrity-gating approach described in §4.2 is based on an active research project at the Rochester Institute of Technology led by the author. Additionally, the author may seek investment to support it. Some investors potentially interested in that work may also have an interest in TripleID, whose hardware-based approach is assessed in §4.3. The author currently has no financial relationship with TripleID.

References

  1. Plato. Sophist.
  2. Baudrillard, Jean. Simulacres et simulation. Paris: Galilée, 1981.
  3. European Data Protection Supervisor. “Synthetic Data.” https://www.edps.europa.eu/press-publications/publications/techsonar/synthetic-data_en
  4. Searle, John R. The Construction of Social Reality. New York: Free Press, 1995.
  5. Gettier, Edmund L. “Is Justified True Belief Knowledge?” Analysis 23, no. 6 (1963): 121-123. https://courses.physics.illinois.edu/phys419/sp2021/Gettier.pdf
  6. Plantinga, Alvin. Warrant and Proper Function. New York: Oxford University Press, 1993.
  7. ——. “Précis of Warrant: The Current Debate and Warrant and Proper Function.” Philosophy and Phenomenological Research 55, no. 2 (1995): 393-396. https://andrewmbailey.com/ap/Precis_Warrant.pdf
  8. Schwartz, Christopher, and Adam Arthur. “Deceiving the Machine: The Case for Synthetic Evidence as a Cybersecurity Category.” Under review, 2026.
  9. National Disease Registration Service, NHS England. The Simulacrum (synthetic cancer dataset). https://digital.nhs.uk/ndrs/data/data-outputs/cancer-publications-and-tools/simulacrum
  10. Martinez, Antonio Lopo. “Synthetic Evidence: Documentary Deepfakes and the Future of Truth in Brazilian Legal Proceedings.” Revista Eletrônica de Direito Processual 27, n. 2 (2026). https://doi.org/10.2139/ssrn.5479486
  11. Durand, Maxime. “When Evidence Becomes Synthetic: Admissibility, Authentication, and the Legal Crisis of AI-Generated Proof.” LexAI Journal, January 12, 2026. https://lexai.sa.utoronto.ca/when-evidence-becomes-synthetic-admissibility-authentication-and-the-legal-crisis-of-ai-generated-proof/
  12. European Union. Regulation (EU) 2016/679 (General Data Protection Regulation). Official Journal of the European Union 119, 2016.
  13. Bartholdy, Matthias. “Positioning Synthetic Data under EU Data Protection Law.” Computer Law & Security Review 61 (2026): 106310. https://doi.org/10.1016/j.clsr.2026.106310
  14. Tari, Henry and Adriana Iamnitchi. “Measuring Privacy vs. Fidelity in Synthetic Social Media Datasets.” arxiv.org/abs/2603.03906
  15. European Union. Regulation (EU) 2024/1689 (Artificial Intelligence Act). Official Journal of the European Union, 2024.
  16. National Institute of Standards and Technology. Reducing Risks Posed by Synthetic Content: An Overview of Technical Approaches to Digital Content Transparency. NIST AI 100-4. Gaithersburg, MD: NIST, 2024. https://doi.org/10.6028/NIST.AI.100-4
  17. California. AI Transparency Act, S.B. 942 (2024), as amended by A.B. 853 (2025).
  18. California. Generative Artificial Intelligence: Training Data Transparency, A.B. 2013 (2024).
  19. CEN-CENELEC Joint Technical Committee 21, Artificial Intelligence, under European Commission standardisation request M/593 (2023).
  20. Coalition for Content Provenance and Authenticity. https://spec.c2pa.org/specifications/specifications/2.4/index.html
  21. GB 45438-2025, Cybersecurity Technology – Labeling Method for Content Generated by Artificial Intelligence (网络安全技术 人工智能生成合成内容标识方法). Mandatory national standard. Standardization Administration of China, released 14 March 2025, effective 1 September 2025. https://www.codeofchina.com/standard/GB45438-2025.html
  22. Cyberspace Administration of China, Ministry of Industry and Information Technology, Ministry of Public Security, and National Radio and Television Administration. Measures for Labeling AI-Generated Synthetic Content (人工智能生成合成内容标识办法). Issued 14 March 2025, effective 1 September 2025. English translation: China Law Translate. https://www.chinalawtranslate.com/en/ai-labeling/
  23. Schwartz, Christopher, Justin Pelletier, David I. Schwartz, Matthew Wright, and Andrea Hickerson. “Deepfakes in Narrative Warfare.” In Artificial Intelligence and International Security. Eds. Alena Vysotskaya Guedes Vieira, Arshin Adib-Moghaddam and Mohammad Eslami. Manchester University Press, 2026. https://manchesteruniversitypress.co.uk/9781526196163/
  24. National Science Foundation. Cybersecurity Innovation for Cyberinfrastructure (CICI). https://www.nsf.gov/funding/opportunities/cici-cybersecurity-innovation-cyberinfrastructure
  25. National Institute of Standards and Technology. Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations. NIST AI 100-2e2025. Gaithersburg, MD: NIST, 2025. https://doi.org/10.6028/NIST.AI.100-2e2025
  26. Federal Communications Commission. Implications of Artificial Intelligence Technologies on Protecting Consumers from Unwanted Robocalls and Robotexts. Declaratory Ruling, CG Docket No. 23-362, FCC 24-17. Adopted 2 February 2024, released 8 February 2024. 39 FCC Rcd 1783. https://docs.fcc.gov/public/attachments/FCC-24-17A1.pdf
  27. The DOI Foundation. https://www.doi.org
  28. Telephone Consumer Protection Act of 1991, Pub. L. No. 102-243, 105 Stat. 2394 (codified at 47 U.S.C. § 227).
  29. NIST Collaborative Research Cycle / SDNist. National Institute of Standards and Technology. https://pages.nist.gov/privacy_collaborative_research_cycle/
  30. DataCebo. SDMetrics. https://docs.sdv.dev/sdmetrics/
  31. Offenhuber, Dietmar. “Shapes and Frictions of Synthetic Data.” Big Data & Society 11, no. 2 (2024). https://doi.org/10.1177/20539517241249390
  32. Bordes, Florian, Quentin Garrido, Justine T. Kao, Adina Williams, Michael Rabbat, and Emmanuel Dupoux. “IntPhys 2: Benchmarking Intuitive Physics Understanding in Complex Synthetic Environments.” arXiv:2506.09849 (2025). https://arxiv.org/abs/2506.09849
  33. Bansal, Hritik, Clark Peng, Yonatan Bitton, Roman Goldenberg, Aditya Grover, and Kai-Wei Chang. “VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation.” In The Fourteenth International Conference on Learning Representations (ICLR 2026), 2026. https://videophy2.github.io

Fairness inside and out: A situated approach to algorithmic allocation in complex sociotechnical systems

Abstract

This article outlines a framework for modeling and simulating complex sociotechnical systems in which an allocation mechanism acts as the interface (and sometimes a barrier) between the public and institutions. Two examples contextualise algorithmic allocation challenges: the Amsterdam school choice and organ allocation for transplant patients. We highlight how algorithmic fairness does not guarantee systemic fairness, and propose a situated approach with institutional modeling and social simulations. With the increasing adoption of AI in decision making, a situated simulation approach provides an alternative to opaque institutional governance and decision making.

The mechanism, institutions, and people

Sociotechnical systems are formed of technical and social components; in present-day societies, most technology is used within a social environment. Mindful of algorithmic discrimination cases, researchers have sought ways to measure the fairness of technical systems. The impact of a technology on its surrounding environment, however, is not always clear, especially when complex social dynamics are at play. Designing a “fair” technical component is not a guarantee of bringing fairness to the system as a whole. Selbst et al. argue that this is due to a focus on solutions (i.e. “make the outcome fair”) instead of a focus on the process (i.e.”make the system fair-er”) [1]. We now need ways to understand the impact of a technical system on fairness, within the technology, and without it.

Resource allocation problems are routinely faced by public institutions, be it in the form of determining who gets access to benefits, or which infrastructure to renovate among a set of proposals. The decision making process forms in institutions, bound by laws of fairness and non-discrimination. However, institutional decisions change the dynamics of the system before they take action, while they are communicated, and after they take hold, altering the behaviour of the social system.

Social simulations can estimate the effect of policy changes by dynamically modeling the social and institutional aspects of sociotechnical systems [2]. The field of institutional modeling investigates the interplay of individual and institutional agents within complex sociotechnical systems [3].  Social and institutional simulations might thus guide the process of creating fairness throughout the system and not only within the mechanism. We use the framing of simulations to explore allocation mechanisms and their impact on complex sociotechnical systems.

Algorithmic allocation

We propose an approach to modeling sociotechnical systems under specific conditions: resource allocation with a registry (here called mechanism) facing the public in one direction and based on rules determined by one or multiple institutions. The communication between public and institution occurs through the mechanism, which is designed to optimize for a certain outcome. Value preferences inform the outcome, but are not always explicit, nor uniform across parties. The institution designs the rules of the allocation mechanism based on (i) its internal principles, and (ii) a fair outcome for the public. Without communication between public and institutions, the rules are decided based on inferred preference models, i.e., the institution designs the mechanism to please the assumed value preference of the people and reach a “fair” outcome (Fig. 1).

In this setting, which we contextualise in two examples below, the mechanism design affects the behaviour of agents independently of its inherent degree of algorithmic fairness. Specifically, we encountered two cases that pose what seem to be “already-solved” allocation problems. Upon further scrutiny, we found underlying, more complex dynamics.

Figure 1. Motivating setup: A sociotechnical system consisting of agents, a mechanism, and policy actors. The agents provide an input to the mechanism, and are affected by its output, while a policy actor sets the rules for the mechanism.

Figure 1. Motivating setup: A sociotechnical system consisting of agents, a mechanism, and policy actors. The agents provide an input to the mechanism, and are affected by its output, while a policy actor sets the rules for the mechanism.

As shown in the examples, the mismatch between the allocation rules and the preference model of the population influences the mechanism’s efficacy, prompting gamification for those who have the resources to invest in understanding and playing the mechanism’s rules. In the second example, the complexity of conflicting institutional values limits the mechanism’s fairness and efficiency.

This raises questions of prosociality, trust, governance and fairness outside algorithmic bounds. How do people respond to a mechanism that does not align with their preference model (and perhaps their values)? How can institutions consider the longitudinal effects of algorithmic allocation within the process of policymaking?

Amsterdam school choice

The Amsterdam school choice system is an especially useful example of a sociotechnical system where the deployment of algorithmic allocation led to unexpected changes in the behavior of the social system, causing continuous updates to the algorithm and subsequent degradation of trust in the system. It is also a good example of when a poorly designed “fair” mechanism leads to diminishing prosocial behavior.

In Amsterdam, there is an open-choice policy when moving from primary to secondary school [4]. Group 8 students (equivalent to 6th grade in the US) are allowed to pick any school within their education level (e.g., vocational, general secondary, pre-university, etc.) anywhere in the city, unrestricted by their zone of residence. Every year, students are asked to rank 8 to 12 schools (based on education level), and then a centralized matching algorithm automatically matches students to schools.

The system is communicated to be based on the famous Deferred Acceptance algorithm [5], a New York based school choice algorithm which won its author the Nobel prize in Economics in 2012 [6]. However, the algorithm is modified in several key places to fit the open-choice policy in Amsterdam.

Firstly, the Deferred Acceptance algorithm is a two-sided matching method, meaning that students and schools both have preferences over each other and are then matched accordingly. In Amsterdam, schools do not have a preference over students and therefore this preference is simulated with the means of a random lottery number. This choice was also grounded in fairness, as with a random lottery, every student has an equal chance of getting a good lottery number. Secondly, the Deferred Acceptance algorithm does not specify a fixed number of schools to rank. In Amsterdam, however, a policy named “placement guarantee” was introduced, where students are guaranteed a position if they rank a fixed number of schools. This guarantee is ensured by increasing the capacity of schools by a marginal amount in a second round of allocation, where students who listed the required number of schools in the first round and still did not get a placement are eligible, and are then allocated using the increased capacity in the second round. These changes unsurprisingly led to the matching algorithm performing very differently from its Nobel-prize-winning counterpart, creating problems of inefficiency and inequality.

Random lottery numbers led to an inefficient allocation of schools, increasing dissatisfaction among students and their parents [7]. Moreover, the “placement guarantee” policy led to strategic gaming from the parents, where they are incentivized to report schools they do not prefer just so they can ensure eligibility for the second round [8]. This further degraded the system’s efficiency, as well as its fairness, as parents from privileged backgrounds are better able to apply strategies than parents from marginalized communities.

The authorities responsible for the school choice system have repeatedly come under public scrutiny for continuous changes to the system without successful results [9]. This has led to a degradation of trust in the institution.

Underneath this allocation problem is a complex sociotechnical issue, one that requires:

  1. Making clear what values the system is actually trying to serve, and whether these align with what parents themselves care about.
  2. Understanding how the rules of the mechanism change social behavior, so that reported preferences are seen not just as choices, but also as responses to risk, incentives, and unequal access to information.
  3. Identifying problems that are easy to miss if we only look at fairness within the mechanism itself, such as distorted preferences, unequal strategic burden, and declining trust in the system.
  4. Looking beyond the matching rule alone, and instead supporting better communication, better alignment between stakeholders, and more situated policy design.

This requires a multidisciplinary sociotechnical approach, one in which a social simulation model can help make the problem and its tradeoffs easier to communicate and discuss. It also requires a systematic way of eliciting stakeholder values and using those discussions to work toward a shared understanding of what the system should aim to achieve.

Organ allocation systems

Fairness plays an important role in organ allocation systems. For patients on the waiting list to receive a transplant, waiting time is a strong determinant of short-term survival, and long-term quality of life. Multiple factors influence the structure of the waiting list and the distribution of donated organs, particularly those from deceased donors. Initially, biological compatibility between donor and recipient seems to provide a baseline allocation rule: the most compatible donor-recipient match (considering blood type and HLA presence) should be prioritised to ensure the best graft survival outcomes. However, the length of the waiting list and persistent scarcity of organs complicates the allocation problem.

Cold ischemia time restrictions (meaning how long an organ is deprived of blood flow before its quality deteriorates beyond utility for transplantation) brings into consideration logistics, donor and recipient location and a corollary of geographical factors such as regional allocation rules, coordination practices and transplant capacity. Allocation policies have been studied via simulations and algorithmic optimization for decades, with awareness of the complex interplay of medical, economic, political and legal factors.

The other component of the system is the prioritization of patients within the waiting list structure. The state of the patients is dynamic, with probabilities of becoming too ill to receive a transplant. The waiting list design attempts to accommodate for this by modeling the disease progression and setting thresholds for transplant eligibility.

In the U.S., Organ Procurement Organizations (OPO) and Transplant Centers’ performance is evaluated competitively based on how many organs were retrieved and how many successful transplants were performed [10], adding economic incentives to the mix.  This influences the local center’s decision to accept available organs or reject them in the hope of a better quality alternative.

Even seemingly aligned bioethical values can create conflicting interests, especially if multiple institutional actors are pushing for their preferred outcomes. Beside the responsibility to ensure “fairness” in allocation, the meaning of “do no harm” manifests differently for Transplant Centers (i.e. reject lower quality organ offers) and OPO (i.e. provide as many quality organs as possible), exacerbating inefficiencies. Concerning the value preferences of the patients themselves, little is known. Without a channel of communication to the algorithmic rule setters, and with conflicting interests among institutions, we risk optimizing for less relevant values.

Most importantly, the algorithm of allocation can be gamed by multilisting. Multilisting is the practice of assigning one patient to multiple waiting lists across Transplant Centers, which initially seemed to alleviate the length of the waiting list at the national scale in the U.S. However, this came at the cost of individual fairness (not everyone can be listed in multiple registries as not everyone can afford to quickly travel across states to receive a life-saving transplant) and collective fairness: while the national wait time decreased, regional variance in wait time can increase [11]. Multiple organ offers also seem to produce a utilitarian gain in efficiency by reducing organ wastage and time to transplant, but may do so at the cost of societal trust in the system [12].

Within this critical healthcare system, the allocation mechanisms endlessly attempt to reconcile fairness and efficiency. The value prioritization shifts from face-value fairness (e.g. centralised FIFO allocation) to utilitarian fairness (prioritizing patients with “the most to gain” from a transplant) or need-based fairness (prioritizing the most sick patients based on disease models with thresholds for exclusion), or introducing market logic to the system (monetary rewards to centers, multilisting, multioffer). Formalizing any value-rule into algorithmic allocation mechanisms creates, as in the example above, new ways for the system to be strategised upon, shifting the locus of inequality and obscuring its methods.

Decontextualised fairness as an attribute of technical (sub)systems has already been criticized [1]. The algorithms designed for organ allocation systems present a clear example of the Framing and Formalism abstraction traps, in which fairness evaluation stops at the technical part of the sociotechnical system and its formalization generates new, unaccounted behaviours in the system. Some literature acknowledges these limitations, suggesting we might expand our abstraction boundary, or even consider other processes beside algorithmic allocation [13,14]. In the absence of a consensus of what “fair” organ allocation is, we might want to shift our focus to a process approach. Social and institutional modelling provide the best toolset for this endeavour.

Our proposal

We propose a four-step approach for studying complex sociotechnical systems in which an allocation mechanism mediates between the public and institutions. The goal is to understand how to embed algorithmic allocation mechanisms in a wider system of stakeholder values, behavioral adaptation, and institutional feedback. As the Amsterdam school choice reminds us, the algorithmic allocation must be designed to accommodate a response to the mechanism itself. To this end, we consider social/institutional simulations as the most eligible, situated approach.

The four steps are as follows:

  1. Value elicitation: The first step is to determine which values the mechanism is meant to serve, and whether those values are actually shared by the stakeholders affected by it. Mechanisms are often designed around an inferred preference model: institutions assume what matters to the population and encode those assumptions into rules and objectives. But in high-stakes systems, these assumptions may diverge from the lived priorities of the people who interact with them. A situated approach therefore begins by making these values explicit and contextable.
    For example, in the Amsterdam school choice, policymakers may prioritize procedural fairness and avoiding unassigned students, while parents may care more about genuine access to preferred schools, reduced strategic burden, transparency, and trust. This gap matters for how the system performs in practice. Within organ allocation, value elicitation is needed to clarify which purpose the allocation system serves within the transplantation system as a whole, and what systemic fairness means to stakeholders. Although this might not result in a univocal “solution”, it lights the fire to improve procedural awareness. The value elicitation should start from Question 0, meaning without the assumption that an allocation mechanism is mandatory [15].
  2. Modeling and simulation: The second step is modeling and simulation. The simulation model is not the end goal, but a structured approach to facilitate and open communication around assumptions, tradeoffs, and possible interventions. If the response of the population to policy is modeled, strategic gamification can be accounted for in the simulated scenarios. In systems such as school choice and organ allocation, the mechanism does not act on fixed inputs. The inputs themselves change in response to the mechanism. Parents adapt their reported preferences strategically; OPOs and patients respond to waiting list structures, matching rules, and institutional incentives. Social simulation makes it possible to examine these interactions explicitly and to ask not only what outcomes a rule produces, but how that rule changes behavior across the wider system.
  3. Identifying bottlenecks and blind spots: The third step is to identify systemic bottlenecks and blind spots. Mechanisms are often evaluated in terms of fairness or efficiency within the allocation procedure itself, but this can obscure deeper problems elsewhere. A mechanism may be fair on paper while placing informational or strategic burdens unevenly across groups. It may also optimize one part of the process while ignoring upstream or downstream failures. A situated simulation framework helps uncover these blind spots by tracing feedback loops across the system: strategic adaptation, unequal ability to navigate rules or exploit advantages, changing interpretations of fairness, and degradation of trust over time. The modeling of heterogeneous agents and institutions allows the representation of multiple values and goals.
  4. Policy recommendations: The final step is policy recommendations. These recommendations should go beyond a technical redesign of the mechanism itself, as the system is sociotechnical; interventions may also need to address communication, participation, institutional coordination, trust, and policy stability. As the approach is situated within a community of stakeholders (institutions and population), the policy recommendations are tailored to local needs. This step can be expressed in policy briefs, appointment of committee and representatives to coordinate across stakeholder groups.

The approach is intended to be iterative. Fixed time intervals between iterations allow for policy to take action, and structure the process of altering it. Moreover, they facilitate participation of stakeholders that are not institutional, constructing the communication channel that was lacking in the initial setting (Fig. 1).

Conclusions

Algorithmic allocation mechanisms are a component of critical sociotechnical systems where institutions and populations interact. We argue that their fairness should be evaluated inside the algorithms and outside of them, encompassing their effect on the system they become a part of. With the increasing use of AI to automate allocation processes, aligning the values of stakeholders and opening communication channels between the population and institutions becomes our priority. A four-step situated approach with social and institutional simulations is presented. The approach shifts the focus from a fair outcome to a fairer process. While we do not claim that the approach is perfect, it provides a step forward in the integration of algorithmic allocation and fosters a procedural, situated view of fairness.

References

[1] Selbst, Andrew D., danah boyd, Sorelle A. Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019. “Fairness and Abstraction in Sociotechnical Systems.” In Proceedings of the Conference on Fairness, Accountability, and Transparency, 59–68. New York: Association for Computing Machinery. https://doi.org/10.1145/3287560.3287598.

[2]  Lorig, Fabian, Fabris, Bertilla, Tucker, Jason. 2025. “Hybrid-Human Policy Modeling: Enhancing Decision-Making Using Social Simulations” Frontiers in Artificial Intelligence and Applications. Doi: 10.3233/FAIA250675

[3] Ghorbani, Amineh. 2022. “Institutional Modelling: Adding Social Backbone to Agent-Based Models.” MethodsX 9: 101801. https://doi.org/10.1016/j.mex.2022.101801.

[4] Ruijs, Nienke, and Hessel Oosterbeek. “School choice in Amsterdam: Which schools are chosen when school choice is free?.” Education Finance and Policy 14, no. 1 (2019): 1-30.
https://doi.org/10.1162/edfp_a_00237.

[5] Abdulkadiroğlu, Atila, Parag A. Pathak, and Alvin E. Roth. 2005. “The New York City High School Match.” American Economic Review 95 (2): 364–67. https://doi.org/10.1257/000282805774670167.

[6] Nobel Prize Outreach. 2012. “The Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel 2012.” NobelPrize.org. https://www.nobelprize.org/prizes/economic-sciences/2012/summary/.

[7] Het Parool. 2026. “Weer Minder Achtstegroepers Kunnen Naar Eerste Voorkeursschool, Volgend Jaar Wordt Het Beter.” Parool.nl. https://www.parool.nl/amsterdam/weer-minder-achtstegroepers-kunnen-naar-eerste-voorkeursschool-volgend-jaar-wordt-het-beter~b0ec8fd5f/.

[8] Tasnim, Mayesha, Paul Verhagen, Tobias Blanke, Erman Acar, and Sennay Ghebreab. 2025. “Modeling Strategic Risk in School Choice: A Case for Transparent Design”. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 8 (3):2470-79. https://doi.org/10.1609/aies.v8i3.36731

[9] Het Parool. 2026. “Loting voor Middelbare School Weer Terug naar Oude Systeem: 12 in Plaats van 9 Voorkeursscholen.” Parool.nl. https://www.parool.nl/amsterdam/loting-voor-middelbare-school-weer-terug-naar-oude-systeem-12-in-plaats-van-9-voorkeursscholen~b7972bda/.

[10] Washburn, Kirt. 2012. “Maximizing Donor Potential: Evolving Organ Procurement Organization Metrics and Optimizing Organ Distribution and Allocation in the United States.” Liver Transplantation 18 (suppl. 2): S1–S4. https://doi.org/10.1002/lt.23507.

[11] Harvey, C., and J. R. Thompson. 2016. “Exploring Advantages in the Waiting List for Organ Donations.” In 2016 Winter Simulation Conference (WSC), 2006–17. Washington, DC: IEEE. https://doi.org/10.1109/WSC.2016.7822245.

[12] Erazo, Ignacio, David Goldsman, Pinar Keskinocak, and Joel Sokol. 2022. “A Simulation-Optimization Framework to Improve the Organ Transplantation Offering System.” In 2022 Winter Simulation Conference (WSC), 1009–20. IEEE. https://doi.org/10.1109/WSC57314.2022.10015431.

[13] Thompson, David, Larry Waisanen, Robert Wolfe, Robert M. Merion, Keith McCullough, and Ann Rodgers. 2004. “Simulating the Allocation of Organs for Transplantation.” Health Care Management Science 7 (4): 331–38. https://doi.org/10.1007/s10729-004-7541-3.

[14] Feccia, Mariano, Arianna Freda, Maurizio Naldi, Gaia Nicosia, and Andrea Pacifici. 2026. “Modelling and Simulating the Organ Donation Process Using Bootstrap and Event-Driven Process Chain Representation.” Journal of Simulation 20 (2): 135–51. https://doi.org/10.1080/17477778.2025.2486709.

[15] Dahlgren Lindström, Adam, Dignum, Virginia, Ericson, Petter, Titareva, Tatjana and Tucker, Jason. 2025. “Responsible AI Self-assessment Workshop: Start with Question Zero”. https://aipolicylab.se/2025/09/05/responsible-ai-self-assessment-workshop-start-with-question-zero/ AI Policy Lab, Published September 5, 2025. Accessed April, 2026.

Rethinking the Digital Omnibus’ Impact on the EU AI Act: Simplification or Dilution? 

Introduction

The adoption of the European Union AI Act (hereafter AI Act) marks a milestone in the union’s ambition to shape trustworthy, human-centric artificial intelligence (AI). It reflects an effort to ground innovation in fundamental human rights – an approach that has positioned the EU as a global standard-setter in digital regulation. The proposed Digital Omnibus represents an effort to streamline and harmonise an increasingly complex regulatory landscape. 

However, as the Digital Omnibus Regulation Proposal directly affects the implementation of the AI Act and related data governance frameworks, it raises a number of critical cross-cutting concerns. In particular, certain simplification measures – introduced with the intention to reduce administrative burdens, facilitate compliance procedures, and improve regulatory coherence – may have unintended consequences on the consistency, traceability, and risk-sensitivity of the EU’s digital regulatory ecosystem. These include potential reductions in the consistency of regulatory application, limitations in the traceability of data and AI systems, and a weakening of the granularity required for effective risk assessment within the EU’s digital framework. Moreover, while these measures are designed to ease obligations for AI providers and developers, they may have the effect of shifting complexity downstream onto deployers and end-users. This could result in increased uncertainty for those responsible for the use of AI systems in practice, particularly in high-risk contexts where clear allocation of responsibilities and robust risk assessment remain essential. 

These interactions are especially relevant where data processing rules, access to large-scale datasets, and incident reporting mechanisms intersect with the criteria used to assess and classify high-risk AI systems. Ensuring that these instruments remain coherent in their application is therefore essential to preserving both the safeguards and the credibility of the EU’s risk-based approach. 

In this context, the AI Policy Lab at Umeå University seeks to contribute constructively to the ongoing discussion by highlighting specific areas of concern and putting forward targeted recommendations. The main objective is to ensure that the EU’s regulatory ecosystem remains both effective and future-proof. 

1. Amendments to the GDPR for AI training, Article 3 of the Digital Omnibus proposal 

The Digital Omnibus proposal introduces clarifications regarding the legal bases and conditions under which personal data may be processed for the development and training of AI systems. In particular, it elaborates on the use of legitimate interest as a legal basis and introduces specific derogations for the processing of special categories of data (i.e., sensitive data such as health, biometric, or political information). 

While these changes primarily concern data protection law, they have indirect but significant implications for the application of the AI Act – especially Article 6, which governs the classification of high-risk AI systems. This is due to the fact that the scope, nature, and volume of data used in training are key elements in assessing the risks associated with AI systems. 

Several aspects of the proposal may inadvertently weaken the link between data governance and AI risk classification: 

  • The introduction of derogations for the processing of so-called “residual” sensitive data during model training (§33) risks expanding the volume of sensitive data that can be used without prior scrutiny. This could reduce the ability of regulators to accurately assess whether an AI system should be classified as high-risk under Article 6 of the AI Act. 
  • The broadening of legal bases for data processing in the context of AI training (§§30-31) may weaken the connection between the actual risks posed by a system and its regulatory classification, particularly in the absence of coordinated interpretative guidelines between data protection authorities – such as the European Data Protection Board (henceforth EDPB) – and AI governance bodies – such as the AI Office. 
  • The simplification of transparency obligations, including information notices and Data Protection Impact Assessments (henceforth DPIAs) (§36, §40), may reduce the level of detail available to regulators. This, in turn, could hinder a proper assessment of whether a system meets the criteria for high-risk classification. 

Our Recommendations: 

  1. Introduce a specific notification requirement for AI models trained using the “residual sensitive data” exemption (§33) to preserve traceability of training datasets and support risk classification under the AI Act.    
  2. Provide a DPIA section dedicated to AI systems. This could help preventing the simplification of DPIA (single EU lists) from reducing the granularity necessary to assess AI risk.  

2. Prevent the merger of the Data Act, Digital Governance Act, and Open Data Act from creating “shortcuts” for high-risk AI systems 

The Digital Omnibus seeks to consolidate several existing legislative instruments – the Data Act, the Data Governance Act (henceforth DGA), and the Open Data Directive – into a single, more coherent framework governing access to and reuse of data, including data held by public authorities. This consolidation is intended to facilitate access to large datasets, including non-personal data, and to promote data sharing across sectors. While this can significantly support innovation and the development of AI systems, it may also have implications for how such systems are classified under the AI Act. Among these, it is important to highlight the following:  

  • Easier access to large volumes of data – including datasets that can be combined or enriched – risks enabling the development of AI systems whose purpose or context of use would place them within the scope of Article 6 of the AI Act (high-risk AI systems).  
  • There is a risk that the simplification of data access mechanisms could be interpreted, in practice, as a justification for lowering the perceived risk level of such systems. In other words, increased data availability risks inadvertently being used as an argument to downgrade regulatory scrutiny.  

Our Recommendations: 

  1. Clearly establish that simplified access to data does not affect the criteria for high-risk classification under the AI Act. The availability of data should not be considered a mitigating factor in the assessment of risk. 
  2. Introduce an ex ante assessment requirement for cases where public or publicly accessible data is reused for AI systems that are likely to operate in high-risk domains (such as employment, education, healthcare, or access to essential services). 
  3. Require public administrations, when authorising the reuse of data for AI development, to explicitly indicate whether the intended use is likely to fall within the scope of Article 6 of the AI Act. This would provide greater legal clarity for developers and strengthen regulatory consistency.  

3. Single-entry point for incident reporting (Article 6 and 9 of the Digital Omnibus proposal) 

The proposal to establish a single European entry point for incident reporting constitutes a significant step towards simplifying and harmonising reporting obligations across multiple regulatory frameworks, including NIS2 (cybersecurity), GDPR (data protection), DORA (financial sector resilience), eIDAS (digital identity), and the Critical Entities Resilience (CER) Directive. By centralising notifications through a platform managed at the EU level – specifically by the European Union Agency for Cybersecurity (ENISA) – the proposal aims to reduce administrative burdens for operators and improve the efficiency of information sharing across authorities. 

However, the centralisation of notifications on a single platform managed by ENISA raises some critical issues that deserve careful consideration (in order to ensure the effectiveness of the system and the protection of operators subject to reporting obligations):  

  • Expanding ENISA’s mandate to manage a unified reporting platform may lead to capacity constraints, given the anticipated volume of notifications. Any delays in processing or triaging reports could negatively affect incident response times and overall system resilience. 
  • The single-entry point is designed as a hub rather than a replacement for Member States’ national competent authorities. However, without seamless technical and procedural interoperability with existing national systems, there is a risk of duplication, inefficiencies, or increased administrative complexity.  
  • The proposal does not sufficiently clarify the allocation of responsibilities between operators, ENISA, and national authorities in cases of system malfunction, delays, or errors. This lack of clarity risks exposing operators to legal consequences for circumstances beyond their control.  
  • The centralisation of incident-related data at the EU level also raises questions regarding data governance and technological sovereignty. In the absence of clear guarantees on data localisation, secure infrastructure, and Member State oversight, there is a risk that sensitive operational information – potentially critical for national security – may be insufficiently protected or subject to dependencies on non-EU technological backbones. 

Our Recommendations: 

  1. Establish an independent annual audit mechanism to assess the functioning of the single-entry point. This audit should evaluate: (i) the system’s capacity to handle notification volumes, (ii) the timeliness and accuracy of information processing, (iii) the cybersecurity of the platform, and (iv) its level of interoperability with national systems. Such an audit would help to ensure transparency, reliability and continuous improvement of the system. 
  2. Introduce a mandatory fallback protocol to be activated in the event of technical unavailability. This should include: (i) alternative reporting channels, (ii) clear criteria for demonstrating compliance efforts by operators, and (iii) automatic suspension of notification deadlines during system outages. This would prevent operators from incurring violations due to circumstances beyond their control.  
  3. Clarify the liability framework by explicitly defining the division of responsibilities between ENISA, national authorities, and reporting entities. This should specify: (i) situations in which failures are attributable to the central system, (ii) the implications for operators’ legal obligations, and (iii) the safeguards available in cases of technical malfunction. Clear and predictable rules are essential to ensure legal certainty and the consistent application of reporting obligations across the European Union. 
  4. Introduce explicit requirements ensuring that the infrastructure supporting the single-entry point is based on secure, EU-controlled technological backbones, with clear provisions on data localisation, access control, and Member States’ oversight. This could include, for instance, reliance on trusted European cloud frameworks or “no non-EU backbone” requirements for particularly sensitive categories of incident data. Such safeguards would strengthen trust in AI systems and ensure alignment with broader EU objectives on digital sovereignty. 

Conclusion 

Overall, the Digital Omnibus proposal reflects a necessary and timely effort to streamline an increasingly complex regulatory framework and to facilitate its practical implementation across sectors. At the same time, the analysis above highlights the importance of maintaining a careful balance between simplification and regulatory integrity. Across the areas examined – namely AI training data governance, access to public and non-personal data, and incident reporting mechanisms – there is a common need to preserve traceabilityensure risk-sensitive oversight, and safeguard legal certainty, while also reinforcing the Union’s strategic autonomy. The recommendations above by the AI Policy Lab at Umeå University are intended to support this balance by addressing specific gaps and ambiguities without undermining the overall objectives of the proposal. In doing so, they aim to contribute to a coherent, robust, and future-proof EU regulatory framework for AI and the data economy. 

References  

Directive (EU) 2022/2555 of the European Parliament and of the Council of 14 December 2022 on measures for a high common level of cybersecurity across the Union (NIS2), OJ L 333, 27.12.2022, p. 80–152.  

Directive (EU) 2022/2557 of the European Parliament and of the Council of 14 December 2022 on the resilience of critical entities (CER), OJ L 333, 27.12.2022, p. 164–198. 

Proposal for a Regulation of the European Parliament and of the Council establishing a Digital Omnibus for the simplification of Union digital legislation, COM(2025) XXX final. 

Regulation (EU) No 910/2014 of the European Parliament and of the Council of 23 July 2014 on electronic identification and trust services (eIDAS), OJ L 257, 28.8.2014, p. 73–114 (as amended).  

Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 (General Data Protection Regulation – GDPR), OJ L 119, 4.5.2016, p. 1–88. 

Regulation (EU) 2022/2554 of the European Parliament and of the Council of 14 December 2022 on digital operational resilience for the financial sector (DORA), OJ L 333, 27.12.2022, p. 1–79. 

Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act), OJ L, 12.7.2024. 

AI first to purpose first: Rethinking Europe’s AI strategy

Abstract

This paper examines the European Commission ‘AI First’ strategy, arguing that it places acceleration and economic competitiveness above democratic values, societal benefit, and human-centric innovation. While substantial investment in AI is welcome when it promotes sustainable, equitable, and responsible innovation, the authors warn that policy is shifting from governance to unchecked deployment, risking fragmentation, dependency, and misaligned priorities. Rather than asking how AI can be applied, the paper urges policymakers to ask why, advocating a “People First” approach grounded in societal needs, digital sovereignty, and responsible innovation. The authors argue that Europe’s AI leadership should be shaped not by speed, but by principled direction, inclusivity, and a commitment to long-term public value.

AI First

AI is increasingly being framed as a strategic imperative for economic growth, competitiveness and innovation. Yet, this purpose is often at odds with a more fundamental question: what should the purpose of AI be, and under which conditions does it genuinely add value to society? Following the recent launch of The European Commission’s (2025a,b) Apply AI Strategy and the ambitious InvestAI Programme, aimed at building pan-European AI “gigafactories”, (European Commission 2025c), heralded by the Commission’s President as a cornerstone for Europe’s AI competitiveness, the policy discourse has shifted from governance to acceleration. This rhetoric of Europe becoming the “Continent of AI’” however may signal a worrying departure from Europe’s longstanding commitment to human-centric and responsible innovation.

The timing and framing of the European Commission (2025a) “AI First” narrative appears to be closely aligned with the recommendations of the Draghi Report (European Commission 2025d), which emphasises digital investment and competitiveness as central to Europe’s economic renewal. While the substantial funding and incentives for AI research and innovation are welcome, the framing of “AI First” ignores a deeper set of concerns, including the limited evidence, if any, of substantial productivity and societal gains from AI use ((Estrada 2025; Wearden 2025). As such the shift to “AI First’” not only threatens to erode the foundations of Europe’s long-standing commitment to human-centric and rights-based innovation, leaving citizens, both in Europe and beyond, as the ultimate losers.

Full Steam Ahead, But What’s The Heading?

Despite Europe’s foundational focus on trustworthy and human-centric AI, recent Commission announcements, and public statements from its leadership, suggest a radical shift away from precaution, governance, and shared responsibility on AI, to a position of acceleration and competitiveness. AI is seen as a means to bolster economic growth through a highly ambitious industrial policy. What this perspective overlooks, both in Europe and globally, is a clear “people first’” perspective: recognition that technology must serve human and societal goals, not the other way around. The “AI First’” approach glosses over this vital point. While, lip service is paid to an assessment of the benefits and risks of the technology, these are framed as checks and balances, and fail to asks, for example, if a non-AI solution may be better or safer.

This acceleration approach also is in direct contravention of the explicit instructions of the EU Parliament (2024), which called for stronger precautionary measures, transparency, and accountability in the design and deployment of digital technologies to safeguard human rights, democratic oversight, and consumer protection within the EU single market. On the other hand, the EU has recently been on the receiving end of considerable criticism from key industry actors in Europe and beyond, who claim overregulation is killing competition, supposedly leaving industry vulnerable and driving skilled professionals to Silicon Valley (Haeck 2025). At the same time, concerns about a potential generative AI bubble burst have been raised by industry leaders and governments (Makortoff 2025), sowing fear of an economic collapse. The AI First policy can thus be understood as a response to mounting pressure to increase investment, reduce regulatory constraints, and accelerate AI deployment across society. In doing so, the European Commission has effectively adopted a full-steam-ahead approach to AI, yet without the coherence, governance frameworks, and people-centric orientation necessary to ensure that such acceleration aligns with Europe’s foundational values and long-term public interests. The European Commission must also recognise that framing AI development as a global race is both misguided and counterproductive, because such a narrative reduces a complex societal transformation to a contest of speed, rather than a question of direction, purpose, and public value. Moreover, Europe will not win any AI race. The US is too dominant in the currently popular massive, centralised approaches to AI, with the EU being too dependent on the US for the tech stack that allows the most pervasive forms of AI to function. AI leadership and digital sovereignty will not come from a fragmented approach where Europeans are told to see if and where AI can be wedged into sectors and society at large. Strategic leadership, a focus on key areas of innovation, how Europe’s limited resources can be used to maximise both economic growth and social good are key. An exploratory and human rights-driven alternative is more suitable and aligned with Europe’s values and aims than trying to keep pace with the US at any cost. An AI First policy will only further fragmentation, increase inefficiencies, undermine the EU’s competitive advantage and increase its dependency on non-European actors.

This is a pivotal moment to reflect not only on how we govern AI in the EU, but why we are developing and deploying it in the first place. Too often, we see technology placed before purpose, and innovation before inclusion. So, if not AI First, what is the right question? And how can poorly resourced actors, such as SMEs, civil society, universities, small EU countries and those in the global south with limited AI literacy make this assessment?

Not AI First, But AI Where It Is The Best Solution

Rather than presuming that AI, as claimed by the Commission’s President Ursula von den Leyen, will inevitably deliver “smarter, faster, and more affordable solutions’” (von der Leyen 2025), Europe must first determine where, and whether, AI genuinely serves societal needs.

That is, we must start with Question Zero: Why AI? (Lindstrom 2025). What problem are we trying to solve? Is AI truly the right or only solution for each case where it is being applied or promoted? Who benefits, and who bears the costs? By asking these simple questions, we quickly realise that sometimes, not always, AI is the answer. This approach offers a quick, low-cost way to assess AI’s relevance, especially useful for poorly resourced actors, who are often expected to adopt AI without sufficient AI literacy, resources, or support.

Europe As An AI Leader

AI is not inevitable, nor is its current trajectory predetermined. The EU has real choices to make. As such, the EU need to focus their efforts on actively navigating the correct path forward, rather than assuming that the choices have been made for them, and the only thing they can do is try to catch up. This ability to make choices is what digital sovereignty really means. Having the ability to decide over our futures. While the Commission’s suggestion of AI First may miss the mark, the EU retains the power to define when and how AI should be used, and, vitally, when it should not. By doing so, the EU can lead not through speed, but through purpose, setting a global example of responsible innovation that strengthens independence, upholds democratic values, and turns digital sovereignty into a shared regional strength.

References

Adam Dahlgren Lindström, Virginia Dignum, Petter Ericson, Tatjana Titareva, and Jason Tucker. 2025. Responsible AI Self-assessment Workshop: Start with Question Zero. https://aipolicylab.se/2025/09/05/responsible-ai-self-assessment-workshop-start-with-question-zero/ AI Policy Lab, Published September 5, 2025. Accessed October 9, 2025.

European Commission. 2025a. Apply AI Strategy. Shaping Europe’s Digital Future. https://digital-strategy.ec.europa.eu/en/policies/apply-ai Accessed October 9, 2025.

European Commission. 2025b. Communication from the commission to the European Parliament and the Council. Apply AI Strategy (Brussels, 8.10.2025, COM(2025) 723 final). https://ec.europa.eu/newsroom/dae/redirection/document/120429

European Commission. 2025c. EU launches InvestAI initiative to mobilise €200 billion of investment in artificial intelligence. https://digital-strategy.ec.europa.eu/en/news/eu-launches-investai-initiative-mobilise-eu200-billion-investment-artificial-intelligence Last update: 12 February 2025; Accessed October 9, 2025.

European Commission. 2025d. The future of European competitiveness. Part A: A competitiveness strategy for Europe. https://commission.europa.eu/document/download/97e481fd-2dc3-412d-be4c-f152a8232961_en?filename=ThefutureofEuropeancompetitiveness_AcompetitivenessstrategyforEurope.pdf European Parliament. 2024. Addictive design of online services and consumer protection in the EU single market (P9 TA(2023)0459). Official Journal of the European Union C/2024/4164 (2024). https://eur-lex.europa.eu/eli/C/2024/4164/oj/eng Accessed October 9, 2025.

Wearden, Graeme. 2025. Entry-level workers face AI ‘job-pocalypse’; US probes Tesla’s self-driving system – as it happened. The Guardian (2025). https://www.theguardian.com/business/live/2025/oct/09/water-customers-bill-hike-winter-blackouts-risk-falls-stock-markets-pound-ftse-business-live-news Retrieved October 9, 2025.

Makortoff, Kalyeena. 2025. Bank of England warns of growing risk that AI bubble could burst. The Guardian (2025). https://www.theguardian.com/business/2025/oct/08/bank-of-england-warns-of-growing-risk-that-ai-bubble-could-burst Accessed October 2025.

Haeck, Pieter. 2025. Dutch chips giant ASML executive Roger Dassen slams EU AI overregulation. https://www.politico.eu/article/dutch-chips-giant-asml-executive-roger-dassen-slams-eu-ai-overregulation/ Accessed October 9, 2025.

Estrada, Sergio. 2025. MIT Report: 95% of Generative AI Pilots at Companies Are Failing. Fortune (2025). https://fortune.com/2025/08/18/mit-report-95-percent-generative-ai-pilots-at-companies-failing-cfo/ Retrieved October 9, 2025.

Von der Leyen, Ursula. 2025. From now on it’s “AI First” — today the European Commission launches its new approach to artificial intelligence. LinkedIn post. Available at: https://www.linkedin.com/posts/ursula-von-der-leyen_from-now-on-its-ai-first-today-the-activity-7381720419516452864-2aUp.