Operationalising AI Ethical Principles in Higher Education with the UNESCO Recommendation on the Ethics of Artificial Intelligence by Policy Makers and Global Universities

Executive summary

The rapid use of artificial intelligence (AI) tools in higher education creates new governance challenges. Universities worldwide face growing concerns about safety, privacy, unfair algorithms and threats to learning process and academic integrity. Without proper governance systems, AI tools’ adoption risks increasing inequalities, reducing learning quality and violating the fundamental rights of students and educators.

The UNESCO’s “Recommendation on the Ethics of Artificial Intelligence” adopted in 2021 represents the first global standard on AI ethics and applies to 194 UNESCO member states. It provides a universal framework of values, principles and actions to guide development and use of AI systems and to protect human rights in the AI era. However, putting these principles into practice requires approaches that reflect the diverse social, cultural and political realities across different countries and institutions.

This policy brief examines AI governance strategies across 16 countries on five continents: South America (Chile, Colombia, Ecuador), Africa (Egypt, Ghana, Kenya, Morocco, Rwanda, South Africa, Tunisia), Asia (China, Kazakhstan, United Arab Emirates), Europe (Sweden,  Latvia) and North America (United States). The analysis maps national regulations and institutional practices against UNESCO’s ten core ethical principles. It shows how governments and higher education institutions are implementing global standards in practice in ways relevant to their local contexts.

This analysis leads to five priority recommendations for policymakers and higher education leaders. First, adopt human rights-based AI frameworks with clearly defined red lines. These should align with international human rights standards and link to higher education institutions accreditation requirements. Second, maintain responsible data governance applying the principle of privacy-by-design. Prohibit unauthorised entry of personal data into third-party tools. Conduct impact assessments for high-risk systems. Third, strengthen human oversight, transparency and accountability. Ensure all important AI decisions undergo human review. Fourth, make AI literacy universal and relevant across all disciplines. Embed ethics, critical thinking and responsible use across all programmes both in technical and non-technical disciplines. Support this with funded educator training programmes. Fifth, conduct ethical procurement and evaluation throughout the AI lifecycle. Contracts should include transparency clauses and provisions to prohibit the use of systems that violate ethical standards.

The ethical deployment of AI tools in higher education demands a multi-level commitment: users must possess adequate understanding of these tools, institutions must establish transparent guidelines and enforce them through monitoring mechanisms, and governments must provide resources for capacity development. This necessitates integrated action combining bottom-up institutional initiatives with top-down government legislation and technical support. Implementation requires coordinated action from policymakers, institutional leaders, administrators, faculty, students and community partners. Shared governance and collaboration across disciplines are essential. 

Background. The adoption of artificial intelligence (AI) tools in higher education is at the forefront of conversations worldwide. Universities have become testbeds for generative AI (GenAI) tools in teaching, research and administration (European Commission, 2022). The growing debate on AI in education spans a continuum from innovation to regulation: at one end, those who advocate rapid implementation of AI tools; at the other, those who raise alarms about the existential risks of AI. In the middle, a third group of voices advocates ethical principles and academic standards for the adoption of AI systems that are fair, safe and inclusive for all higher education stakeholders. 

Drawing on diverse voices of higher education institutions (HEI) leaders and researchers from five continents, this policy brief examines AI governance strategies, policies and practices across 16 countries. At the macro level, we review the most prominent national AI regulations, standards and ethical guidelines informing HEIs governance. At the micro level, some exemplary practices at selected HEIs within each country. Recognising that AI governance reflects the unique local social, cultural and political realities of each country, the cases presented aim to encourage reflection and dialogue on what is relevant and has been adopted across different contexts.  

Problem statement. The rapid adoption of AI systems, especially GenAI tools, in higher education presents complex global challenges. Leaders in HEIs are increasingly confronting the reality of AI-assisted work in classrooms and administrative operations. Applications range from personalised content to meet learners’ needs to tutoring support, enrollment processing and AI-enabled chatbots for student support (European Commission, 2022). 

Rising concerns about safety, disinformation, privacy, bias, the absence of transparency, accountability and human oversight pose challenges for governance, equity and inclusion (UNESCO, 2025).  The use of AI-enabled tools in the classroom is associated with mental health impacts, reduced trust in institutions and overreliance on AI systems, with effects on learning quality and student retention (World Bank, 2025). The use of GenAI tools raises concerns about the authenticity of learners’ work, the erosion of their critical thinking skills, the lack of depth in their learning and the loss of human connectedness (Bozkurt et al., 2024). Broader risks to inclusivity arise when AI tools are trained on data from a narrow set of contexts, which can erase diverse perspectives and reinforce stereotypes rooted in specific geographic regions (Miao and Holmes, 2023). The absence of governance mechanisms to ensure that AI tool selection aligns with educational goals, safety and fairness poses an additional challenge for equity and inclusion (OECD, 2024). 

Policy analysis. The implementation of AI ethics in higher education institutions worldwide is as diverse as the countries in which they are located, the types of HEIs, their constituencies and their socio-economic contexts. While national governments and international organisations are increasingly adopting global policy and ethical frameworks to govern AI (CAIDP, 2026), the implementation of these frameworks into ethical principles, policies and regulations reflects their own governance structures, cultural norms and digital readiness. 

The purpose of this policy brief is to share global examples reflecting on: 

(1) How are policymakers and HEIs leaders operationalising the UNESCO Recommendations on the Ethics of AI? 

(2) How can AI ethics, safety and inclusivity in higher education be both globally informed and locally grounded?  

The 16 countries under study constitute a non-exhaustive sample across diverse continents. The analysis is organised according to the ten core principles of the Recommendation and maps national laws, policies and university guidelines. The analysis also considers actions undertaken by countries or specific HEIs, within Policy Area 8: Education and Research, which underscore commitments to adequate AI literacy to enable individuals to make informed decisions about the use of AI systems (para. 101), development of AI ethics curricula (para. 106), support for AI ethics research (para. 107), training of AI researchers on AI ethics, and ensuring critical evaluation of AI research through independent scientific research (para. 110-111) (UNESCO, 2022).

Principle 1: Proportionality and Do No Harm operationalised as Human Rights-Based Frameworks for Harm Prevention 

The use of AI systems must be proportionate to the legitimate aim, should not infringe on human rights and should be scientifically based and appropriate to the context. Provisions must be in place for rigorous risk assessment to prevent and mitigate harm, with final human determination in irreversible or life-and-death decisions (UNESCO, 2022, para. 25 – 26). Ethical  values  and  principles  can  help  develop  and  implement  rights-based  policy measures and legal norms.  

Risk classification systems were enacted in the “European Union (EU) Artificial Intelligence Act” (EU AI Act), the first legally binding AI regulation globally relevant to the EU member states (European Union, 2024). The use of AI systems in admissions, assessment and student monitoring is classified as high risk, ‘with strict requirements, including risk-mitigation systems, high-quality data sets, logging of activity, detailed documentation, clear user information, human oversight, and a high level of robustness, accuracy and cybersecurity’ (European Commission, 2024, para. 4). 

In Ecuador, the proposed bill  “Proyecto de  Ley Orgánica de Regulación y Promoción de la Inteligencia Artificial en Ecuador” sets out a four-tier AI risk classification system, including a ban on extreme-risk systems and on the processing of personal data for purposes incompatible with human rights. The Bill proposes the requirement for public registries for high-risk AI applications (Asamblea Nacional, 2024). In China, the ”Ethical Norms for the New Generation of Artificial Intelligence” (2021) include requirements for the strengthening of risk monitoring systems, the establishment of user exit and feedback mechanisms (Ministry of Science and Technology of the People’s Republic of China, 2021). 

Principle 2: Safety and Security operationalised as Requirements for Technical Robustness and Contingency Plans

Plans must be in place to prevent and address unwanted harms (safety risks) and vulnerabilities to attacks (security risks) throughout the technology life cycle, thereby establishing sustainable, privacy-protective data frameworks for the training and validation of AI models (UNESCO, 2022, para. 27).

In the EU, high-risk AI systems require technical robustness and resilience plans to prevent harmful or undesirable behaviours that adversely affect fundamental rights. This includes fail-safe plans to interrupt the operation when anomalies exist (EU AI Act, 2024, Art. 15).  In Ecuador, the proposed bill  “Proyecto de  Ley Orgánica de Regulación y Promoción de la Inteligencia Artificial en Ecuador” (Asamblea Nacional, 2024, Art. 4d) includes mandates for rigorous tests, human rights impact assessments and safeguards against unintended use and adverse effects. Morocco’s national cybersecurity strategy requires regular penetration testing and intrusion detection systems to prevent harm from misuse or attacks (Hespress, 2025). In China, the ”Ethical Norms for the New Generation of Artificial Intelligence” requires AI systems to be verifiable, auditable, supervisable, traceable, predictable, trustworthy (Art 12), as well as tasks organisations to research and develop emergency mechanisms and loss compensation plans (Ministry of Science and Technology of the People’s Republic of China, 2021, Art 17). 

Principle 3: Right to Privacy and Data Protection operationalised as Adequate Personal Data Protection Measures

The use of AI systems must align with appropriate data protection and governance mechanisms for the collection, use, sharing, archiving and deletion of personal data.  In education, provisions must be in place to ensure that the use of AI supports learning without the extraction of sensitive information or the misuse, misappropriation or criminal exploitation of data collected in interactions with users (UNESCO 2022, para. 32, 33, 34 and 104). 

In Morocco, the General Secretariat of the Government is reviewing the “Digital X.0”, a draft framework law that will govern AI, data and digital identity, with three core priorities: data governance, digital identity and interoperability (Secrétariat Général du Gouvernement, 2025). Digital X.0 will overhaul the provisions of the existing personal data law09-08 (DGSSI, 2009). In Kazakhstan, the new “AI Law” ‘bans harmful practices such as behavioral manipulation, exploitation of vulnerabilities, emotion recognition without consent, social scoring and illegal data collection’ (Omirgazy, 2025a). In Sweden and Latvia, university policies prohibit uploading personal or copyrighted data to third-party AI tools, in accordance with the General Data Protection Regulation (GDPR) requirements (KTH Royal Institute, 2025a; University of Latvia, 2024/2026).

Principle 4: Multi-stakeholder and Adaptive Governance & Collaboration operationalised as Multi-level Collaborative Governance Approaches and Mechanisms

Diverse stakeholders must be involved in governance of AI throughout the AI system life cycle, including governments, intergovernmental organisations, the technical community, civil society, researchers and academia, media, education, policy-makers, private sector, human rights institutions, anti-discrimination monitoring bodies and groups for youth and children. Measures must be in place to enable meaningful participation by marginalised groups, communities, and individuals, and to respect Indigenous Peoples’ self-governance of their data (UNESCO, 2022, para. 47).

The government of Kazakhstan engaged 13 state bodies in developing the first “AI law” to ensure the safe, transparent and ethical use of AI (Omarova, 2025). In Ecuador, the Ministry of Information and Telecommunications (MINTEL) proposed the creation of a collegiate entity with multisectoral representation and administrative independence to govern decisions on AI capability, data management and competitiveness (Albornoz & Gualavisi, 2025). In Morocco, in preparation for the “Draft Law 59.24 on Higher Education and Research”, the Ministry of Higher Education involved universities, research centres, regulatory bodies and NGOs in a coordinated, participatory, and flexible oversight of technology and the broader research ecosystem (Islah, 2025). The University of Los Andes and the Externado University in Colombia host multi-stakeholder roundtables on AI regulation with the participation of academia, civil society and government (University de los Andes, 2025). 

Principle 5: Responsibility and Accountability operationalised as Accountability and Liability Guidelines

The ethical responsibility and liability for decisions and actions based on an AI system should ultimately be attributable to AI actors, in proportion to their roles in the AI system’s lifecycle (UNESCO, 2022, para. 42). Oversight, impact assessments, audit, due diligence, and whistleblower protections must be in place to ensure accountability (para. 43). In learning, strict requirements for monitoring, assessing abilities and predicting learners’ behaviours must be in place (para. 104). Rigorous, independent, multidisciplinary scientific research must ensure a critical evaluation of AI research and proper monitoring of potential misuses or adverse effects (para. 110).

In China, the ”Ethical Norms for the New Generation of Artificial Intelligence”  prescribe that humans are the ultimately responsible entities and must ‘comprehensively heighten awareness of responsibility, and exercise self-reflection and self-discipline at every stage of the AI life cycle’ (Ministry of Science and Technology of the People’s Republic of China, 2021, Art. 3). In the UAE,  theCharter for the Development and Use of Artificial Intelligence(2024) defines accountability as a policy objective and the UAE International Stance on AI Policy (2024) emphasises the need for accountability mechanisms to address any potential violations in AI use (priority 2). 

In Kenya, Aga Khan University established GenAI guidelines to guide the transparent and responsible integration of AI tools into academic and administrative processes (Aga Khan University, 2023). In Kazakhstan, Nazarbayev University establishes the levels of responsibility for the integration of GenAI into learning and teaching, including response, position, acceptable use statements, dissemination, supervision and reporting (Nazarbayev University, 2023). In Latvia, the Guidelines for the Use of AI in the University of Latvia(2024, updated in 2026) define that authors of AI-assisted work ‘still are and remain responsible for the originality of content, truthfulness of the facts and validity of the statements’ (p. 8). In Colombia, Universidad de los Andes states that users bear the ‘responsibility to verify the pertinence, accuracy and veracity of the material generated by GenAI’ (Universidad de los Andes, 2024, section 2.2.1). 

Principle 6: Transparency and Explainability operationalised as Mandatory Disclosure and Contestability Frameworks

AI actors must ensure transparency and explainability as fundamental requirements for accountability and governance. AI systems must operate in ways that are traceable and understandable to those they affect, with disclosure of when and how systems are used, and with provisions to challenge decisions and outcomes that affect their safety or human rights (UNESCO, 2022, para. 37 – 41).

In China, the ”Ethical Norms for the New Generation of Artificial Intelligence” establish the obligation to improve transparency, explainability, understandability in the design, implementation and application of algorithms (Ministry of Science and Technology of the People’s Republic of China, 2021, Art. 12). In Morocco, AI processing of personal data is governed by the “Data Protection Law 09-08”  (CNDP, 2009) which grants citizens the right to seek recourse against automated decisions. CNDP holds hearings with scientific organisations to advance the implementation of the law (CNDP, 2025). 

In Kazakhstan, the guidelines of “Nazarbayev University’s Response to Generative Artificial Intelligence in Learning and Teaching” request faculty and students to disclose the use of AI tools and their details in the design, delivery and assessment of courses, as well as assignment completion (Nazarbayev University, 2023). In Sweden, KTH Royal Institute’s “GenAI Guidelines for studying with GenAI” encourage students to be transparent in whether, how and why students use the AI tools (KTH Royal Institute, 2025a). 

Principle 7: Human Oversight and Determination operationalised as Non-Negotiable Human Determination

Human oversight requires the attribution of ethical and legal responsibility at any stage of the AI lifecycle, as well as the provision of remedies to physical persons or existing legal entities. In cases where humans decide to rely on AI systems for efficacy, the decision to cede control remains with humans. AI systems can never replace ultimate human responsibility and accountability. Life-and-death decisions should not be ceded to AI systems (UNESCO, 2022, para. 35-36).

In the UAE, the “Charter for the Development and Use of AI” (2024) protects the ‘irreplaceable value of human judgment and human oversight over AI to correct any errors or biases that may arise’ (Principle 6). In China, the ”Ethical Norms for the New Generation of Artificial Intelligence” (2021) stipulate the principle of ‘strengthening responsibility’, whereby humans are the ultimate responsible party and should not avoid accountability review or evade responsibilities (Ministry of Science and Technology of the People’s Republic of China, 2021, Art 3). 

In Sweden, the “GenAI Guidelines for studying with GenAI” purposes at KTH Royal Institute establish the ‘human-in-the-loop’ principle whereby students are ‘responsible for all decisions and assessments of what is appropriate and correct’ (KTH Royal Institute, 2025a, Guideline 3), as well as human educators should be the final decision makers about the AI systems used in educational institutions (Jonkoping University, 2023).

Principle 8: Sustainability operationalised as Sustainable Development Goals (SDG) Alignment 

Continuous assessments of the human, social, cultural, economic and environmental impacts of AI technology must be in place, aligned with the evolving goals of the Sustainable Development Goals (SDGs) (UNESCO 2022, para. 31). Data, energy and resource-efficient AI methods must be prioritised. Should there be disproportionate “negative impacts on the environment, AI should not be used” (para. 86). Rigorous and independent scientific research that incorporates interdisciplinary perspectives beyond science, technology, engineering and mathematics must be ensured (UNESCO, 2022, para. 110). 

In Sweden, the KTH Royal Institute (2025b) provides resources on the energy consumption and ecological footprint of generative AI. In China, the discussion of AI and SDGs is active through academic research projects, conferences and reports (The Chinese University of Hong Kong, Shenzhen, n.d.). In Colombia, the guidelines for the use of generative AI (“Lineamientos para el uso de inteligencia artificial generativa (IAG)”) at the University de los Andes stipulate that the use of GenAI must be coherent with and contribute to the SDGs (Universidad de los Andes, 2024)

Principle 9: AI Awareness and Literacy operationalised as Transdisciplinary AI Literacy Initiatives

Governments, academia, civil society, media and the private sector must work jointly on accessible education, civic engagement and AI literacy initiatives so that members of society can make informed decisions and be protected from the undue influence of AI systems (para. 44). Learning must focus on the impact of AI on human rights and access to rights, as well as on the environment and ecosystems (para. 45). Pre-requisite skills for AI education must include an AI ethics curriculum (para. 101-102), aligned with national education programmes and traditions, developed in local and indigenous languages and accessible to persons with disabilities (para. 106) (UNESCO, 2022).

The “EU AI Act” requires AI literacy for providers and deployers (European Union, 2024, Art. 4). In the UAE, the Charter for Development and Use of AI” (2024) stipulates as one of the priorities and key components the promotion of AI awareness (p. 11). In Morocco, “National Strategy “Digital Morocco 2030” invites scientists to ‘engage in multidisciplinary research (digital, legal, sociological, etc.) to reflect on the development of Al and its impact on society’, as well as ‘include digital education as early as primary school’ (Ministry of Digital Transition and Administrative Reform, Morocco, 2024, p. 12 – 13). The UAE Education, Human Development, and Community Development Council established a compulsory AI curriculum across K-12, with 25% of the content devoted to AI ethics (Rasheed, 2025).   Kazakhstan is integrating AI into national curricula at multiple levels (Omirgazy, 2025b). In Rwanda, the National AI Policy includes 21st-century skills and high-level literacy as enablers of the strategy (Republic of Rwanda, Ministry of ICT and Innovation, 2022). Public AI literacy is a key element of Kenya’s national AI strategy (Kenya AI Strategy 2025 – 2030). 

Tsinghua University in China launched an internal AI Literacy website that focuses on the fundamentals and applications of generative AI in teaching and learning, with a dedicated section on the “Responsible Use of Generative AI” with key principles for academic and ethical practice (Tsinghua University, 2025). In the United States, the State Council for Higher Education in Virginia (SCHEV) awarded a grant to a consortium led by Virginia State University, a historically Black institution, Northern Virginia Community College, a major community college, and Brightpoint Community College, to expand AI instruction in dual-enrollment and transfer-pathway programmes. The programme includes AI micro-credentials for professional advancement and open educational resources for wider use across the state’s higher education sector (Northern Virginia Community College, 2025; Virginia State University, 2025). 

Principle 10: Fairness and Non-Discrimination Operationalised as Ex-ante Human Rights Impact Assessments and Remediation

AI actors must promote social justice and safeguard fairness and non-discrimination in line with international law. Inclusive approaches in AI development must consider the specific needs of diverse constituencies to the benefits of AI, regardless of race, colour, gender, age, disability or other grounds. AI actors must make all reasonable efforts to minimise and avoid reinforcing discriminatory or biased applications and outcomes and ensure effective remedies against algorithmic harms (UNESCO, 2022, para. 28-30).

The “African Union Continental Strategy” includes plans for legal protection against algorithmic bias and discrimination, including the need to ‘compensate for bias and discrimination’ (p. 21-22). Member States committed to develop and roll-out AI assessments including the UNESCO Ethical Impact Assessment (EIA) (African Union, Continental AI Strategy, 2024). In Ecuador, the draft bill “Proyecto de  Ley Orgánica de Regulación y Promoción de la Inteligencia Artificial en Ecuador”, proposes the prohibition of algorithmic discrimination against vulnerable groups and historically disadvantaged groups (Asamblea Nacional, 2024, Art. 4b and 29). In Morocco, ethical risk-assessment mechanisms aligned with the UNESCO Recommendation and the “EU AI Act”, will be part of the national AI roadmap announced by the Ministry of Digital Transition and Administrative Reform (Amer, 2025).

Ethical impact assessments are recommended by the University of Latvia, before the adoption of AI-enabled products, to protect users against manipulation and embedded biases (University of Latvia, 2024/2026). In the United States, the Center for Responsible AI (CRAI) at Virginia State University, focuses on the ethical adoption of AI to ensure that ‘AI amplifies opportunity and inclusion, not division’ (n.d.). 

Recommendations

Building on the policy analysis above, in this section we offer five recommendations for policymakers and higher education stakeholders. They are grounded in the Recommendation. The aim is to translate high-profile principles into practice with consideration that ethical AI in higher education has to be both globally informed and locally relevant.

1) Adopt human rights-based AI frameworks with clearly defined red lines. We encourage policymakers to align national or regional frameworks with international human rights standards, and universities to provide clear guidance on the classification of the educational uses of AI tools according to levels of risk. Higher education stakeholders can implement this through institutional risk registers, ethical impact assessments and responsible procurement processes. It is important to ensure that no AI system is adopted without adequate human oversight or harm prevention mechanisms. These measures should be linked to HEIs accreditation standards and public reporting requirements. Universities can look for existing templates and guidance for institutional self-assessment for responsible AI adoption (AI Policy Lab, Umea University, 2025; UNESCO, 2023).

2) Maintain responsible data governance and privacy by design. Protecting personal and sensitive information must be a non-negotiable foundation of any ethical AI policy in higher education. Policymakers should update national privacy guidance for education. It should guarantee that the privacy protection mechanisms go beyond legal compliance. Universities should prohibit the entry of personal or copyrighted data into third-party AI tools unless approved and secure. They should also carry out data protection impact assessments for high-risk AI systems used in teaching, assessment and administrative functions. Additionally, HEIs need to promote secure internal tools, train staff and students on privacy and maintain transparency on how data are collected, stored and deleted. 

3) Reinforce human oversight, transparency and accountability. All high-stakes AI-assisted decisions, including admissions, grading or disciplinary procedures should be subject to human review. Students and staff must be informed when AI tools are used and have the right to contest and request explanations for decisions affecting them. Institutions can lead by example of transparency by publishing databases of the AI systems they use, including information about their purpose, model design and safeguards. 

4) Make digital literacy, including AI literacy, universal and contextual across disciplines. AI literacy should not be limited to computer science but embedded in all programmes with an emphasis on critical thinking, ethics, data protection and the psychological effects of AI on learning and mental health. This requires coordinated policy support to fund educator training, micro-credentials and professional development programmes. Universities should design curricula that integrate AI ethics and responsible innovation into existing courses. It is important to stimulate educational stakeholders to learn to question, interpret and responsibly use AI outputs. 

5) Conduct ethical procurement and evaluation of AI systems throughout the AI lifecycle. Procurement offices should act as gatekeepers of trustworthy AI systems, conducting ethical impact assessments before purchase and throughout the use (Hickok, 2024; UNESCO, 2023). Contracts with AI vendors should include transparency, access and termination clauses. It is important that institutions can discontinue systems that no longer meet ethical standards or where human oversight is impossible. National and regional policymakers can strengthen this process by tying funding and accreditation to proof of ethical procurement and periodic evaluation.

Conclusion

AI tools in higher education cannot be ethical unless users understand how to use them, institutions set clear expectations (guidelines), enforce and monitor them, as well as governments support capacity building in educational settings. This requires both top-down (government legislation and technical support) and bottom-up (institutional action) approaches. For example, using ethical impact assessments before purchasing AI tools, as well as shared agency among faculty, administrators, students and community partners. Ethical AI in higher education must be inclusive, adaptive and globally informed, but locally grounded.

Acknowledgements. This policy brief was produced under the United Nations Higher Education Sustainability Initiative (HESI) as part of the Implementation Group (IG) on “AI Ethics, Safety, and Inclusivity in Higher Education”. The authors appreciate the support of Ms Arianna Valentini and Lily Liu Bosen. The research findings in this policy brief reflect the work of the authors named above. They do not represent the opinion or position of any other organisation, including the United Nations. 

 

References

African Union. 2024, July 20. Continental Artificial Intelligence Strategy. Harnessing AI for Africa’s Development and Prosperity. Available at: https://au.int/sites/default/files/documents/44004-doc-EN-_Continental_AI_Strategy_July_2024.pdf (Accessed: 19 December 2025)

‌Aga Khan University. 2023. The Use of Generative AI in Higher Education at AKU (Draft Guidelines). (n.d.). Available at: https://www.aku.edu/admissions/Documents/policy-use-of-generative-ai.pdf (Accessed: 19 December 2025)

Albornoz, M. B. and Gualavisi, A. Dictamen Técnico sobre la Política Gubernamental aprobada por MINTEL. [Technical Assessment about the Government Policy approved by MINTEL]. Available at: https://unesdoc.unesco.org/ark:/48223/pf0000394822 (in Spanish) (Accessed: 20 December 2025)

Amer, O. M. 2025, Dec. Morocco Accelerates AI Strategy with New Regulations, JAZARI Institutes and Sovereign Cloud Push. Available at https://www.moroccoworldnews.com/2025/12/271574/morocco-accelerates-ai-strategy-with-new-regulations-jazari-institutes-and-sovereign-cloud-push (Accessed: 6 January 2026) 

American Association of University Professors (AAUP). 2025. Artificial Intelligence and Academic Professions. Available at https://www.aaup.org/reports-publications/aaup-policies-reports/topical-reports/artificial-intelligence-and-academic (Accessed: 8 October 2025)

Asamblea Nacional República del Ecuador. 2024, June 20. Proyecto de Ley Orgánica de Regulación y Promoción de la IA en Ecuador. [Draft Organic Law for the Regulation and Promotion of AI in Ecuador]. Available at: https://www.asambleanacional.gob.ec/ (in Spanish) (Accessed: 19 December 2025)

Bozkurt, A., Xiao, J., Farrow, R., Bai, J.Y., Nerantzi, C., Moore, S., Dron, J., Stracke, C.M., Singh, L., Crompton, H. Koutropoulos, A. and others. 2024. The manifesto for teaching and learning in a time of generative AI: A critical collective stance to better navigate the future. Open Praxis, Vol. 16, No. 4, pp. 487-513. https://search.informit.org/doi/pdf/10.3316/informit.T2025011300014191575969861

Center for AI & Digital Policy (CAIDP). 2026. AI and Democratic Values (CAIDP Index) 2026. Available at https://www.caidp.org/reports/caidp-index-2026/ (Accessed: 17 August 2026)

Commission Nationale de Contrôle de la Protection des Données à Caractère Personnel (CNDP). 2025. Communiqué : IA et protection des données à caractère personnel. Available at: https://www.cndp.ma/wp-content/uploads/2025/03/CNDP-Communique-IA-et-protection-des-donnees-a-caractere-personnel-FR.pdf (in French) (Accessed: 19 December 2025)

Commission Nationale de Contrôle de la Protection des Données à Caractère Personnel (CNDP). 2009. Loi n° 09-08 relative à la protection des personnes physiques à l’égard du traitement des données à caractère personnel [Law No. 09-08 relating to the protection of natural persons with regard to the processing of personal data.]. Available at https://www.cndp.ma/loi-09-08/ (in French) (Accessed: 19 December 2025)

General Directorate of Information Systems Security (DGSSI). 2009. Loi n° 09-08 relative à la protection des personnes physiques à l’égard du traitement des données à caractère personnel [Law No. 09-08 relating to the protection of natural persons with regard to the processing of personal data]. Available at: https://www.dgssi.gov.ma/fr/loi-09-08-relative-la-protection-des-personnes-physiques-legard-du-traitement-des  (in French) (Accessed: 19 December 2025)

Government of the UAE. 2024, June 10. UAE Charter for the Development and Use of Artificial Intelligence. UAE Legislation. Available at: https://uaelegislation.gov.ae/en/policy/details/the-uae-charter-for-the-development-and-use-of-artificial-intelligence (Accessed: 19 December 2025)

Government of the UAE. 2024, September 2. UAE’s International Stance on Artificial Intelligence Policy. UAE Legislation. Available at: https://uaelegislation.gov.ae/en/policy/details/uae-s-international-stance-on-artificial-intelligence-policy (Accessed: 19 December 2025)

Grūzītis, N., Skadiņa, I., Baranova, S. and Nerenberga, J.  Updated in 2026, January. The Guidelines for the Use of Artificial Intelligence at the University of Latvia.  University of Latvia. Available at: https://www.lu.lv/fileadmin/user_upload/LU.LV/www.lu.lv/Studijas/guidelines_for_
AI_use_at_UL_study_process_2026.pdf
(Accessed: 17 August 2026)

Hespress. 2025, April 29. Morocco doubles down on cybersecurity and AI as digital push gains speed. Available at: https://en.hespress.com/109307-morocco-doubles-down-on-cybersecurity-and-ai-as-digital-push-gains-speed.html  (Accessed: 19 December 2025)

Hickok, M. 2024. Public procurement of artificial intelligence systems: new risks and future proofing. AI & Society, Vol. 39, No. 3, pp.1213-1227. https://link.springer.com/article/10.1007/s00146-022-01572-2 

European Commission, Directorate-General for Education, Youth, Sport and Culture. 2022. Ethical guidelines on the use of artificial intelligence (AI) and data in teaching and learning for educators. Publications Office of the European Union. Available at: https://data.europa.eu/doi/10.2766/153756 (Accessed: 15 October 2025)

European Commission. 2024. European Artificial Intelligence Act comes into force. Available at: https://ec.europa.eu/commission/presscorner/detail/en/ip_24_4123 (Accessed: 15 October 2025)

European Union. 2024. ‌ Artificial Intelligence Act (Regulation (EU) 2024/1689), Official Journal of the European Union. Version of 13 June 2024. Interinstitutional File: 2021/0106(COD). Available at: https://eur-lex.europa.eu/eli/reg/2024/1689/oj  (Accessed: 19 December 2025)

Islah, S. 2025. Enseignement supérieur : El Midaoui fait le point sur les nouveautés, les chantiers et les défis de l’Université. [Higher education: El Midaoui provides an update on the latest developments, projects, and challenges facing the University].  Le Matin. Available at: https://lematin.ma/enseignement/nouveautes-2025-2026-de-lenseignement-superieur/314556 (in French) (Accessed: 19 December 2025)

Jonkoping University, 2023, June 19. Artificial Intelligence Ethical Guidelines. Available at: https://ju.se/portal/educate/en/guides/artificial-intelligence/ethical-guidelines.html  (Accessed: 19 December 2025)

Kenya Ministry of Information, Communications and the Digital Economy. 2025, March. Kenya AI Strategy 2025 – 2030. Available at: https://ict.go.ke/sites/default/files/2025-03/Kenya%20AI%20Strategy%202025%20-%202030.pdf (Accessed: 19 December 2025)

Kingdom of Morocco. Ministry of Digital Transition and Administrative Reform (MMSP). 2024. National StartegyDigital Morocco 2030”. Available at: https://www.mmsp.gov.ma/sites/default/files/2024-09/PlaquetteInstitutionnel_18092024_Ang.pdf (Accessed: 19 December 2025)

KTH Royal Institute of Technology. 2025a, June 24. Considerations for studying with generative AI. Available at: https://www.kth.se/en/student/studier/kurs/generativ-ai/avvagningar-1.1410997 (Accessed: 19 December 2025)

KTH Royal Institute of Technology. 2026, July 17. Generative AI in higher education. Available at: https://intra.kth.se/en/utbildning/systemstod/generativ-ai/oversikt-1.1253698  (Accessed: 17 August 2026)

KTH Royal Institute of Technology. 2025b, April 10. AI and Sustainability- Towards a Fair Future. Documentation from Lunch’n’ Learn. Available at: https://intra.kth.se/en/utbildning/e-larande/e-larande-nyheter/ai-och-hallbarhet-mot-en-rattvis-framtid-1.1398761 (Accessed: 22 December 2025)

Miao, F. and Holmes, W. 2023. Guidance for generative AI in education and research. UNESCO.  Available at: https://unesdoc.unesco.org/ark:/48223/pf0000386693  (Accessed: 19 December 2025)

‌Ministry of Science and Technology of the People’s Republic of China. 2021. 《新一代人工智能伦理规范》发布. [Ethical Norms for the New Generation of Artificial Intelligence].  Available at:  http://www.most.gov.cn/kjbgz/202109/t20210926_177063.html  (in Chinese Mandarin) (Accessed: 19 December 2025)

Nazarbayev University. Nazarbayev University’s Response to Generative Artificial Intelligence in Learning and Teaching. 2023. The Center of Innovation in Learning and Teaching Office of the Provost. Available at: https://regulations.nu.edu.kz/bitstream/123456789/1030/1/NU%20Response%20to%20
Integrating%20Generative%20AI%20into%20Learning%20%26%20Teaching.pdf#page=3.08

(Accessed: 19 December 2025)

Northern Virginia Community College (NOVA). 2025, February 21. NOVA Collaborates on Computer Science Pathway with an AI Focus. Available at: https://www.nvcc.edu/about/news/featured-articles/2025/AI-grant-computer-science.html (Accessed: 19 December 2025)

OECD. 2024. The Potential Impact of Artificial Intelligence on Equity and Inclusion in Education. OECD Artificial Intelligence Papers. No. 23. Available at: https://www.oecd.org/content/dam/oecd/en/publications/reports/2024/08/the-potential-impact-of-artificial-intelligence-on-equity-and-inclusion-in-education_0d7e9e00/15df715b-en.pdf (Accessed: 15 November 2025)

Omarova, Z. 2025, April 8. Kazakhstan Seeks to Attract 10,000 AI Specialists Annually. The Astana Times. Available at: https://astanatimes.com/2025/04/kazakhstan-seeks-to-attract-10000-ai-specialists-annually/ (Accessed: 19 December 2025)

Omirgazy, D. 2025a, November 17. Kazakhstan Adopts Central Asia’s First AI Law. The Astana Times. Available at: https://astanatimes.com/2025/11/kazakhstan-adopts-central-asias-first-ai-law/ (Accessed: 19 December 2025)

Omirgazy, D. 2025b, September 18. Kazakhstan Establishes National Standards for Artificial Intelligence in Education. Available at: https://astanatimes.com/2025/09/kazakhstan-establishes-national-standards-for-artificial-intelligence-in-education (Accessed: 19 December 2025)

Rasheed, A. 2025, August 23. UAE schools to introduce AI curriculum in new academic year. Available at: https://gulfnews.com/uae/education/uae-schools-to-introduce-ai-curriculum-in-new-academic-year-1.500242831 (Accessed: 19 December 2025)

Republic of Rwanda. Ministry of ICT & Innovation (MICT). 2022. The National AI Policy. Available at: https://www.minict.gov.rw/fileadmin/user_upload/minict_user_upload/Documents/Policies/
Artificial_Intelligence_Policy.pdf
(Accessed: 19 December 2025)

Riga Stradins University. 2024. Artificial Intelligence in Higher Education Guidelines [English version]. Available at: https://www.rsu.lv/en/artificial-intelligence-higher-education  (Accessed: 19 December 2025) 

Secrétariat Général du Gouvernement (SGG). 2025, November 8. Journal de la semaine du 03 novembre 2025: Actualités nationales du numérique [Journal of the week of November 3, 2025: National digital news]. Available at: https://www.sgg.gov.ma/Portals/1/veille_juridique/actualite_numerique/aj_03112025.pdf (Accessed: 19 December 2025)

‌The Chinese University of Hong Kong in Shenzhen (n.d.). AI与可持续发展展望. [AI and Sustainable Development Outlook]. Available at: https://www.zhaojunhua.org/reports/AI_SUSTAINABLE_DEVELOPMENT_OUTLOOK_ZH.pdf (in Chinese Mandarin) (Accessed: 19 December 2025)

Titareva T., Tucker J., Carli R., Movchan V., Dignum V. 2026. Question Zero: Why Responsible AI Begins Before AI Adoption. AI Policy Exchange Forum (AIPEX). Available at: https://aipolicylab.se/aipex-contributions/question-zero-why-responsible-ai-begins-before-ai-adoption/ (Accessed: 17 August 2026)

Tsignhua University. 2025. Responsible Use of Generative AI. Available at: https://yuketang.tsinghua.edu.cn/ai/learning-center?selectNodeIds=60876,60878 (for internal use only). (Accessed: 19 December 2025)

UNESCO (n.d.). A human rights approach to AI. Ten core principles lay out a human-rights-centred approach to the Ethics of AI. Available at: https://www.unesco.org/en/artificial-intelligence/recommendation-ethics (Accessed: 19 December 2025)

UNESCO. 2022. Recommendation on the Ethics of Artificial Intelligence. Available at: https://unesdoc.unesco.org/ark:/48223/pf0000381137 (Accessed: 19 December 2025)

UNESCO. 2023. Ethical impact assessment: a tool of the Recommendation on the Ethics of Artificial Intelligence. Available at:  https://unesdoc.unesco.org/ark:/48223/pf0000386276 (Accessed: 19 December 2025)

UNESCO. 2025. AI and Education: Protecting the Rights of Learners. Available at: https://unesdoc.unesco.org/ark:/48223/pf0000395373 (Accessed: 19 December 2025)

Universidad de los Andes. 2024, October. Lineamientos para el uso de inteligencia artificial generativa (IAG) en la Universidad de los Andes. [Guidelines for the use of generative artificial intelligence (GAI) at the University of the Andes]. Available at: https://secretariageneral.uniandes.edu.co/images/documents/lineamientos-uso-inteligencia-artificial-generativa-IAG-uniandes.pdf (in Spanish) (Accessed: 19 December 2025)

Universidad de los Andes. 2025, August. El futuro del trabajo en Colombia en la era de la IA: hallazgos de la décima mesa multiactor sobre regulación de la IA. [Future of Work in Colombia in the AI era. Findings of the 10th Multi-stakeholder roundtable about AI regulation]. Available at: https://gobierno.uniandes.edu.co/el-futuro-del-trabajo-en-colombia-en-la-era-de-la-ia-hallazgos-de-la-decima-mesa-multiactor-sobre-regulacion-de-la-ia/ (Accessed: 19 December 2025)

Virginia State University (VSU). 2025, September 30. Virginia State University Awarded $2 Million Grant From Thurgood Marshall College Fund For AI-Enabled Digital Transformation.  Available at: https://www.vsu.edu/news/2025/vsu-awarded-tmcf-ai.php (Accessed: 19 December 2025)

Virginia State University. (n.d.). Center for Responsible AI (CRAI). Available at https://www.vsu.edu/about/leadership/vp-for-academic-affairs/crai/ (Accessed: 20 December 2025)

World Bank (2024). Global Trends in AI Governance. Available at: https://documents1.worldbank.org/curated/en/099120224205026271/pdf/P1786161ad76ca0ae
1ba3b1558ca4ff88ba.pdf
(Accessed: 19 December 2025)

 

Didactics before tools: Redesigning teaching and assessment in the age of generative AI

Abstract

Research and policy have become good at naming the cognitive risks that generative AI (GenAI) poses to learning. Namely, detrimental cognitive offloading, metacognitive laziness, and the illusion of competence. What this literature does much less well is tell teachers what to do with that knowledge in their everyday practice. This article argues that alongside AI literacy, regulation and safeguards protecting educational stakeholders, one of the central challenges facing education in the age of GenAI is related to didactic design. When a finished text or answer no longer reveals what a student has understood, struggled with, revised, verified or learned, teaching and assessment have to be redesigned so that the process of thinking is made visible and necessary again. We argue that this redesign should be approached through the classic didactic questions: why, what, how, when and for whom. We connect these questions with Question Zero (Q0), an approach to responsible technology adoption, through the dimensions of why, who, what, how and where. Both didactic and Q0 perspectives and underlying questions move the starting point from the technology to educational purpose, stakeholders and the conditions under which the use of selected technology is well-designed and justified. We argue that current AI strategies, guidelines and institutional frameworks in education leave a conspicuous gap precisely here. We translate these didactic and Q0 perspective questions into a concrete, phased design for teaching and assessment, whose guiding principle is not to forbid cognitive offloading but to regulate it didactically. We also argue that making the learning process visible is not in itself a neutral act, and that its design must attend to who carries its costs. This challenge recurs across education systems globally and cannot be solved by top-down regulation alone. Instead, translating high-profile principles of ethical AI in education into pedagogically sound use of GenAI requires the contextual and professional judgement of educators.

From cognitive risk to didactic response

Recent reports synthesise the evidence on AI and learning. A particularly useful contribution is the distinction between beneficial and detrimental cognitive offloading (Dunn & Risko, 2016; Risko & Gilbert, 2016; Lodge & Loble, 2026). AI can manage extraneous load and free working memory for the intrinsic work of learning, or it can bypass that intrinsic work altogether. Equally useful is the distinction between a task-completion goal and a learn-the-concept goal (Kalyuga & Plass, 2025). These distinctions give us a more precise language for what is at stake.

The problem is not simply that students might cheat, or that they might produce better answers than they could have produced alone. The more serious issue is that generative AI (GenAI) can interfere with the cognitive and metacognitive processes through which students learn (Fan et al., 2025). Students may complete the task without developing the knowledge, judgement or self-regulated learning capacity the task was meant to support. The most recent international synthesis reaches the same conclusion. The OECD (2026) finds that when students lean too heavily on GenAI, metacognitive engagement falls and task performance becomes misaligned with genuine learning.

The current empirical data and preliminary findings describe the risk well and give much less guidance on what teachers are supposed to do with this knowledge. How should they design teaching and assessment when the finished product no longer tells us very much about what the student has understood, revised, verified or learned? This is the question we want to place at the centre.

The pedagogical problem beyond AI literacy

Much of current education policy responds to GenAI by emphasising AI literacy and competence. We agree with their importance. But they do not constitute a holistic pedagogical response to cognitive outsourcing. Treating them as such risks diverting attention and resources from the difficult didactic work required (Weidlich et al., 2025). Knowing what a large language model is, and how to prompt it, does not by itself help a teacher decide how to assess understanding when fluent, confident output is available on demand. A recent article makes a parallel point: conventional, skills-based AI literacy frameworks remain anchored in functional competence and say little about the metacognitive and epistemic demands of probabilistic, opaque systems (Shapiro et al., 2026). That argument moves toward expanding literacy itself, into a critical, metacognitive and civic practice, whereas we argue the missing piece is the appropriate redesign of teaching and assessment.

In this opinion piece, we argue that the challenge of AI in education is fundamentally pedagogical rather than solely technological. The same tool produces opposite effects depending on the instructional design and the learner’s goals. A systematic review of 67 empirical studies confirms the pattern. Generative AI’s cognitive effects are not intrinsic to the tool but dependent on pedagogical framing, with scaffolded, inquiry-oriented use supporting higher-order thinking and unstructured use triggering cognitive offloading (Li et al., 2026). A risk taxonomy, however accurate, does not contain its own remedy. Naming “metacognitive laziness” tells a teacher what to fear, not how to handle it in the classroom.

The tension is structural rather than rhetorical. Cognitive offloading is, at its core, a way of reducing mental effort (Risko & Gilbert, 2016), and generative AI lowers the cost of such offloading further than any earlier technologies. Education rests on people working their way through uncertainty and complexity. This is not a superficial tension but a conflict between what the tool is built to optimise and what learning requires. Decades of research on desirable difficulties suggest that effort, struggle, and even the unpleasantness of thinking are often what produce durable learning (Bjork & Bjork, 2015; David et al., 2024), which is precisely the work that a tool designed to minimise effort invites students to skip. Early empirical work points the same way. In a study of 666 participants, more frequent use of AI tools was associated with weaker critical thinking, an association largely mediated by cognitive offloading (Gerlich, 2025).

If AI offers the fastest route to a finished product, then teaching has to make the process of thinking (i.e., explaining, revising, verifying, judging) more explicit. This is not a small adjustment. It touches the design of assignments, the role of feedback, assessment practices, the relationship between teacher and student, and students’ motivation to engage in the difficult and sometimes frustrating work that learning requires.

Several strands of assessment scholarship that predate generative AI are especially relevant here. Work on assessment reform for an age of GenAI argues for distinguishing between assessment that secures and validates individual achievement and assessment that develops capability through authentic, process-rich tasks (Lodge et al., 2023; Dawson, 2021). Work on formative assessment and feedback shows how assessment can be designed to drive self-regulated learning rather than merely certify a product (Nicol & Macfarlane-Dick, 2006; Carless & Boud, 2018; Hathcoat et al., 2026). And the notion of evaluative judgement (i.e., the capacity to judge the quality of work, one’s own and others’) names precisely the capability that detrimental offloading erodes and that teaching now has to cultivate deliberately (Tai et al., 2018; Bearman et al., 2024). The OECD’s synthesis points in the same direction. Effective integration, it argues, may require teachers to emphasise process, how students think and learn, rather than student output, and finds that hybrid systems pairing GenAI with explicit pedagogical models, such as structured tutoring or evidence-centred assessment design, show more promise than general-purpose chatbots (OECD, 2026). A 2025 UNESCO anthology reaches a similar conclusion, devoting a major section to assessment and arguing that standardised, output-focused testing is being destabilised by generative AI in favour of more human-centred, process-oriented forms of knowing (UNESCO, 2025b).

Making learning visible, but for whom?

There is a cautionary precedent worth recalling. Earlier efforts to make learning visible through digital tools did not automatically translate into changed teaching. A study of the introduction of mandatory personal development plans and digital documentation in Swedish compulsory schools found that, although the reform was meant to centre each student’s learning, students’ own goals had little effect on actual teaching. Responsibility for reaching them was largely devolved to the individual student, and the process drifted towards documentation, behaviour management and the production of the “model student” (Mårell-Olsson, 2012). Teachers, working under pressure, rationalised the task. For instance, teachers copied and adapted written assessments across students. The lesson is relevant to the current stage of GenAI adoption in education. Mandating a digital tool without redesigning teaching and assessment tends to create an administrative routine rather than reshape the learning process it was meant to support.

It is also important to be careful with the word ‘visible’. Making the learning process visible is not a neutral or automatically benign act. The study showed how documentation practices can turn into surveillance, control and the disciplining of students towards a prevailing norm, a dynamic that Bernstein’s distinction between visible and invisible pedagogy helps to name (Mårell-Olsson, 2012). Later studies of one-to-one computing classrooms found a similar dynamic, showing how teachers’ enacted didactical design decisions carry symbolic power and control over students (Bergström et al., 2019). Thus, designing for process-visible learning has to attend to the ‘for whom’. Visibility can support metacognition and self-regulation, or it can become a mechanism of compliance that weighs most heavily on the students with the least resources to navigate it.

The aim, therefore, is not visibility for its own sake but tasks designed so that the thinking serves learning rather than control, and so that students have a reason to understand the concept and apply it in the real world, rather than complete the task (see Lodge et al., 2023).

A didactic reframing: why, what, how, when, for whom

A further observation sharpens the design task. The evidence suggests that GenAI works best as a scaffold, not as a substitute. When it is used for Socratic questioning, feedback, alternative examples, concept explanations and support for self-regulation, it can support learning. When it is used to produce finished texts, solutions or analyses before the student has done their own cognitive work, the risk of shallow learning rises. A synthesis of research on curiosity and metacognition draws this distinction directly, warning that large language models’ default interaction modes can act as a cognitive shortcut but when redesigned, they can instead become a partner for sustained epistemic development (Desvaux et al., 2026). The design question, therefore, is not AI or no AI, but when in the learning process AI is used, for what purpose, and with what degree of student responsibility. That reframing is  what the didactic questions are built to handle.

We propose that AI in education be approached through the questions educators have long asked of any teaching decision. Why something should be taught, what learners need to know, how it should be taught, when in the course of learning, and for whom. These are the didactic questions of the Didaktik tradition (Klafki, 2000; Hopmann, 2007; Uljens, 1997). This tradition has already been extended to digital tools through, for example, the concept of digital didactical design, which provides an empirical framework for analysing how teachers’ design decisions shape technology-mediated learning (Jahnke et al., 2017).

Approached this way, GenAI becomes one more factor in teaching design, to be shaped by the same questions rather than settled by a blanket rule:

  • The why recovers the purpose of an assignment, which is often what makes offloading either harmless or harmful. This mirrors, at the level of the classroom, a broader argument that Europe’s strategy should move from an AI first to a purpose first posture, placing societal benefit and human-centric values ahead of acceleration for its own sake (Dignum et al., 2025). What purpose first is to AI policy, the didactic why is to teaching.
  • The what forces a distinction between knowledge that must be built in the student and knowledge that may legitimately be offloaded.
  • The how and when concern where in a learning sequence friction is desirable and where it is merely unnecessary.
  • The for whom is not incidental. The cognitive risks of AI fall hardest on novices and on students with weaker self-regulation, raising the prospect of a widening equity gap (Bastani et al., 2025). It is the same population, and the same equity concern, raised by the caution about visibility above. Making the process visible can support these students or, if designed carelessly, weigh most heavily on them. This is also why frameworks aimed only at upgrading teachers’ technical competence, important as they are (Redecker, 2017; UNESCO, 2024a, 2024b), need to be complemented by frameworks for pedagogical and content-specific judgement (Mishra & Koehler, 2006; Mishra et al., 2023).

Next, we introduce the Question Zero tool from the policy world with five overarching questions-dimensions related to thoughtful and responsible AI adoption and in what ways they overlap and complement the didactic questions.

Question Zero: why, who, what, how, where

Question Zero (Q0) from the policy world asks: “Under what conditions should an AI system be adopted, if at all?” (Dignum et al., 2026; Titareva et al., 2026).  Its purpose is to resist an “AI First” logic in which the technology is perceived as the only next step. And problems are identified later for the technology to solve. Instead, the problem and available alternatives (why?), affected stakeholders (who?), type of technology to solve a specific problem at hand (what?), and conditions of adoption (how and where?) should be considered before an AI tool is selected as the solution.

In education, a similar mistake is to begin with the assumption that generative AI will be used and then impose the decision on teachers and students to use it without pedagogically sound guidance. A didactic approach requires an earlier question: why we are using the tool and what educational problem are we trying to solve?

This is where Q0 and the classical didactic questions, coming from different disciplines and angles, become connected. Q0 is a governance question about responsible and thoughtful pre-adoption stage; the didactic why is a question about educational purpose. Both questions move the focus from technology as their starting point to the purpose. Q0 requires organisations to describe honestly the existing problem (not the solution/tool), consider human and non-AI alternatives, and justify why AI is appropriate before adopting it. The didactic why for teaching asks what an activity is intended to achieve (desirable teachers and learners’ goals) before the stakeholders decide what tools to use.

The connection between didactic questions and Q0 dimensions moves beyond why. Q0 operationalises responsible adoption through five iterative dimensions: why, who, what, how and where. These dimensions include: motivation for using the tool starting from the real problem identification and alternatives, affected stakeholders, the type and implications of the AI system, the adoption process, and infrastructure and monitoring. Didactic questions ask what happens when technology is imposed on teaching practices: what knowledge has to remain with the learner, when assistance should be provided, how much cognitive work may be offloaded, and for whom a particular design encourages or limits learning. The two approaches and questions are complementary and help make informed and weighted decisions prior to technology adoption in core activities, including in teaching and assessment.

Table 1

Didactic Questions vs. Q0 Dimensions

Didactic QuestionsQuestion Zero dimensions
Why?Why?
What?Who?
How?What?
When?How?
For whom?Where?

Why top-down regulation is not enough

It is tempting to respond to the resulting uncertainty with detailed, centralised rules, which is important for such complex systems and environments as education.  Indeed, the lack of concrete guidance does create a large space for interpretation, which will likely produce considerable variation in how AI is understood and used across classrooms and institutions (Mårell-Olsson, in press). However, education cannot be governed as if teaching was mainly a compliance task. Teachers need pedagogical freedom and the exercise of professional evaluated judgement, because the right use of AI is contextual. It depends on the subject, the stage, the goal and the student.

The current high-profile policy documents vividly demonstrate it. High-level, rights-based guiding documents such as UNICEF’s “Guidance on AI and Children” (2025) and UNESCO’s “AI and education: protecting the rights of learners” (2025a) are valuable precisely because they are general enough to apply across very different contexts. However, their authors are explicit that these guidelines must remain high-level, and they do not and cannot be easily applied at the level of each classroom. Even the European Commission’s dedicated, recently updated “Ethical guidelines on the use of AI and data in teaching and learning for educators” (European Commission, 2026), which now incorporates the AI Act and GDPR and offers core principles, guiding questions and classroom scenarios, is framed around ethical and responsible use and risk awareness rather than around the teaching redesign that the cognitive-offloading problem demands. The same is true of most national AI strategies and sectoral guidelines. They identify principles and risks. They do not, and arguably should not, prescribe teaching design. That work has to be done with and by teachers, not to them.

From high-profile principles to practice: designing teaching and assessment in GenAI times

Assessment has to begin by measuring process, judgement and independence, not only the finished product. The research and policy debate on AI and assessment points towards combinations of AI-free and AI-integrated components (Lodge et al., 2023; Dawson, 2021). AI-free components are needed to secure basic individual understanding. AI-integrated components are needed to assess how a student uses AI critically, ethically and in a subject-responsible way. These are complementary, not alternatives. Below we propose a framework for putting didactic/Q0 questions into practice in pedagogical settings in the age of generative AI.  


We propose designing teaching in phases:

Stage 1. Own understanding, without AI (i.e., securing initial cognitive work): The purpose is to protect the cognitive work through which initial understanding is built. The student must first read, think, formulate and test ideas on their own.

Stage 2. AI as dialogue partner, critic or feedback resource (i.e., extending and challenging thinking):  GenAI is used only to extend thinking rather than replace it. AI is introduced to question, challenge, exemplify and provide feedback.

Stage 3. Verification (i.e., developing disciplinary and epistemic judgement): AI use is conditional. The student must compare GenAI tool’s output against course literature, empirical evidence, theory or their own data, thereby turning passive acceptance into active judgement.

Stage 4. Accountability (i.e., making the use of AI transparent and reflective): The student reports what AI was used for, what was accepted, what was rejected, and why.

    Stage 5. Monitoring: Assessment can then ask a different set of questions: Can the student explain this without AI? Can the student use AI without losing subject-matter judgement? Can the student identify errors, uncertainties and limitations? Can the student defend their choices, orally or in writing? Can the student show how their understanding developed over time?

    Table 2

    5-Stage AI Pedagogy Framework (Mårell-Olsson & Titareva, 2026)

    StageFocus & PurposeAI StatusKey Student & Instructor Actions
    1. Own UnderstandingSecure initial cognitive workNo AIStudent reads, thinks, formulates, and tests initial concepts independently.
    2. Dialogue & ChallengeExtend thinkingActive AI PartnerAI acts as critic / Socratic dialogue partner to question and exemplify.
    3. VerificationDevelop epistemic judgementConditional AIStudent cross-references AI outputs against peer-reviewed literature, theory, and data.
    4. AccountabilityTransparency & reflectionTransparent AIStudent documents AI prompts, accepted/rejected output, and rationale.
    5. MonitoringAssessment of core masteryEvaluated MasteryAssessment tests independent defence, error detection, and student growth over time.

    We are not claiming to offer a novel solution to designing teaching and assessment for GenAI reality. A recent framework for higher education proposes similar principles, preserving cognitive friction, positioning AI as a provisional thinking partner, embedding evaluation throughout, and sequencing AI-free with AI-mediated phases (Vendrell & Johnston, 2026). Wazan (2026) builds an assessment design and examination system around the same logic. Students must commit to an initial answer before consulting GenAI, then interrogate the model’s output (including deliberately flawed responses) and justify a refined answer, scored by a rubric that assesses reasoning as a process rather than a product. That two such proposals, developed from cognitive psychology, critical pedagogy and epistemology tradition, converge on the same sequence is significant in itself.  What we add is the didactic account of why it works, the why, what, how, when and for whom, its extension across schooling as well as higher education, and the caution developed above that making the process visible is not automatically benign.

    To illustrate our framework in practice, consider a written assignment that requires students to analyse a problem using course material. In the first phase, students draft their analysis in class, without AI, so that the initial reading and reasoning are their own. In the second stage, students use AI to interrogate their draft, asking it to challenge the argument, surface counterexamples or offer alternative framings, and then deciding what to keep. In the third phase, students check every AI-generated claim, source and example against the literature and the evidence, and correct what does not hold. In the fourth stage, students submit a short account of how AI was used, what they accepted or rejected and why, and are examined on whether they can defend the analysis without it. In the fifth stage, the educator assesses how students can explain their analysis without AI, show critical thinking and demonstrate the growth of their own thought and knowledge over the process of completing the assignment.

    These phases are a design logic rather than a fixed timetable. They may recur within a longer task, and where friction is most valuable depends on the students and the subject. A novice may need a longer, more protected first phase, while an advanced student may begin closer to verification.

    We believe that, rather than forbidding cognitive offloading, education in the future must regulate it pedagogically. AI may reduce extraneous load. It must not be allowed to take over the load that builds understanding. This principle, drawn from the distinction between beneficial and detrimental offloading, is helpful in guiding teaching and assessment design in GenAI times.

    Implications for professional development and a different kind of guidance

    We offer two crucial implications. First, education systems face a very large need for professional development, from schools to higher education globally. This goes beyond AI literacy. Teachers need practical pedagogical support in designing teaching and assessment that reduces detrimental offloading and metacognitive laziness, while still allowing beneficial offloading, and in helping students understand why the goal is to learn the concept and its applicability to the real world, not merely to complete the task (see Lodge et al., 2023). Policy recommendations are starting to reflect this need. The Council of the European Union’s 2026 “Conclusions on teachers in the era of artificial intelligence (AI)” place teachers at the centre, recognising that AI may reshape how they design, deliver and assess instruction and calling for guidance and supportive frameworks for their continuous development (Council of the European Union, 2026). Characteristically, though, such instruments invite and recognise rather than specify what that development should contain, which is  the gap we attempt to cover in this article.

    Second, the kind of guidance that policy produces needs to pivot. The most useful instruments for teachers and classrooms will not be longer lists of risks or prohibitions, but resources that help teachers translate risks into concrete decisions about assignment design, feedback, and assessment (working examples, design patterns, and exemplars of process-visible tasks) while preserving professional judgement. Better AI tools may help at the margins, such as Socratic tutoring models, tools designed to scaffold metacognition rather than provide complete answers, and teacher-facing applications that augment rather than replace an expert (Wang et al., 2024; OECD, 2026). The OECD frames this as a choice between replacement, complementarity and augmentation, and argues that augmentation (e.g., teacher and AI refining each other’s work, while professional judgement is preserved) holds the most promise (OECD, 2026). But tools are not the fundamental issue. The fundamental issue is how generative AI changes the design of teaching and the assessment of students’ knowledge.

    Conclusion

    Many current AI policies and reports have grown adept at identifying the risks generative AI poses to learning. They are useful but give teachers very little practical support in translating those risks into teaching and assessment design. We argue that closing that gap is the central educational task of this moment, and it is a shared task across education systems globally. Just as AI policy is being urged to put purpose before acceleration (Dignum et al., 2025), education should put didactics before tools. The argument is not merely that didactics matter, but that the didactic questions can be turned into a concrete, phased design for teaching and assessment, one whose guiding principle is to regulate cognitive offloading rather than forbid it. Closing the gap, therefore, depends a lot on the design work, and on investing, at scale, in the professional judgement of educators.

    In conclusion, earlier we introduced Question Zero (Q0) as a way of asking the right questions before adoption: “Under what conditions should an AI system be adopted, if at all?” Q0 includes dimensions of why, who, what, how and where. What real problem are we trying to solve? Who might be affected? What kind of technology, if any, is appropriate to solve a problem at hand? How should it be adopted and governed? And where, and under what conditions should it operate? The didactic questions ask: why, what, how, when and for whom? The two approaches converge on one overarching premise: the adoption of the technology does not start with the tool, instead it starts with the purpose, i.e., answering why and what do we want to achieve? Both invite us to think from a perspective of purpose, context, people and the goals and conditions in which the tool is planned to be applied.

    References

    Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26), Article e2422633122. https://doi.org/10.1073/pnas.2422633122

    Bearman, M., Tai, J., Dawson, P., Boud, D., & Ajjawi, R. (2024). Developing evaluative judgement for a time of generative artificial intelligence. Assessment & Evaluation in Higher Education, 49(6), 893–905. https://doi.org/10.1080/02602938.2024.2335321

    Bergström, P., Mårell-Olsson, E., & Jahnke, I. (2019). Variations of symbolic power and control in the one-to-one computing classroom: Swedish teachers’ enacted didactical design decisions. Scandinavian Journal of Educational Research, 63(1), 38–52. https://doi.org/10.1080/00313831.2017.1324902

    Bjork, E. L., & Bjork, R. A. (2015). Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning. In M. A. Gernsbacher & J. R. Pomerantz (Eds.), Psychology and the real world: Essays illustrating fundamental contributions to society (2nd ed., pp. 59–68). Worth.

    Carless, D., & Boud, D. (2018). The development of student feedback literacy: Enabling uptake of feedback. Assessment & Evaluation in Higher Education, 43(8), 1315–1325. https://doi.org/10.1080/02602938.2018.1463354

    Council of the European Union. (2026). Council conclusions on teachers in the era of artificial intelligence (AI) (9003/26). https://data.consilium.europa.eu/doc/document/ST-9003-2026-INIT/en/pdf

    David, L., Vassena, E., & Bijleveld, E. (2024). The unpleasantness of thinking: A meta-analytic review of the association between mental effort and negative affect. Psychological Bulletin, 150(9), 1070–1093. https://doi.org/10.1037/bul0000443

    Dawson, P. (2021). Defending assessment security in a digital world: Preventing e-cheating and supporting academic integrity in higher education. Routledge. https://doi.org/10.4324/9780429324178

    Desvaux, C., Abdelghani, R., Oudeyer, P.-Y., & Sauzéon, H. (2026). Curiosity and metacognition: Towards a unified framework for learning and education in the age of AI [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2604.25648

    Dignum, V., Carli, R., Ericson, P., Titareva, T., & Tucker, J. (2025). “AI first” to “purpose first”: Rethinking Europe’s AI strategy. AI Policy Lab (AIPEX), Umeå University. https://doi.org/10.63439/LPOU6506

    Dignum, V., Carli, R., Dahlgren Lindström, A., Ericson, P., Titareva, T., & Tucker, J. (2026 June). Question Zero for Explainability and Vice Versa: The Case of the EU’s AI First Strategy. In International Conference on Human-Computer Interaction (pp. 17-31). Cham: Springer Nature Switzerland.

    Dunn, T. L., & Risko, E. F. (2016). Toward a metacognitive account of cognitive offloading. Cognitive Science, 40(5), 1080–1127. https://doi.org/10.1111/cogs.12273

    European Commission. (2026, May 12). Guidelines on the ethical use of artificial intelligence and data in teaching and learning. European School Education Platform. https://school-education.ec.europa.eu/en/discover/publications/guidelines-ethical-use-ai-and-data-teaching-and-learning

    Fan, Y., Tang, L., Le, H., Shen, K., Tan, S., Zhao, Y., Shen, Y., Li, X., & Gašević, D. (2025). Beware of metacognitive laziness: Effects of generative artificial intelligence on learning motivation, processes, and performance. British Journal of Educational Technology, 56(2), 489–530. https://doi.org/10.1111/bjet.13544

    Gerlich, M. (2025). AI tools in society: Impacts on cognitive offloading and the future of critical thinking. Societies, 15(1), Article 6. https://doi.org/10.3390/soc15010006

    Hathcoat, J. D., Slotnick, R., & Miller, W. (2026). The End of Assessment? Disruption and Transformation in the Age of AI. Research & Practice in Assessment, 21(3), 5-26 (Special Issue–AI in Assessment).

    Hopmann, S. (2007). Restrained teaching: The common core of Didaktik. European Educational Research Journal, 6(2), 109–124. https://doi.org/10.2304/eerj.2007.6.2.109

    Jahnke, I., Bergström, P., Mårell-Olsson, E., Häll, L., & Kumar, S. (2017). Digital didactical designs as research framework: iPad integration in Nordic schools. Computers & Education, 113, 1–15. https://doi.org/10.1016/j.compedu.2017.05.006

    Kalyuga, S., & Plass, J. L. (2025). Rethinking cognitive load theory. Oxford University Press. https://doi.org/10.1093/9780190078539.001.0001

    Klafki, W. (2000). Didaktik analysis as the core of preparation of instruction. In I. Westbury, S. Hopmann, & K. Riquarts (Eds.), Teaching as a reflective practice: The German Didaktik tradition (pp. 139–159). Lawrence Erlbaum Associates.

    Li, C., Cui, H., & Hagedorn, L. S. (2026). The cognitive impact of ChatGPT in higher education: A systematic review of critical and creative thinking outcomes. Computers and Education: Artificial Intelligence, 10, Article 100571. https://doi.org/10.1016/j.caeai.2026.100571

    Lodge, J. M., Howard, S., Bearman, M., Dawson, P., & Associates. (2023). Assessment reform for the age of artificial intelligence. Tertiary Education Quality and Standards Agency (TEQSA). https://www.teqsa.gov.au/sites/default/files/2023-09/assessment-reform-age-artificial-intelligence-discussion-paper.pdf

    Lodge, J.M., Loble, L. (2026). Artificial intelligence, cognitive offloading and implications for education. University of Technology Sydney. https://doi.org/10.71741/4pyxmbnjaq.31302475

    Mårell-Olsson, E. (2012). Att göra lärandet synligt? Individuella utvecklingsplaner och digital dokumentation [Making learning visible? Personal development plans and digital documentation] [Doctoral dissertation, Umeå University]. DiVA. http://urn.kb.se/resolve?urn=urn:nbn:se:umu:diva-59420

    Mårell-Olsson, E. (in press). AI literacy, governance, and equity in Swedish education policy. In T. Titareva, A. Csonska-Stambekova, & E. Mårell-Olsson (Eds.), AI ethics in global education: Building a future grounded in local realities. CRC Press.

    Mishra, P., & Koehler, M. J. (2006). Technological pedagogical content knowledge: A framework for teacher knowledge. Teachers College Record, 108(6), 1017–1054. https://doi.org/10.1111/j.1467-9620.2006.00684.x

    Mishra, P., Warr, M., & Islam, R. (2023). TPACK in the age of ChatGPT and generative AI. Journal of Digital Learning in Teacher Education, 39(4), 235–251. https://doi.org/10.1080/21532974.2023.2247480

    Nicol, D. J., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199–218. https://doi.org/10.1080/03075070600572090

    OECD. (2026). OECD digital education outlook 2026: Exploring effective uses of generative AI in education. OECD Publishing. https://doi.org/10.1787/062a7394-en

    Redecker, C. (2017). European framework for the digital competence of educators: DigCompEdu (Y. Punie, Ed.). Publications Office of the European Union. https://doi.org/10.2760/159770

    Risko, E. F., & Gilbert, S. J. (2016). Cognitive offloading. Trends in Cognitive Sciences, 20(9), 676–688. https://doi.org/10.1016/j.tics.2016.07.002

    Shapiro, H., Souto-Otero, M., & Watermeyer, R. (2026). Metacognitive AI literacy: Going beyond the AI skills gap agenda. Learning, Media and Technology. https://doi.org/10.1080/17439884.2026.2652638

    Tai, J., Ajjawi, R., Boud, D., Dawson, P., & Panadero, E. (2018). Developing evaluative judgement: Enabling students to make decisions about the quality of work. Higher Education, 76(3), 467–481. https://doi.org/10.1007/s10734-017-0220-3

    Titareva T., Tucker J., Carli R., Movchan V., Dignum V. (2026). Question Zero: Why Responsible AI Begins Before AI Adoption. AI Policy Exchange Forum (AIPEX). https://doi.org/10.63439/WCEZ7337

    Uljens, M. (1997). School didactics and learning: A school didactic model framing an analysis of pedagogical implications of learning theory. Psychology Press. https://doi.org/10.4324/9780203304778

    UNESCO. (2024a). AI competency framework for teachers. https://doi.org/10.54675/ZJTE2084

    UNESCO. (2024b). AI competency framework for students. https://doi.org/10.54675/JKJB9835

    UNESCO. (2025a). AI and the future of education: Disruptions, dilemmas and directions. https://doi.org/10.54675/KECK1261

    UNESCO (2025). AI and education: protecting the rights of learners.  https://doi.org/10.54675/ROQH4287

    UNICEF. (2025b). Policy guidance on AI for children (Version 3.0). https://www.unicef.org/innocenti/reports/policy-guidance-ai-children

    Vendrell, M., & Johnston, S.-K. (2026). Scaffolding critical thinking with generative AI: Design principles for integrating large language models in higher education. Computers and Education: Artificial Intelligence, 10, Article 100572. https://doi.org/10.1016/j.caeai.2026.100572

    Wang, R. E., Ribeiro, A. T., Robinson, C. D., Loeb, S., & Demszky, D. (2024). Tutor CoPilot: A human-AI approach for scaling real-time expertise [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2410.03017

    Wazan, A. S. (2026). Strategies for creating uncertainty in the AI era to trigger students’ critical thinking: Pedagogical design, assessment rubric, and exam system [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2602.00026

    Weidlich, J., Gašević, D., Drachsler, H., & Kirschner, P. (2025). ChatGPT in education: An effect in search of a cause. Journal of Computer Assisted Learning, 41(5), Article e70105. https://doi.org/10.1111/jcal.70105

    AI data labelling: a pathway or peril to refugee self-reliance?

    Executive summary

    Humanitarian budgets are collapsing at the same moment that AI systems are generating major demand for high-quality human-labelled data. Considering flexibility and low entry requirements of AI data labelling, social enterprises have moved quickly to position refugees at this intersection, framing this digital work as a scalable, dignified pathway to economic self-reliance. Drawing on organisational documents, social enterprise impact reports, and first-hand worker testimonies, the policy brief examines whether data labelling constitutes a meaningful livelihood solution for displaced populations or whether it reproduces new forms of digital precarity within global AI supply chains.

    The analysis focuses on four interrelated dynamics: the dual identity of social enterprises, which pitch their impact-oriented services to tech clients while selling the same work as empowerment to humanitarian donors; the self-reliance narrative promoted by social enterprises; and the lived realities behind this narrative, including both the positive aspects of giving refugees access to AI work and structural labour harms embedded in this labour.  While social enterprises are providing hope and some insulation for refugees entering the labelling market, they cannot overcome structural harms of the market, leaving refugees to bear the cost of that failure. This paper recommends that UNHCR publish binding operational guidelines for social enterprises, drawing on the Fairwork Framework, to ensure that AI-related digital work can support refugee livelihoods without entrenching the very exploitation it claims to address.

    1. Context

    Global emergencies are intensifying: the number of state-based armed conflicts reached a 70-year record in 2024 – numbering 61 (PRIO 2025). At the same time, global warming neared a 1.3 anthropogenic increase (World Weather Attribution 2025), and authoritarian repressions keep rising under the flag of a right-wing turn (Human Rights Watch 2024), leading to growing displacement all over the world. Despite this increasing polycrisis, in 2025 UNHCR’s budgets have diminished by 30% year to year and are expected to further fall by 15% in 2026 (Le Poidevin 2026). The greatest impact of these shortfalls has been felt on refugees themselves, whose numbers soared to over 36 million in 2025 (UNHCR n.d.).

    By 2023, two-thirds of refugees lived in poverty, and 62% resided in host countries that severely restrict labour market access in practice, (UNHCR 2023, 1) despite the 1951 Refugee Convention’s formal guarantee of the right to work. The International Labour Organization (ILO) thus framed the gig economy as a “stepping stone to jobs and income”, (ILO 2025a, 19) while UNHCR highlights that web-based economic activities afford displaced individuals the vital opportunity to earn a living (Easton-Calabria, Hackl 2023, 7). In 2022 UNHCR and ILO launched an 18-month project in 8 countries called “Promotion, Inclusion and Protection of Refugees in the Gig Economy,” aiming to equip displaced populations with new skills and knowledge connected to digital hygiene and cyber risks (ILO n.d.).

    Remote digital work like AI data labelling is thus uniquely attractive because it bypasses physical borders and discriminatory national labour laws, offering low barriers to entry and flexible, piece-rate work (Clark 2021, 770). For displaced populations, it theoretically offers a “virtual entry” into the global workforce, presenting an immediate income path for those routinely blocked from stable, local jobs (Altenried 2024, 1120). For tech companies, this transnational digital matching grants access to an “extremely scalable workforce” on a “pay-as-you-go” basis (De Stefano 2016, 4-5).

    2. The hidden infrastructure of AI: what data labelling is

    “AI isn’t magic; it’s a pyramid scheme of human labour” – Adio Dinika, a researcher at the Distributed AI Research Institute explained to The Guardian (Bansla 2025). AI market has been rapidly expanding with UNCTAD projecting for it to grow by 25 times – from 189 billion to 4.8 trillion USD between 2023 to 2033 (UNCTAD 2025). Microsoft estimated that one in six people are using generative AI worldwide with Global North leading the way with almost a quarter of the population being GenAI users (Microsoft 2026, 2). Its major branches like large language models (LLMs), developed for chatbots like ChatGPT, and computer vision systems, used to identify objects in self-driving cars, require massive amounts of high-quality data. If AI is trained on raw, unlabelled data, it absorbs inaccurate and different in format information, ruining its output (Jenkins 2024). To prevent a drop in quality, tech giants rely on millions of human data labellers, who are important for teaching AI systems to do tasks that are hard to describe clearly, such as finding emotions or bias in text (CWA Union 2025). These workers draw boxes around objects in images, transcribe audio, tag particular words and mistakes in AI-generated text and review datasets to ensure consistency and accuracy (SuperAnnotate 2025). Labelling is quite easy to complete and only requires language knowledge (often non-English languages on a native level), basic computer literacy and common sense (Kaneti 2025, 611).

    3. The social enterprise model and its self-reliance pitch

    Social enterprises entered the humanitarian sector in 2010-s, offering humanitarian agencies a mechanism to embrace market logics while maintaining moral legitimacy (Portales 2019, 57). By presenting themselves as hybrid entities that combine the efficiency and innovation of for-profit businesses with the passion and values of non-profits, they now act as a “bridge” between these two worlds, creating a “blended value” that appeals to both corporate clients and humanitarian donors (Battisti 2019, 140).

    3.1 Hybrid structure and promise of social enterprises

    If anyone visits the website of a social enterprise with AI data labelling programs, they find a platform created to speak to two entirely different worlds simultaneously. To tech giants, mostly based in the Global North, the enterprise pitches seamless, high-quality data labelling services at competitive rates, promising to build the future of AI. To humanitarian donors and foundations the pitch is more about resilience and showing refugees economically and mentally benefiting by entering the global digital economy.[i] To succeed in disseminating both pitches, social enterprises make their whole structure hybrid. The first half is a commercial agency that targets corporate tech contracts, showing they can successfully work in a private setting. The second one is a non-profit foundation that benefits from tax reductions and charity donors that give out grants specifically to NGOs. It is responsible for the social mission part and takes on digital literacy training and providing infrastructure, like laptops, to their refugee workers.[ii]

    By utilizing these two faces, the social enterprise positions itself as a cushion between the client AI data labelling platform and refugees. However, the main website page of the corporate agency of EmpowerAI does not make major emphasis on datasets being produced by refugees: the visitor would need to scroll and click to learn about this.[iii] The reason behind it is that the social impact pitch is deemed unattractive to corporate clients that are “accustomed to no compensation floor and thriving on a rampant race to the bottom”. When one enterprise tried to establish stricter rules on fair compensation, they faced so many difficulties in getting new clients that the hired consultants recommended it to “tone it [social impact narrative] down” (Kaneti 2025, 618, 620).

    3.2 Selling “self-reliance” to the market and the worker

    The narrative of self-reliance has been present in refugee support field, being grounded by UNHCR in their 2005 “Handbook of Self-Reliance”, aiming to empower refugees “to meet essential needs <…> in a sustainable manner and with dignity” (UNHCR 2005, 1) It has later become a running theme in most social enterprise reports under an umbrella narrative of “work instead of aid” (Sama 2022, 12). They set an inspiring tone by making the title “Empowering Change in Conflict-Affected Communities” and featuring “inspiring case studies” that “captured positive changes our initiatives brought to communities in need” (Humans in the Loop 2024, 1). Social enterprise operations are thus driven by a goal of “economic self-sufficiency of disadvantaged populations,” while a longer-term vision is “refugees [being] able to overcome economic & social barriers and become pioneers in the tech sector” (Subul 2024, 10-11). Here social enterprises reframe refugees from “tech user[s] to tech empowered,” (Techfugees n.d.) actively promoting a narrative change of the role refugees can play in digital economy. Considering the ambition, the effects are supposedly going beyond giving short-term solutions, instead aspiring to provide “financial independence and career development”, serving as an opportunity to “re-start their careers” (Humans in the Loop 2024, 3). However, analysed social impact reports provide no longitudinal data on whether participants transition to stable employment. Instead, it highlights “success stories” of workers who leveraged their annotation gigs with the only “ripple effect” being that participating refugees started “investing more into their families, education and communities after securing digital work” (Sama 2022, 24).

    4. The realities of AI data labelling programs for refugees

    To understand how social enterprises succeed or fail in fulfilling their promise of self-reliance to refugees, it is necessary to consider both the bright and the dark side of AI data labelling programs.

    4.1 The bright(-er) side of data labelling

    Hope given through market access

    As it was mentioned above, in many host countries displaced people routinely confront obstacles, including the denial of work permits, unrecognized educational credentials, language barriers, widespread discrimination and geographic isolation in camps or peripheral urban areas (Altenried 2024, 1120). Hence, AI data labelling can give much-needed accessibility to income that does not necessarily require a 9-5 office presence. It makes it accessible to refugees caring for families, those with physical or mental disabilities or facing religious restrictions on mixed-gender workplaces (CWA Union 2025). Another benefit is transportability of data labelling work as regardless of refugee’s next destination, it will still be available online, which is a ray of stability important for ever-changing livelihood of displaced people (De Stefano 2015, 5). Moreover, from the psychological perspective, earning money independent of social allowance can in itself be empowering with refugees feeling “on top of the world” when they first got it (Berhane Gebrekidan 2024, 7). A job in a prospective field of AI serves as motivation and a point of pride for people who lost their homes and had to restart their lives in a new country.

    Social enterprise mediation

    Importantly, social enterprise model offers an additional insulation for refugees that would otherwise be thrown into the ocean of data labelling work without any safety nets. Refugees are usually unaware about how to register and engage on digital platforms or do not even possess the needed infrastructure (Kaneti 2025, 616). Most social enterprises are at least trying to negotiate better conditions for their workers, for instance, setting fairer pricing models, arranging clearer task scopes and deadlines, and implementing specific ethical guidelines (Ibid, 615). They are also taking on most of the administrative burden: finding clients and negotiating connection with refugees in spite of their documentary situation. This support is especially vital for refugees who often lack official documents or bank accounts due to host state restrictions and other bureaucratic obstacles. So, social enterprises focus on mediating bank transfers from the clients, which refugees themselves call “very convenient”. While it might take some time: “We sometimes do not receive our work allowances until after two months” (Al-Hammada 2024, 11), this is still much better than not receiving anything at all. In case of office work in host countries, social enterprises can also offer the space and infrastructure (computers, Wi-Fi etc.) needed for data labelling. Finally, as mentioned above, they are conducting trainings that go beyond data labelling, important for those without education, computer or soft skills, which can thus improve refugees’ employability and future work opportunities.[iv]

    4.2 The dark side of AI data labelling for refugees

    Non-transparency and communication hurdles

    When a refugee starts work on the first project, the enterprise gives instructions from the client. They are individual for each project and can be 30-40 pages which workers must learn to avoid mistakes. However, workers have no direct line to the client, so any unclear cases are communicated back through a long chain: enterprise team manager to senior staff to client (Ibid, 9). This layered communication frequently produces misalignment between what the written policy says, what the trainer explained, and what the client actually wanted. The consequences then fall on refugees. When a completed and submitted project fails to meet client expectations, workers are expected to redo the work under revised guidelines, importantly doing it for free. As one Syrian refugee data worker in Lebanon put it: “Why do we have to show flexibility and re-implement the work even though it is not our fault, as the instructions were not clear from the beginning? Is it fair to work twice on a project for one wage?” (Ibid, 10).

    Non-disclosure agreements (NDAs) remain an issue for most data labellers as well, as they are forced to sign these documents to formally protect data of the client company. NDAs function as instruments of forced silence and intimidation, making it punishable to form unions to try improving working conditions of labellers. While some social enterprises are trying to take on more advocacy functions and negotiate the unionization of workers, it remains extremely difficult due to clients justifying NDAs as necessary to protect their intellectual property and corporate confidentiality (Equidem 2025, 42).

    Algorithmic control and NDAs

    The mental strain of meeting targets is serious – in most projects the lowest quality threshold stands at 90-97% (al-Hammada 2024, 7). A score below this could trigger warnings, lost incentives or even placement on a “performance improvement plan,” which workers described less as support than as a formal corridor toward dismissal (Berhane Gebrekidan 2024). To make matters worse, workers might have to face triggering text or pictures, often connected to their war-torn homelands, as they are often hired to label in their native language. While most times social enterprises are trying to avoid such projects, they are often left unaware about concrete tasks and end goals of their clients (Kaneti 2025, 617). “I hope this isn’t for weapons” is a valid concern among labellers, however the non-transparency of end client makes it impossible to be certain about the usage of labelled data. It might indeed be for weapons (Miceli 2024).

    The time pressure remains massive as well. If the worker takes too much time on a task, their rating will fall, resulting in less or lower-paid tasks in the future (Hao, Hernandez 2022). It leaves them in a vicious cycle of trying to complete tasks fast but retain the highest quality. One data worker summarized this pressure: “How can you read this essay, grade it accurately, and then provide feedback in five minutes? The time per task was not adding up, and then the timer put on pressure” (CWA Union 2025). This is usually the point that clients of social enterprises are not willing to negotiate on, as famously said: “time is money,” and the labelled datasets are expected by the company on time, regardless of how much work and rework it actually requires (al-Hammada 2024, 7).

    Psychological effects

    In an attempt to mitigate negative effects on workers, most social enterprises introduce wellness breaks and support sessions with a psychologist. Despite flexibility being one of the appeals of this work, in reality, those working in the office have a standard 8-hour day and highly restricted wellness breaks. Workers in similar facilities reported having just “a 20-minute wellness break twice a week” and noted that they “have to beg for a wellness break on the group chat for it to be granted” (Berhane Gebrekidan 2024, 14). Psychological support is also unprofessional. Instead of receiving help, workers state: “I always go to the counsellors and come out more triggered and more crazy” because some counsellors “think you are faking it or trying to manipulate them so you can get a leave permit when you cry” (Ibid, 32-33).

    Economic precarity and dependence in the absence of choice

    When there is “only one project every few months <…> Many annotators feel compelled to accept any available work” (al-Hammada 2024, 32-33) The task can also come in the middle of the night or when they are unavailable. Some workers resort to using a “browser extension that sounds an alarm when a new task appears” and keep it “on loud even when [they] sleep, to wake [them] up in the middle of the night” (Hao, Hernandez 2022). However, this additional payment may be fundamental for the whole family, so missing or declining it is not an option. Yet this work is still poorly paid. Labelling objects on 10,000 images over several days is compensated with just 140 euros – an amount that barely covers ten days of food in a country like Lebanon (al-Hammada 2024, 9). In a year the “salary”, already positively negotiated by one data labelling social enterprise, amounted to just a bit above 2000 euro (Kaneti 2025, 615), which is far from enough for survival of even one person.

    Non-transferable skills

    Finally, the long-term effects of trainings conducted by social enterprises for refugees might not be as useful as promoted. Instead of acquiring high-level coding or web development skills that foster genuine upward mobility, the social enterprise only taps into these spheres on a surface level, primarily focusing on training refugees to carry out routine, low-skill microwork. The trainings range from turning on a computer to English language or soft skills, however they rarely go into developing any of these skills higher than pre-intermediate level (Humans in the Loop 2025, 15). While this is important for some displaced data labellers, many other are not only literate but also have a higher education (al-Hammada 2024, 8). Transferability of these skills thus remains doubtful, trapping refugees in the cage of labelling microtasks for specific clients who have narrow rules and skills needed for usually short-term projects.

    5. International policy landscape: Emerging good practices

    The regulatory vacuum that historically enabled these harms is beginning to fill. The following instruments constitute a growing toolkit that UNHCR can leverage in its frameworks for refugee platform work. These examples were chosen to represent relevant policies in different regions and are not meant to cover the entire regulatory landscape of platform work and are listed by scale of coverage.

    Who introduced Measure / Instrument Relevance for Refugee Contexts
    ILO Convention concerning decent work in the platform economy, ILC.114/Convention No. 193: recognizes workers’ rights to decent labour and occupational health and safety; requires human involvement in algorithmic management changes; mandates disclosure of monitoring and decision-making systems; classifies workers based on the terms of their work  (ILO 2026). Provides UNHCR with a multilateral standard to cite in partnership negotiations and advocate for in UN digital governance fora.
    EU Platform Work Directive 2024/2831 (EUR-LEX 2024): mandates employment status clarification, algorithmic transparency, human oversight of automated decisions and worker representative consultation. Directly applicable to BPO and platform-mediated data work. Creates enforceable transparency standards UNHCR can require of EU-headquartered partners.
    Malaysia Gig Workers Act 2025 (Malaysia Parliament 2025): national legal framework regulating digital platform workers; prohibits unfair wage deductions, mandates payment transparency, ensures algorithmic transparency, introduces occupational safety requirements and creates a grievance tribunal (ILO 2025c). First comprehensive national gig worker law in Southeast Asia. Provides a model for host-country advocacy in regions where UNHCR has operational presence.
    Australia Fair Work (Digital Labour Platform Deactivation Code) 2024 (Department of Employment and Workplace Relations 2024): minimum standards for worker deactivations; requires warning notices, opportunity to respond, review processes, and record-keeping. Applies to all “regulated workers” on digital platforms (ILO 2024). Addresses the arbitrary termination dynamic documented in refugee data labelling contexts. Model for NDA-alternative grievance pathways.
    India (Bihar) Bihar Platform Based Gig Workers (Registration, Safety and Welfare) Act 2025 (PRS 2025): Welfare Board, Social Security Fund, registration, payment regulation, data protection, algorithmic transparency, grievance redressal (ILO 2025b). Demonstrates that emerging economies can legislate binding digital labour protections. Directly relevant to UNHCR operations in South Asia considering large numbers of gig workers in the region.

    Considering the existing legislation, it is also important for UNHCR to take into account special vulnerabilities of refugees when developing own policy for work conditions of social enterprise programmes in data labelling, recommendation for which will be presented in the last part of this paper.

    6. Policy recommendations

    1. Prioritize integrated pathways to sustainable livelihoods

    To strengthen the long-term impact of digital livelihood programs, UNHCR should move beyond short-term output metrics and tie recommendations to longitudinal evidence that refugee participants progress toward stable, formal employment. Innovation grants can be impactful if directed toward programs that build genuinely transferable skills, such as advanced coding, ensuring that refugees, including those with higher education, are supported toward meaningful economic mobility instead of entry-level microwork.

    1. Adopt minimum labour standards via the Fairwork Framework

    Current UN initiatives have made valuable progress on digital literacy and cyber risk awareness. To build on this foundation and ensure that digital livelihood programs deliver on their self-reliance promise, UNHCR should formally endorse the Fairwork Framework (Fairwork n.d.) as a baseline for recommendations to social enterprises across five dimensions:

    • Fair pay

    UNHCR should work with social enterprises towards replacing piece-rate structures that produce poverty-level earnings with living wage thresholds adapted to host-country cost of living. Enterprises should be required to publish transparent, disaggregated income data as a condition of humanitarian endorsement, replacing the current practice of curated success stories with longitudinal evidence that the work actually generates economic self-reliance.

    • Fair conditions

    Because employer-provided mental health support has been documented as often inadequate and even harmful for workers, UNHCR should mandate that trauma-informed mental health care for refugee data laborers be funded and managed entirely independently of employer control with particular attention to the double exposure risk for refugees labelling content from their own countries of origin.

    • Fair contracts

    UNHCR is encouraged to work with social enterprises to review the use of NDAs in refugee data labelling contexts, exploring whether worker protections and client confidentiality can be balanced through alternative grievance mechanisms. Proactive disclosure of client identities and dataset purposes, where feasible, would help workers make informed decisions about the work they undertake.

    • Fair management

    UNHCR should call on social enterprises to introduce meaningful human oversight of algorithmic performance systems, including the 90-97% accuracy thresholds enforced under time pressure, and to ensure that no worker faces income reduction or dismissal risk for raising concerns about task content. Task timelines should be set in consultation with workers, instead of dictated by client delivery schedules alone in order to improve both working conditions and output quality.

    • Fair representation

    UNHCR should require social enterprises to build worker representation into their governance structures, viewing them as central voices with real influence over pricing negotiations, task standards and client selection. These should include, among others, regular anonymous worker surveys, introduction of Beneficiary Advisory boards which can receive concerns from workers, as well as allowing to form unions.

    7. Literature list

    1. Al-Hammada, R. (2024). “If I Had Another Job, I Would Not Accept Data Annotation Tasks.”
    2. Altenried, M. (2024). Mobile workers, contingent labour: Migration, the gig economy and the multiplication of labour. Economy and Space.
    3. Bansla, V. (2025). How thousands of ‘overworked, underpaid’ humans train Google’s AI to seem smart. Internet source. URL: https://www.theguardian.com/technology/2025/sep/11/google-gemini-ai-training-humans
    4. Battisti, S. (2019). Digital Social Entrepreneurs as Bridges in Public-Private Partnerships. Journal of Social Entrepreneurship.
    5. Berhane Gebrekidan, F. (2024). Content moderation: The harrowing, traumatizing job that left many African data workers with mental health issues and drug dependency. Data Workers Inquiry.
    6. Clark, T. (2021). The Gig Is Up: An Analysis of the Gig-Economy and an Outdated Worker Classification System in Need of Reform, 19 Seattle J. Soc. Just.
    7. CWA Union. (2025). Ghost Workers in the AI Machine: U.S. Data Workers Speak Out About Big Tech’s Exploitation. Internet source. URL: https://cwa-union.org/ghost-workers-ai-machine#workers-value
    8. De Stefano, V. (2015). The Rise of the “Just-in-Time Workforce”: On-Demand Work, Crowdwork, and Labor Protection in the “Gig Economy”.
    9. De Stefano, V. (2016). The Rise of the “Just-in-Time Workforce”: On-Demand Work, Crowdwork, and Labor Protection in the “Gig Economy”. Comparative labor law.
    10. Department of Employment and Workplace Relations. (2024). Fair Work (Digital Labour Platform Deactivation Code) Instrument 2024.
    11. Easton-Calabria, E., & Hackl, A. (2023). Refugees in the digital economy: The future of work among the forcibly displaced. Journal of Humanitarian Affairs, 4(3).
    12. Equidem. (2025). Scroll. Click. Suffer. The Hidden Human Cost of Content Moderation and Data Labelling.
    13. EUR-LEX. (2024). Directive (EU) 2024/2831 of the European Parliament and of the Council of 23 October 2024 on improving working conditions in platform work.
    14. Fairwork. Principles. URL: https://fair.work/en/fw/principles/
    15. Hao, K., Hernandez, A. (2022). How the AI industry profits from catastrophe. Internet source. URL: https://www.technologyreview.com/2022/04/20/1050392/ai-industry-appen-scale-data-labels/
    16. Human Rights Watch. (2024). World report 2025. New York.
    17. Humans in the Loop. (2024). Impact Report 2023.
    18. Humans in the Loop. (2025). Impact Report 2024.
    19. ILO. (2024). Fair Work (Digital Labour Platform Deactivation Code) Instrument 2024 under the Fair Work Act. Internet source. URL: https://digitallabour.ilo.org/legislation/fair-work-digital-labour-platform-deactivation-code-instrument-2024-under-fair-work-act
    20. ILO. (2025a). Exploring the gig economy: Challenges and opportunities A self-guided resource.
    21. ILO. (2025b). The Bihar Platform Based Gig Workers (Registration, Safety and Welfare) Act, 2025. Internet source. URL: https://digitallabour.ilo.org/legislation/bihar-platform-based-gig-workers-registration-safety-and-welfare-act-2025.
    22. ILO. (2025c). Gig Workers Act 2025. Internet source. URL: https://digitallabour.ilo.org/legislation/gig-workers-act-2025
    23. ILO (2026). Convention concerning decent work in the platform economy. ILC.114/Convention No. 193. URL: https://www.ilo.org/sites/default/files/2026-06/ILC114-Instrument%20C.193-EN.pdf.
    24. ILO. Promotion, inclusion and protection of refugees and host communities in the digital and gig economy. Internet source. URL: https://www.ilo.org/projects-and-partnerships/projects/promotion-inclusion-and-protection-refugees-and-host-communities-digital.
    25. Jenkins, T. (2024). Garbage in Garbage out. Internet source. URL: https://www.applogicnetworks.com/blog/garbage-in-garbage-out
    26. Kaneti, M. (2025). Social Enterprise Solutions for Migrants, Microwork, and AI Ethics: The Case of HiTL. Human Service Organizations: Management, Leadership & Governance, 49(5), 609-627.
    27. Le Poidevin, O. (2026). Exclusive: Cash-strapped UN refugee agency to cut more jobs, even as crises mount. Internet source. URL: https://www.reuters.com/world/europe/cash-strapped-un-refugee-agency-cut-more-jobs-even-crises-mount-2026-05-18/
    28. Malaysia Parliament. (2025). Rang Undang-undang Pekerja Gig 2025. D.R. 27/2025.
    29. Miceli, M. (2024). “I hope this isn’t for weapons.” How Syrian data workers train AI. Internet source. URL: https://untoldmag.org/i-hope-this-isnt-for-weapons-how-syrian-data-workers-train-ai/
    30. Microsoft. (2026). AI Diffusion Report. Global AI Adoption in 2025—A Widening Digital Divide.
    31. Portales, L. (2019). Basics, Characteristics, and Differences of Social Entrepreneurship.
    32. PRIO (2025). New data shows conflict at historic high as U.S. signals retreat from world stage. Internet source. URL: https://www.prio.org/news/3616
    33. PRS. (2025). The Bihar Platform Based Gig Workers (Registration, Safety and Welfare) Act. No 8.
    34. Sama. (2022). Environmental & Social Impact Report 2022.
    35. Subul. (2024). Annual Impact Report 2024.
    36. SuperAnnotate (2025). What is data labeling? The ultimate guide. Internet source. URL: https://www.superannotate.com/blog/guide-to-data-labelling
    37. Techfugees. From tech user to tech empowered (#Empowerment). Internet source. URL: https://www.notion.so/covidrefugees/2-From-tech-user-to-tech-empowered-Empowerment-11392e4039c8492d8cef375417c01166
    38. UNCTAD. (2025). 2025 Technology and Innovation report. Internet source. URL: https://unctad.org/publication/technology-and-innovation-report-2025
    39. UNHCR. (2005). Handbook for Self-Reliance. Geneva.
    40. UNHCR. (2023). Refugees’ Access to Jobs and Financial Services. Background Guide. Challenge Topic #3.
    41. UNHCR. Refugee Data Finder. Internet source. URL: http://unhcr.org/refugee-statistics.
    42. World Weather Attribution. (2025). Unequal evidence and impacts, limits to adaptation: Extreme weather in 2025. Internet source. URL: https://www.worldweatherattribution.org/unequal-evidence-and-impacts-limits-to-adaptation-extreme-weather-in-2025/

    Endnotes

    [i] The difference is mostly visible in these two websites of the same social enterprise: The NGO part: https://humansintheloop.org/impact/foundation/, the corporate part: https://humansintheloop.org/

    [ii] See, for example, two entities of this social enterprise:  Na’amal. URL: https://naamal.org/, Na’amal Agency. URL: https://agency.naamal.org/.

    [iii] See the website: HITL. URL: https://humansintheloop.org/.

    [iv] Look, for instance at the following social enterprise sources: Techfugees. Inclusion. URL: https://techfugees.com/inclusion/,  Medium. Na’amal, DOT, and The Conrad N. Hilton Foundation Launch Follow-Up to Successful Digital Skills Programme in Ethiopia. Internet source. URL: https://na3amal.medium.com/naamal-dot-and-the-conrad-n-9c2268fd56b2

    Question Zero: Why Responsible AI Begins Before AI Adoption

    Abstract

    The Question Zero (Q0) Self-Assessment Tool for Responsible AI, developed by the AI Policy Lab at Umeå University, supports organisations in posing foundational questions before adopting AI. Grounded in the concept of Question Zero: “Under what conditions should an AI system be adopted, if at all?”, the tool offers support for cross-functional team discussions covering motivation, stakeholder mapping, system type, adoption process and infrastructure. It is designed for public institutions, civil society organisations, policymakers and other actors looking for responsible AI decision-making approaches. With the help of the tool, we argue that purposeful, problem-led assessment must precede any procurement and deployment decisions.

    Keywords: Question Zero, Q0, Responsible AI, AI Self-Assessment, AI Governance, EU AI First Strategy, AI Policy, AI Adoption Motivation, Stakeholder Mapping, Explainability, AI Policy Lab, Umeå University

     

    The Problem with Starting from the Answer

    Public and private organisations face growing pressure to adopt artificial intelligence (AI). Governments across the EU and beyond are committing vast public resources to AI acceleration. Organisations, large and small, feel urged to integrate AI into their operations for fear of being perceived as falling behind. The European Commission’s Apply AI Strategy (European Commission, 2025) has formalised this pressure into policy direction, framing AI as the default first response to organisational and societal challenges. The same trend can be seen at a national level, as well as across sectors and organisations.

    However, this urgency leads to a fundamental disruption of sound decision-making. Rather than beginning with a problem and asking what solutions are available, an “AI First” logic begins with the technology and asks where it can be applied. As we have argued elsewhere, the result is that the question of whether AI is appropriate in a given context is hardly ever properly posed (Dignum et al., 2025). The consequences of skipping this step are not abstract.

    The Dutch childcare benefit scandal, in which an algorithmic system wrongly flagged tens of thousands of parents as committing fraud, disproportionately affecting ethnic minority and low-income families (Amnesty International, 2021), is one of the examples of what might happen when AI is deployed without adequate prior deliberation about necessity, appropriateness and who might be harmed. Similarly, the UK’s A-level grading algorithm reinforced existing inequalities, putting students from smaller schools and lower socio-economic backgrounds at a disadvantage by relying on historical data to determine results (Kolkman, 2020). In the United States, ProPublica’s investigation into the COMPAS recidivism algorithm revealed significant racial biases, with Black defendants nearly twice as likely as their white counterparts to be wrongly flagged as high risk (Larson et al., 2016). Australia’s Robodebt scandal exposed the dangers of deploying flawed automated decision-making systems at scale, showing how vulnerable citizens can be unjustly penalised without adequate oversight, transparency and safeguards (Royal Commission into the Robodebt Scheme, 2023).

    Together, these cases illustrate how without robust AI governance, such systems can deepen discrimination against marginalised groups and undermine public trust in key institutions. While these cases differ in context and technical form, they share an important feature: technology was introduced without adequately addressing Question Zero (Q0), meaning the prior scrutiny of necessity, appropriateness, accountability and social impact was weak or absent.

    The question Q0 (“Under what conditions should an AI system be adopted, if at all?”) we pose, is a call for rigour and strategic foresight. It asks organisations to define the problem before considering technological solutions, to assess alternatives, to map who benefits and who bears risks and to ensure that any decision on how to proceed is genuinely justified. Only sometimes, not always, is AI the right answer. Thus, the purpose of the Q0 Self-Assessment Tool is to facilitate a structured pause to allow for better decision making about responsible AI.

    What is the Q0 Self-Assessment Tool?

    The Q0 Self-Assessment Tool is a free assessment tool designed not as a compliance checklist or technical audit, but as a structured self-reflection instrument for cross-functional groups within organisations. Its primary goal is to support decision-makers in working through the foundational questions of responsible AI governance before procurement or deployment decisions are finalised (Titareva et al., 2026).

    The tool is organised into five sections: Why, Who, What, How and Where, each targeting a distinct dimension of responsible AI governance. These sections are not a linear sequence; organisations are encouraged to move between them iteratively, as answers in one area frequently raise additional questions in others.

    Section A: Why (Motivation) asks organisations’ decision-makers and employees to articulate the problem they are attempting to solve, the reasons they are considering AI, the alternatives they have considered (including human and non-AI technical solutions) and why and whether AI is the best solution for the existing problem. This is the core of Q0: encouraging explicit justification before any further investment of time or resources into AI procurement and deployment.

    Section B: Who (Stakeholder Mapping) asks organisations to identify the stakeholders who may be directly or indirectly affected by the adoption of an AI system. This includes colleagues, employees, customers, and members of marginalised or underrepresented groups who are often overlooked in initial assessments (as well as at later stages). The section also prompts organisations to consider who currently performs the tasks the AI system is intended to undertake, who is likely to benefit, who could be harmed and whether meaningful opt-out options exist. By mapping the full stakeholder landscape, organisations can assess how different groups may experience the adoption of AI and ensure that no group is disproportionately burdened, excluded or deprived of meaningful choice.

    Section C: What (Type of AI System) asks organisations to define and describe the specific AI system, method or tool under consideration (e.g., predictive, generative, categorising or hybrid), as well as whether it will be procured off the shelf, developed in-house or built for specific needs of an organisation. This section encourages organisations to ensure that the chosen system fits the problem they want to solve and that the system’s complexity, development approach and flexibility match its intended use.

    Section D: How (Adoption Process) asks organisations to consider how the AI system would be deployed, how existing workflows would change, how outputs and performance would be monitored, how data security would be ensured and how affected users would be able to understand the basis of the system’s conclusions. It also prompts organisations to define how complaints and errors would be handled and who would be accountable for oversight throughout the system’s lifecycle. This section emphasises that accountability must be embedded from the beginning rather than added retrospectively, and that explainability should be treated as an ongoing organisational practice rather than merely a compliance requirement.

    Section E: Where (Infrastructure and Control) asks organisations to consider where the AI system will operate, where data will be stored, where the provider is based and where the training data originates. These questions address issues of digital sovereignty, data jurisdiction and the extent to which organisations can maintain meaningful control over the AI systems they adopt. This section emphasises that effective governance and accountability depend on understanding where the system operates, who controls it and how data is collected, stored and managed. It therefore encourages organisations to ensure transparency regarding infrastructure, models and training data in order to meet legal, security and ethical responsibilities and requirements.

    Each section concludes with a confidence rating on a five-point scale, allowing organisations to reflect on how confidently they can answer the questions within that dimension. Rather than measuring compliance or assigning a level of AI maturity, the ratings are intended to stimulate discussion, identify areas of uncertainty and highlight where further information, stakeholder engagement or organisational preparation may be needed. The aggregated scores are visualised in a radar diagram, providing an overview of confidence across all five dimensions. This visualisation enables organisations to compare results across teams, over time or between different AI use cases, thus supporting internal learning, reflection and continuous improvement. The ratings are not intended for external accreditation, benchmarking or audit, but as a self-assessment tool that helps organisations recognise strengths, identify gaps and guide responsible AI governance.

    A final section of concluding questions invites decision-makers and employees of organisations to reflect on the insights gained throughout the self-assessment process by evaluating any pilot experiences, considering the expected benefits of the proposed AI system alongside its possible risks and harms and identifying appropriate next steps. Rather than prescribing decisions or producing a pass-fail outcome, the tool is designed to support informed, context-sensitive judgement, recognising that responsible AI adoption depends on an organisation’s objectives, values, legal obligations and operational context. As a decision-support instrument, the Q0 tool helps structure deliberation, surface assumptions and strengthen transparency, but responsibility for interpreting the findings and deciding how to act upon them remains with the organisation and its employees as part of its internal AI governance.

    Figure 1.Question Zero Conceptual Operationalisation
                              Figure 1. Question Zero Conceptual Operationalisation

    Positioning the Q0 Self-Assessment Tool Among Existing AI Assessment Frameworks

    The Q0 self-assessment tool builds on a growing ecosystem of AI governance and assessment instruments. At the same time, it addresses a different stage of the AI lifecycle. Existing frameworks such as the European Commission’s Assessment List for Trustworthy Artificial Intelligence (ALTAI), UNESCO’s Ethical Impact Assessment (EIA), the UNESCO Readiness Assessment Methodology (RAM), the UNDP Human Rights Impact Assessment Toolkit and similar international and national frameworks support responsible AI through structured reflection on legal, ethical, technical and organisational issues. Although these tools have different purposes and target different audiences, they share a common goal: They help governments or organisations identify, manage and govern the risks associated with AI systems and promote transparency, accountability, human rights and public trust.

    The Q0 tool shares many of these principles. Like ALTAI, it addresses human oversight, transparency, accountability, fairness and data governance. Similar to UNESCO’s Ethical Impact Assessment, it encourages multidisciplinary participation and considers the wider social impacts of AI alongside technical issues. The UNDP Human Rights Impact Assessment Toolkit also emphasises the importance of identifying rights holders and affected communities. UNESCO’s Readiness Assessment Methodology takes a broader perspective by examining institutional capacity, governance, infrastructure and organisational readiness. Q0 tool follows a similar approach by treating responsible AI governance as an institutional challenge rather than only a technical one.

    The main difference is not the questions themselves, but when they are asked. The assessment tools mentioned above often assume that an organisation has already decided to develop, procure or deploy an AI system. Their purpose is to assess whether that system is trustworthy, legally compliant or ethically governed. Some frameworks do include questions about necessity or proportionality during the pre-adoption stage. However, these questions usually form one part of a broader impact or compliance assessment when the AI system’s adoption project has already been initiated.

    The Q0 tool starts one step earlier. It asks whether AI should be adopted at all, and if yes, under what conditions. This is the central purpose of Question Zero. The tool encourages organisations to define the problem first, consider non-AI alternatives and examine governance, stakeholder impacts and organisational readiness before resources are committed or procurement begins. It also places greater emphasis on collective deliberation and participation. It recommends cross-functional participation, the inclusion of affected stakeholders where possible, iterative use of the tool, documentation of disagreements and integration with existing governance processes. The goal is not a single assessment but an ongoing process of organisational learning and reflection.

    A second difference is the intended use of the tool. Many existing assessment instruments support regulatory compliance, formal impact assessment or risk management. The Q0 tool has a different purpose. It is a self-assessment instrument. It does not generate a compliance score or certify that an AI system or case is trustworthy. The confidence ratings are intended to stimulate discussion, reveal uncertainty and identify gaps in knowledge. The radar diagram allows organisations to compare discussions across teams, projects or points in time. This helps identify differences in understanding and areas that require further attention.

    The Q0 Self-Assessment Tool is intended to complement, not replace, existing governance frameworks. Organisations that decide AI is appropriate can then carry out more detailed assessments, such as ethical impact assessments, human rights impact assessments, data protection impact assessments or assessments required under the national or regional legislation (e.g., the EU AI Act and others). Q0 tool therefore supports an earlier stage of decision-making. It aims to improve the quality of the decision that determines whether these later assessments will be necessary.

    The Q0 tool also has limitations. It is a self-assessment tool. Its value depends on organisations’ employees engaging with the questions and including a diversity of perspectives. It cannot remove organisational bias or guarantee that all relevant stakeholders are represented. It cannot ensure that recommendations will be implemented. It is also not a substitute for legal, technical or human rights assessments where these are required. Finally, no questionnaire can capture the full complexity of every organisational context. The Q0 tool should therefore be seen as a governance aid rather than a decision-making mechanism. It supports better judgement, but responsibility for the final decision always remains with the organisations’ leadership and employees.

    Example of the Q0 Self-Assessment Tool in Practice

    A university department is considering introducing a generative AI assistant to answer students’ questions about courses, deadlines and administrative procedures. Instead of immediately assuming that an AI chatbot is the right solution, the department first uses the Q0 Self-Assessment Tool. Addressing the questions in the section Why”, university employees from diverse departments and functional areas define the problem, for example, long response times during busy periods. They also consider other options, such as improving online guidance, extending office hours or employing more student assistants. AI is treated as one possible solution, not the starting point.

    The remaining Q0 tool’s sections help to broaden the discussion. Under the section Who, the participants identify as many categories of stakeholders as possible, who may be affected, including students, lecturers, administrative staff, IT services and others. It also considers students with disabilities, international students and others who may be disadvantaged by digital-only support. Under What, participants compare different AI systems and ask whether a simpler non-AI or rule-based solution could meet the need. Under the sections Howand Where, they discuss governance, accountability, transparency, data security, hosting arrangements and whether the university keeps appropriate control over its data.

    At the end of the process, the group gives a confidence rating for each section. The radar diagram shows where participants feel confident and where more work is needed. In this example, confidence could be high for the motivation behind the project but lower for infrastructure, data governance and long-term accountability. These results do not determine whether AI should be adopted. Instead, they help the department identify where further discussion, expert advice or pilot testing is needed before making a final decision.

    Conclusion

    AI is not inevitable. Its adoption is not inherently beneficial. Responsible AI governance begins before procurement, development or deployment. It begins by asking whether AI is the best solution for your current problem, if so, under what conditions. This is the purpose of Question Zero.

    The Q0 Self-Assessment Tool provides organisations with a practical way to support this early reflection. It does not replace existing governance frameworks or assessments. Instead, it complements them by addressing a stage of decision-making that is often overlooked. It focuses on motivations, stakeholders, system choice, adoption, infrastructure and organisational control.

    The value of the Q0 Self-Assessment Tool lies not in providing answers, but in improving the quality of the questions organisations ask before adopting AI systems. It creates space for discussion, makes assumptions explicit and helps identify issues that require further attention. In the end, the responsibility for deciding whether to adopt an AI system remains with the organisations’ employees. The Q0 process helps ensure that this decision is more informed, transparent and accountable.

    Access the Tool

    The Q0 Self-Assessment Tool (Version 3, April 2026) is available: HERE 

     

    References

    Ala-Pietilä, P., Bonnet, Y., Bergmann, U., Bielikova, M., Bonefeld-Dahl, C., Bauer, W., … & Van Wynsberghe, A. (2020). The assessment list for trustworthy artificial intelligence (ALTAI). European Commission.

    Amnesty International (2021, October 2025). Xenophobic machines: Discrimination through unregulated use of algorithms in the Dutch childcare benefits scandalhttps://www.amnesty.org/en/documents/eur35/4686/2021/en/

    Dignum, V., Carli, R., Ericson, P., Titareva, T., & Tucker, J. (2025). ‘AI First’ to ‘Purpose First’: Rethinking Europe’s AI Strategy. AI Policy Exchange Forum (AIPEX)https://doi.org/10.63439/LPOU6506

    Dignum, V., Carli, R., Dahlgren Lindström, A., Ericson, P., Titareva, T., & Tucker, J. (2026, June). Question Zero for Explainability and Vice Versa: The Case of the EU’s AI First Strategy. In International Conference on Human-Computer Interaction (pp. 17-31). Cham: Springer Nature Switzerland. https://doi.org/10.1007/978-3-032-29456-2_2 

    European Commission (2026, March 27). Apply AI Strategy. Shaping Europe’s Digital Future. https://digital-strategy.ec.europa.eu/en/policies/apply-ai

    Kolkman, D. (2020, August 26). “f**k the algorithm?”: What the world can learn from the UK’s A-level grading fiasco. The London School of Economics and Political Science (LSE) Impact.  https://blogs.lse.ac.uk/impactofsocialsciences/2020/08/26/fk-the-algorithm-what-the-world-can-learn-from-the-uks-a-level-grading-fiasco/#:~:text=%E2%80%9CF**k%20the%20algorithm%E2%80%9D?:%20What%20the%20world%20can,address%20through%20making%20their%20algorithms%20more%20explainable.

    Larson, J., Mattu, S., Kirchner, L., & Angwin, J. (2016, May 23). How we analyzed the COMPAS recidivism algorithm. ProPublica. https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm

    Royal Commission into the Robodebt Scheme (2023, July 7). The Report of the Royal Commission into the Robodebt Scheme to the Governor-General, His Excellency, General the Honourable David Hurley AC DSC (Retd). https://robodebt.royalcommission.gov.au/publications/report

    Titareva, T., Carli, R., Dahlgren Lindström, A., Dignum, V., Fabris, B., Ericson, P., & Tucker, J. (2026). Q0 Self-Assessment Tool with Guiding Questions for Responsible AI Approach by AI Policy Lab (AIPL). Test version 3 from April 2026. Umeå University.

    UNESCO. (2023). Ethical Impact Assessment: A tool of the Recommendation on the Ethics of Artificial Intelligence. https://doi.org/10.54678/YTSA7796

    UNESCO. (2023). Readiness Assessment Methodology: a Tool of the Recommendation on the Ethics of Artificial Intelligence. https://doi.org/10.54678/YHAA4429

    United Nations Development Programme. (2025). Human Rights Impact of AI Assessment Tool.  https://www.undp.org/eurasia/publications/human-rights-impact-ai-assessment-toolkit

    Fairness inside and out: A situated approach to algorithmic allocation in complex sociotechnical systems

    Abstract

    This article outlines a framework for modeling and simulating complex sociotechnical systems in which an allocation mechanism acts as the interface (and sometimes a barrier) between the public and institutions. Two examples contextualise algorithmic allocation challenges: the Amsterdam school choice and organ allocation for transplant patients. We highlight how algorithmic fairness does not guarantee systemic fairness, and propose a situated approach with institutional modeling and social simulations. With the increasing adoption of AI in decision making, a situated simulation approach provides an alternative to opaque institutional governance and decision making.

    The mechanism, institutions, and people

    Sociotechnical systems are formed of technical and social components; in present-day societies, most technology is used within a social environment. Mindful of algorithmic discrimination cases, researchers have sought ways to measure the fairness of technical systems. The impact of a technology on its surrounding environment, however, is not always clear, especially when complex social dynamics are at play. Designing a “fair” technical component is not a guarantee of bringing fairness to the system as a whole. Selbst et al. argue that this is due to a focus on solutions (i.e. “make the outcome fair”) instead of a focus on the process (i.e.”make the system fair-er”) [1]. We now need ways to understand the impact of a technical system on fairness, within the technology, and without it.

    Resource allocation problems are routinely faced by public institutions, be it in the form of determining who gets access to benefits, or which infrastructure to renovate among a set of proposals. The decision making process forms in institutions, bound by laws of fairness and non-discrimination. However, institutional decisions change the dynamics of the system before they take action, while they are communicated, and after they take hold, altering the behaviour of the social system.

    Social simulations can estimate the effect of policy changes by dynamically modeling the social and institutional aspects of sociotechnical systems [2]. The field of institutional modeling investigates the interplay of individual and institutional agents within complex sociotechnical systems [3].  Social and institutional simulations might thus guide the process of creating fairness throughout the system and not only within the mechanism. We use the framing of simulations to explore allocation mechanisms and their impact on complex sociotechnical systems.

    Algorithmic allocation

    We propose an approach to modeling sociotechnical systems under specific conditions: resource allocation with a registry (here called mechanism) facing the public in one direction and based on rules determined by one or multiple institutions. The communication between public and institution occurs through the mechanism, which is designed to optimize for a certain outcome. Value preferences inform the outcome, but are not always explicit, nor uniform across parties. The institution designs the rules of the allocation mechanism based on (i) its internal principles, and (ii) a fair outcome for the public. Without communication between public and institutions, the rules are decided based on inferred preference models, i.e., the institution designs the mechanism to please the assumed value preference of the people and reach a “fair” outcome (Fig. 1).

    In this setting, which we contextualise in two examples below, the mechanism design affects the behaviour of agents independently of its inherent degree of algorithmic fairness. Specifically, we encountered two cases that pose what seem to be “already-solved” allocation problems. Upon further scrutiny, we found underlying, more complex dynamics.

    Figure 1. Motivating setup: A sociotechnical system consisting of agents, a mechanism, and policy actors. The agents provide an input to the mechanism, and are affected by its output, while a policy actor sets the rules for the mechanism.

    Figure 1. Motivating setup: A sociotechnical system consisting of agents, a mechanism, and policy actors. The agents provide an input to the mechanism, and are affected by its output, while a policy actor sets the rules for the mechanism.

    As shown in the examples, the mismatch between the allocation rules and the preference model of the population influences the mechanism’s efficacy, prompting gamification for those who have the resources to invest in understanding and playing the mechanism’s rules. In the second example, the complexity of conflicting institutional values limits the mechanism’s fairness and efficiency.

    This raises questions of prosociality, trust, governance and fairness outside algorithmic bounds. How do people respond to a mechanism that does not align with their preference model (and perhaps their values)? How can institutions consider the longitudinal effects of algorithmic allocation within the process of policymaking?

    Amsterdam school choice

    The Amsterdam school choice system is an especially useful example of a sociotechnical system where the deployment of algorithmic allocation led to unexpected changes in the behavior of the social system, causing continuous updates to the algorithm and subsequent degradation of trust in the system. It is also a good example of when a poorly designed “fair” mechanism leads to diminishing prosocial behavior.

    In Amsterdam, there is an open-choice policy when moving from primary to secondary school [4]. Group 8 students (equivalent to 6th grade in the US) are allowed to pick any school within their education level (e.g., vocational, general secondary, pre-university, etc.) anywhere in the city, unrestricted by their zone of residence. Every year, students are asked to rank 8 to 12 schools (based on education level), and then a centralized matching algorithm automatically matches students to schools.

    The system is communicated to be based on the famous Deferred Acceptance algorithm [5], a New York based school choice algorithm which won its author the Nobel prize in Economics in 2012 [6]. However, the algorithm is modified in several key places to fit the open-choice policy in Amsterdam.

    Firstly, the Deferred Acceptance algorithm is a two-sided matching method, meaning that students and schools both have preferences over each other and are then matched accordingly. In Amsterdam, schools do not have a preference over students and therefore this preference is simulated with the means of a random lottery number. This choice was also grounded in fairness, as with a random lottery, every student has an equal chance of getting a good lottery number. Secondly, the Deferred Acceptance algorithm does not specify a fixed number of schools to rank. In Amsterdam, however, a policy named “placement guarantee” was introduced, where students are guaranteed a position if they rank a fixed number of schools. This guarantee is ensured by increasing the capacity of schools by a marginal amount in a second round of allocation, where students who listed the required number of schools in the first round and still did not get a placement are eligible, and are then allocated using the increased capacity in the second round. These changes unsurprisingly led to the matching algorithm performing very differently from its Nobel-prize-winning counterpart, creating problems of inefficiency and inequality.

    Random lottery numbers led to an inefficient allocation of schools, increasing dissatisfaction among students and their parents [7]. Moreover, the “placement guarantee” policy led to strategic gaming from the parents, where they are incentivized to report schools they do not prefer just so they can ensure eligibility for the second round [8]. This further degraded the system’s efficiency, as well as its fairness, as parents from privileged backgrounds are better able to apply strategies than parents from marginalized communities.

    The authorities responsible for the school choice system have repeatedly come under public scrutiny for continuous changes to the system without successful results [9]. This has led to a degradation of trust in the institution.

    Underneath this allocation problem is a complex sociotechnical issue, one that requires:

    1. Making clear what values the system is actually trying to serve, and whether these align with what parents themselves care about.
    2. Understanding how the rules of the mechanism change social behavior, so that reported preferences are seen not just as choices, but also as responses to risk, incentives, and unequal access to information.
    3. Identifying problems that are easy to miss if we only look at fairness within the mechanism itself, such as distorted preferences, unequal strategic burden, and declining trust in the system.
    4. Looking beyond the matching rule alone, and instead supporting better communication, better alignment between stakeholders, and more situated policy design.

    This requires a multidisciplinary sociotechnical approach, one in which a social simulation model can help make the problem and its tradeoffs easier to communicate and discuss. It also requires a systematic way of eliciting stakeholder values and using those discussions to work toward a shared understanding of what the system should aim to achieve.

    Organ allocation systems

    Fairness plays an important role in organ allocation systems. For patients on the waiting list to receive a transplant, waiting time is a strong determinant of short-term survival, and long-term quality of life. Multiple factors influence the structure of the waiting list and the distribution of donated organs, particularly those from deceased donors. Initially, biological compatibility between donor and recipient seems to provide a baseline allocation rule: the most compatible donor-recipient match (considering blood type and HLA presence) should be prioritised to ensure the best graft survival outcomes. However, the length of the waiting list and persistent scarcity of organs complicates the allocation problem.

    Cold ischemia time restrictions (meaning how long an organ is deprived of blood flow before its quality deteriorates beyond utility for transplantation) brings into consideration logistics, donor and recipient location and a corollary of geographical factors such as regional allocation rules, coordination practices and transplant capacity. Allocation policies have been studied via simulations and algorithmic optimization for decades, with awareness of the complex interplay of medical, economic, political and legal factors.

    The other component of the system is the prioritization of patients within the waiting list structure. The state of the patients is dynamic, with probabilities of becoming too ill to receive a transplant. The waiting list design attempts to accommodate for this by modeling the disease progression and setting thresholds for transplant eligibility.

    In the U.S., Organ Procurement Organizations (OPO) and Transplant Centers’ performance is evaluated competitively based on how many organs were retrieved and how many successful transplants were performed [10], adding economic incentives to the mix.  This influences the local center’s decision to accept available organs or reject them in the hope of a better quality alternative.

    Even seemingly aligned bioethical values can create conflicting interests, especially if multiple institutional actors are pushing for their preferred outcomes. Beside the responsibility to ensure “fairness” in allocation, the meaning of “do no harm” manifests differently for Transplant Centers (i.e. reject lower quality organ offers) and OPO (i.e. provide as many quality organs as possible), exacerbating inefficiencies. Concerning the value preferences of the patients themselves, little is known. Without a channel of communication to the algorithmic rule setters, and with conflicting interests among institutions, we risk optimizing for less relevant values.

    Most importantly, the algorithm of allocation can be gamed by multilisting. Multilisting is the practice of assigning one patient to multiple waiting lists across Transplant Centers, which initially seemed to alleviate the length of the waiting list at the national scale in the U.S. However, this came at the cost of individual fairness (not everyone can be listed in multiple registries as not everyone can afford to quickly travel across states to receive a life-saving transplant) and collective fairness: while the national wait time decreased, regional variance in wait time can increase [11]. Multiple organ offers also seem to produce a utilitarian gain in efficiency by reducing organ wastage and time to transplant, but may do so at the cost of societal trust in the system [12].

    Within this critical healthcare system, the allocation mechanisms endlessly attempt to reconcile fairness and efficiency. The value prioritization shifts from face-value fairness (e.g. centralised FIFO allocation) to utilitarian fairness (prioritizing patients with “the most to gain” from a transplant) or need-based fairness (prioritizing the most sick patients based on disease models with thresholds for exclusion), or introducing market logic to the system (monetary rewards to centers, multilisting, multioffer). Formalizing any value-rule into algorithmic allocation mechanisms creates, as in the example above, new ways for the system to be strategised upon, shifting the locus of inequality and obscuring its methods.

    Decontextualised fairness as an attribute of technical (sub)systems has already been criticized [1]. The algorithms designed for organ allocation systems present a clear example of the Framing and Formalism abstraction traps, in which fairness evaluation stops at the technical part of the sociotechnical system and its formalization generates new, unaccounted behaviours in the system. Some literature acknowledges these limitations, suggesting we might expand our abstraction boundary, or even consider other processes beside algorithmic allocation [13,14]. In the absence of a consensus of what “fair” organ allocation is, we might want to shift our focus to a process approach. Social and institutional modelling provide the best toolset for this endeavour.

    Our proposal

    We propose a four-step approach for studying complex sociotechnical systems in which an allocation mechanism mediates between the public and institutions. The goal is to understand how to embed algorithmic allocation mechanisms in a wider system of stakeholder values, behavioral adaptation, and institutional feedback. As the Amsterdam school choice reminds us, the algorithmic allocation must be designed to accommodate a response to the mechanism itself. To this end, we consider social/institutional simulations as the most eligible, situated approach.

    The four steps are as follows:

    1. Value elicitation: The first step is to determine which values the mechanism is meant to serve, and whether those values are actually shared by the stakeholders affected by it. Mechanisms are often designed around an inferred preference model: institutions assume what matters to the population and encode those assumptions into rules and objectives. But in high-stakes systems, these assumptions may diverge from the lived priorities of the people who interact with them. A situated approach therefore begins by making these values explicit and contextable.
      For example, in the Amsterdam school choice, policymakers may prioritize procedural fairness and avoiding unassigned students, while parents may care more about genuine access to preferred schools, reduced strategic burden, transparency, and trust. This gap matters for how the system performs in practice. Within organ allocation, value elicitation is needed to clarify which purpose the allocation system serves within the transplantation system as a whole, and what systemic fairness means to stakeholders. Although this might not result in a univocal “solution”, it lights the fire to improve procedural awareness. The value elicitation should start from Question 0, meaning without the assumption that an allocation mechanism is mandatory [15].
    2. Modeling and simulation: The second step is modeling and simulation. The simulation model is not the end goal, but a structured approach to facilitate and open communication around assumptions, tradeoffs, and possible interventions. If the response of the population to policy is modeled, strategic gamification can be accounted for in the simulated scenarios. In systems such as school choice and organ allocation, the mechanism does not act on fixed inputs. The inputs themselves change in response to the mechanism. Parents adapt their reported preferences strategically; OPOs and patients respond to waiting list structures, matching rules, and institutional incentives. Social simulation makes it possible to examine these interactions explicitly and to ask not only what outcomes a rule produces, but how that rule changes behavior across the wider system.
    3. Identifying bottlenecks and blind spots: The third step is to identify systemic bottlenecks and blind spots. Mechanisms are often evaluated in terms of fairness or efficiency within the allocation procedure itself, but this can obscure deeper problems elsewhere. A mechanism may be fair on paper while placing informational or strategic burdens unevenly across groups. It may also optimize one part of the process while ignoring upstream or downstream failures. A situated simulation framework helps uncover these blind spots by tracing feedback loops across the system: strategic adaptation, unequal ability to navigate rules or exploit advantages, changing interpretations of fairness, and degradation of trust over time. The modeling of heterogeneous agents and institutions allows the representation of multiple values and goals.
    4. Policy recommendations: The final step is policy recommendations. These recommendations should go beyond a technical redesign of the mechanism itself, as the system is sociotechnical; interventions may also need to address communication, participation, institutional coordination, trust, and policy stability. As the approach is situated within a community of stakeholders (institutions and population), the policy recommendations are tailored to local needs. This step can be expressed in policy briefs, appointment of committee and representatives to coordinate across stakeholder groups.

    The approach is intended to be iterative. Fixed time intervals between iterations allow for policy to take action, and structure the process of altering it. Moreover, they facilitate participation of stakeholders that are not institutional, constructing the communication channel that was lacking in the initial setting (Fig. 1).

    Conclusions

    Algorithmic allocation mechanisms are a component of critical sociotechnical systems where institutions and populations interact. We argue that their fairness should be evaluated inside the algorithms and outside of them, encompassing their effect on the system they become a part of. With the increasing use of AI to automate allocation processes, aligning the values of stakeholders and opening communication channels between the population and institutions becomes our priority. A four-step situated approach with social and institutional simulations is presented. The approach shifts the focus from a fair outcome to a fairer process. While we do not claim that the approach is perfect, it provides a step forward in the integration of algorithmic allocation and fosters a procedural, situated view of fairness.

    References

    [1] Selbst, Andrew D., danah boyd, Sorelle A. Friedler, Suresh Venkatasubramanian, and Janet Vertesi. 2019. “Fairness and Abstraction in Sociotechnical Systems.” In Proceedings of the Conference on Fairness, Accountability, and Transparency, 59–68. New York: Association for Computing Machinery. https://doi.org/10.1145/3287560.3287598.

    [2]  Lorig, Fabian, Fabris, Bertilla, Tucker, Jason. 2025. “Hybrid-Human Policy Modeling: Enhancing Decision-Making Using Social Simulations” Frontiers in Artificial Intelligence and Applications. Doi: 10.3233/FAIA250675

    [3] Ghorbani, Amineh. 2022. “Institutional Modelling: Adding Social Backbone to Agent-Based Models.” MethodsX 9: 101801. https://doi.org/10.1016/j.mex.2022.101801.

    [4] Ruijs, Nienke, and Hessel Oosterbeek. “School choice in Amsterdam: Which schools are chosen when school choice is free?.” Education Finance and Policy 14, no. 1 (2019): 1-30.
    https://doi.org/10.1162/edfp_a_00237.

    [5] Abdulkadiroğlu, Atila, Parag A. Pathak, and Alvin E. Roth. 2005. “The New York City High School Match.” American Economic Review 95 (2): 364–67. https://doi.org/10.1257/000282805774670167.

    [6] Nobel Prize Outreach. 2012. “The Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel 2012.” NobelPrize.org. https://www.nobelprize.org/prizes/economic-sciences/2012/summary/.

    [7] Het Parool. 2026. “Weer Minder Achtstegroepers Kunnen Naar Eerste Voorkeursschool, Volgend Jaar Wordt Het Beter.” Parool.nl. https://www.parool.nl/amsterdam/weer-minder-achtstegroepers-kunnen-naar-eerste-voorkeursschool-volgend-jaar-wordt-het-beter~b0ec8fd5f/.

    [8] Tasnim, Mayesha, Paul Verhagen, Tobias Blanke, Erman Acar, and Sennay Ghebreab. 2025. “Modeling Strategic Risk in School Choice: A Case for Transparent Design”. Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society 8 (3):2470-79. https://doi.org/10.1609/aies.v8i3.36731

    [9] Het Parool. 2026. “Loting voor Middelbare School Weer Terug naar Oude Systeem: 12 in Plaats van 9 Voorkeursscholen.” Parool.nl. https://www.parool.nl/amsterdam/loting-voor-middelbare-school-weer-terug-naar-oude-systeem-12-in-plaats-van-9-voorkeursscholen~b7972bda/.

    [10] Washburn, Kirt. 2012. “Maximizing Donor Potential: Evolving Organ Procurement Organization Metrics and Optimizing Organ Distribution and Allocation in the United States.” Liver Transplantation 18 (suppl. 2): S1–S4. https://doi.org/10.1002/lt.23507.

    [11] Harvey, C., and J. R. Thompson. 2016. “Exploring Advantages in the Waiting List for Organ Donations.” In 2016 Winter Simulation Conference (WSC), 2006–17. Washington, DC: IEEE. https://doi.org/10.1109/WSC.2016.7822245.

    [12] Erazo, Ignacio, David Goldsman, Pinar Keskinocak, and Joel Sokol. 2022. “A Simulation-Optimization Framework to Improve the Organ Transplantation Offering System.” In 2022 Winter Simulation Conference (WSC), 1009–20. IEEE. https://doi.org/10.1109/WSC57314.2022.10015431.

    [13] Thompson, David, Larry Waisanen, Robert Wolfe, Robert M. Merion, Keith McCullough, and Ann Rodgers. 2004. “Simulating the Allocation of Organs for Transplantation.” Health Care Management Science 7 (4): 331–38. https://doi.org/10.1007/s10729-004-7541-3.

    [14] Feccia, Mariano, Arianna Freda, Maurizio Naldi, Gaia Nicosia, and Andrea Pacifici. 2026. “Modelling and Simulating the Organ Donation Process Using Bootstrap and Event-Driven Process Chain Representation.” Journal of Simulation 20 (2): 135–51. https://doi.org/10.1080/17477778.2025.2486709.

    [15] Dahlgren Lindström, Adam, Dignum, Virginia, Ericson, Petter, Titareva, Tatjana and Tucker, Jason. 2025. “Responsible AI Self-assessment Workshop: Start with Question Zero”. https://aipolicylab.se/2025/09/05/responsible-ai-self-assessment-workshop-start-with-question-zero/ AI Policy Lab, Published September 5, 2025. Accessed April, 2026.