Will university teachers become obsolete in times of AI?

Elin Kvist (Department of Sociology, Umeå University)


Published on 22 May 2025
PDF
Cite

We live in times of endless crisis alarms, a general state of uncertainty, combined with an orchestrated and intentional undermining of established academic institutions. Science and academic research are under questioning around the world, and universities’ position as a legitimate source of knowledge and critical thinking are under attack. Adding to this sense of instability and uncertainty, there is an ongoing digital and technological development that some argue has become a major risk for technical unemployment – fortelling the “end of work” (Brynjolfsson & McAfee, 2014; Danaher, 2017).

Automation, history and current threats

Historically, we have seen how new technology has changed the way we work and our everyday work practices. Machines have revolutionised and increased efficiency in agriculture and industrial production, and reduced the number of workers demanded in the process. In light of today’s ever-growing improvements in computer power, artificial intelligence and robotics, the gloomiest predictors are now once again convinced that we are moving towards a jobless future (Brynjolfsson & Mcafee, 2014). This time, technology is substituting more cognitively advanced and emotionally demanding jobs – ones previously performed by professionals in technical and managerial professions, including university teachers (Autor, 2015).

However, the past two centuries of automatisation and technological change have not made human labour obsolete. Even though unemployment rates have fluctuated cyclically, there has been no long-run increase in unemployment (Autor, 2015). Through governmental programs of re-education and reorientation, most exempted workers have been able to move to other forms of labour, and new areas of work have opened up in the wake of technological transformation. In previous technological automation processes the focus have been on replacing the most dangerous, bodily harming, and repetitive tasks, and in doing so contributing to a more human friendly labour market.

The digital transformation of academic work

University teachers have seen their work tasks and everyday work practices changed dramatically with digitalisation. Through digital aids and tools, they book rooms, coordinate and organise lectures and seminars, examine and grade students digitally, do research, apply for research funding and ethical approval, publish in academic outlets, manage conference bookings, calculate budgets for research proposals and develop data management plans. These are just a few examples. Their everyday work practices include a significant amount of digital administration, interacting with large numbers of different digital platforms. In the name of efficiency, an increasing number of tasks that have been previously assigned to administrative employees, have gradually been reassigned to the university teachers.

However, when trying to understand the consequences of technological changes of the everyday work situation for university employees, it is important to also take into consideration that during the same time these organisations have also been subject to New Public Management (Thomas & Davies, 2002). Which has also entailed increases in the number of students with diverse needs, less preparation time for teaching, and continuous monitoring of performance through audits and performance evaluations. In result, this left each university teacher with less time to do research due to increased demands and shrinking resources. Consequently, this has spurred even more administrative work as researchers constantly need to apply for research grants in highly competitive, complex and time-consuming funding processes, resulting in additional time and resource-consuming processes. This is also important to take into consideration when trying to understand the implications of genAI on the future of university teachers, illustrating the importance of moving beyond a techno-deterministic understanding (Lindberg et al., 2022). Automation and digitalisation are often presented as neutral, a consequence of technological progress, and as something inevitable. The ideological and material consequences remain hidden (Lindgren, 2024).

AI’s role and data dependency

We have to understand what distinguishes AI technology from previous types of technology, and in doing so, understand what consequences it will have on the everyday practices of university teachers. First, how can they use AI in their everyday work? What tasks do they have that could be suitable for genAI tools? Tasks such as compiling large amounts of text or getting an overview of a new research field for teaching or research, writting summaries, and polishing research applications, compiling CVs and creating concise bios, conducting  text analysis of ethnographic materials, supporting peer-review and expert processes, when assessing exams and essays, supporting development of lectures, conference and seminar presentations. The possibilities are endless. In the modern universities that the New Public Management have constructed, with its endless rounds of evaluations, constant applications and assessments, genAI tools can function to support and facilitate the everyday administrative work, making the work practices more manageable. However, it is also important to keep in mind that AI needs large amounts of data to be able to learn from the environment in which it will operate. To be able to help the teachers in their professional practices, genAI needs access to information and data, and the employees must assist and train the systems. Algorithmic systems depend on humans performing a certain kind of digital work, data labeling and moderation, breaking down the work into smaller components for autonomous decisions (Lindgren, 2024). This work is not always visible or even seen as actual work (Moore & Woodcock, 2021).

Hidden labour and digital capitalism

In digital capitalism, we all are involved in generating this type of data. When we move around in digital environments, we perform a lot of work for free that contributes to generating profits for the system, often without us being aware of it. As users we contribute to training the AI systems. Those who are involved in creating content online leave behind data traces, and it is these traces that the large digital media giants (Meta, X, etc.) exploit and capitalise on. The work that people do in the borderland between AI and society is often hidden (Lindgren, 2024; Moore & Woodcock, 2021; Taylor, 2018). For example, when you order a pizza for home delivery via an app, you might perceive it as a digital process. However, the actual physical work behind is invisible. Someone is standing and making the pizza. Another person is delivering it to your home. Foodora and Deliveroo’s apps are part of the complex socio-technical ecosystem of digital society. The pizza delivery people use their own bicycles to deliver the pizza. Foodora does not own the bicycles. Therefore, the company is not responsible for them. The companies claim to offer “flexible and free work”. The couriers can work whenever they want. The work is clearly fragmented, and the workers are interchangeable. The work schedule is individualised. The workers have to deal on their own with all the challenges, including icy roads, angry customers, unclear directions, and other issues. The digital platforms pay for the result, not for the time in-between. In many ways, these working conditions resemble those at the beginning of industrialisation, before union mobilisation, labor protection, sick leave pay, and the right to vacation (Ilsøe & Söderqvist, 2023). What is presented as high-tech and new, is in fact a regression in labour law. Historically, we have seen how every technological leap favors the emergence of armies of marginalised workers, who would take jobs that are not considered jobs anymore. In this respect, automation processes are often much less impressive than the big tech companies and large digital platforms want us to believe. Some tasks may disappear and wages will be reduced, though people continue working alongside the machines for lower pay or even sometimes without pay (Taylor, 2018).

To understand work under digital capitalism, we need to go back to the basic question formulated within  the socialist feminist tradition “What is work?” (Ferguson, 2020). How digital capitalism has not only survived but prospered while certain types of work have been hidden and unpaid (Fraser, 2016). The unrecognised work performed by most of us in the borderline between genAI and society, mirrors digital capitalism’s historical and current approach to reproductive work (Jarrett, 2018). The indispensable affective and material labour, mostly cast as “women’s work”, often performed without recognition and pay. In other words, a work that is not regarded as work and is not considered as having any social or economic value. This work in practice is extremely important, while ideologically seen as completely unimportant. Departing from this reasoning, we can conclude that the capitalist system have an inherent desire to devalue and hide socially important work. As participants of the digital capitalism, we often ignore the work that takes place behind the applications, and buy the myth of “success”. This way, we give automation more credence than it deserves. We ignore the work that lies behind the shiny facades of digitalisation. Making the machines appear smarter than they are (Taylor, 2018).

If the discussion about technology continues to focuse only on the narrative that technology drives humanity’s development forward and humans have to keep up, there is an imminent risk of missing the social contexts in which these technical devices are created. When considering the consequences of genAI on university teachers and their daily work, it is important to understand that it is not the technology that will make the teachers obsolete. Technology is developed within a specific economic and social system, where certain resourceful organisations and individuals invest in developing technology that will benefit their personal interests, including control, power, and immense financial returns.  The technology is designed to replace human labor to some extent, but it is developed within a digital capitalism that thrives on making people feel that they are constantly replaceable and vulnerable.

Conclusion and final reflections

To conclude, will genAI make university teacher obsolete? There is a need to acknowledge both the advantages and disadvantages of these tools. With the current university climate orchestrated by the New Public Management with its constant demands for auditing, evaluations, counting, and compiling information, daily tasks of university teachers might become more manageable with the help of genAI. In a better of world, genAI tools might be used to feed the insatiable New Public Management system, freeing up time for research, ciritical thinking and teaching. On the other hand, AI needs data and other resources to be able to learn from the environment in which it will operate. When university teachers participate in training the algorithmic systems, the digital capitalism thrives. This work is often not recognised as work. Digital capitalism wants us to believe that technological development is unstoppable and that we need to accept that our work is exploited. As educators and citizens, we need to be aware that there is an inherent mechanism in the system that actively benefits from hiding work tasks and treats them as non-work.

References

Autor, D. H. (2015). Why Are There Still So Many Jobs? The History and Future of Workplace Automation. Journal of Economic Perspectives, 29(3), 3–30. https://doi.org/10.1257/jep.29.3.3

Brynjolfsson, E., & McAfee, A. (2014). The Second Machine Age: Work, Progress, and Prosperity in a Time of Brilliant Technologies. W. W. Norton & Company.

Danaher, J. (2017). Will Life Be Worth Living in a World Without Work? Technological Unemployment and the Meaning of Life. Science and Engineering Ethics, 23(1), 41–64. https://doi.org/10.1007/s11948-016-9770-5

Ferguson, S. J. (2020). Women and work: Feminism, labour, and social reproduction. Between the lines.

Fraser, N. (2016). Contradictions of Capital and Care. New Left Review, 100, 99–117.

Ilsøe, A., & Söderqvist, C. F. (2023). Will there be a Nordic model in the platform economy? Evasive and integrative platform strategies in Denmark and Sweden. Regulation & Governance, 17(3), 608–626. https://doi.org/10.1111/rego.12465

Jarrett, K. (2018). Laundering women’s history: A feminist critique of the social factory. First Monday. https://doi.org/10.5210/fm.v23i3.8280

Lindberg, J., Kvist,E., & Lindgren, S. (2022). The Ongoing and Collective Character of Digital Care for Older People: Moving Beyond Techno-Determinism in Government Policy. Journal of Technology in Human Services, 40(4), 357–378. https://doi.org/10.1080/15228835.2022.2144588

Lindgren, S. (2024). AI – ett kritiskt perspektiv (Upplaga 1). Studentlitteratur.

Moore, P. V., & Woodcock, J. (2021). Augmented Exploitation: Artificial Intelligence, Automation, and Work. For Work / Against Work. Pluto Press. https://onwork.edu.au/bibitem/2021-Moore,Phoebe+V-Woodcock,Jamie-Augmented+Exploitation+Artificial+Intelligence,Automation,and+Work/

Taylor, A. (2018). The Automation Charade. Logic(s) Magazine. https://logicmag.io/failure/the-automation-charade/

Thomas, R., & Davies, A. (2002). Gender and New Public Management: Reconstituting Academic Subjectivities. Gender, Work & Organization, 9(4), 372–397. https://doi.org/10.1111/1468-0432.00165

Keywords:

Technological unemployment, genAI, work, gender, digital capitalism, university teachers, reproductive work

How to cite this article:

Kvist E. (2025). Will university teachers become obsolete in times of AI? AI Policy Exchange Forum (AIPEX). https://doi.org/10.63439/CBGU8049

AI Technologies in Public Service: A Workshop for Identifying Needs, Challenges, and Solutions


Date & Location: April 9, 2025, Umeå University, Västerbotten, Sweden

The workshop was organised by the AI Policy Lab in collaboration with the AI Technologies for Sustainable Public Service Co-creation (AICOSERV) project members.

Overview

The workshop brought together more than 50 stakeholders from the public and private sectors, as well as academia, to explore the relationship between barriers to AI adoption in public services and the skills and expertise required to overcome them.

A central theme of the workshop was the “question zero”, the fundamental inquiry of whether AI should be used at all in a given context. As AI technologies continue to advance and expand into complex public sector tasks, the assumption that AI is always the right or necessary solution must be critically examined. The workshop challenged participants to consider not only how AI can be implemented, but to question whether it should be, emphasizing that responsible adoption begins with questioning the appropriateness and desirability of AI in specific domains.

This foundational concern set the tone for broader discussions about trust, governance, transparency, and the skillsets required to navigate the opportunities and risks of AI in public service.

Keynote Address

Professor Virginia Dignum, Director of the AI Policy Lab, opened the workshop with a keynote titled “Governing AI: Why, What, How?”

She addressed the societal and governance implications of AI, focusing on the need to critically evaluate when and how AI should be integrated into public service contexts. Her talk stressed the importance of not overlooking ethical, legal, and operational challenges in the rush to adopt AI.

Regional Case Study: AI in Västerbotten

Considering the wide range of public services open to AI adoption, a recurring set of challenges consistently emerges. Whether deploying AI-driven diagnostic tools in healthcare or implementing predictive analytics within smart city infrastructures, public and private sector actors, and community stakeholders face diverse barriers. Henry Lopez-Vega, fellow at the AI Policy Lab, presented on the challenges of AI adoption in the Västerbotten region in his session titled “What are the challenges with AI (in Västerbotten)?”

His research identified three core barriers to building a responsible AI ecosystem:

  • Technological infrastructure and processes within organisations
  • Organisational culture and resistance to change
  • Lack of clarity around AI governance and ownership

Group Discussions: Skills and Stakeholder Engagement

In the second half of the workshop, participants engaged in group discussions focusing on organisational challenges, key stakeholders, and barriers to implementation. Each group then mapped the skills and knowledge needed for responsible AI adoption in their contexts.

For example, in the case of AI-supported recruitment processes, participants identified several critical barriers:

  • Lack of transparency in AI decision-making
  • Biases in training data
  • Limited legal and ethical guidelines for automated hiring

To address these issues, participants emphasized the need for professionals with a blend of competences, including:

  • Knowledge of data protection and anti-discrimination legislation
  • Skills in evaluating and auditing AI systems
  • Awareness of ethical considerations in algorithmic decision-making

Findings and Framework

The increasing efforts to implement AI across various public sector domains have led to a critical question: what types of professionals, equipped with what specific skills and competences, should lead the integration of responsible AI? Defining the essential set of skills, knowledge, and professional competences required for the effective and ethical deployment of AI in both public and private sector services becomes a key priority.

A key outcome of the workshop was the identification of a recurring set of challenges affecting. These include:

  • Low levels of trust
  • Lack of transparency
  • Unclear ownership and responsibility
  • Insufficient stakeholder awareness
  • Limited AI literacy and governance skills

To address these challenges, we propose the conceptual framework depicted in Figure 1. This framework highlights the urgent need to cultivate professionals who combine technical expertise, ethical sensitivity, and domain-specific knowledge to lead responsible AI integration. It maps the interconnected layers that influence the deployment and responsible use of AI technologies in public services, including:

  • Contextual Challenges – such as low trust, resistance to change, and limited organisational readiness
  • Structural Barriers – including unclear project ownership, inadequate governance frameworks, and insufficient infrastructure
  • Skill and Knowledge Gaps – highlighting the lack of AI literacy, ethical awareness, and domain-specific competences
  • Stakeholder Roles – outlining the importance of identifying and engaging relevant actors (e.g. policymakers, IT professionals, legal advisors, and citizens) throughout the AI lifecycle

This framework is intended to guide structured reflection and planning around AI deployment, helping ensure that technologies are not only functional but also trustworthy, inclusive, and aligned with public values. As such, it can serve as a practical tool for organisations seeking to integrate AI technologies responsibly. It encourages a systemic perspective, one that moves beyond technical feasibility to consider broader organisational, social, and ethical dimensions.

By applying this framework, decision-makers and project leads can:

  • Identify context-specific challenges before adopting AI
  • Map key stakeholders and clarify roles and responsibilities
  • Recognise skill and competence gaps that must be addressed
  • Design AI initiatives that align with principles of transparency, accountability, and fairness

Figure 1. The framework summarising the workshop’s thematic discussions

In conclusion, the workshop underscored that a stakeholder-oriented, challenge-driven approach is key to enabling responsible AI adoption. By starting with specific domain needs and mapping corresponding skills and knowledge, organisations can more effectively navigate the complex landscape of AI integration.

Civil Sector Vulnerabilities and NATO’s Strategic Role: The Case for International AI Governance

Jason Tucker (Researcher, Institute for Futures Studies, Sweden. Adjunct Associate Professor, AI Policy Lab, Department of Computing Science, Umeå University)


Published on 14 May 2025
PDF
Cite

Adapted from a presentation given at the NATO Science for Peace and Security Programme, Advanced Research Workshop “Clicking the Pause: The Role of Transatlantic Cooperation in AI Supervision”, Salamanca, Spain, 8-9 May 2025.

As AI becomes increasingly embedded in critical societal functions, the need for robust, internationally coordinated governance grows more urgent. While some national and regional regulation of AI is emerging, applications in defence and international security often remain exempt from these initiatives. This historical separation between civil and defence sectors is understandable given the unique operational requirements of the military. However, it risks creating a false dichotomy—suggesting that AI use in civil domains is largely divorced from international security concerns. However, the geopolitical implications of AI in the civil sector are profound and escalating (Schaake, 2024).

To illustrate this, healthcare provides a concrete and urgent example. Across NATO members and partners, localized and largely disconnected decisions are being made to adopt small-scale AI solutions in healthcare. With states having limited capacity to develop in-house solutions, they often turn to external actors. Doing so means that they are then subject to a complex and opaque web of global supply chains and international actors. This poses substantial risks, including vulnerabilities to cyber-attacks, dependencies on potentially hostile states or corporations, and strain on critical infrastructure to support its adoption.

The growing instability of the international order compounds these challenges. The United States has recently exhibited unpredictability in both its Administration and its corporate tech sector. Even if diplomatic relations are maintained, trust at the local level is harder to rebuild. Working with partners whose long-term reliability is in question introduces significant risk, and other non-traditional partners become more appealing. Where these actors are not aligned with NATO, this could be a vulnerability.

Moreover, the adoption of AI in the civil sector has been driven by techno-solutionism — the prioritisation of technological fixes that neglects broader societal and security trade-offs, as well as potentially more appropriate non-technical solutions. It glosses over the reality that AI, as a socio-technical system is embedded in cultural, institutional, and ethical contexts and requires participation from a broad range of actors to function at its best.

Healthcare systems are particularly susceptible to this narrative (Strange and Tucker, 2024). They face resource constraints that limit the capacity to develop, implement, and secure AI technologies. Combined with the dominant discourse being that AI is the only and best way to solve a broad range of healthcare issues, everyday actors in healthcare are facing pressure to adopt AI where they can. At the same time, NATO’s security infrastructure is drawing from the same limited resource pool—particularly in terms of skills, energy, data infrastructure, and cybersecurity capacity. Without careful coordination, this could lead to a zero-sum scenario, undermining societal resilience and military advantage.

Cybersecurity threats to healthcare are well documented. The World Health Organization has recognized that cyber-attacks targeting health systems have considerable consequences in terms of public health and international security (WHO, 2024). In 2021, WHO reported that one-third of global healthcare institutions had suffered at least one ransomware attack in the preceding year (Mishra, 2024). The European Union reported that in 2023, healthcare was the most targeted critical sector in cyber-attacks (WHO, 2024). During the COVID-19 pandemic, healthcare was not just a target but a vector for disinformation and destabilisation by state and non-state actors alike. Given these risks, decisions about AI adoption in critical civil sectors like healthcare cannot be made in isolation from geopolitical and security considerations. Yet most local actors are not equipped to understand or navigate these complex dynamics. The absence of coherent guidance or frameworks linking AI adoption to national and international security exacerbates vulnerability, weakens societal resilience, and increases dependence on untrustworthy partners.

Global AI governance is essential. It can establish the guardrails necessary to manage these risks and guide responsible adoption of AI technologies across sectors. NATO has a critical role to play here. By integrating civil sector AI governance into its strategic thinking, and engaging with the Allies on this, NATO can help ensure that AI adoption enhances—not undermines—resilience and collective security. This will allow for a more realistic assessment of the trade-offs involved in AI adoption, especially in sectors like healthcare that are both vital to public well-being, are particularly vulnerable to attack and a conduit for hostile actors to cause societal disruption. NATO’s role here should be seen as complementing other international AI governance efforts, such as those by UNESCO, OECD and the EU etc. This would ensure that these governance structures do not become dominated by military priorities and bridge the gap between the defense and civil sector. Democratic safeguards, such as civil society oversight or public reporting, for any NATO-related initiatives affecting the civil sector, would also be essential. As would multidimensional and multidisciplinary views on civil resiliency frameworks.

AI in the civil sector is not a technical or administrative matter alone—it is a strategic issue with implications for the stability, security, and cohesion of NATO members’ and partners’ societies. Only through coordinated, international governance, can we navigate this new terrain with the prudence and foresight it demands.

References

Mishra, V., (2024) Cyberattacks on healthcare: A global threat that can’t be ignored, UN News, https://news.un.org/en/story/2024/11/1156751.

World Health Organization., (2024), Ransomware Attacks on Healthcare Sector ‘Pose a Direct and Systemic Risk to Global Public Health and Security’, Executive Tells Security Council, https://press.un.org/en/2024/sc15891.doc.htm.

Schaake, M., (2024). The Tech Coup. Princeton University Press.

Strange, M. and Tucker, J., 2024. Global governance and the normalization of artificial intelligence as ‘good’ for human health. AI & SOCIETY, 39(6), pp.2667-2676.

Further Information

This article is part of the Politics of AI & Health: From Snake Oil to Social Good funded by The Wallenberg AI, Autonomous Systems and Software Program – Humanity and Society (WASP-HS).

Keywords (comma separated):

NATO, Civil Sector, Artificial Intelligence, Security, Healthcare, Governance

Related URL (if any):

https://www.iffs.se/en/research/research-projects/politics-of-ai-health-from-snake-oil-to-social-good/

How to cite this article:

Tucker J. (2025). Civil Sector Vulnerabilities and NATO’s Strategic Role: The Case for International AI Governance. AI Policy Exchange Forum (AIPEX). https://doi.org/10.63439/UJML2174

Responsible AI Retreat at Lövånger 

On March 17-20, 2025 at Lövånger, AI Policy Lab (AIPL), Responsible AI group and Research Group for Socially Aware AI had a 3.5-day retreat with 15 participants from both units and Francien Dechesne from Leiden University, Netherlands.

Participants discussed responsible AI from multiple perspectives – technological, ethical, and social. Below we share the key messages from the retreat. We hope to inspire future multidisciplinary discussions, workshops and projects related to responsible AI research, literacy, and practical solutions for diverse stakeholders in Sweden and internationally.

Question Zero & Human Responsibility

When we hear about new AI ventures, we need to ask the question zero: “Is the adoption of an AI tool the best solution for our current problem?” with assessments including the environmental costs of running this AI system and understanding of what specific aspects of operations it improves. This includes critically assessing who is participating, what is the provenance and the management of data, what are the bases for modelling and design choices, how will results and impact be evaluated. AI does not happen to us, we [humans] design and/or adopt and use the AI systems. 

  • Responsible AI development should follow a compositional approach, where verified datasets and models with clear principles can be combined to create new systems. This framework emphasises the need to balance ethical considerations and accuracy while accounting for differences across sectors, industries, and global value systems. The approach prioritises sustainability, transparency, control, regulation, participation, inclusiveness, trust, and fairness, with a focus on measuring societal implications and ensuring equal representation and identification. 
  • True AI fairness goes beyond mathematical equality, considering historical disparities and contextual factors, and diverse definitions of fairness, as mathematical fairness alone can still create discriminatory outcomes in complex social settings. 
  • Participation, where people meaningfully contribute to (design) decisions, strengthens democracy and supports responsible design, development, and use of AI. However, participation alone does not guarantee better outcomes; thoughtful design is needed to prevent manipulation and address blind spots by incorporating diverse perspectives. 
  • Technological change is ecological, not additive, in that it is transformative and systemic rather than simply incremental, fundamentally reshaping existing environments and society. Drawing from Neil Postman’s (1995) framework, technological innovations fundamentally transform existing systems rather than simply adding new capabilities, creating trade-offs, winners and losers, and often leading to reframing how we perceive the world. 
  • Structural challenges. Several “traps” hinder responsible AI research: the framing trap (failure to model entire systems), portability trap (ignoring context sensitivity), formalism trap (oversimplifying social concepts), ripple effect trap (missing ecological impacts), and solutionism trap (overreliance on technological fixes). 
  • AI Art. The concept of “AI as a mirror” raises important questions about artistic expression, with significant differences between the inner experience of human artistic creation versus AI-assisted art generation, challenging traditional perspectives on IP rights, creativity (human only, genAI only, human-AI hybrid), intention, and artistic value. 

If the ideas above resonate with you, we encourage to check out the AI Policy Lab LinkedIn page for common cooperation opportunities. 

Beyond the AI race: why global governance is the greatest innovation

Virginia Dignum (Wallenberg Chair, Professor Responsible Artificial Intelligence, Director of AI Policy Lab at Umeå University)


Published on 8 April 2025
PDF
Cite

As artificial intelligence (AI) evolves rapidly, the urgency to govern it responsibly becomes more pressing by the day. We are standing at a pivotal moment, one where the choices we make today will shape not just technological outcomes, but also the foundations of our societies, economies, and planetary well-being.

Even as the UN and other international agencies advocate for global AI regulation, major players – particularly the US, UK, and China – seem increasingly hesitant to fully commit. While the US and UK are currently moving towards light-touch, innovation-driven approaches that prioritize industry leadership over binding rules, China leans toward a state-controlled model aligned with its national priorities. Their reluctance undermines efforts to build the effective, inclusive governance frameworks we urgently need, and may encourage others to also sideline global cooperation in favour of fragmented, self-serving strategies.

But AI governance is not optional, it is essential. It protects rights, upholds global values, and ensures long-term economic stability and sustainable innovation. Without global governance, we open the door to a race to the bottom, marked by short-term thinking, ethical shortcuts, and growing global inequality. We cannot allow geopolitical competition to derail the collective responsibility required to ensure AI serves the common good. Now is the moment to strengthen our commitment to global, values-driven governance, not to stall it. Meanwhile, AI governance shouldn’t chase every new technology, but instead follow clear principles: transparency, fairness, explainability, and accountability. These form a foundation for adaptable policy that protects rights and safety. Tools like regulatory sandboxes, public engagement, and stronger international coordination support this flexible yet high-standard approach as AI evolves.

The best competitive advantage is not ruthless speed, but wise collaboration, especially when the stakes include trust, stability, and the health of our planet. In this context, the European Union’s €200 billion investment in regulated, human-centered AI stands out. This visionary approach demonstrates how regulation can act not as a brake on innovation but as a stepping stone for it. The EU’s commitment to ethics, inclusion, and sustainability offers a powerful alternative to the more narrowly competitive models pursued by the US and China. Yet, funding alone is not enough. Investment must be accompanied by sustainable practices, equitable access, and strengthened social cohesion. Other countries—Canada, Japan, Brazil—are also making important strides. But this is not a race with a single winner. It’s a collective effort, and meaningful progress depends on a globally aligned framework that ensures AI serves all of humanity.

Still, I am deeply concerned about the growing competition to dominate the AI landscape. China, the US, and others are increasingly viewing AI as a tool of economic and military supremacy. This race risks concentrating power in a handful of nations or corporations, sidelining most of the world and worsening inequality. China, the US, and others view AI through the lens of strategic dominance. But AI is not a zero-sum game. True progress requires transparency, ethical alignment, and shared governance.

One of the greatest ethical challenges today is the erosion of human agency through opaque, unaccountable AI systems. That’s why I advocate for Earth alignment, as we introduced in a recent article in Nature Sustainability. This framework emphasizes the need for AI governance to be anchored in environmental sustainability, global justice, and societal cohesion. These goals cannot be achieved in isolation or through regional silos. They require a shared commitment to values that transcend borders, and the democratization of governance. A small group of governments and companies cannot be allowed to shape society through their control over AI development, nor solely through the lens of existential threats and geopolitical rivalry.

Responsible development requires systemic change, not just technical fixes. Ethics must be embedded from the outset, but we must do so through systemic change, not fear-mongering. This is not just a question of innovation; it’s a matter of justice. AI should not be a tool that widens global divides or undermines democracy and social cohesion. It should be a force for empowerment, equity, and resilience. That’s only possible through shared governance, transparency, and ethical alignment across all borders, including those of the most powerful players.

This is why global governance of AI is not optional, it is urgent.

Looking to the future, what excites me most about AI is its potential to empower us, not to replace us. If we govern it well, AI could become one of our most powerful tools for addressing climate change, improving healthcare and education, and advancing equity and social cohesion. But that future is not guaranteed. It depends on the choices we make now. The future of AI is not just about building smarter machines and software, it is about working together towards a wiser humanity. One that values cooperation over competition, solidarity over supremacy. One that uses AI not to dominate, but to heal and uplift.

There is no alternative: in the long run, only responsible AI will lead to innovation that truly benefits society. Anything else will not only undermine trust and human rights but will also lead to technically weaker systems and a loss of true innovation. Irresponsible AI may promise short-term advantages, but it will cost us our long-term future.

Responsible AI is not the finish line. It is the only viable path forward.

Keywords (comma separated):

AI governance, global cooperation, responsible AI, sustainability, transparency, regulation

How to cite this article:

Virginia D. (2025). Beyond the AI race: why global governance is the greatest innovation. AI Policy Exchange Forum (AIPEX). https://doi.org/10.63439/LNQA3726

Potential impact of the EU Platform Work Directive on AI labelers

Mariia Lesina (Lund University)


Published on 25 March 2025
PDF
Cite

Introduction

The gig economy has revolutionized labor markets, introducing high flexibility while simultaneously raising critical concerns about workers’ rights and protections. Across the European Union (EU), 3% of people aged 15-64 rely on digital platforms to access employment, yet many remain trapped in a legal grey zone, lacking social security, job stability, and transparency in how their labor is managed (Eurostat 2024).
While the attention has been largely drawn to the impact of AI algorithms on the workflows of such gig workers, most notably drivers or delivery people, one group has been largely overlooked – the invisible workforce behind AI. These workers perform essential tasks such as data tagging, annotation, and content moderation, which help train AI models (Muldoon et al. 2024, 9-10). Their work is often fragmented and distributed through digital labor platforms, making them susceptible to unfair algorithmic control, precarious employment conditions, and opaque remuneration structures. AI labelers work asynchronously across different time zones and are also subject to algorithmic management, where automated systems monitor their accuracy, determine their pay, and can even suspend or terminate their accounts without clear justification (Ibid, 12-15). Evidence of this can be easily found on community discussions on Reddit where stories of such workers are widespread, telling how their work was declined and payment delayed or cancelled altogether (Nagaraj Rao et al. 2025, 23). One of the recent attempts to address this systematic issue was taken by the EU via the Platform Work Directive, features and implications of which will be analyzed in this article.

EU Platform Work Directive

In 2024 the EU introduced the Platform Work Directive, a regulatory framework designed to enhance labor protections, redefine employment classifications, and establish oversight mechanisms for AI-driven algorithmic management. The EU Directive recognizes AI labeling as a form of platform work if it is conducted through a digital platform within the EU and based on a contractual relationship. Article 19 of Introduction specifically mentions tagging as a form of crowd work that can be conducted remotely. This recognition aims at allowing AI labelers to benefit from the Directive’s employment presumption when platforms exercise control over their workflows, task assignments, or performance evaluations. More importantly, it states that it is platforms that have to prove employment status of their platform staff, as the latter usually lacks the means and leverage to do so (Articles 30, 31).
The core of the regulations is that “platform work should be provided with rights, with a view to promoting transparency, fairness, human oversight, safety and accountability” (Article 14). These goals will be achieved through a number of legal changes, translated into the national legislation from the EU level. To begin with, Article 10 of Chapter III enforces human oversight in algorithmic decision-making and mandates greater transparency in how these systems operate, granting AI labelers the right to request detailed explanations of algorithmic decisions that impact their work. This means that platforms cannot solely rely on automated systems to suspend or deactivate an AI labeler’s account. Instead, these decisions must “ensure human oversight and regularly carry out an evaluation of the impact of individual decisions taken or supported by automated monitoring systems”: essentially demanding human in the loop, ensuring that workers are not unfairly penalized by flawed algorithms (Article 47). Additionally, AI labelers now have the right to contest algorithmic decisions, demand explanations, and request human reviews of automated rulings that affect their employment status, pay, or continued access to work (Article 8). The Directive also states that platform workers’ representatives “should be involved in the evaluation process” of these automated systems (Article 44).
The mental and physical well-being of platform workers is another crucial aspect addressed by a separate Article 12 of Chapter III. Due to the nature of their work – repetitive tasks, exposure to harmful content, and tight deadlines – AI labelers face unique risks to their mental and physical health. The Directive requires platforms to assess and mitigate these risks, ensuring that AI systems do not “put undue pressure on platform workers or otherwise puts at risk their safety and physical and mental health”. Platforms must now provide effective information and consultation for workers, while Member States – ensure that digital labour platforms take “preventive measures, including providing for effective reporting channels” (Article 12 of Chapter III).

Possible shortcomings of the Directive

While the Directive provides a theoretically effective legal framework, its success is based on effective enforcement. One major concern is that companies will exploit national law loopholes, restructuring their business models to avoid classifying workers as employees. This issue has been explored by an organization called Fairwork, which evaluates the work conditions of digital labour platforms, and whose extensive analysis highlights large reliance on contextual enforcement in the country-specific legislation. Therefore, as Fairwork experts state, in member states where the “power of labour unions is undermined”, like, for instance, in Italy, the workers can remain unprotected and “self-employed” (Fairwork Project 2024, 2).
Moreover, as experts from International Labour Organization (ILO) explore, outsourcing work to foreign workers in countries where the cost of labour is lower is common as it enables businesses to optimize their costs (Rani et al. 2021, 22). Hence, the Directive, which only protects workers within the EU, does not address the issue of exploitation of non-EU based platform staff.

Conclusion and further discussion

The EU Platform Work Directive is a crucial step toward recognizing AI labelers as platform workers, granting them employment protection, transparency, and human oversight in algorithmic management. However, its impact depends on consistent enforcement across Member States and preventing platforms from exploiting loopholes or outsourcing labor beyond the EU’s reach.
As the December 2026 deadline for national implementation approaches, the fight for fair AI labor practices is only just beginning. With platforms already pushing back, arguing that regulation stifles innovation and contradicts the inherent advantage of flexibility that digital work provides (Copenhagen Economics 2021, 24), the real test will be in Directive’s practical integration. Whether the EU emerges as a pioneer in ethical AI labor governance or struggles with unintended consequences will depend on the ability of each state to implement the Directive in a way that is both principled and pragmatic. Hence, the most topical question remains: “Will this policy set a global precedent for fair AI labor practices, or will fragmented enforcement and corporate resistance weaken its impact?”
To achieve the first outcome, EU policymakers, researchers, and labor rights advocates will need to continue pushing for a regulatory environment that ensures that this overlooked groups in the gig economy – AI labelers – receive the protections they deserve.

References
1. Copenhagen Economics. The value of Flexible work for local delivery couriers. Study for Delivery Platforms Europe. November 2021. 28 p.
2. European Parliament and Council Directive (EU) 2024/2831 of 23 October 2024 on improving working conditions in platform work [2024] OJ L283/1. Internet source. URL: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32024L2831 (accessed 7 March 2025).
3. Eurostat. Experimental statistics on digital platform employment. 18 July 2024. Internet source.URL: https://ec.europa.eu/eurostat/web/products-eurostat-news/w/ddn-20240718-1#:~:text=In20223.0ofpeople,countriesand1EFTAcountry (accessed 7 March 2025).
4. Fairwork Project. Fairwork’s Response to the EU Directive on Platform Work. March 2024. Internet source. URL: https://fair.work//srv/www/wp-content/test.aipolicylab.se/uploads/sites/17/2024/03/Fairworks-Response-to-the-EU-Directive-on-Platform-Work.pdf (accessed 7 March 2025).
5. Muldoon, J., Graham, M., Cant, C. Feeding the Machine: The Hidden Human Labour Powering AI. Canongate Books. 2024. 288 p.
6. Nagaraj Rao, V. Dalal, S., Agarwal, E., Calacci, D., and Monroy-Hernández, A. Rideshare Transparency: Translating Gig Worker Insights on AI Platform Design to Policy. ACM Hum.Comput. Interact. No. 9, 2. April 2025. Pp. 1-49.
7. Rani, U., Rishabh, K.D., Furrer, M. G?bel, N. Moraiti, A. and Cooney, S. World employment and social outlook: the role of digital labour platforms in transforming the world of work. Geneva: International Labour Office. 2021. 283 p.
8. Silberman, M.S., Adams-Prassl, J., Abraha, H. and Suresh, R., Doth the Platform Protest Too Much? Uber, Employment Status, and the EU’s Proposed Platform Work Directive. Oxford Law Blogs, 28 September 2023. Internet source. URL: https://blogs.law.ox.ac.uk/oblb/blog-post/2023/09/doth-platform-protest-too-much-uber-employment-status-and-eus-proposed (accessed 7 March 2025).

Keywords:
EU Platform Work Directive, AI labeling, Gig Economy, Human Rights

How to cite this article:

Lesina M. (2025). Potential impact of the EU Platform Work Directive on AI labelers. AI Policy Exchange Forum (AIPEX). https://doi.org/10.63439/APJD5342

Metadata/README elements for synthetic structured data made with GenAI: Recommendations to data repositories to encourage transparent, reproducible, and responsible data sharing

Ericka Johnson (Dept of Thematic Studies, Linköping University)
David Rayner (Swedish National Data Service, University of Gothenburg)
Jools Kasmire (University of Manchester), Victor Hennetier (Dept of Thematic Studies, Linköping University), Saghi Hajisharif (Dept of Science & Technology, Linköping University), Helene Ström (Fair AI Data)


Published on 20 March 2025
PDF
Cite

Introduction

Publication of AI-generated synthetic structural data in data repositories is beginning to reveal the specific documentation elements that need to accompany synthetic datasets so as to ensure reproducibility and enable data reuse.
 
This document identifies actions that research repositories can take to encourage users to provide AI-generated synthetic datasets with appropriate structure and documentation. The recommendations are specifically for AI generated data, not (for example) data produced using pre-configured models or missing data created by statistical inference. Additionally, this document discusses metadata/README elements for synthetic structured datasets (tabular and multi-modal) and not textual data from LLMs or images for computer vision. 

The document is the result of a workshop held on 23rd January 2025, with participants from the Swedish National Data Service, Linköping University and Manchester University. It also draws on survey responses about current practice from 17 data repositories and a review of existing metadata and README requirements. 

Background

AI-generated synthetic structured datasets are generated using machine learning techniques with the aim of reproducing the essential elements of an existing dataset (Guépin et al., 2024 Jacobsen, 2023; Li et al., 2023; Offenhuber, 2024; Savage 2023). Synthetic data generation may be driven by the need to ensure privacy or to expand, enhance or substitute for real-world datasets which may be insufficient or non-existent. Sometimes synthetic data is produced to create a portable or shareable dataset that is considered safe for open access, for example to share via a data repository. 

While synthetic structured data may reproduce the essential elements of an original dataset, it will also inevitably introduce “intersectional hallucinations”, which refer to anomalous inter-attribute relations within a dataset (Lee, Hajisharif & Johnson 2025). AI generated synthetic data also have a known tendency to minimize minority elements and amplify majority elements (Chen et al., 2024; Johnson & Hajisharif 2024). Thus, knowing in what ways a synthetic dataset demonstrates fidelities and in what ways it is ‘different’ from the original data is essential for successful and responsible re-use of synthetic data. Given that the goal of many data repositories is to provide access to data that is replicable and/or reusable, there is a clear need to establish protocols for documenting synthetic data.

Primary recommendations

Our primary recommendations are:

a) that data repositories establish a standardized way to label data as synthetic data, and that this information is prompted-for or required in the metadata or READMEs associated with synthetic datasets. 

b) that data repositories provide users with a guide that explains how to properly document synthetic data. The extent to which documentation should be provided with the dataset or provided in associated articles or publications linked to the data will depend on the policies of the repository. An example is the guide provided by the Swedish National Data Service (2025). 

c) that domain experts be prompted to document the context and motivation for generating synthetic data.

Documenting synthetic data – process and product

Reusability often refers to the data as a product. In the case of synthetic structured data, however, it may be the method of data generation (the data as a process) that  is reusable, not the data itself. We therefore suggest that data repositories require information about both the process of data generation and details about the actual synthetic data.

Data as a process

The following elements should be included to describe the technical details of the synthetic structured data generation process: 

  • A description of the workflow.
  • The generative model used (i.e. GAN, Diffusion, etc.). As techniques are constantly evolving, these requirements should be formulated in such a way to allow for and capture new techniques. The structure and hyperparameters (learning rate, number of epochs, etc.) of the generative model are also important factors for reproducibility and should be included.
  • What raw data or inputs, if any, were used, including its mode of collection. A link to the source of the raw data should be provided where appropriate.
  • Which (random) seeds were used.
  • If a subset of raw data were reserved for testing, how was this subset selected?
  • Versions of the software and packages used.
  • Operating system information, values for relevant environment variables.
  • A link to the source code (we suggest keeping code in a separate repository so it can be reviewed, improved, and re-released) and if appropriate a link to the weights of the trained model.
  • Citation details (including DOIs) for related documents or the release versions of code.

Additionally, some cases of synthetic data are not based on raw data (e.g. agent based modeling/multi agent systems, digital twins). In such cases, this should also be clearly stated in the description of the data generation process. 

If a repository considers that publishing the data generation model is out-of-scope, we suggest providing information on how models can be deposited in either a more generic repository or in a specific repository for models. Links to the model can then be provided in the dataset metadata and/or README.

Product

Synthetic structured datasets inevitably contain stochastic variability, meaning that different datasets can be obtained by running the same code multiple times with different random seeds. We therefore suggest that metadata/READMEs also contain information about:

  • whether the dataset is entirely synthetic or augmented. If it is augmented, what are the proportions of real and synthetic data?
  • missing edge cases at the single-attribute level and inter-attribute level. 
  • inter-attribute combinations in the raw data that have diminished frequency in the synthetic data.
  • inter-attribute hallucinations that have been observed in the synthetic data. 
  • details of the verification/validation process: how was the model tested, etc.
  • how the synthetic data are structured at the file-level: are the input data in a folder marked “raw” or “input”, and output in an “output” folder?

Privacy and specific circumstances

A common use-case for synthetic data is when privacy assurance is necessary for sensitive data. In such cases, we recommend the metadata/README contain information about disclosure risk, indication risk, reidentification risks, K-anonymity, etc. This type of synthetic data requires extra care and should only be made freely available if specific individuals cannot be re-identified by any reasonably likely means.

We also suggest that repositories include instructions on creating the metadata/README that will prompt domain experts to explain the specific circumstances of their synthetic data. Why was it generated? What is the fundamental hypothesis behind the synthetic dataset’s use? What is its subject and purpose(s)? Data creators should be encouraged to disclose, for example, if the dataset was created for exploratory research, to represent sensitive data, to allow for work by a distributed team, to enable data portability, to create categories or support classification decisions, etc.. Encourage data submitters to consider sensitive areas and intersections within the data, as well as how many relational intersections are valuable to combine when using the dataset for new research purposes.

Summary and discussion

Synthetic structured data may be produced where scientific research requires data with no personal information, data that are portable and shareable, data which are not obtainable for practical or ethical reasons, or large datasets for machine learning. However, the details of the generation process and the variations inherent in synthetic data need to be documented, either in a dataset’s metadata/README or in the articles accompanying the dataset.

Many aspects of synthetic data are still emerging, and in some cases, we lack established routines or even vocabularies for them. We hope the recommendations in this policy document will serve as a starting point for further discussions. In particular, we aim to encourage those working with data repositories to collectively establish best practices for managing synthetic data and developing vocabularies to describe them. For example, we might promote an accepted keyword or subheading, such as SYNTHETIC_DATA, or suggest appending “_synth” to filenames containing synthetic data. Additionally, controlled vocabularies should include subcategories to distinguish between fully synthetic and blended/augmented data.

With a well-defined vocabulary and clear metadata guidelines, repositories can help researchers to describe both their datasets and the processes used to create them in an open, transparent, and reproducible manner, ensuring responsible data sharing within the scientific community.

References

??Chen W, Yang K, Yu Z, et al. (2024) A survey on imbalanced learning: latest research, applications and future directions. Artificial Intelligence Review 57(6): 137.

Guépin F, et al. (2024) Synthetic Is All You Need: Removing the Auxiliary Data Assumption for Membership Inference Attacks Against Synthetic Data. In: Katsikas, S., et al. Computer Security. ESORICS 2023 International Workshops. ESORICS 2023. Lecture Notes in Computer Science, vol 14398. Springer, Cham. https://doi.org/10.1007/978-3-031-54204-6_10.

Jacobsen BN (2023). Machine learning and the politics of synthetic data. Big Data & Society. 10(1).  

Johnson E and Hajisharif S (2024) The intersectional hallucinations of synthetic data. AI & Society. https://doi.org/10.1007/s00146-024-02017-8. 

Lee, Hajisharif & Johnson (2025) The ontological politics of synthetic data: normalities, outliers, and intersectional hallucinations. Big Data & Society.

Li X, Wang K, Gu X, Deng F, Wang FY (2023) Parallel eye pipeline: An effective method to synthesize images for improving the visual intelligence of intelligent vehicles. IEEE Transactions on Systems, Man, and Cybernetics: Systems. 53(9), 5545-5556.

Offenhuber D (2024) Shapes and Frictions of Synthetic Data. Big Data & Society. 11 (2): 20539517241249390. https://doi.org/10.1177/20539517241249390.

Savage, N (2023) Synthetic data could be better than real data. Nature Machine Intelligence. doi: https://doi.org/10.1038/d41586-023-01445-8.

Swedish National Data Service. (2025). Managing and publishing synthetic research data (Version 1). Zenodo. https://doi.org/10.5281/zenodo.14887525 

How to cite this article:

Johnson, E., Rayner, D., Kasmire, J., Hennetier, V., Hajisharif, S., & Ström, H. (2025). Metadata/README elements for synthetic structured data made with GenAI: Recommendations to data repositories to encourage transparent, reproducible, and responsible data sharing. AI Policy Exchange Forum (AIPEX). https://doi.org/10.63439/MPEW5336

Towards Successful Industrial Policy on AI in Healthcare: Establishing the Conditions for Future Public Benefit

Erez Maggor (Assistant Professor, Ben-Gurion University and the Institute for Futures Studies). 
Jason Tucker (Researcher, Institute for Futures Studies, Adjunct Associate Professor, AI Policy Lab @Umeå University and Visiting Research Fellow, AI & Society, LTH, Lund University).

Published on 18 March 2025
PDF
Cite

Abstract

This paper explores how pro-active government policies could promote artificial intelligence (AI) in healthcare for the public good. Building on insights from the literature on industrial policy, we argue that without clear conditions and guardrails to ensure future public benefit, state assistance and subsidies will be nothing more than corporate welfare with unpredictable, if any, societal benefit. We provide a few concrete examples of this and then conclude by reflecting on how industrial policy can be a useful conceptual lens to challenge techno-solutionism, increase accountability and situate AI healthcare policies in the broader political-economy.

Keywords: Industrial Policy, Corporate Welfare, Artificial Intelligence, Healthcare, Public Interest, Conditionality, Futures.

Introduction

States and regions are increasingly turning to interventionist policies to try and realise the benefits of the development and deployment of artificial intelligence (AI). This is highlighted by the growing number of national and regional AI strategies, where bold visions for how AI will solve a plethora of societal problems are set out (OECD, 2025). For example, just recently the EU announced the 200 billion Euro regional InvestAI, the largest AI investment in history “to make Europe the continent of AI” (European Commission, 2025). This optimism is possible as AI is a useful empty signifier for various visions of technological innovation (Kak, 2024). These visions are often built upon previous AI policies and initiatives, thus reflecting a continuation of socio-technical imaginaries of AI (Bareis & Katzenbach, 2022). 

Healthcare has featured prominently in these strategies (Hoff, 2023). States are promising that the technology can solve a range of societal issues related to health, fix crumbling and underfunded healthcare systems, and improve our individual health (Tucker, 2023). Often, existing policies are market-based, which requires states to collaborate with the private sector. To get private firms on board, governments provide various incentives, including increased public funding of R&D and procurement, access to public health infrastructure and data, and cutting so-called red tape that allegedly “stifles” innovation. Regarding AI in healthcare, the recent increase in interest in industrial policies coincides with fluctuations in private sector investment in the medical and health sector – which rose sharply between 2019-2021, then plummeted afterwards (North, 2025). The sharp decline post 2021 means that relying on the private sector to address healthcare challenges without incentives is not likely to achieve states´ promised future visions. 

Such policies represent a gamble for states, as these technoscientific futures, which are based on the notoriously unpredictable development trajectory of AI, also rely on the alignment of private and public interests if this is realised. This begs the question of how we can improve the odds for these policies to succeed? To answer this question, we draw lessons from the literature on industrial policy. One of the central insights of this body of work is that when government subsidies are provided without proper conditionalities they are unlikely to produce desired socially beneficial results (Bulfone et al, 2023; Bulfone et al, 2024; Mazzucato & Rodrik, 2023).  We argue, therefore, that a critical element to the success of AI policies is the centralizing of the public good in the conditions of future benefit of AI. Without this in place these policies will end up as nothing more than corporate welfare in the guise of public interest innovation. 

Healthcare is arguably one of the sectors with the highest risk and reward in this regard. We have seen an “AI turn” in global and national health discourses (Strange & Tucker, 2024). AI is posited as being the best, and often only means, to address a broad range of individual, societal and systemic healthcare issues. Health is also intricately intertwined with social stability and democracy (Johnson & Longmore, 2023). In addition, the potential benefits of AI in the healthcare sector is often used to justify broader political agendas on AI, as we saw in the case of the EU’s InvestAI. As such, the success or failure of industrial policy on AI in healthcare has ramifications well beyond the sector. 

Industrial Policies versus Corporate Welfare

The historical record of industrial policy has been mixed. States like Japan, South Korea, Taiwan, Israel, France, and, most recently, China, have had remarkable success using industrial policies to upgrade their industries and ‘catch-up’ to more economically developed nations (Amsden, 1989; Johnson, 1982; Zysman, 1984; Wade, 2004; Maggor, 2021; Ang, 2018). However, in India, Turkey, and across Latin America, industrial policies are considered to have been a relative failure, leading to waste and corruption and resulting in these countries failing to advance economically or, at best, becoming stuck in the ‘middle-income trap’ (Doner & Schneider, 2016). The main lesson emerging from this mixed comparative-historical experience has been that rather than representing a dangerous or wrongheaded economic policy – as many on the ideological right often argue – industrial policies are, first and foremost, a daunting political challenge. 

While profit-maximizing firms will gladly accept various government subsidies and assistance that often accompany industrial policy efforts, they will always prefer this assistance be provided with limited strings and conditions. Policymakers, on the other hand, understand that subsidies without conditionalities are a form of corporate welfare, i.e., a gift from the public to the private sector. As a result, the main political challenge for policymakers is designing industrial policies that incorporate institutional mechanisms that dictate clear conditions regarding the future benefit of such public-private collaborations, ensuring the public benefit is protected, and equipping the state with the capacity to ‘discipline’ uncooperative firms (Amsden, 1989; Chibber, 2023; Maggor, 2021). Crucially, this needs to be implemented from the outset, as this is the stage at which policymakers have the greatest leverage over the private sector. To ground and contextualise this we can turn to two recent cases, the development of the COVID-19 vaccine in the USA and the NHS Google-DeepMind health data scandal in the UK. 

Developing the COVID-19 Vaccine in the USA 

One useful example to demonstrate the aforementioned logic is the development of vaccines for COVID-19. In 2020, in response to the pandemic, the Trump White House approved “Operation Warp Speed,” to develop an mRNA vaccine. In addition to federal support for the promotion of R&D, the program also provided government subsidies for scaling-up manufacturing (in the case of Moderna), offered strategic government procurement (in the case of Pfizer), and delivered various government assistance to industry across multiple sectors and regions (Adler, 2021). On the one hand, this experience represented an effective and comprehensive industrial policy program that saw the government partner with the private sector to produce a significant social good– a safe and effective vaccine that helped end a global health emergency. On the other hand, it showed that when state-supported innovation is not governed for the common good via strict terms and conditionalities, many people remain excluded from its benefits. 

For example, even though Moderna used public investments and research to develop its vaccine, they refused to share intellectual property and know-how with less-developed countries (Mazzucato, 2023). The experience  also demonstrates that without imposing strict limitations on profit-taking, private firms that succeeded due to public assistance are likely to retain “astronomical and unconscionable profits”, in this case due to their monopolies of mRNA COVID vaccines — upwards of 69% profit margins in the case of Moderna and BioNTech (Wilson, 2021; Emergency USA, 2021).

The NHS Google-DeepMind Health Data Scandal

There are also examples of industrial policies on AI and health more specifically, such as the UK National Health Service (NHS) Google-DeepMind case. In 2015 the Royal Free London NHS Foundation Trust shared the personal health records of 1.6 million patients with Google’s AI firm DeepMind. This was to support the development of a system that could potentially better diagnose acute kidney injury. Successive UK governments have long courted Google to try and attract more of the firm’s investment in the UK market, so this public-private partnership came as no surprise. 

However, while the diagnostic application showed early signs of success in detecting kidney injury, the partnership hit the headlines in 2017 due to a significant public backlash. There were serious concerns about data privacy and the lack of transparency in how the sharing of publicly funded health data with DeepMind was decided upon. It was also unclear what DeepMind was using the data for, as well as there being an uncertain public benefit from the data transfer (Dickens, 2021). The Information Commissioner’s Office eventually ruled that the NHS had failed to comply with the Data Protection Act by transferring the data (Information Commissioner’s Office, nd), though a subsequent class action lawsuit against DeepMind itself failed in the UK courts. With the introduction of the UK’s AI Opportunities Action plan in January 2025, the case has raised its head once again (Milmo & Stacey, 2025). The legacy of the NHS DeepMind scandal thus lives on, impacting public trust in the next generation of industrial policies on AI. 

Establishing the Conditions for Future Public Benefit

The unpredictable nature of AI development poses challenges in terms of establishing the conditions of future benefits in industrial policies. However, these are not insurmountable, and as argued above, need to be overcome if the social benefit of the technological investment is to be realised. We should remember that conditions do not always need to be very specific (such as the quick development of a vaccine), or indeed directly related to healthcare. For example, supporting a flourishing MedTech sector that creates high skilled jobs, and pays corporation tax into the state coffers, could be seen as an acceptable outcome. Yet, there needs to be clear guardrails about how the public are to benefit from any technological innovation that an industrial policy has facilitated. Assuming private actors will act in the public good if a discovery is made has proven to be wishful thinking time and time again. 

The economist Mariana Mazzucato and her colleagues have outlined several conditionalities that could ensure the public shares the returns of public investment in health. These include charging royalties from companies who profit from technologies developed with public funding, with funds earmarked to finance future innovation. Another strategy could be for states to retain a “golden share” of patents developed with public assistance while incorporating weak and narrow (rather than strong and broad) intellectual property protections to ensure greater access for marginalised members of the community as well as developing nations. Finally, rather than paying exorbitant prices, public health systems should pay prices that reflect their contribution to the development of new therapeutic or health technologies (Mazzucato & Li, 2019; Mazzucato & Roy, 2019).

One should also remember that industrial policies do not only support the growth of certain sectors but can also be used to reduce redundant or harmful ones. Indeed, this insight has been highlighted in the context of the green transition, as policymakers seek to promote green technologies while, at the same time, phasing out the carbon economy (Ergen & Schmitz, 2023). Industrial policies on AI in health can be considered in a similar fashion by situating them in relation to the broader healthcare sector. Healthcare actors, patient groups, unions, and other civil society actors should play a key role here in deciding where to advance and reduce different public health infrastructures and services. This raises the issue of defining “public interest” and classifying the “public(s)”, which is a process of “power, politics and truth seeking” in AI (Sieber et al 2024, p634). How this impacts the establishment of future public interest in AI industrial policy requires context specific interrogation.    

Finally, with this piece we hope to encourage the further use of industrial policy as a conceptual lens to analyse the increase in interventionist policies around AI and healthcare. While this was only an initial foray into the potential of doing so, industrial policy proved to be a useful approach for several reasons. First, the need for conditionality in the future public benefit to be established pushes back against techno-solutionist narratives. It also facilitates a broader understanding of the purpose of these policies within the wider political economy. Thus, we are better able to wrestle with the reality that policies about AI in healthcare are not just about AI or indeed healthcare. Rather, they are about prioritising certain health futures, closing down others and advancing broader national and regional AI visions. 

References

Adler, D. (2021). Inside Operation Warp Speed: A new model for industrial policy. American Affairs, 5(2), 3-32.

Amsden, A. H. (1989). Asia’s next giant: South Korea and late industrialization. Oxford University Press.

Amsden, A. H. (1989). Asia’s next giant: South Korea and late industrialization. Oxford University Press.

Ang, Y. Y. (2018). How China escaped the poverty trap. Cornell University Press.

Bareis, J., & Katzenbach, C. (2022). Talking AI into being: The narratives and imaginaries of national AI strategies and their performative politics. Science, Technology, & Human Values, 47(5), 855-881. https://doi.org/10.1177/01622439211030007

Bulfone, F., Ergen, T., & Kalaitzake, M. (2023). No strings attached: Corporate welfare, state intervention, and the issue of conditionality. Competition & change, 27(2), 253-276.

Bulfone, F., Ergen, T., & Maggor, E. (2024). The political economy of conditionality and the new industrial policy (No. 24/6). MPIfG Discussion Paper.

Chibber, V. (2003). Locked in Place: State-building and Late Industrialization in India. Princeton University Press;

Dickens, AD (2021) The right to health implications of data-driven health research partnerships. PhD thesis, University of Essex.

Doner, R. F., & Schneider, B. R. (2016). The middle-income trap: More politics than economics. World Politics, 68(4), 608-644.

Emergency USA. (2021, September). Pharmaceutical companies reaping immoral profits from COVID vaccines, yet paying low tax rates. Emergency USA. https://www.emergencyusa.org/2021/09/pharmaceutical-companies-reaping-immoral-profits-from-covid-vaccines-yet-paying-low-tax-rates/?doing_wp_cron=1739796679.7930979728698730468750

Ergen, T., & Schmitz, L. (2023). The sunshine problem: Climate change and managed decline in the European Union (No. 23/6). MPIfG Discussion Paper.

European Commission. (2025, February 18). EU 200 billion Euros for AI, largest AI investment in history. European Commission. https://ec.europa.eu/commission/presscorner/detail/en/ip_25_467

Hoff, J. L. (2023). Unavoidable futures? How governments articulate sociotechnical imaginaries of AI and healthcare services. Futures, 148, 103131. https://doi.org/10.1016/j.futures.2023.103131

Information Commissioner’s Office. (n.d.). Google DeepMind and class action lawsuit. Information Commissioner’s Office. https://ico.org.uk/for-the-public/ico-40/google-deepmind-and-class-action-lawsuit/

Johnson, C., (1982). MITI and the Japanese Miracle: The Growth of Industrial Policy: 1925–1975, Stanford: Stanford University Press.

Johnson, D., & Longmore, S. (2023) How Healthy Is Democracy: The Role of Healthcare and Social Equity Considerations in the Governance of People. In Rethinking Democracy and Governance (pp. 345-369). Routledge.

Kak, A. (2024). What does “AI for the public good” really mean? AI Now Institute. https://ainowinstitute.org/general/ai-nationalisms-executive-summary#h-what-does-ai-for-the-public-good-really-mean 

Maggor, E. (2021). Sources of state discipline: lessons from Israel’s developmental state, 1948–1973. Socio-economic review, 19(2), 553-581.

Maggor, E. (2021). The politics of innovation policy: Building Israel’s “neo-developmental” state. Politics & society, 49(4), 451-487.

Mazzucato, M. (2023). Health for All: Transforming economies to deliver what matters. bmj, 381.

Mazzucato, M., & Li, H. L. (2019). Health innovation re-imagined to deliver public value.

Mazzucato, M., & Rodrik, D. (2023). Industrial policy with conditionalities: a taxonomy and sample cases.

Mazzucato, M., & Roy, V. (2019). Rethinking value in health innovation: from mystifications towards prescriptions. Journal of Economic Policy Reform, 22(2), 101-119.

Milmo, D & Stacey, K, (2025, January 13). Labour AI action plan for NHS patient data: Why it’s causing concern. The Guardian. https://www.theguardian.com/politics/2025/jan/13/labour-ai-action-plan-nhs-patient-data-why-causing-concern

North, M. (2025, January 22). 5 ways AI is transforming healthcare. World Economic Forum. https://www.weforum.org/stories/2025/01/ai-transforming-global-health/.

OECD. (2025). National strategies, agendas, and plans. OECD AI Policy Observatory. https://oecd.ai/en/dashboards/policy-instruments/National_strategies_agendas_and_plans

Sieber, R., Brandusescu, A., Adu-Daako, A., & Sangiambut, S. (2024). Who are the publics engaging in AI? Public Understanding of Science, 33(5), 634–653. https://doi.org/10.1177/09636625231219853 p.634

Strange, M., Tucker, J. Global governance and the normalization of artificial intelligence as ‘good’ for human health. AI & Soc 39, 2667–2676 (2024). https://doi.org/10.1007/s00146-023-01774-2

Tucker, J. (2023). The future vision (s) of AI health in the Nordics: comparing the national AI strategies. Futures, 149, 103154. https://doi.org/10.1016/j.futures.2023.103154

Wade, R. (2004). Governing the Market: Economic Theory and the Role of Government in East Asian Industrialization, Princeton University Press.

Wilson, M R. (2021, November 4). Vaccine manufacturers are profiteering. History shows how to stop them. Politico. https://www.politico.com/news/magazine/2021/11/04/vaccine-manufacturers-are-profiteering-history-shows-how-to-stop-them-519504;

Zysman, J., (1984). Governments, Markets, and Growth: Financial Systems and the Politics of Industrial Change. Ithaca: Cornell University Press.

Other

The authors contributed equally to this work.

Funding

JT’s contribution was made possible with the support of the Wallenberg AI, Autonomous Systems and Software Program – Humanity and Society (WASP-HS) funded by the Marianne and Marcus Wallenberg Foundation and the Marcus and Amalia Wallenberg Foundation.

How to cite this article:

Maggor, E., & Tucker, J. (2025), Towards Successful Industrial Policy on AI in Healthcare: Establishing the Conditions for Future Public Benefit, AI Policy Exchange Forum (AIPEX), https://doi.org/10.63439/PFRX3762

Bridging the Gap Between AI Research and Policy

The future of Artificial Intelligence (AI) lies not just in its technical advancements but in its responsible governance, underpinned by human-centered principles and policies. As such, AI policy research is an urgently needed area of focus, not just AI research, not just policy research, but a deliberate intersection of the two. This realization was at the core of the recent AI Policy Summit, a collaborative platform bringing together researchers from around the world and co-organised by MILA and the AI Policy Lab that I have the previlege to direct.  This was not just an event but a pivotal step toward shaping the trajectory of AI policy and governance. As AI technologies increasingly permeate every aspect of society, their potential to drive progress must be balanced with safeguards to ensure they align with human-centered values. This balance cannot be achieved by technical or legislative approaches alone; it demands the collaborative efforts of researchers, policymakers, and civil society.

The AI Policy Summit provided a unique platform for representatives from independent research organizations, spanning academia and civil society from diverse national contexts to engage in an open, informal environment that enable deep and heated exchange of ideas. A panel discussion with policymakers from multiple countries added depth and diversity to the discussions. Their contributions underscored the varying challenges and opportunities faced across different governance frameworks. Policymakers from Sweden, Tanzania, Canada, the Netherlands, and Portugal shared insights into their regional experiences with AI regulation, highlighting both shared objectives—such as transparency and accountability—and the unique cultural and legislative nuances that influence AI governance.

I was also especially encouraged by Marietje Schaake’s keynote, which highlighted the critical role of researchers in engaging with policymakers through building lasting relationships, providing actionable insights like policy briefs, and actively contributing to both the creation and implementation of legislation, all while acknowledging the challenges both sides face.

During the two days, exchanges between the participants emphasized the critical need for localized approaches to AI policy that are informed by global best practices. The summit fostered an environment where academic and civil society researchers could present evidence-based findings while gaining a firsthand understanding of the practical realities policymakers face. This interaction not only enriched the dialogue but also set the foundation for future collaborations aimed at shaping inclusive, effective, and context-sensitive AI policies.

Why AI Policy Research Matters

The development and governance of Artificial Intelligence (AI) are complex, interconnected challenges that demand a dedicated focus on AI policy research, a field distinct yet integrative of AI technology research and policy governance. This emerging discipline addresses gaps that neither AI research nor policy alone can resolve, ensuring that governance frameworks are not only informed by cutting-edge science but also aligned with societal needs and values. While AI research focuses on advancing technology and policy research on governance frameworks, neither can address the multifaceted impacts of AI in isolation:  

  1. AI advancements without Governance: Left unchecked, rapid AI innovation can deepen societal inequalities, exacerbate environmental damage, and consolidate power among a few, undermining public trust and equitable access?.
  2. Policy without AI research: Policies uninformed by empirical evidence or understanding of AI’s dynamic landscape risk becoming outdated, excessively restrictive, or misaligned with technological realities, stifling innovation and public benefits.

AI policy research as foundation for Responsible AI

Responsible AI begins well before algorithms are written or systems deployed: it starts with the fundamental questions: What problems are we solving? For whom? And with what consequences? What are the most suitable solutions? Is it AI? Addressing these questions requires a nuanced interplay between policy and research. The summit highlighted the growing need for this alignment to ensure that AI technologies foster societal progress, uphold human rights, and contribute to global sustainability goals.

At its heart, AI policy research navigates complex trade-offs. Fostering innovation while mitigating societal inequities requires a framework that ensures AI benefits are equitably distributed, particularly to those most vulnerable to its disruptions. AI policy research creates a vital bridge between these domains by focusing on actionable, evidence-based governance. It emphasizes transparency, accountability, and sustainability while ensuring equitable outcomes. By addressing issues such as inclusivity, environmental trade-offs, and regulatory foresight, AI policy research supports:

  • Proactive governance: Anticipating the implications of AI advancements demands foresight-driven policies that anticipate potential risks and societal impacts before they arise. By proactively identifying challenges—such as biases, security vulnerabilities, or unintended social consequences—governance frameworks can mitigate harm and establish safeguards that evolve alongside technological innovation.
  • Cross-sector collaboration: Effective AI policy requires a united effort from academia, industry, and government. Collaborative frameworks enable the sharing of expertise, aligning research insights with regulatory needs and industrial priorities. This synergy fosters the creation of policies that are both practical and evidence-based, ensuring comprehensive oversight and adaptability.
  • Responsible innovation: Encouraging the use of AI only when its benefits outweigh costs and align with ethical standards?. That is, AI should be deployed only when its advantages clearly outweigh associated costs and risks. Responsible innovation emphasizes ethical design, sustainability, and equitable access, ensuring that AI systems contribute to societal well-being without exacerbating inequalities or environmental harm.

The AI Policy Summit’s Contribution

The recent AI Policy Summit brought together global policymakers, academic researchers, and civil society actors to highlight this integrative approach. Discussions focused on immediate and long-term goals, such as fostering global accountability standards, developing foresight mechanisms, and crafting practical tools for inclusive governance. By emphasizing a shared roadmap and cross-sectorial expertise, the summit illuminated how AI policy research can drive actionable solutions for the responsible development of AI technologies?. This collective effort underscores the urgency of AI policy research as a means to guide innovation and governance toward equitable, sustainable outcomes. It is a field poised not only to mitigate the risks of AI but to maximize its potential as a force for societal good. Building on insights from the summit, several ideas were proposed to solidify the role of AI policy research, including:

  • Establish Visiting AI Policy Fellowships: These programs at different research institutes connect researchers with policymakers, fostering mutual understanding and collaboration?.
  • Launch an AI Policy Research Network: A global platform to share best practices, insights, and resources for evidence-based policymaking.
  • Develop AI Policy Briefs: Translating research findings into actionable insights tailored for policymakers is essential for informed decision-making.
  • Focus on Education and Capacity Building: Initiatives like student exchanges and Erasmus programs can cultivate a new generation of leaders at the intersection of AI and governance?).

A Shared Responsibility

AI policy research is not just a necessity, it is an opportunity to ensure that AI serves humanity rather than shaping societies in ways that exacerbate inequities or environmental harm. By combining the rigor of scientific inquiry with the pragmatism of governance, this field provides a pathway to align AI innovation with ethical, human-centered values.

The AI Policy Summit marked the beginning of a critical journey, one that bridges the gap between technological innovation and governance to ensure AI serves humanity responsibly. This initiative is more than a conference or a network; it is a call to action for researchers, policymakers, and civil society to collaborate in shaping an equitable and sustainable AI future.

Looking ahead, the true measure of its success will be our ability to foster lasting impact. This includes creating actionable frameworks, building trust through transparency and accountability, and policy instruments that ensure that the benefits of AI are accessible to all. As AI continues to evolve, our collective efforts must remain grounded in shared principles of fairness, sustainability, and human-centered development.

The challenges are immense, but so too is our collective potential. By uniting diverse perspectives and expertise, we can navigate the complexities of AI with integrity and purpose. Together, we have the opportunity not only to mitigate risks but to redefine AI as a tool for societal good—one that reflects the values and aspirations of all. The journey is just beginning, but the urgency is clear. I welcome you all to join us to #InformAIpolicy, a joint commitment to building a future where AI contributes to societal progress, respects the planet, and ensures equity for all.

Multilevel oversight of AI systems in line with the AI Act

Diana M. Popa (Delft University of Technology)
PDF
Cite

Abstract

The AI “oversight – control” nexus is a matter of debate in both scientific literature and policy papers, given the complex and disruptive nature of the technology and the intricate legislative systems regulating the deployment of AI systems. Most approaches give precedence to the need for a multilayered governance model, while in the same time taking into account the lifecycle stages of the AI product. Oversight over the ecosystem in which the AI is deployed (the governance layer) should not be confused with the process level human oversight. At process level, human oversight is needed in the case of “high-risk systems”, and AI literacy is a precondition for this oversight to be relative. Effective oversight requires many resources at both ecosystem and organisational levels. Additionally, oversight of AI systems is also highly regulated in democratic societies and therefore can be seen as a reflection of the importance that democratic values have in the way disruptive technologies are deployed within society.   

The need for control and human oversight of AI systems is acknowledged and set down in both legislation texts and research and policy papers, but operationalisation of both concepts is either broad or diverse and in practice address different levels of applicability in relation to the given AI system. Control and oversight of or over the functioning of the AI system itself should not be equalled to how the legislative framework address measures of control and oversight of the ecosystem in which the AI systems are deployed. The multitude of initiatives aiming to establish generally accepted definitions and determine the practical limitations of the “meaningful human oversight” – control nexus (Verdiesen, et al. 2021) underline the difficulty of determining  the control capacity over the AI system. Efforts for identifying the correct balance of human oversight and control are meant to counter the “black box” effect that the disruptive nature of the technology brings with itself, especially when used as a decision support tool with a high impact on the individuals.

The Artificial Intelligence Act (AIA) itself mentions both control and human oversight as risk mitigation measures, using terms such as “relevant” / “appropriate”/ “meaningful” human oversight. What the specific implementation measures are remain at a general level (Article 14 primarily) while in the same time, the scientific literature addresses the fact that there is ambiguity in the way human oversight is operationalized and implemented, even more so when it comes to “meaningful human oversight”. This is not necessarily a drawback of the legislation, given the fact that the AI Act was just approved by the European Parliament and also considering that, like in the case of the GDPR, the AI Act wants to be an overarching legislation, leaving room for personalized application at Member State level and remaining broad enough so that the AIA itself needs not reviewing or updating every year, taking into consideration also the alert rhythm of technological development in the field.

From the policy and research approaches, different initiatives have addressed the operationalization of human oversight and the “control – oversight” relationship, either as a nexus or as opposed measures, with approaches depending on the applications field of the concepts. Both control and oversight are defined and embedded within the given broader ecosystem in which the AI model is implemented, and most approaches adhere to a three layer ecosystem: the governance layer (the supra-national and/or national ecosystem in which the AI applications are developed), the socio-technical or organisational layer (based on internal regulations and sector rules) and the technical or process layer (also addressing the product safety regulations) (Verdiesen et al., 2021; Adams et al., 2024; Novelli et al.,2024). Oversight takes place at European level, within dedicated structures (still in development), at national level, with data protection authorities, at organizational level and at process level. Multilayer oversight overlaps (even if one might argue not perfectly) with a multistakeholder governance system of AI systems implementation within a certain regional or national context. Oversight is implemented both vertically and horizontally: the former within national supervisory authorities, expanding their structures to take on formal oversight and control roles with sanctioning capabilities and the latter within the organisations deploying the AI system themselves.

From a component perspective, building blocks for an effective oversight system include good functioning of the democratic rule of law principles, sound legislative frameworks accompanied by binding powers (both for the pre and post deployment phase), appropriate financial and human resources in all three layers and technical capabilities (technological maturity). In line with democratic principles, an oversight system should address the principles of transparency, accountability and responsibility regarding the way the AI systems are used in public context and incorporate human rights standards. Binding involvement of oversight bodies (in the sense of consultation before the implementation of a tool) and binding powers post factum are necessary for oversight to be effective (Wetzling, 2024), the latter with supervisory/ investigative and sanctioning powers.

Not addressing here prohibited practices that are equally regulated by the legislation, in the case of high-risk or impactful AI systems, the risk approach managed through measures and degrees of oversight is also influenced by the type of deployer, the sector it activates in and the process in question. While in the case of internal processes, it is a reflection of the risk appetite of an organisation, in the case of processes that have an impact on individuals outside the organisation, such as is the case with AI used by public authorities in the execution of the public administration act, oversight is tighter regulated from the governance level, since it should also be in line with the social values of the system it operates in. Although actual alignment with public values is not required by the legislation, it is a practice in line with democratic values that increases transparency of the governing act and public decision making and is a reflection of the governing style of a certain nation state, such as is the case of the Netherlands.

The ecosystem division is also often overlapped with the stages of the product’s lifecycle, given that AI is a product deployed on the unique market and also has to comply with product safety regulations, approach which identifies key points during the life of the AI system when human control or oversight is needed, also in relation to the inherent risks of that certain stage. Therefore, within each layer, different control and oversight measures are put in place, at different moments in the AI product’s lifecycle: before deployment, during deployment and after deployment.

A well-structured oversight model therefore includes both ex-ante and ex-post elements. Expert assessment in the form of expert bodies should be included in both stages, with pre-deployment expert advice going in the design phase of the system and in the high level regulating frameworks and with ex-post oversight in the form of expert assessment and democratic scrutiny (Oetheimer, 2024). These recommendations are also valid in the case of regulatory AI frameworks.

From this temporal perspective, control is implemented at three different points in time: ex-ante or pro-active control, on-going or simultaneous control, and ex-post control. Ex-ante or pro-active controls are (or should be) implemented by way of the AIA or by the means of the AI national strategy or organisational policies and practices through: 

  • bans on prohibited (high risks) systems (AIA);
  • enforcement through market surveillance and control (AI Act);
  • (quick) scans/ risk analysis for selection of trustworthy suppliers and safe acquisitions (NCTV, 2024) and identification of supply chain risks (Bluebird & Hawk BV, et. al, 2024);
  • DPIAs and FRIAs, (AI Act, National AI strategy, GDPR, Organisational policies);
  • attribution of supervisory roles and compartmentalisation;
  • establishment of internal ethics committees addressing issues such as data ethics compliance or data “pedigree”.

Ongoing or simultaneous control:

  • supervisory authorities at national and international levels with sanctioning powers;
  • Meaningful human oversight within the process (organisational policies);
  • Process- related activity of ethics committees for deviating cases.

Equally relevant, for oversight to be effective, practical and technical capabilities giving external parties (either supervisory authorities of the broader public) relevant insight into log frames to evaluate how the data processing took place, details on the way the training data was processed and altered at every step should also be made available. These are forms of ex-post control, giving external parties the possibility to get insight into the system, either through information made public by the deployer beforehand or through standard access requests. A different form of ex-post control would be the requirement to make the AI models FAIR (Findable, Accessible, Interoperable and Reusable) and have metadata or a more extensive documentation package made available, either openly or upon request. This is partially addressed by the requirement to have high-risk AI systems registered in the European data base for High Risk systems that is under development (Article 71 of the AIA).

Article 14 of the AIA specifically addresses human oversight measures at the process level, placing responsibilities on both developer and deployer: on the developer during the design phase (therefore an ex-ante measure) for the identified risks, and for the deployer for the attribution of the actual human oversight during the process to human resources with the appropriate AI literacy. For high-risk systems, technical provisions include measures similar to kill switches (Article 14, point 4, e.) and oversight measures on supervision of two separate natural persons with appropriate training, competence and authority. Thus, for human oversight to be meaningful, AI literacy is a precondition (Recital 20, Articles 14, 26, 91). Adequate human and financial resources at eco-system and socio-technical levels are also paramount for a functioning oversight system. This is where fragmentation at European level is foreseen to remain high, For example, the Dutch Data Protection Authority (AP), as the dedicated supervisory authority for the use of AI systems that have an impact on personal data, has had an increased budget of 3.6 million euro for 2026 and 2027 (AP, 2024) to address the new tasks under its coordination streaming from the upcoming of the AIA. Yet, despite the budget increase, the Dutch AP is still challenged by finding the necessary numbers of trained specialists for the exercise of the given tasks. While the oversight and governance systems involve multiple entities, data protection authorities, as the responsible oversight entities of the AI also have the role of curbing the data appetite of public authorities. At the implementation level, human oversight remains both challenging and fragmented, influenced by level of digital maturity within a certain society, financial capabilities of each deployer (hyper-scalers having the higher ground), the availability of a trained workforce or lack thereof, the difficulty to keep up with technological and legislative developments, efforts by central government to harmonize implementation of EU legislation in a top bottom approach within national context  and uniformise/change already established daily practices.

The fact that AI is a disruptive technology is also reflected by the intense debates and analyses in legislative and policy-making circles in order to set it into a unified, comprehensive and EU level actionable text. Oversight of AI systems is highly regulated in democratic societies and while democratic states spend years and a great deal of financial resources to harmonize these legislative frameworks, less democratic states or ones that rather focus on market  competition principles seem more inclined to invest in and focus on the development and implementation of the technology self, focusing on capitalizing on the benefits without the restrictions of ethical and robust legislative frameworks, and as such gaining larger market shares in the world competition stage. 

Addressing the “why” of the need for human oversight includes the fact that AI is a disruptive technology, the level of trust in technology in general (at social level for example – for example the World Values Survey) and AI technology in particular, and the approach within a certain society towards rules and regulations. As Adams (2024) suggests, AI governance does not translate into responsible AI. Oversight structures are a form of foresight and a practice of a healthy ecosystem, in which decision makers, weather at supra-national, national or sectorial and even company levels implement checks and balances in a proactive manner.  Foresight is limited though to the overview of the known variables of the given environment at a certain moment and unforeseen uses remain a reality and very much in the control and responsibility of the deployer. Taking the discussion on when human oversight is needed further, it comes down to the cases and moments of high risk situations. Despite the fact that automated instruments for decision making have been hailed for the benefits they bring regarding time saving with repetitive tasks, it is in “life or death” situations where they failed and when ultimate responsibility was passed to the human. Identifying such “life or death” situations should be addressed in the design phase of the AI system, despite the limitations or impossibility of mapping out all possible real life situations in a design/ test phase or lab setting.  And while the AIA does address ex-ante or pre-deployment controls, lists prohibited AI practices, and identify highly regulated sectors and exceptions, in practice, specific sector level knowledge and also case by case analysis are needed in order to identify what qualifies as such and to identify the correct moment and forms of human oversight.

Empirical case studies involving the use of responsible AI in line with the AIA within different contexts are much awaited. Paradoxically, despite extensive regulatory systems, what gives rise to the need for case by case approaches is the uniqueness stemming from the intersection of legislations (for an overview see Annex 1 of the AI Act: list of Union Harmonization legislation), technological regulations, ethics, organizational settings and target groups affected by the AI. Harmonizing legislation attempts such as those in the case of products for the single market are underway. However, if the harmonization process will not work, it is foreseeable that the EU commission will come with its own standards, with the consideration that technical safety is easier to implement and standardize, whereas human oversight is more difficult both to regulate and ethical implications even more difficult to evaluate, reflecting also political choices. As practice evolves, it is expected that in the next years the scientific literature will abound with these case studies. It will be interesting to see how early adaptors, independently of their size, will set trends, how fast good practices will be absorbed and, as was the case with the GDPR, how case law will evolve. These will also serve as measures for measuring the effectiveness of the human oversight and control nexus, currently a challenge that will take the form of consistent practices in the coming years.  

References

Adams, R., Adeleke, F., Florido, A., de Magalhaes Santos, L.G., Grossman, N., Junck, L., Stone, K. (2024) Global Index on Responsible AI 2024  (1st Edition). South Africa: Global Center on AI Governance.

Autoriteit Persoonsgegevens. (2024). Directie Coordinatie Algoritmes Werkagenda coördinerend algoritmetoezicht in 2024. Accessed Mey 2024 at: https://www.autoriteitpersoonsgegevens.nl/documenten/werkagenda-coordinerend-algoritmetoezicht-2024

Bluebird & Hawk BV., De Nederlandse Vereniging van Banken, ICT Group, Nederlandse Spoorwegen en Technolution;  Nationaal Coördinator Terrorismebestrijding en Veiligheid (NCTV), Nationaal Cyber Security Centrum (NCSC); Algemene Inlichtingen en Veiligheidsdienst (AIVD); CIO Rijk (2024). Cybercheck: ook jij hebt supply chain risico’s. Available at: https://www.ncsc.nl/documenten/publicaties/2024/april/18/cybercheck-ook-jij-hebt-supply-chain-risicos

Oetheimer, M. (2024) Presentation during the European Data Protection Summit. Rethinking data in a democratic society. Session 3: Zooming out onto democracy and the rule of law. How to build a functioning democratic oversight. Available at:  https://20years.edps.europa.eu/en/summit/media

Nationaal Coordinator Terrorismebestrijding en Veiligheid (2024). Quick Scan nationale veiligheid bij inkoop en aanbesteding. Accessed June 2024 at: https://www.nctv.nl/onderwerpen/economische-veiligheid/documenten/publicaties/2024/02/01/quick-scan-nationale-veiligheid-bij-inkoop-en-aanbesteding

Novelli, Claudio and Hacker, Philipp and Morley, Jessica and Trondal, Jarle and Floridi, Luciano, A Robust Governance for the AI Act: AI Office, AI Board, Scientific Panel, and National Authorities (May 5, 2024). Available at SSRN: https://ssrn.com/abstract=4817755 or http://dx.doi.org/10.2139/ssrn.4817755

Verdiesen, I.; Santoni de Sio, F., Dignum, V. (2021).  Accountability and Control Over Autonomous Weapon Systems: A Framework for Comprehensive Human Oversight. Minds and Machines (2021) 31:137–163. https://doi.org/10.1007/s11023-020-09532-9

Wetzling, T. (2024) Presentation during the European Data Protection Summit. Rethinking data in a democratic society. Session 3: Zooming out onto democracy and the rule of law. How to build a functioning democratic oversight. Available at:  https://20years.edps.europa.eu/en/summit/media

Keywords: , , , , , ,

How to cite this article:

Popa, D. M. (2024). Multilevel oversight of AI systems in line with the AI Act. AI Policy Exchange Forum (AIPEX). https://doi.org/10.63439/XEFY3802

A reappraisal of the AI Act in light of the qualitative requirements of the Law

Inês Neves (Faculty of Law, University of Porto / CIJ - Centre for Interdisciplinary Research on Justice)

Published on 29 June 2024
PDF
Cite

As the Artificial Intelligence Act (‘AI Act’ or ‘Regulation’) approaches its entry into force, it is tempting to conclude that the “job is (well) done”. However, a closer examination of the final text (signed on 13 June 2024) reveals a more nuanced reality. In addition to being implemented gradually over time and extending until 2026 (Article 113), the Regulation forms an integral part of the “New Legislative Framework” (‘NLF’) (Recitals 9, 46, 64, 83, 84). As such, it operates similarly to a ‘framework’ law, setting out a minimum set of essential requirements and high-level obligations applicable to a defined group of AI systems (those deemed high risk and specific risk).
As a result, before the Regulation is implemented in an administrative and jurisdictional manner, a process of normative creation must be initiated. In short, the actual law-making process – whereby the essential requirements of the AI Act will be specified and concretised through standards and other instruments, specifying the broad and open-texture wording of the AI Act through technical or quality specifications – is yet to commence (along with the governance structure and paraphernalia).

Nonetheless, it would be erroneous to assume that no progress has been made thus far or that nothing existed before the AI Act. There are ethical guidelines, standards that have already been published or are currently under development, and of course, there is all the European Union (‘EU’) and national legislation that is not affected by the AI Act (in terms of consumer rights, product safety, labour law, privacy and data protection, competition law, digital services, financial services, among others). Indeed, the existing legislation even raises doubts about the usefulness of the AI Act, and the EU legislator is not agnostic about the expectations and challenges of misalignment and coherence (see, among others, Recitals 46, 64, 158, and Articles 8(2), 72(4)).
While it is accurate to reflect that the AI Act is not intended to reinvent the wheel, the existing framework is deemed insufficient to provide a comprehensive framework of essential requirements specifically applicable to AI systems (high risk and specific risk). With this in mind, the AI Act follows the logic of product safety legislation, appearing as a hybrid between the need to promote trade and reduce barriers to innovation and the need to guarantee the protection of fundamental rights in the face of the specific risks of AI.

The AI Act thus exhibits a shared genetic identity with other acts of the NLF. However, it can be neither summarised nor equivalated to product safety legislation. On the one hand, this is due to the evolution and dynamism of AI systems, which make them very different from products typically associated with this type of legislation (toys, radio equipment, medical devices, machinery, among others). On the other hand, in addition to concerns for safety and health, the Regulation expressly includes fundamental rights among its objectives.
While the connection with fundamental rights does not extend to the point of turning the AI Act into an instrument that enshrines or guarantees new or specific fundamental rights per se, this link requires us to question whether greater demands on the applicable legal frameworks shall be ensured.

As is well known, the principles of legal certainty and legality demand that the law prescribing duties and obligations fulfil specific requirements of accessibility, foreseeability and precision. These requirements ensure that those obliged to (binding) legal frameworks understand what is expected of them in order to comply with such demands. Of course, these qualitative or substantive requirements do not go so far as to demand absolute certainty, which is incompatible with the recurrence of indeterminate concepts and vague terms (and the pace of technology). Furthermore, they do not negate the potential need for legal counsel or case law to comprehend the obligations prescribed by the legal system comprehensively.
However, if this is the case, there will be limits to the openness of the law, mainly when it is associated with burdens and requirements that restrict fundamental rights. This is the case of the freedom to conduct a business (Article 16 of the Charter of Fundamental Rights of the European Union), which is limited by the requirements and obligations applicable to the systems covered by the AI Act. Alternatively, the Regulation provides for sanctions that may be manifestly harmful to public interests, such as innovation and the very realisation of fundamental rights (whose realisation, at least in the current era, may depend on AI solutions).

A review of the AI Act’s recitals reveals a clear awareness of the necessity for legal certainty (see, for example, Recitals 3, 12, 83, 84, 97, 139 and 177) and the importance of defining its terms and requirements in a way that is consistent with the broader regulatory framework. However, in contrast to the limited references to this general principle of EU law, searching for the qualifier “appropriate” yields 230 results, which clearly demonstrates the broad and generic way in which the requirements for operators and systems are laid down and designed in the AI Act.
The necessity to reconsider the limitations of openness and the high-level approach adopted by the AI Act is highlighted by the fact that the AIA challenges the traditional remit of the requirements of legal certainty and the quality of the law, essentially on two levels.
Firstly, by referring to guidelines, codes of conduct, standards and common specifications, the question arises as to whether this connection between basic-fundamental law and implementation, not just by the European legislator but by other bodies, including private ones, will comply with the requirements of legitimacy and substance imposed on law in the material sense.
Secondly, insofar as the AI Act does not appear in a regulatory vacuum but instead leaves relevant European legislation (equally imposing obligations) untouched, it is essential to question whether the Regulation’s entry into force will have an impact on the predictability of what is expected of operators (in the face of perhaps contradictory expectations arising from different legal acts). It is believed that potential misalignments may be solved through a revision of the legislation in force, to identify conflicts and, if necessary, ensure that the combination of sectoral (old) and transversal (new) obligations and requirements does not result in greater entropy than advantage.

In this piece, we focus on the first area of concern.
Indeed, the Regulation leaves the implementation of a significant portion of requirements to the European Commission through a variety of means, including guidelines (Article 96), delegated acts (Article 97), and implementing acts (Articles 41(1) and 50(7)), among which codes of practice, where the AI Office acquires prominence (Articles 50(7) and 56).
The Commission is but one of many relevant players, however. Among other actors (including at the national level), the European Artificial Intelligence Board (‘AI Board’) will have the task of issuing recommendations and written opinions on issues relating to the implementation of the AI Act, aimed at ensuring its consistent and effective application (Article 66(e)). Furthermore, the role of the Advisory Forum (Article 67) and the Scientific Panel of Independent Experts (Article 68) must not be neglected.

In this regulatory landscape, harmonised standards must be recognised as primus inter pares (Article 40), as they will provide detailed technical specifications on how to comply with and meet the public interest objectives and the high-level requirements in the AI Act. The shortcomings of this co-regulation or delegation become evident when one considers that the acts in question are not those of EU institutions, bodies, offices, or agencies, and thus are not subject to judicial scrutiny (Article 263 TFEU). Additionally, fundamental rights are not a subject that can be technically specified. Finally, the fact that the institutions responsible for law-making are private entities and do not offer adequate guarantees of inclusion, representativeness and transparency creates the risk of regulatory capture and reinforces competitive foreclosure (which is already a consequence of the exclusion of AI systems already placed on the market or put into service (Article 111(2)).
In addition to standardisation, the Regulation also provides for common specifications (Article 41) in the event of i) non-acceptance of the standardisation request by any of the European standardisation organisations; ii) insufficient coverage by the harmonised standard of fundamental rights concerns, or iii) failure by the standard to meet the requirements of the request (promptly).
In both cases, voluntary compliance with harmonised standards or common specifications is associated with a presumption of conformity, particularly relevant for high-risk AI systems and general-purpose AI models. Although this presumption contributes to legal certainty, it merely guarantees a reversal of the burden of proof. The presumption is rebuttable, and the degree of legal certainty depends on the material content of the standards and specifications that will be adopted.
The preceding analysis leads to the conclusion that, in terms of meeting the requirements of accessibility and foreseeability, the AI Act does not represent an exemplar piece of legislation. The model adopted here is collaborative, with the intervention of various other players, and where the law and technology are inextricably linked. This does not imply that it is inherently inadmissible.
In fact, this way of co-regulation may be the only viable means of ensuring that the law aligns with reality. However, if this is indeed the case, the requirements for legitimising the processes for drawing up the intervening acts and actors should ensure high(er) levels of inclusion, participation, representativeness, and expertise. About the adoption of standards, it is crucial to bear in mind that the representativeness of smaller players and civil society may be impeded by factual inequalities (resources and expertise), even when abstract equality is provided for.

The interconnection between the AI Act and the fundamental rights renders these demands particularly urgent. Indeed, in this regard, the EU Regulation can be seen as both a virtue and a vice. On the one hand, its concern with fundamental rights is a distinctive feature, setting it apart from other acts within the NLF. On the other hand, however, this approach presents a tremendous challenge, as it is not aligned with the complexity and non-technical nature of fundamental rights, which are best understood as positions of value that neither the vagueness of the AIA nor the loopholes of standardisation fully consider.


In our view, there is hope in the subsidiary role of common specifications (Article 41(1)(a)(iii)). These are tailored to address the shortcomings of harmonized standards with regard to fundamental rights concerns and should be drawn up by the European Commission (an EU institution) in consultation with the consultative forum (Article 67). According to the provisions of the AI Act, this forum should represent a balanced selection of stakeholders, including industry, start-ups, SMEs, civil society and academia. Furthermore, it should be balanced about commercial and non-commercial interests, and within the category of commercial interests, with regard to SMEs and other businesses. In addition to the European Union Agency for Cybersecurity (‘ENISA’), the European Committee for Standardisation (‘CEN’), the European Committee for Electrotechnical Standardisation (‘CENELEC’), and the European Telecommunications Standards Institute (‘ETSI’), the Fundamental Rights Agency is also a permanent member of the advisory forum.

In our view, common specifications provide an additional and important safeguard. Firstly, they ensure that gaps in protecting fundamental rights are filled. Secondly, their adoption addresses the lack of representation and inclusiveness in the standardisation process, including regarding the participation of business and civil society. Thirdly, implementing acts (just as delegated acts) are subject to judicial review by the Court of Justice of the EU, as a result of which there is no lack of control.

As we can see, the Artificial Intelligence Act is not without flaws. It introduces particular challenges to the basic principles of the rule of law and the substantive requirements applicable to legal acts. This is not only because of the regulatory model it adopts but also because of the (lack of) legitimacy and democratic deficit of the players and procedures participating in the normative creation entailed by the AI Act.

It is acknowledged that the law and legislative procedures are not impervious to technological advancement and that they must, therefore, be capable of adapting frameworks and embracing new procedures and instruments (of a technical nature if needed). The key is to ensure that, in this process of openness, the law is not conflated with technology, but that technology is shaped to align with the requirements and fundamental principles of the (rule of) law.

Keywords: , , , , ,

How to cite this article

Neves, I. (2024). A reappraisal of the AI Act in light of the qualitative requirements of the law. AI Policy Exchange Forum (AIPEX). https://doi.org/10.63439/TJZR2589

Welcome to our first fellows!

During spring the first call for AI Policy Fellows was launched. This first call was open for researchers working at all faculties at Umeå University. Many strong submissions were made and after a review process, we are now happy to announce the appointment of the inaugural fellows of the AI Policy Lab. The fellows will join the lab in August and during their time with us the fellows will lead research projects that investigates and address critical intersections of artificial intelligence and societal impact. The selected fellows are:

Elin Kvist

Elin Kvist will address the profound implications of data-driven and algorithmic systems on work environments and workers’ rights. Kvist aims to explore how AI influences workers’ abilities to mobilize and organize, crucially examining the landscape of political attitudes and union preparedness. This initiative seeks to empower workers amidst evolving AI-driven work dynamics.

Lars Norqvist project focuses on AI’s transformative role in leadership practices within educational settings. His interdisciplinary approach aims to unravel the complexities of AI’s impact on leadership autonomy, accountability, and decision-making processes. By employing qualitative methods, Norqvist seeks to provide actionable insights that can inform policy at various governance levels, aiming to bridge theoretical constructs with practical applications in educational leadership.

Lars Norqvist

Henry Lopez Vega research undertakes the challenge of shaping AI policies in healthcare delivery across the region of Västerbotten. Collaborating with regional healthcare stakeholders, Lopez Vega’s research aims to evaluate how AI regulations can enhance service delivery and ecosystem emergence in healthcare. This project will analyze policy implications and foster innovation through the adoption of AI technologies like chatbots and healthcare platforms.

The AI Policy Lab remains committed to advancing cutting-edge research that addresses critical societal challenges posed by AI technologies. Each fellow’s work aligns closely with the lab’s vision to inform policy and practice, ensuring AI’s responsible integration into society.