AIPEX Publications

Read, comment and share the latest contributions to AIPEX – the AI Policy Exchange Forum!

ARTICLES

  • Civil Sector Vulnerabilities and NATO’s Strategic Role: The Case for International AI Governance

    Jason Tucker (Researcher, Institute for Futures Studies, Sweden. Adjunct Associate Professor, AI Policy Lab, Department of Computing Science, Umeå University)


    Published on 14 May 2025
    PDF
    Cite

    Adapted from a presentation given at the NATO Science for Peace and Security Programme, Advanced Research Workshop “Clicking the Pause: The Role of Transatlantic Cooperation in AI Supervision”, Salamanca, Spain, 8-9 May 2025.

    As AI becomes increasingly embedded in critical societal functions, the need for robust, internationally coordinated governance grows more urgent. While some national and regional regulation of AI is emerging, applications in defence and international security often remain exempt from these initiatives. This historical separation between civil and defence sectors is understandable given the unique operational requirements of the military. However, it risks creating a false dichotomy—suggesting that AI use in civil domains is largely divorced from international security concerns. However, the geopolitical implications of AI in the civil sector are profound and escalating (Schaake, 2024).

    To illustrate this, healthcare provides a concrete and urgent example. Across NATO members and partners, localized and largely disconnected decisions are being made to adopt small-scale AI solutions in healthcare. With states having limited capacity to develop in-house solutions, they often turn to external actors. Doing so means that they are then subject to a complex and opaque web of global supply chains and international actors. This poses substantial risks, including vulnerabilities to cyber-attacks, dependencies on potentially hostile states or corporations, and strain on critical infrastructure to support its adoption.

    The growing instability of the international order compounds these challenges. The United States has recently exhibited unpredictability in both its Administration and its corporate tech sector. Even if diplomatic relations are maintained, trust at the local level is harder to rebuild. Working with partners whose long-term reliability is in question introduces significant risk, and other non-traditional partners become more appealing. Where these actors are not aligned with NATO, this could be a vulnerability.

    Moreover, the adoption of AI in the civil sector has been driven by techno-solutionism — the prioritisation of technological fixes that neglects broader societal and security trade-offs, as well as potentially more appropriate non-technical solutions. It glosses over the reality that AI, as a socio-technical system is embedded in cultural, institutional, and ethical contexts and requires participation from a broad range of actors to function at its best.

    Healthcare systems are particularly susceptible to this narrative (Strange and Tucker, 2024). They face resource constraints that limit the capacity to develop, implement, and secure AI technologies. Combined with the dominant discourse being that AI is the only and best way to solve a broad range of healthcare issues, everyday actors in healthcare are facing pressure to adopt AI where they can. At the same time, NATO’s security infrastructure is drawing from the same limited resource pool—particularly in terms of skills, energy, data infrastructure, and cybersecurity capacity. Without careful coordination, this could lead to a zero-sum scenario, undermining societal resilience and military advantage.

    Cybersecurity threats to healthcare are well documented. The World Health Organization has recognized that cyber-attacks targeting health systems have considerable consequences in terms of public health and international security (WHO, 2024). In 2021, WHO reported that one-third of global healthcare institutions had suffered at least one ransomware attack in the preceding year (Mishra, 2024). The European Union reported that in 2023, healthcare was the most targeted critical sector in cyber-attacks (WHO, 2024). During the COVID-19 pandemic, healthcare was not just a target but a vector for disinformation and destabilisation by state and non-state actors alike. Given these risks, decisions about AI adoption in critical civil sectors like healthcare cannot be made in isolation from geopolitical and security considerations. Yet most local actors are not equipped to understand or navigate these complex dynamics. The absence of coherent guidance or frameworks linking AI adoption to national and international security exacerbates vulnerability, weakens societal resilience, and increases dependence on untrustworthy partners.

    Global AI governance is essential. It can establish the guardrails necessary to manage these risks and guide responsible adoption of AI technologies across sectors. NATO has a critical role to play here. By integrating civil sector AI governance into its strategic thinking, and engaging with the Allies on this, NATO can help ensure that AI adoption enhances—not undermines—resilience and collective security. This will allow for a more realistic assessment of the trade-offs involved in AI adoption, especially in sectors like healthcare that are both vital to public well-being, are particularly vulnerable to attack and a conduit for hostile actors to cause societal disruption. NATO’s role here should be seen as complementing other international AI governance efforts, such as those by UNESCO, OECD and the EU etc. This would ensure that these governance structures do not become dominated by military priorities and bridge the gap between the defense and civil sector. Democratic safeguards, such as civil society oversight or public reporting, for any NATO-related initiatives affecting the civil sector, would also be essential. As would multidimensional and multidisciplinary views on civil resiliency frameworks.

    AI in the civil sector is not a technical or administrative matter alone—it is a strategic issue with implications for the stability, security, and cohesion of NATO members’ and partners’ societies. Only through coordinated, international governance, can we navigate this new terrain with the prudence and foresight it demands.

    References

    Mishra, V., (2024) Cyberattacks on healthcare: A global threat that can’t be ignored, UN News, https://news.un.org/en/story/2024/11/1156751.

    World Health Organization., (2024), Ransomware Attacks on Healthcare Sector ‘Pose a Direct and Systemic Risk to Global Public Health and Security’, Executive Tells Security Council, https://press.un.org/en/2024/sc15891.doc.htm.

    Schaake, M., (2024). The Tech Coup. Princeton University Press.

    Strange, M. and Tucker, J., 2024. Global governance and the normalization of artificial intelligence as ‘good’ for human health. AI & SOCIETY, 39(6), pp.2667-2676.

    Further Information

    This article is part of the Politics of AI & Health: From Snake Oil to Social Good funded by The Wallenberg AI, Autonomous Systems and Software Program – Humanity and Society (WASP-HS).

    Keywords (comma separated):

    NATO, Civil Sector, Artificial Intelligence, Security, Healthcare, Governance

    Related URL (if any):

    https://www.iffs.se/en/research/research-projects/politics-of-ai-health-from-snake-oil-to-social-good/

    How to cite this article:

    Tucker J. (2025). Civil Sector Vulnerabilities and NATO’s Strategic Role: The Case for International AI Governance. AI Policy Exchange Forum (AIPEX). https://doi.org/10.63439/UJML2174

  • Beyond the AI race: why global governance is the greatest innovation

    Virginia Dignum (Wallenberg Chair, Professor Responsible Artificial Intelligence, Director of AI Policy Lab at Umeå University)


    Published on 8 April 2025
    PDF
    Cite

    As artificial intelligence (AI) evolves rapidly, the urgency to govern it responsibly becomes more pressing by the day. We are standing at a pivotal moment, one where the choices we make today will shape not just technological outcomes, but also the foundations of our societies, economies, and planetary well-being.

    Even as the UN and other international agencies advocate for global AI regulation, major players – particularly the US, UK, and China – seem increasingly hesitant to fully commit. While the US and UK are currently moving towards light-touch, innovation-driven approaches that prioritize industry leadership over binding rules, China leans toward a state-controlled model aligned with its national priorities. Their reluctance undermines efforts to build the effective, inclusive governance frameworks we urgently need, and may encourage others to also sideline global cooperation in favour of fragmented, self-serving strategies.

    But AI governance is not optional, it is essential. It protects rights, upholds global values, and ensures long-term economic stability and sustainable innovation. Without global governance, we open the door to a race to the bottom, marked by short-term thinking, ethical shortcuts, and growing global inequality. We cannot allow geopolitical competition to derail the collective responsibility required to ensure AI serves the common good. Now is the moment to strengthen our commitment to global, values-driven governance, not to stall it. Meanwhile, AI governance shouldn’t chase every new technology, but instead follow clear principles: transparency, fairness, explainability, and accountability. These form a foundation for adaptable policy that protects rights and safety. Tools like regulatory sandboxes, public engagement, and stronger international coordination support this flexible yet high-standard approach as AI evolves.

    The best competitive advantage is not ruthless speed, but wise collaboration, especially when the stakes include trust, stability, and the health of our planet. In this context, the European Union’s €200 billion investment in regulated, human-centered AI stands out. This visionary approach demonstrates how regulation can act not as a brake on innovation but as a stepping stone for it. The EU’s commitment to ethics, inclusion, and sustainability offers a powerful alternative to the more narrowly competitive models pursued by the US and China. Yet, funding alone is not enough. Investment must be accompanied by sustainable practices, equitable access, and strengthened social cohesion. Other countries—Canada, Japan, Brazil—are also making important strides. But this is not a race with a single winner. It’s a collective effort, and meaningful progress depends on a globally aligned framework that ensures AI serves all of humanity.

    Still, I am deeply concerned about the growing competition to dominate the AI landscape. China, the US, and others are increasingly viewing AI as a tool of economic and military supremacy. This race risks concentrating power in a handful of nations or corporations, sidelining most of the world and worsening inequality. China, the US, and others view AI through the lens of strategic dominance. But AI is not a zero-sum game. True progress requires transparency, ethical alignment, and shared governance.

    One of the greatest ethical challenges today is the erosion of human agency through opaque, unaccountable AI systems. That’s why I advocate for Earth alignment, as we introduced in a recent article in Nature Sustainability. This framework emphasizes the need for AI governance to be anchored in environmental sustainability, global justice, and societal cohesion. These goals cannot be achieved in isolation or through regional silos. They require a shared commitment to values that transcend borders, and the democratization of governance. A small group of governments and companies cannot be allowed to shape society through their control over AI development, nor solely through the lens of existential threats and geopolitical rivalry.

    Responsible development requires systemic change, not just technical fixes. Ethics must be embedded from the outset, but we must do so through systemic change, not fear-mongering. This is not just a question of innovation; it’s a matter of justice. AI should not be a tool that widens global divides or undermines democracy and social cohesion. It should be a force for empowerment, equity, and resilience. That’s only possible through shared governance, transparency, and ethical alignment across all borders, including those of the most powerful players.

    This is why global governance of AI is not optional, it is urgent.

    Looking to the future, what excites me most about AI is its potential to empower us, not to replace us. If we govern it well, AI could become one of our most powerful tools for addressing climate change, improving healthcare and education, and advancing equity and social cohesion. But that future is not guaranteed. It depends on the choices we make now. The future of AI is not just about building smarter machines and software, it is about working together towards a wiser humanity. One that values cooperation over competition, solidarity over supremacy. One that uses AI not to dominate, but to heal and uplift.

    There is no alternative: in the long run, only responsible AI will lead to innovation that truly benefits society. Anything else will not only undermine trust and human rights but will also lead to technically weaker systems and a loss of true innovation. Irresponsible AI may promise short-term advantages, but it will cost us our long-term future.

    Responsible AI is not the finish line. It is the only viable path forward.

    Keywords (comma separated):

    AI governance, global cooperation, responsible AI, sustainability, transparency, regulation

    How to cite this article:

    Virginia D. (2025). Beyond the AI race: why global governance is the greatest innovation. AI Policy Exchange Forum (AIPEX). https://doi.org/10.63439/LNQA3726

  • Potential impact of the EU Platform Work Directive on AI labelers

    Mariia Lesina (Lund University)


    Published on 25 March 2025
    PDF
    Cite

    Introduction

    The gig economy has revolutionized labor markets, introducing high flexibility while simultaneously raising critical concerns about workers’ rights and protections. Across the European Union (EU), 3% of people aged 15-64 rely on digital platforms to access employment, yet many remain trapped in a legal grey zone, lacking social security, job stability, and transparency in how their labor is managed (Eurostat 2024).
    While the attention has been largely drawn to the impact of AI algorithms on the workflows of such gig workers, most notably drivers or delivery people, one group has been largely overlooked – the invisible workforce behind AI. These workers perform essential tasks such as data tagging, annotation, and content moderation, which help train AI models (Muldoon et al. 2024, 9-10). Their work is often fragmented and distributed through digital labor platforms, making them susceptible to unfair algorithmic control, precarious employment conditions, and opaque remuneration structures. AI labelers work asynchronously across different time zones and are also subject to algorithmic management, where automated systems monitor their accuracy, determine their pay, and can even suspend or terminate their accounts without clear justification (Ibid, 12-15). Evidence of this can be easily found on community discussions on Reddit where stories of such workers are widespread, telling how their work was declined and payment delayed or cancelled altogether (Nagaraj Rao et al. 2025, 23). One of the recent attempts to address this systematic issue was taken by the EU via the Platform Work Directive, features and implications of which will be analyzed in this article.

    EU Platform Work Directive

    In 2024 the EU introduced the Platform Work Directive, a regulatory framework designed to enhance labor protections, redefine employment classifications, and establish oversight mechanisms for AI-driven algorithmic management. The EU Directive recognizes AI labeling as a form of platform work if it is conducted through a digital platform within the EU and based on a contractual relationship. Article 19 of Introduction specifically mentions tagging as a form of crowd work that can be conducted remotely. This recognition aims at allowing AI labelers to benefit from the Directive’s employment presumption when platforms exercise control over their workflows, task assignments, or performance evaluations. More importantly, it states that it is platforms that have to prove employment status of their platform staff, as the latter usually lacks the means and leverage to do so (Articles 30, 31).
    The core of the regulations is that “platform work should be provided with rights, with a view to promoting transparency, fairness, human oversight, safety and accountability” (Article 14). These goals will be achieved through a number of legal changes, translated into the national legislation from the EU level. To begin with, Article 10 of Chapter III enforces human oversight in algorithmic decision-making and mandates greater transparency in how these systems operate, granting AI labelers the right to request detailed explanations of algorithmic decisions that impact their work. This means that platforms cannot solely rely on automated systems to suspend or deactivate an AI labeler’s account. Instead, these decisions must “ensure human oversight and regularly carry out an evaluation of the impact of individual decisions taken or supported by automated monitoring systems”: essentially demanding human in the loop, ensuring that workers are not unfairly penalized by flawed algorithms (Article 47). Additionally, AI labelers now have the right to contest algorithmic decisions, demand explanations, and request human reviews of automated rulings that affect their employment status, pay, or continued access to work (Article 8). The Directive also states that platform workers’ representatives “should be involved in the evaluation process” of these automated systems (Article 44).
    The mental and physical well-being of platform workers is another crucial aspect addressed by a separate Article 12 of Chapter III. Due to the nature of their work – repetitive tasks, exposure to harmful content, and tight deadlines – AI labelers face unique risks to their mental and physical health. The Directive requires platforms to assess and mitigate these risks, ensuring that AI systems do not “put undue pressure on platform workers or otherwise puts at risk their safety and physical and mental health”. Platforms must now provide effective information and consultation for workers, while Member States – ensure that digital labour platforms take “preventive measures, including providing for effective reporting channels” (Article 12 of Chapter III).

    Possible shortcomings of the Directive

    While the Directive provides a theoretically effective legal framework, its success is based on effective enforcement. One major concern is that companies will exploit national law loopholes, restructuring their business models to avoid classifying workers as employees. This issue has been explored by an organization called Fairwork, which evaluates the work conditions of digital labour platforms, and whose extensive analysis highlights large reliance on contextual enforcement in the country-specific legislation. Therefore, as Fairwork experts state, in member states where the “power of labour unions is undermined”, like, for instance, in Italy, the workers can remain unprotected and “self-employed” (Fairwork Project 2024, 2).
    Moreover, as experts from International Labour Organization (ILO) explore, outsourcing work to foreign workers in countries where the cost of labour is lower is common as it enables businesses to optimize their costs (Rani et al. 2021, 22). Hence, the Directive, which only protects workers within the EU, does not address the issue of exploitation of non-EU based platform staff.

    Conclusion and further discussion

    The EU Platform Work Directive is a crucial step toward recognizing AI labelers as platform workers, granting them employment protection, transparency, and human oversight in algorithmic management. However, its impact depends on consistent enforcement across Member States and preventing platforms from exploiting loopholes or outsourcing labor beyond the EU’s reach.
    As the December 2026 deadline for national implementation approaches, the fight for fair AI labor practices is only just beginning. With platforms already pushing back, arguing that regulation stifles innovation and contradicts the inherent advantage of flexibility that digital work provides (Copenhagen Economics 2021, 24), the real test will be in Directive’s practical integration. Whether the EU emerges as a pioneer in ethical AI labor governance or struggles with unintended consequences will depend on the ability of each state to implement the Directive in a way that is both principled and pragmatic. Hence, the most topical question remains: “Will this policy set a global precedent for fair AI labor practices, or will fragmented enforcement and corporate resistance weaken its impact?”
    To achieve the first outcome, EU policymakers, researchers, and labor rights advocates will need to continue pushing for a regulatory environment that ensures that this overlooked groups in the gig economy – AI labelers – receive the protections they deserve.

    References
    1. Copenhagen Economics. The value of Flexible work for local delivery couriers. Study for Delivery Platforms Europe. November 2021. 28 p.
    2. European Parliament and Council Directive (EU) 2024/2831 of 23 October 2024 on improving working conditions in platform work [2024] OJ L283/1. Internet source. URL: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32024L2831 (accessed 7 March 2025).
    3. Eurostat. Experimental statistics on digital platform employment. 18 July 2024. Internet source.URL: https://ec.europa.eu/eurostat/web/products-eurostat-news/w/ddn-20240718-1#:~:text=In20223.0ofpeople,countriesand1EFTAcountry (accessed 7 March 2025).
    4. Fairwork Project. Fairwork’s Response to the EU Directive on Platform Work. March 2024. Internet source. URL: https://fair.work/wp-content/uploads/sites/17/2024/03/Fairworks-Response-to-the-EU-Directive-on-Platform-Work.pdf (accessed 7 March 2025).
    5. Muldoon, J., Graham, M., Cant, C. Feeding the Machine: The Hidden Human Labour Powering AI. Canongate Books. 2024. 288 p.
    6. Nagaraj Rao, V. Dalal, S., Agarwal, E., Calacci, D., and Monroy-Hernández, A. Rideshare Transparency: Translating Gig Worker Insights on AI Platform Design to Policy. ACM Hum.Comput. Interact. No. 9, 2. April 2025. Pp. 1-49.
    7. Rani, U., Rishabh, K.D., Furrer, M. Gőbel, N. Moraiti, A. and Cooney, S. World employment and social outlook: the role of digital labour platforms in transforming the world of work. Geneva: International Labour Office. 2021. 283 p.
    8. Silberman, M.S., Adams-Prassl, J., Abraha, H. and Suresh, R., Doth the Platform Protest Too Much? Uber, Employment Status, and the EU’s Proposed Platform Work Directive. Oxford Law Blogs, 28 September 2023. Internet source. URL: https://blogs.law.ox.ac.uk/oblb/blog-post/2023/09/doth-platform-protest-too-much-uber-employment-status-and-eus-proposed (accessed 7 March 2025).

    Keywords:
    EU Platform Work Directive, AI labeling, Gig Economy, Human Rights

    How to cite this article:

    Lesina M. (2025). Potential impact of the EU Platform Work Directive on AI labelers. AI Policy Exchange Forum (AIPEX). https://doi.org/10.63439/APJD5342

  • Metadata/README elements for synthetic structured data made with GenAI: Recommendations to data repositories to encourage transparent, reproducible, and responsible data sharing

    Ericka Johnson (Dept of Thematic Studies, Linköping University)
    David Rayner (Swedish National Data Service, University of Gothenburg)
    Jools Kasmire (University of Manchester), Victor Hennetier (Dept of Thematic Studies, Linköping University), Saghi Hajisharif (Dept of Science & Technology, Linköping University), Helene Ström (Fair AI Data)


    Published on 20 March 2025
    PDF
    Cite

    Introduction

    Publication of AI-generated synthetic structural data in data repositories is beginning to reveal the specific documentation elements that need to accompany synthetic datasets so as to ensure reproducibility and enable data reuse.
     
    This document identifies actions that research repositories can take to encourage users to provide AI-generated synthetic datasets with appropriate structure and documentation. The recommendations are specifically for AI generated data, not (for example) data produced using pre-configured models or missing data created by statistical inference. Additionally, this document discusses metadata/README elements for synthetic structured datasets (tabular and multi-modal) and not textual data from LLMs or images for computer vision. 

    The document is the result of a workshop held on 23rd January 2025, with participants from the Swedish National Data Service, Linköping University and Manchester University. It also draws on survey responses about current practice from 17 data repositories and a review of existing metadata and README requirements. 

    Background

    AI-generated synthetic structured datasets are generated using machine learning techniques with the aim of reproducing the essential elements of an existing dataset (Guépin et al., 2024 Jacobsen, 2023; Li et al., 2023; Offenhuber, 2024; Savage 2023). Synthetic data generation may be driven by the need to ensure privacy or to expand, enhance or substitute for real-world datasets which may be insufficient or non-existent. Sometimes synthetic data is produced to create a portable or shareable dataset that is considered safe for open access, for example to share via a data repository. 

    While synthetic structured data may reproduce the essential elements of an original dataset, it will also inevitably introduce “intersectional hallucinations”, which refer to anomalous inter-attribute relations within a dataset (Lee, Hajisharif & Johnson 2025). AI generated synthetic data also have a known tendency to minimize minority elements and amplify majority elements (Chen et al., 2024; Johnson & Hajisharif 2024). Thus, knowing in what ways a synthetic dataset demonstrates fidelities and in what ways it is ‘different’ from the original data is essential for successful and responsible re-use of synthetic data. Given that the goal of many data repositories is to provide access to data that is replicable and/or reusable, there is a clear need to establish protocols for documenting synthetic data.

    Primary recommendations

    Our primary recommendations are:

    a) that data repositories establish a standardized way to label data as synthetic data, and that this information is prompted-for or required in the metadata or READMEs associated with synthetic datasets. 

    b) that data repositories provide users with a guide that explains how to properly document synthetic data. The extent to which documentation should be provided with the dataset or provided in associated articles or publications linked to the data will depend on the policies of the repository. An example is the guide provided by the Swedish National Data Service (2025). 

    c) that domain experts be prompted to document the context and motivation for generating synthetic data.

    Documenting synthetic data – process and product

    Reusability often refers to the data as a product. In the case of synthetic structured data, however, it may be the method of data generation (the data as a process) that  is reusable, not the data itself. We therefore suggest that data repositories require information about both the process of data generation and details about the actual synthetic data.

    Data as a process

    The following elements should be included to describe the technical details of the synthetic structured data generation process: 

    • A description of the workflow.
    • The generative model used (i.e. GAN, Diffusion, etc.). As techniques are constantly evolving, these requirements should be formulated in such a way to allow for and capture new techniques. The structure and hyperparameters (learning rate, number of epochs, etc.) of the generative model are also important factors for reproducibility and should be included.
    • What raw data or inputs, if any, were used, including its mode of collection. A link to the source of the raw data should be provided where appropriate.
    • Which (random) seeds were used.
    • If a subset of raw data were reserved for testing, how was this subset selected?
    • Versions of the software and packages used.
    • Operating system information, values for relevant environment variables.
    • A link to the source code (we suggest keeping code in a separate repository so it can be reviewed, improved, and re-released) and if appropriate a link to the weights of the trained model.
    • Citation details (including DOIs) for related documents or the release versions of code.

    Additionally, some cases of synthetic data are not based on raw data (e.g. agent based modeling/multi agent systems, digital twins). In such cases, this should also be clearly stated in the description of the data generation process. 

    If a repository considers that publishing the data generation model is out-of-scope, we suggest providing information on how models can be deposited in either a more generic repository or in a specific repository for models. Links to the model can then be provided in the dataset metadata and/or README.

    Product

    Synthetic structured datasets inevitably contain stochastic variability, meaning that different datasets can be obtained by running the same code multiple times with different random seeds. We therefore suggest that metadata/READMEs also contain information about:

    • whether the dataset is entirely synthetic or augmented. If it is augmented, what are the proportions of real and synthetic data?
    • missing edge cases at the single-attribute level and inter-attribute level. 
    • inter-attribute combinations in the raw data that have diminished frequency in the synthetic data.
    • inter-attribute hallucinations that have been observed in the synthetic data. 
    • details of the verification/validation process: how was the model tested, etc.
    • how the synthetic data are structured at the file-level: are the input data in a folder marked “raw” or “input”, and output in an “output” folder?

    Privacy and specific circumstances

    A common use-case for synthetic data is when privacy assurance is necessary for sensitive data. In such cases, we recommend the metadata/README contain information about disclosure risk, indication risk, reidentification risks, K-anonymity, etc. This type of synthetic data requires extra care and should only be made freely available if specific individuals cannot be re-identified by any reasonably likely means.

    We also suggest that repositories include instructions on creating the metadata/README that will prompt domain experts to explain the specific circumstances of their synthetic data. Why was it generated? What is the fundamental hypothesis behind the synthetic dataset’s use? What is its subject and purpose(s)? Data creators should be encouraged to disclose, for example, if the dataset was created for exploratory research, to represent sensitive data, to allow for work by a distributed team, to enable data portability, to create categories or support classification decisions, etc.. Encourage data submitters to consider sensitive areas and intersections within the data, as well as how many relational intersections are valuable to combine when using the dataset for new research purposes.

    Summary and discussion

    Synthetic structured data may be produced where scientific research requires data with no personal information, data that are portable and shareable, data which are not obtainable for practical or ethical reasons, or large datasets for machine learning. However, the details of the generation process and the variations inherent in synthetic data need to be documented, either in a dataset’s metadata/README or in the articles accompanying the dataset.

    Many aspects of synthetic data are still emerging, and in some cases, we lack established routines or even vocabularies for them. We hope the recommendations in this policy document will serve as a starting point for further discussions. In particular, we aim to encourage those working with data repositories to collectively establish best practices for managing synthetic data and developing vocabularies to describe them. For example, we might promote an accepted keyword or subheading, such as SYNTHETIC_DATA, or suggest appending “_synth” to filenames containing synthetic data. Additionally, controlled vocabularies should include subcategories to distinguish between fully synthetic and blended/augmented data.

    With a well-defined vocabulary and clear metadata guidelines, repositories can help researchers to describe both their datasets and the processes used to create them in an open, transparent, and reproducible manner, ensuring responsible data sharing within the scientific community.

    References

    ​​Chen W, Yang K, Yu Z, et al. (2024) A survey on imbalanced learning: latest research, applications and future directions. Artificial Intelligence Review 57(6): 137.

    Guépin F, et al. (2024) Synthetic Is All You Need: Removing the Auxiliary Data Assumption for Membership Inference Attacks Against Synthetic Data. In: Katsikas, S., et al. Computer Security. ESORICS 2023 International Workshops. ESORICS 2023. Lecture Notes in Computer Science, vol 14398. Springer, Cham. https://doi.org/10.1007/978-3-031-54204-6_10.

    Jacobsen BN (2023). Machine learning and the politics of synthetic data. Big Data & Society. 10(1).  

    Johnson E and Hajisharif S (2024) The intersectional hallucinations of synthetic data. AI & Society. https://doi.org/10.1007/s00146-024-02017-8. 

    Lee, Hajisharif & Johnson (2025) The ontological politics of synthetic data: normalities, outliers, and intersectional hallucinations. Big Data & Society.

    Li X, Wang K, Gu X, Deng F, Wang FY (2023) Parallel eye pipeline: An effective method to synthesize images for improving the visual intelligence of intelligent vehicles. IEEE Transactions on Systems, Man, and Cybernetics: Systems. 53(9), 5545-5556.

    Offenhuber D (2024) Shapes and Frictions of Synthetic Data. Big Data & Society. 11 (2): 20539517241249390. https://doi.org/10.1177/20539517241249390.

    Savage, N (2023) Synthetic data could be better than real data. Nature Machine Intelligence. doi: https://doi.org/10.1038/d41586-023-01445-8.

    Swedish National Data Service. (2025). Managing and publishing synthetic research data (Version 1). Zenodo. https://doi.org/10.5281/zenodo.14887525 

    How to cite this article:

    Johnson, E., Rayner, D., Kasmire, J., Hennetier, V., Hajisharif, S., & Ström, H. (2025). Metadata/README elements for synthetic structured data made with GenAI: Recommendations to data repositories to encourage transparent, reproducible, and responsible data sharing. AI Policy Exchange Forum (AIPEX). https://doi.org/10.63439/MPEW5336

  • Towards Successful Industrial Policy on AI in Healthcare: Establishing the Conditions for Future Public Benefit

    Erez Maggor (Assistant Professor, Ben-Gurion University and the Institute for Futures Studies). 
    Jason Tucker (Researcher, Institute for Futures Studies, Adjunct Associate Professor, AI Policy Lab @Umeå University and Visiting Research Fellow, AI & Society, LTH, Lund University).

    Published on 18 March 2025
    PDF
    Cite

    Abstract

    This paper explores how pro-active government policies could promote artificial intelligence (AI) in healthcare for the public good. Building on insights from the literature on industrial policy, we argue that without clear conditions and guardrails to ensure future public benefit, state assistance and subsidies will be nothing more than corporate welfare with unpredictable, if any, societal benefit. We provide a few concrete examples of this and then conclude by reflecting on how industrial policy can be a useful conceptual lens to challenge techno-solutionism, increase accountability and situate AI healthcare policies in the broader political-economy.

    Keywords: Industrial Policy, Corporate Welfare, Artificial Intelligence, Healthcare, Public Interest, Conditionality, Futures.

    Introduction

    States and regions are increasingly turning to interventionist policies to try and realise the benefits of the development and deployment of artificial intelligence (AI). This is highlighted by the growing number of national and regional AI strategies, where bold visions for how AI will solve a plethora of societal problems are set out (OECD, 2025). For example, just recently the EU announced the 200 billion Euro regional InvestAI, the largest AI investment in history “to make Europe the continent of AI” (European Commission, 2025). This optimism is possible as AI is a useful empty signifier for various visions of technological innovation (Kak, 2024). These visions are often built upon previous AI policies and initiatives, thus reflecting a continuation of socio-technical imaginaries of AI (Bareis & Katzenbach, 2022). 

    Healthcare has featured prominently in these strategies (Hoff, 2023). States are promising that the technology can solve a range of societal issues related to health, fix crumbling and underfunded healthcare systems, and improve our individual health (Tucker, 2023). Often, existing policies are market-based, which requires states to collaborate with the private sector. To get private firms on board, governments provide various incentives, including increased public funding of R&D and procurement, access to public health infrastructure and data, and cutting so-called red tape that allegedly “stifles” innovation. Regarding AI in healthcare, the recent increase in interest in industrial policies coincides with fluctuations in private sector investment in the medical and health sector – which rose sharply between 2019-2021, then plummeted afterwards (North, 2025). The sharp decline post 2021 means that relying on the private sector to address healthcare challenges without incentives is not likely to achieve states´ promised future visions. 

    Such policies represent a gamble for states, as these technoscientific futures, which are based on the notoriously unpredictable development trajectory of AI, also rely on the alignment of private and public interests if this is realised. This begs the question of how we can improve the odds for these policies to succeed? To answer this question, we draw lessons from the literature on industrial policy. One of the central insights of this body of work is that when government subsidies are provided without proper conditionalities they are unlikely to produce desired socially beneficial results (Bulfone et al, 2023; Bulfone et al, 2024; Mazzucato & Rodrik, 2023).  We argue, therefore, that a critical element to the success of AI policies is the centralizing of the public good in the conditions of future benefit of AI. Without this in place these policies will end up as nothing more than corporate welfare in the guise of public interest innovation. 

    Healthcare is arguably one of the sectors with the highest risk and reward in this regard. We have seen an “AI turn” in global and national health discourses (Strange & Tucker, 2024). AI is posited as being the best, and often only means, to address a broad range of individual, societal and systemic healthcare issues. Health is also intricately intertwined with social stability and democracy (Johnson & Longmore, 2023). In addition, the potential benefits of AI in the healthcare sector is often used to justify broader political agendas on AI, as we saw in the case of the EU’s InvestAI. As such, the success or failure of industrial policy on AI in healthcare has ramifications well beyond the sector. 

    Industrial Policies versus Corporate Welfare

    The historical record of industrial policy has been mixed. States like Japan, South Korea, Taiwan, Israel, France, and, most recently, China, have had remarkable success using industrial policies to upgrade their industries and ‘catch-up’ to more economically developed nations (Amsden, 1989; Johnson, 1982; Zysman, 1984; Wade, 2004; Maggor, 2021; Ang, 2018). However, in India, Turkey, and across Latin America, industrial policies are considered to have been a relative failure, leading to waste and corruption and resulting in these countries failing to advance economically or, at best, becoming stuck in the ‘middle-income trap’ (Doner & Schneider, 2016). The main lesson emerging from this mixed comparative-historical experience has been that rather than representing a dangerous or wrongheaded economic policy – as many on the ideological right often argue – industrial policies are, first and foremost, a daunting political challenge. 

    While profit-maximizing firms will gladly accept various government subsidies and assistance that often accompany industrial policy efforts, they will always prefer this assistance be provided with limited strings and conditions. Policymakers, on the other hand, understand that subsidies without conditionalities are a form of corporate welfare, i.e., a gift from the public to the private sector. As a result, the main political challenge for policymakers is designing industrial policies that incorporate institutional mechanisms that dictate clear conditions regarding the future benefit of such public-private collaborations, ensuring the public benefit is protected, and equipping the state with the capacity to ‘discipline’ uncooperative firms (Amsden, 1989; Chibber, 2023; Maggor, 2021). Crucially, this needs to be implemented from the outset, as this is the stage at which policymakers have the greatest leverage over the private sector. To ground and contextualise this we can turn to two recent cases, the development of the COVID-19 vaccine in the USA and the NHS Google-DeepMind health data scandal in the UK. 

    Developing the COVID-19 Vaccine in the USA 

    One useful example to demonstrate the aforementioned logic is the development of vaccines for COVID-19. In 2020, in response to the pandemic, the Trump White House approved “Operation Warp Speed,” to develop an mRNA vaccine. In addition to federal support for the promotion of R&D, the program also provided government subsidies for scaling-up manufacturing (in the case of Moderna), offered strategic government procurement (in the case of Pfizer), and delivered various government assistance to industry across multiple sectors and regions (Adler, 2021). On the one hand, this experience represented an effective and comprehensive industrial policy program that saw the government partner with the private sector to produce a significant social good– a safe and effective vaccine that helped end a global health emergency. On the other hand, it showed that when state-supported innovation is not governed for the common good via strict terms and conditionalities, many people remain excluded from its benefits. 

    For example, even though Moderna used public investments and research to develop its vaccine, they refused to share intellectual property and know-how with less-developed countries (Mazzucato, 2023). The experience  also demonstrates that without imposing strict limitations on profit-taking, private firms that succeeded due to public assistance are likely to retain “astronomical and unconscionable profits”, in this case due to their monopolies of mRNA COVID vaccines — upwards of 69% profit margins in the case of Moderna and BioNTech (Wilson, 2021; Emergency USA, 2021).

    The NHS Google-DeepMind Health Data Scandal

    There are also examples of industrial policies on AI and health more specifically, such as the UK National Health Service (NHS) Google-DeepMind case. In 2015 the Royal Free London NHS Foundation Trust shared the personal health records of 1.6 million patients with Google’s AI firm DeepMind. This was to support the development of a system that could potentially better diagnose acute kidney injury. Successive UK governments have long courted Google to try and attract more of the firm’s investment in the UK market, so this public-private partnership came as no surprise. 

    However, while the diagnostic application showed early signs of success in detecting kidney injury, the partnership hit the headlines in 2017 due to a significant public backlash. There were serious concerns about data privacy and the lack of transparency in how the sharing of publicly funded health data with DeepMind was decided upon. It was also unclear what DeepMind was using the data for, as well as there being an uncertain public benefit from the data transfer (Dickens, 2021). The Information Commissioner’s Office eventually ruled that the NHS had failed to comply with the Data Protection Act by transferring the data (Information Commissioner’s Office, nd), though a subsequent class action lawsuit against DeepMind itself failed in the UK courts. With the introduction of the UK’s AI Opportunities Action plan in January 2025, the case has raised its head once again (Milmo & Stacey, 2025). The legacy of the NHS DeepMind scandal thus lives on, impacting public trust in the next generation of industrial policies on AI. 

    Establishing the Conditions for Future Public Benefit

    The unpredictable nature of AI development poses challenges in terms of establishing the conditions of future benefits in industrial policies. However, these are not insurmountable, and as argued above, need to be overcome if the social benefit of the technological investment is to be realised. We should remember that conditions do not always need to be very specific (such as the quick development of a vaccine), or indeed directly related to healthcare. For example, supporting a flourishing MedTech sector that creates high skilled jobs, and pays corporation tax into the state coffers, could be seen as an acceptable outcome. Yet, there needs to be clear guardrails about how the public are to benefit from any technological innovation that an industrial policy has facilitated. Assuming private actors will act in the public good if a discovery is made has proven to be wishful thinking time and time again. 

    The economist Mariana Mazzucato and her colleagues have outlined several conditionalities that could ensure the public shares the returns of public investment in health. These include charging royalties from companies who profit from technologies developed with public funding, with funds earmarked to finance future innovation. Another strategy could be for states to retain a “golden share” of patents developed with public assistance while incorporating weak and narrow (rather than strong and broad) intellectual property protections to ensure greater access for marginalised members of the community as well as developing nations. Finally, rather than paying exorbitant prices, public health systems should pay prices that reflect their contribution to the development of new therapeutic or health technologies (Mazzucato & Li, 2019; Mazzucato & Roy, 2019).

    One should also remember that industrial policies do not only support the growth of certain sectors but can also be used to reduce redundant or harmful ones. Indeed, this insight has been highlighted in the context of the green transition, as policymakers seek to promote green technologies while, at the same time, phasing out the carbon economy (Ergen & Schmitz, 2023). Industrial policies on AI in health can be considered in a similar fashion by situating them in relation to the broader healthcare sector. Healthcare actors, patient groups, unions, and other civil society actors should play a key role here in deciding where to advance and reduce different public health infrastructures and services. This raises the issue of defining “public interest” and classifying the “public(s)”, which is a process of “power, politics and truth seeking” in AI (Sieber et al 2024, p634). How this impacts the establishment of future public interest in AI industrial policy requires context specific interrogation.    

    Finally, with this piece we hope to encourage the further use of industrial policy as a conceptual lens to analyse the increase in interventionist policies around AI and healthcare. While this was only an initial foray into the potential of doing so, industrial policy proved to be a useful approach for several reasons. First, the need for conditionality in the future public benefit to be established pushes back against techno-solutionist narratives. It also facilitates a broader understanding of the purpose of these policies within the wider political economy. Thus, we are better able to wrestle with the reality that policies about AI in healthcare are not just about AI or indeed healthcare. Rather, they are about prioritising certain health futures, closing down others and advancing broader national and regional AI visions. 

    References

    Adler, D. (2021). Inside Operation Warp Speed: A new model for industrial policy. American Affairs, 5(2), 3-32.

    Amsden, A. H. (1989). Asia’s next giant: South Korea and late industrialization. Oxford University Press.

    Amsden, A. H. (1989). Asia’s next giant: South Korea and late industrialization. Oxford University Press.

    Ang, Y. Y. (2018). How China escaped the poverty trap. Cornell University Press.

    Bareis, J., & Katzenbach, C. (2022). Talking AI into being: The narratives and imaginaries of national AI strategies and their performative politics. Science, Technology, & Human Values, 47(5), 855-881. https://doi.org/10.1177/01622439211030007

    Bulfone, F., Ergen, T., & Kalaitzake, M. (2023). No strings attached: Corporate welfare, state intervention, and the issue of conditionality. Competition & change, 27(2), 253-276.

    Bulfone, F., Ergen, T., & Maggor, E. (2024). The political economy of conditionality and the new industrial policy (No. 24/6). MPIfG Discussion Paper.

    Chibber, V. (2003). Locked in Place: State-building and Late Industrialization in India. Princeton University Press;

    Dickens, AD (2021) The right to health implications of data-driven health research partnerships. PhD thesis, University of Essex.

    Doner, R. F., & Schneider, B. R. (2016). The middle-income trap: More politics than economics. World Politics, 68(4), 608-644.

    Emergency USA. (2021, September). Pharmaceutical companies reaping immoral profits from COVID vaccines, yet paying low tax rates. Emergency USA. https://www.emergencyusa.org/2021/09/pharmaceutical-companies-reaping-immoral-profits-from-covid-vaccines-yet-paying-low-tax-rates/?doing_wp_cron=1739796679.7930979728698730468750

    Ergen, T., & Schmitz, L. (2023). The sunshine problem: Climate change and managed decline in the European Union (No. 23/6). MPIfG Discussion Paper.

    European Commission. (2025, February 18). EU 200 billion Euros for AI, largest AI investment in history. European Commission. https://ec.europa.eu/commission/presscorner/detail/en/ip_25_467

    Hoff, J. L. (2023). Unavoidable futures? How governments articulate sociotechnical imaginaries of AI and healthcare services. Futures, 148, 103131. https://doi.org/10.1016/j.futures.2023.103131

    Information Commissioner’s Office. (n.d.). Google DeepMind and class action lawsuit. Information Commissioner’s Office. https://ico.org.uk/for-the-public/ico-40/google-deepmind-and-class-action-lawsuit/

    Johnson, C., (1982). MITI and the Japanese Miracle: The Growth of Industrial Policy: 1925–1975, Stanford: Stanford University Press.

    Johnson, D., & Longmore, S. (2023) How Healthy Is Democracy: The Role of Healthcare and Social Equity Considerations in the Governance of People. In Rethinking Democracy and Governance (pp. 345-369). Routledge.

    Kak, A. (2024). What does “AI for the public good” really mean? AI Now Institute. https://ainowinstitute.org/general/ai-nationalisms-executive-summary#h-what-does-ai-for-the-public-good-really-mean 

    Maggor, E. (2021). Sources of state discipline: lessons from Israel’s developmental state, 1948–1973. Socio-economic review, 19(2), 553-581.

    Maggor, E. (2021). The politics of innovation policy: Building Israel’s “neo-developmental” state. Politics & society, 49(4), 451-487.

    Mazzucato, M. (2023). Health for All: Transforming economies to deliver what matters. bmj, 381.

    Mazzucato, M., & Li, H. L. (2019). Health innovation re-imagined to deliver public value.

    Mazzucato, M., & Rodrik, D. (2023). Industrial policy with conditionalities: a taxonomy and sample cases.

    Mazzucato, M., & Roy, V. (2019). Rethinking value in health innovation: from mystifications towards prescriptions. Journal of Economic Policy Reform, 22(2), 101-119.

    Milmo, D & Stacey, K, (2025, January 13). Labour AI action plan for NHS patient data: Why it’s causing concern. The Guardian. https://www.theguardian.com/politics/2025/jan/13/labour-ai-action-plan-nhs-patient-data-why-causing-concern

    North, M. (2025, January 22). 5 ways AI is transforming healthcare. World Economic Forum. https://www.weforum.org/stories/2025/01/ai-transforming-global-health/.

    OECD. (2025). National strategies, agendas, and plans. OECD AI Policy Observatory. https://oecd.ai/en/dashboards/policy-instruments/National_strategies_agendas_and_plans

    Sieber, R., Brandusescu, A., Adu-Daako, A., & Sangiambut, S. (2024). Who are the publics engaging in AI? Public Understanding of Science, 33(5), 634–653. https://doi.org/10.1177/09636625231219853 p.634

    Strange, M., Tucker, J. Global governance and the normalization of artificial intelligence as ‘good’ for human health. AI & Soc 39, 2667–2676 (2024). https://doi.org/10.1007/s00146-023-01774-2

    Tucker, J. (2023). The future vision (s) of AI health in the Nordics: comparing the national AI strategies. Futures, 149, 103154. https://doi.org/10.1016/j.futures.2023.103154

    Wade, R. (2004). Governing the Market: Economic Theory and the Role of Government in East Asian Industrialization, Princeton University Press.

    Wilson, M R. (2021, November 4). Vaccine manufacturers are profiteering. History shows how to stop them. Politico. https://www.politico.com/news/magazine/2021/11/04/vaccine-manufacturers-are-profiteering-history-shows-how-to-stop-them-519504;

    Zysman, J., (1984). Governments, Markets, and Growth: Financial Systems and the Politics of Industrial Change. Ithaca: Cornell University Press.

    Other

    The authors contributed equally to this work.

    Funding

    JT’s contribution was made possible with the support of the Wallenberg AI, Autonomous Systems and Software Program – Humanity and Society (WASP-HS) funded by the Marianne and Marcus Wallenberg Foundation and the Marcus and Amalia Wallenberg Foundation.

    How to cite this article:

    Maggor, E., & Tucker, J. (2025), Towards Successful Industrial Policy on AI in Healthcare: Establishing the Conditions for Future Public Benefit, AI Policy Exchange Forum (AIPEX), https://doi.org/10.63439/PFRX3762

  • Multilevel oversight of AI systems in line with the AI Act

    Diana M. Popa (Delft University of Technology)
    PDF
    Cite

    Abstract

    The AI “oversight – control” nexus is a matter of debate in both scientific literature and policy papers, given the complex and disruptive nature of the technology and the intricate legislative systems regulating the deployment of AI systems. Most approaches give precedence to the need for a multilayered governance model, while in the same time taking into account the lifecycle stages of the AI product. Oversight over the ecosystem in which the AI is deployed (the governance layer) should not be confused with the process level human oversight. At process level, human oversight is needed in the case of “high-risk systems”, and AI literacy is a precondition for this oversight to be relative. Effective oversight requires many resources at both ecosystem and organisational levels. Additionally, oversight of AI systems is also highly regulated in democratic societies and therefore can be seen as a reflection of the importance that democratic values have in the way disruptive technologies are deployed within society.   

    The need for control and human oversight of AI systems is acknowledged and set down in both legislation texts and research and policy papers, but operationalisation of both concepts is either broad or diverse and in practice address different levels of applicability in relation to the given AI system. Control and oversight of or over the functioning of the AI system itself should not be equalled to how the legislative framework address measures of control and oversight of the ecosystem in which the AI systems are deployed. The multitude of initiatives aiming to establish generally accepted definitions and determine the practical limitations of the “meaningful human oversight” – control nexus (Verdiesen, et al. 2021) underline the difficulty of determining  the control capacity over the AI system. Efforts for identifying the correct balance of human oversight and control are meant to counter the “black box” effect that the disruptive nature of the technology brings with itself, especially when used as a decision support tool with a high impact on the individuals.

    The Artificial Intelligence Act (AIA) itself mentions both control and human oversight as risk mitigation measures, using terms such as “relevant” / “appropriate”/ “meaningful” human oversight. What the specific implementation measures are remain at a general level (Article 14 primarily) while in the same time, the scientific literature addresses the fact that there is ambiguity in the way human oversight is operationalized and implemented, even more so when it comes to “meaningful human oversight”. This is not necessarily a drawback of the legislation, given the fact that the AI Act was just approved by the European Parliament and also considering that, like in the case of the GDPR, the AI Act wants to be an overarching legislation, leaving room for personalized application at Member State level and remaining broad enough so that the AIA itself needs not reviewing or updating every year, taking into consideration also the alert rhythm of technological development in the field.

    From the policy and research approaches, different initiatives have addressed the operationalization of human oversight and the “control – oversight” relationship, either as a nexus or as opposed measures, with approaches depending on the applications field of the concepts. Both control and oversight are defined and embedded within the given broader ecosystem in which the AI model is implemented, and most approaches adhere to a three layer ecosystem: the governance layer (the supra-national and/or national ecosystem in which the AI applications are developed), the socio-technical or organisational layer (based on internal regulations and sector rules) and the technical or process layer (also addressing the product safety regulations) (Verdiesen et al., 2021; Adams et al., 2024; Novelli et al.,2024). Oversight takes place at European level, within dedicated structures (still in development), at national level, with data protection authorities, at organizational level and at process level. Multilayer oversight overlaps (even if one might argue not perfectly) with a multistakeholder governance system of AI systems implementation within a certain regional or national context. Oversight is implemented both vertically and horizontally: the former within national supervisory authorities, expanding their structures to take on formal oversight and control roles with sanctioning capabilities and the latter within the organisations deploying the AI system themselves.

    From a component perspective, building blocks for an effective oversight system include good functioning of the democratic rule of law principles, sound legislative frameworks accompanied by binding powers (both for the pre and post deployment phase), appropriate financial and human resources in all three layers and technical capabilities (technological maturity). In line with democratic principles, an oversight system should address the principles of transparency, accountability and responsibility regarding the way the AI systems are used in public context and incorporate human rights standards. Binding involvement of oversight bodies (in the sense of consultation before the implementation of a tool) and binding powers post factum are necessary for oversight to be effective (Wetzling, 2024), the latter with supervisory/ investigative and sanctioning powers.

    Not addressing here prohibited practices that are equally regulated by the legislation, in the case of high-risk or impactful AI systems, the risk approach managed through measures and degrees of oversight is also influenced by the type of deployer, the sector it activates in and the process in question. While in the case of internal processes, it is a reflection of the risk appetite of an organisation, in the case of processes that have an impact on individuals outside the organisation, such as is the case with AI used by public authorities in the execution of the public administration act, oversight is tighter regulated from the governance level, since it should also be in line with the social values of the system it operates in. Although actual alignment with public values is not required by the legislation, it is a practice in line with democratic values that increases transparency of the governing act and public decision making and is a reflection of the governing style of a certain nation state, such as is the case of the Netherlands.

    The ecosystem division is also often overlapped with the stages of the product’s lifecycle, given that AI is a product deployed on the unique market and also has to comply with product safety regulations, approach which identifies key points during the life of the AI system when human control or oversight is needed, also in relation to the inherent risks of that certain stage. Therefore, within each layer, different control and oversight measures are put in place, at different moments in the AI product’s lifecycle: before deployment, during deployment and after deployment.

    A well-structured oversight model therefore includes both ex-ante and ex-post elements. Expert assessment in the form of expert bodies should be included in both stages, with pre-deployment expert advice going in the design phase of the system and in the high level regulating frameworks and with ex-post oversight in the form of expert assessment and democratic scrutiny (Oetheimer, 2024). These recommendations are also valid in the case of regulatory AI frameworks.

    From this temporal perspective, control is implemented at three different points in time: ex-ante or pro-active control, on-going or simultaneous control, and ex-post control. Ex-ante or pro-active controls are (or should be) implemented by way of the AIA or by the means of the AI national strategy or organisational policies and practices through: 

    • bans on prohibited (high risks) systems (AIA);
    • enforcement through market surveillance and control (AI Act);
    • (quick) scans/ risk analysis for selection of trustworthy suppliers and safe acquisitions (NCTV, 2024) and identification of supply chain risks (Bluebird & Hawk BV, et. al, 2024);
    • DPIAs and FRIAs, (AI Act, National AI strategy, GDPR, Organisational policies);
    • attribution of supervisory roles and compartmentalisation;
    • establishment of internal ethics committees addressing issues such as data ethics compliance or data “pedigree”.

    Ongoing or simultaneous control:

    • supervisory authorities at national and international levels with sanctioning powers;
    • Meaningful human oversight within the process (organisational policies);
    • Process- related activity of ethics committees for deviating cases.

    Equally relevant, for oversight to be effective, practical and technical capabilities giving external parties (either supervisory authorities of the broader public) relevant insight into log frames to evaluate how the data processing took place, details on the way the training data was processed and altered at every step should also be made available. These are forms of ex-post control, giving external parties the possibility to get insight into the system, either through information made public by the deployer beforehand or through standard access requests. A different form of ex-post control would be the requirement to make the AI models FAIR (Findable, Accessible, Interoperable and Reusable) and have metadata or a more extensive documentation package made available, either openly or upon request. This is partially addressed by the requirement to have high-risk AI systems registered in the European data base for High Risk systems that is under development (Article 71 of the AIA).

    Article 14 of the AIA specifically addresses human oversight measures at the process level, placing responsibilities on both developer and deployer: on the developer during the design phase (therefore an ex-ante measure) for the identified risks, and for the deployer for the attribution of the actual human oversight during the process to human resources with the appropriate AI literacy. For high-risk systems, technical provisions include measures similar to kill switches (Article 14, point 4, e.) and oversight measures on supervision of two separate natural persons with appropriate training, competence and authority. Thus, for human oversight to be meaningful, AI literacy is a precondition (Recital 20, Articles 14, 26, 91). Adequate human and financial resources at eco-system and socio-technical levels are also paramount for a functioning oversight system. This is where fragmentation at European level is foreseen to remain high, For example, the Dutch Data Protection Authority (AP), as the dedicated supervisory authority for the use of AI systems that have an impact on personal data, has had an increased budget of 3.6 million euro for 2026 and 2027 (AP, 2024) to address the new tasks under its coordination streaming from the upcoming of the AIA. Yet, despite the budget increase, the Dutch AP is still challenged by finding the necessary numbers of trained specialists for the exercise of the given tasks. While the oversight and governance systems involve multiple entities, data protection authorities, as the responsible oversight entities of the AI also have the role of curbing the data appetite of public authorities. At the implementation level, human oversight remains both challenging and fragmented, influenced by level of digital maturity within a certain society, financial capabilities of each deployer (hyper-scalers having the higher ground), the availability of a trained workforce or lack thereof, the difficulty to keep up with technological and legislative developments, efforts by central government to harmonize implementation of EU legislation in a top bottom approach within national context  and uniformise/change already established daily practices.

    The fact that AI is a disruptive technology is also reflected by the intense debates and analyses in legislative and policy-making circles in order to set it into a unified, comprehensive and EU level actionable text. Oversight of AI systems is highly regulated in democratic societies and while democratic states spend years and a great deal of financial resources to harmonize these legislative frameworks, less democratic states or ones that rather focus on market  competition principles seem more inclined to invest in and focus on the development and implementation of the technology self, focusing on capitalizing on the benefits without the restrictions of ethical and robust legislative frameworks, and as such gaining larger market shares in the world competition stage. 

    Addressing the “why” of the need for human oversight includes the fact that AI is a disruptive technology, the level of trust in technology in general (at social level for example – for example the World Values Survey) and AI technology in particular, and the approach within a certain society towards rules and regulations. As Adams (2024) suggests, AI governance does not translate into responsible AI. Oversight structures are a form of foresight and a practice of a healthy ecosystem, in which decision makers, weather at supra-national, national or sectorial and even company levels implement checks and balances in a proactive manner.  Foresight is limited though to the overview of the known variables of the given environment at a certain moment and unforeseen uses remain a reality and very much in the control and responsibility of the deployer. Taking the discussion on when human oversight is needed further, it comes down to the cases and moments of high risk situations. Despite the fact that automated instruments for decision making have been hailed for the benefits they bring regarding time saving with repetitive tasks, it is in “life or death” situations where they failed and when ultimate responsibility was passed to the human. Identifying such “life or death” situations should be addressed in the design phase of the AI system, despite the limitations or impossibility of mapping out all possible real life situations in a design/ test phase or lab setting.  And while the AIA does address ex-ante or pre-deployment controls, lists prohibited AI practices, and identify highly regulated sectors and exceptions, in practice, specific sector level knowledge and also case by case analysis are needed in order to identify what qualifies as such and to identify the correct moment and forms of human oversight.

    Empirical case studies involving the use of responsible AI in line with the AIA within different contexts are much awaited. Paradoxically, despite extensive regulatory systems, what gives rise to the need for case by case approaches is the uniqueness stemming from the intersection of legislations (for an overview see Annex 1 of the AI Act: list of Union Harmonization legislation), technological regulations, ethics, organizational settings and target groups affected by the AI. Harmonizing legislation attempts such as those in the case of products for the single market are underway. However, if the harmonization process will not work, it is foreseeable that the EU commission will come with its own standards, with the consideration that technical safety is easier to implement and standardize, whereas human oversight is more difficult both to regulate and ethical implications even more difficult to evaluate, reflecting also political choices. As practice evolves, it is expected that in the next years the scientific literature will abound with these case studies. It will be interesting to see how early adaptors, independently of their size, will set trends, how fast good practices will be absorbed and, as was the case with the GDPR, how case law will evolve. These will also serve as measures for measuring the effectiveness of the human oversight and control nexus, currently a challenge that will take the form of consistent practices in the coming years.  

    References

    Adams, R., Adeleke, F., Florido, A., de Magalhaes Santos, L.G., Grossman, N., Junck, L., Stone, K. (2024) Global Index on Responsible AI 2024  (1st Edition). South Africa: Global Center on AI Governance.

    Autoriteit Persoonsgegevens. (2024). Directie Coordinatie Algoritmes Werkagenda coördinerend algoritmetoezicht in 2024. Accessed Mey 2024 at: https://www.autoriteitpersoonsgegevens.nl/documenten/werkagenda-coordinerend-algoritmetoezicht-2024

    Bluebird & Hawk BV., De Nederlandse Vereniging van Banken, ICT Group, Nederlandse Spoorwegen en Technolution;  Nationaal Coördinator Terrorismebestrijding en Veiligheid (NCTV), Nationaal Cyber Security Centrum (NCSC); Algemene Inlichtingen en Veiligheidsdienst (AIVD); CIO Rijk (2024). Cybercheck: ook jij hebt supply chain risico’s. Available at: https://www.ncsc.nl/documenten/publicaties/2024/april/18/cybercheck-ook-jij-hebt-supply-chain-risicos

    Oetheimer, M. (2024) Presentation during the European Data Protection Summit. Rethinking data in a democratic society. Session 3: Zooming out onto democracy and the rule of law. How to build a functioning democratic oversight. Available at:  https://20years.edps.europa.eu/en/summit/media

    Nationaal Coordinator Terrorismebestrijding en Veiligheid (2024). Quick Scan nationale veiligheid bij inkoop en aanbesteding. Accessed June 2024 at: https://www.nctv.nl/onderwerpen/economische-veiligheid/documenten/publicaties/2024/02/01/quick-scan-nationale-veiligheid-bij-inkoop-en-aanbesteding

    Novelli, Claudio and Hacker, Philipp and Morley, Jessica and Trondal, Jarle and Floridi, Luciano, A Robust Governance for the AI Act: AI Office, AI Board, Scientific Panel, and National Authorities (May 5, 2024). Available at SSRN: https://ssrn.com/abstract=4817755 or http://dx.doi.org/10.2139/ssrn.4817755

    Verdiesen, I.; Santoni de Sio, F., Dignum, V. (2021).  Accountability and Control Over Autonomous Weapon Systems: A Framework for Comprehensive Human Oversight. Minds and Machines (2021) 31:137–163. https://doi.org/10.1007/s11023-020-09532-9

    Wetzling, T. (2024) Presentation during the European Data Protection Summit. Rethinking data in a democratic society. Session 3: Zooming out onto democracy and the rule of law. How to build a functioning democratic oversight. Available at:  https://20years.edps.europa.eu/en/summit/media

    Keywords: , , , , , ,

    How to cite this article:

    Popa, D. M. (2024). Multilevel oversight of AI systems in line with the AI Act. AI Policy Exchange Forum (AIPEX). https://doi.org/10.63439/XEFY3802