ON EU’s plans for a scientific panel of independent experts

by: Virginia Dignum and Maja Fjaestad

The Artificial Intelligence (AI) Act envisages the establishment of a scientific panel of independent experts to advise on, and assist the AI Office and national market surveillance authorities with, implementing and enforcing the AI Act. The Commission is currently seeking public for input on implementing regulation establishing a scientific panel of independent experts. Here is our response to this request

The AI Policy Lab welcomes the opportunity to provide feedback on the European Commission’s draft regulation establishing a scientific panel of independent experts in artificial intelligence. This initiative is crucial in ensuring robust, transparent, and impartial oversight of AI systems, aligning with EU objectives to foster innovation while safeguarding fundamental rights. We commend the Commission’s focus on multidisciplinary expertise, diversity, and transparency in panel operations. However, to enhance effectiveness, we offer recommendations to streamline procedural workflows and strengthen data security protocols, ensuring the panel’s structure fully supports its mission in this rapidly evolving field.

Strong Points

We commend key features of the proposal that strengthen the panel’s credibility, flexibility, and proactive oversight:

  • Transparency and Conflict of Interest: Requirements for experts to make declarations of interest and to act in the public interest enhance the panel’s credibility and independence
  • Flexible Structure for Task Allocation: The document enables adaptability by allowing specific members to serve as rapporteurs for individual tasks, ensuring expertise aligns with task requirements?
  • Qualified Alerts for AI Risks: The ability of the scientific panel to issue qualified alerts to the AI Office is an innovative mechanism for highlighting potential AI risks.

Recommendations

Several areas for improvement could enhance efficiency, security, and impartiality in the panel’s operations. In particular, we suggest to address the following issues:

  • Complex Bureaucracy: The involvement of multiple administrative bodies, such as the AI Office, Joint Research Centre, and the Commission, could introduce delays and administrative bottlenecks in the panel’s operations. Streamlined procedural workflows and clarified responsibilities for each entity could enhance the panel’s responsiveness and effectiveness in providing timely guidance.
  • Strengthening Data Security and Confidentiality Measures: Although confidentiality is mentioned, the document could benefit from more detailed procedures on data handling to further mitigate risk related to data security?. Adding explicit guidelines for the secure storage, sharing, and destruction of sensitive information would strengthen the protocol for data handling, especially in cases involving sensitive AI data
  • Enhancing Panel Independence through Conflict of Interest Protocols: While the requirement for declarations of interest is a positive step, more rigorous conflict of interest safeguards – such as independent audits or periodic reviews – could reinforce the panel’s impartiality, especially given the rapidly evolving nature of AI and potential industry influences.
  • Streamline Procedural Steps: Simplifying interactions between the AI Office, Joint Research Centre, and the Commission could enhance efficiency without compromising oversight.
  • Equitable selection criteria: Equality is crucial to guarantee diverse input, to have democratic legitimacy, and to avoid bias. Article 3, par 5, Selection criteria and composition of the scientific panel would therefore benefit from a clearer formalation: Instead of “The Commission shall aim to ensure gender balance” a better formulation could be “the commission shall ensure gender balance”. 
  • Multidisciplinary relevance: The importance of humanities and social sciences expertise should be emphasized. For instance, in Article 3, paragraph 3, by removing “scientific or technical expertise” would broaden the focus, avoiding an unnecessary bias toward natural sciences and valuing multidisciplinary insights on rights, equality, and ethics in AI.

Proposals for consideration

Additional measures could increase the panel’s adaptability, responsiveness, and independence in handling evolving AI challenges, as follows:

  • To maintain the panel’s relevance across rapidly evolving AI fields, we propose supplementing the core panel with a flexible pool of specialized experts. These “on-call” experts would offer guidance on niche areas like ethical AI, quantum AI, or specific sectoral applications, allowing the panel to draw on targeted expertise without permanently expanding its membership.
  • Recognizing the potential risks posed by high-stakes AI applications, we recommend a dedicated Rapid Response Protocol within the panel. This would enable the panel to perform expedited assessments of AI models flagged as potentially harmful, particularly those impacting public safety or fundamental rights, ensuring that urgent cases receive timely and focused attention.
  • To safeguard the panel’s impartiality, we suggest enhanced conflict of interest protocols, including independent audits or periodic reviews of expert affiliations and potential biases. This would reinforce trust in the panel’s independence, especially important given AI’s sensitive and influential role across industries.
  • To promote transparency and public trust, the panel could introduce an AI Accountability Dashboard that provides the public with non-sensitive summaries of decisions, recommendations, and qualified alerts issued by the panel. This dashboard could track metrics like panel activity levels, time-to-decision for urgent alerts, and diversity statistics, thus allowing stakeholders to observe the panel’s impact on AI governance.

AI Policy Lab organizes a Summit on Research on AI Policy and Governance in Stockholm

Summit in Stockholm

The AI Policy Lab at Umeå University and MILA – Quebec AI Institute hosted Framing the Future: A Collaborative Summit on Research on AI Policy and Governance on November 14-15, 2024 in Stockholm, Sweden.

The summit gathered experts from leading research centers to discuss AI policy and governance. The attendees engaged in in-depth discussions and collaborative workshops designed to uncover challenges, opportunities, and practical strategies for shaping the future of research on AI policy and governance.

As a result of the event, the organisers and participants presented a collaborative effort by participants of the AI Policy SummitRoadmap for AI Policy Research – which outlines key priorities for advancing research that strengthens AI governance and human-centered AI.

How Europe is Shaping AI for Human Rights

A Comparative Analysis of the EU AI Act and the Council of Europe Framework Convention

The “Council of Europe Framework Convention on Artificial Intelligence and Human Rights, Democracy and the Rule of Law” (CETS 225), approved in May and open for signatures today, 5 September 2024, creates a legal framework with a strong focus on safeguarding human rights, democracy, and the rule of law in AI development and usage. The convention emphasizes key principles such as transparency, accountability, risk management, and special protection for vulnerable groups, aligning in many ways with the European Union AI Act, which is more detailed in its categorization of AI systems by risk levels and introduces specific regulatory mechanisms for high-risk AI applications.

Based on our analysis, it can be said that the EU AI Act excels in its market-centric approach, providing clear regulatory guidelines that ensure a safe, innovation-friendly environment for businesses while protecting consumer rights. Its risk-based framework is well-defined, allowing for differentiated oversight based on the risk posed by AI applications, particularly in high-risk sectors like healthcare and transportation. This precision fosters compliance and encourages AI development within clear ethical boundaries. On the other hand, the Council of Europe Framework Convention, broader in scope, with its primary strength being a robust focus on human rights, democracy, and the rule of law. It emphasizes transparency, accountability, and inclusivity across all sectors, going beyond economic concerns to ensure that AI systems respect fundamental rights. The convention’s commitment to protecting vulnerable groups and fostering international cooperation for global AI governance is another key strength, ensuring AI development aligns with global human rights standards. In short, I describe the similarities and differences between the two approaches as follows:

Similarities

  1. Human Rights Focus: Both the EU AI Act and the Council of Europe Framework Convention emphasize the importance of safeguarding human rights in the development, deployment, and use of AI systems. This includes ensuring that AI systems do not infringe upon fundamental rights such as privacy, freedom of expression, and non-discrimination.
  2. Risk-Based Approach: Both frameworks adopt a risk-based approach to AI regulation. They require measures to be scaled according to the potential risks that AI systems pose to human rights, democracy, and the rule of law. This involves stricter oversight and requirements for high-risk AI systems.
  3. Transparency and Accountability: Transparency is a key principle in both documents, mandating that AI systems be designed and operated in a way that is understandable and explainable. They also emphasize accountability, requiring entities deploying AI systems to take responsibility for their impacts.
  4. Non-Discrimination: Both the EU AI Act and the Convention address the need to prevent and mitigate discrimination that might arise from the use of AI systems, particularly in vulnerable groups, including women, minorities, and people with disabilities.
  5. International Cooperation: Both frameworks recognize the importance of international cooperation in AI governance, aiming to create a harmonized approach across jurisdictions to address the global nature of AI technology.

Differences

  1. Legal Scope and Binding Nature: The EU AI Act is a regulatory framework specific to the European Union and its member states, establishing binding legal obligations for entities operating within the EU. In contrast, the Council of Europe Convention is a treaty that countries can choose to ratify, and it applies to a broader range of countries, not limited to the EU. As a treaty, the Council of Europe Convention provides a broad framework, leaving it to member states to determine how to implement its provisions into their national laws. This gives countries flexibility in adapting the convention to their legal systems while still adhering to its overarching principles.
  2. Focus on Democracy and Rule of Law: The Council of Europe Convention places a stronger emphasis on protecting democratic processes and the rule of law. While the EU AI Act also addresses these issues, it is more focused on market regulation and the safe integration of AI into the internal market.
  3. Implementation Mechanisms: The EU AI Act includes specific enforcement mechanisms, such as fines for non-compliance, and assigns responsibilities to existing national authorities for implementation. The Council of Europe Convention, on the other hand, establishes a Conference of the Parties to oversee implementation and foster cooperation among the signatories. While the Council of Europe document includes explicit exemptions for national security and defense, these are explicitly excluded from coverage in the EU AI Act.
  4. Definitions and Scope: The scope of what constitutes an AI system and the range of activities covered differ slightly. For instance, the Council of Europe Convention includes a broad definition of AI systems and explicitly covers their entire lifecycle, from development to decommissioning. The EU AI Act also refers to the AI lifecycle but is more focused on categorizing AI systems by risk levels.
  5. Public Consultation and Participation: The Convention explicitly requires public consultation and multistakeholder involvement in discussions about AI governance, which is less emphasized in the EU AI Act, where the focus is more on regulatory compliance by businesses and public sector entities. As such, The Council of Europe convention places a notably more emphasis on promoting digital literacy and skills across all populations, which is less prominent in the EU AI Act. Moreover, the Council of Europe convention explicitly calls for measures to address the rights of specific vulnerable groups, such as children and persons with disabilities, which is less explicitly stated in the EU AI Act.
  6. Remedies and Oversight: The Council of Europe convention explicitly calls for accessible remedies for violations of human rights caused by AI systems, which is detailed in Chapter IV. While the EU AI Act also emphasizes accountability, the approach to remedies might differ in terms of implementation mechanisms.

Both the EU AI Act and the Council of Europe Framework Convention provide strong foundations for regulating AI, but they leave certain gaps. One major shortcoming is their lack of specificity on how to adapt to rapid technological advancements.  AI evolves quickly, and both frameworks focus heavily on supporting current innovation, which, while beneficial in the short term, may undermine public trust and hinder broader adoption in the future if societal concerns and risks are not adequately addressed. Additionally, while they emphasize international cooperation, neither framework offers a clear path for integrating their approaches into a broader, global AI governance system. This lack of alignment could result in fragmented regulations across countries, making it harder to establish consistent ethical standards worldwide.  Another critical omission is the ethical use of AI in military and national security contexts. Both frameworks largely sidestep this issue, leaving a significant gap in ensuring that AI applications in these areas respect human rights and ethical principles. Lastly, while both stress accountability and oversight, there are challenges in implementing clear and practical enforcement mechanisms, particularly for cross-border AI applications and private actors outside direct government control. Addressing these issues would enhance the comprehensiveness and effectiveness of both frameworks in governing AI responsibly.

Welcome to our first fellows!

During spring the first call for AI Policy Fellows was launched. This first call was open for researchers working at all faculties at Umeå University. Many strong submissions were made and after a review process, we are now happy to announce the appointment of the inaugural fellows of the AI Policy Lab. The fellows will join the lab in August and during their time with us the fellows will lead research projects that investigates and address critical intersections of artificial intelligence and societal impact. The selected fellows are:

Elin Kvist

Elin Kvist will address the profound implications of data-driven and algorithmic systems on work environments and workers’ rights. Kvist aims to explore how AI influences workers’ abilities to mobilize and organize, crucially examining the landscape of political attitudes and union preparedness. This initiative seeks to empower workers amidst evolving AI-driven work dynamics.

Lars Norqvist project focuses on AI’s transformative role in leadership practices within educational settings. His interdisciplinary approach aims to unravel the complexities of AI’s impact on leadership autonomy, accountability, and decision-making processes. By employing qualitative methods, Norqvist seeks to provide actionable insights that can inform policy at various governance levels, aiming to bridge theoretical constructs with practical applications in educational leadership.

Lars Norqvist

Henry Lopez Vega research undertakes the challenge of shaping AI policies in healthcare delivery across the region of Västerbotten. Collaborating with regional healthcare stakeholders, Lopez Vega’s research aims to evaluate how AI regulations can enhance service delivery and ecosystem emergence in healthcare. This project will analyze policy implications and foster innovation through the adoption of AI technologies like chatbots and healthcare platforms.

The AI Policy Lab remains committed to advancing cutting-edge research that addresses critical societal challenges posed by AI technologies. Each fellow’s work aligns closely with the lab’s vision to inform policy and practice, ensuring AI’s responsible integration into society.

Opening AI Policy Lab

Invitation

On 18 June 2024, from 13.00-17.oo, the AI Policy Lab will officially open!

You are welcome to join us at MIT-huset A400 (see location in mazemap) for an afternoon filled with presentations, informal conversations and the opportunity to explore the premises.

The program is as follows:

13.00 – Opening by Virginia Dignum, director of the AI Policy Lab

13.15 – Bertram Malle, Brown University: Trust in and Trustworthiness of Artificial Agents

Abstract:

If artificial agents should deserve human trust, they must be trust-worthy.  But what makes a system worthy of trust?  I will introduce a model of human trust in other agents (whether persons, institutions, or machines) that specifies five dimensions of trustworthiness: competence, reliability, integrity, transparency, and benevolence.  I will show that ordinary people think of trust as expectations that the other agent has those attributes. I will then explore how an artificial agent might meet those expectations.  

14.00 – Ericka Johnson and Saghi Hajisharif, Linköping University: Bias and representation in synthetic data

Abstract:

Bias is an issue in the real world and for the AIs learning from real world data. But if we know a dataset is biased, one could hoped that making a curated synthetic data would be a way of eliminating that bias. However, while there is quite a bit of work being done to produce more just datasets through synthetic data, at the same time, work we are doing is demonstrating the challenges of caring for intersectional representation when generating synthetic data. This presentation of our findings ends with a question to the audience about how synthetic data should be labelled and regulated.

14.45 – Q&A

15.15 – Snacks and open house

17.00 – End

We look forward to welcoming you at the opening of the AI Policy Lab!


Morning program

In the morning, we are holding presentations from the MMW Project “AI: destroyer or enabler of democracy”

9.30 – Welcome and fika 

10.00 – 12- 00 – Short presentations (10+minutes each, followed by panel Q&A (topics tbc)

·     Privacy and Self-determination (Kalle Grill and Björn Lundgren) 

·     Automated decision-making in the public sector (Andreas Ojehag) 

·     Political studies of automated governing (Malin Rönnblom) 

·     Democracy and self-determination in a participatory design process in the public sector of a virtual coach for behaviour change (Helena Lindgren) 

·     Global AI governance (Virginia Dignum) 

·     Tool/method for exploring enactment of self-determination (Luis Gustavo Ludesher) 

12.00 – Lunch 

LAUNCHING AIPEX – AI policy Exchange Forum

We are thrilled to introduce the AI Policy Exchange Forum (AIPEX), an open online platform dedicated to fostering academic discussion on AI policy and governance. AIPEX bridges the gap between an academic journal and a blog, providing rapid, citable, and lightly-reviewed publications to stimulate global debates on emerging AI issues.

We look forward to your contributions and engagement in meaningful discussions.

Learn more and submit your contributions here: aipolicy.se/aipex.