EDPS Opinion 44/2023 on the Proposal for Artificial Intelligence Act in the light of legislative developments

23 October 2023

Opinion 44/2023

on the Proposal for Artificial Intelligence Act in the light of legislative developments

0

The European Data Protection Supervisor (EDPS) is an independent institution of the EU, responsible under Article 52(2) of Regulation 2018/1725 ‘With respect to the processing of personal data… for ensuring that the fundamental rights and freedoms of natural persons, and in particular their right to data protection, are respected by Union institutions and bodies’, and under Article 52(3)‘…for advising Union institutions and bodies and data subjects on all matters concerning the processing of personal data’.

Wojciech Rafał Wiewiórowski was appointed as Supervisor on 5 December 2019 for a term of five years. Under Article 42(1) of Regulation (EU) 2018/1725, the Commission shall ‘following the adoption of proposals for a legislative act, of recommendations or of proposals to the Council pursuant to Article 218 TFEU or when preparing delegated acts or implementing acts, consult the EDPS where there is an impact on the protection of individuals’ rights and freedoms with regard to the processing of personal data’ and under Article 58(3)(c), the EDPS has the power to ‘to issue, on his or her own initiative or on request, opinions to Union institutions and bodies and to the public on any issue related to the protection of personal data’.

This own initiative Opinion relates to the Proposal for a Regulation of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) and amending certain Union legislative acts . This Opinion aims to provide further recommendations to the EU co-legislators as the negotiations on the Proposal enter their final stage, focusing on the EDPS roles and tasks in the AI Act.

This Opinion is without prejudice to any future action that may be taken by the EDPS in the exercise of his powers pursuant to Regulation (EU) 2018/1725. This Opinion is limited to the provisions of the Proposal that are relevant from a data protection perspective.

Executive Summary

On 21 April 2021, the European Commission issued a Proposal for a Regulation of the European Parliament and of the Council laying down harmonised rules on artificial intelligence. The present Opinion aims to provide further recommendations to the EU co-legislators as the negotiations on the Proposal enter their final stage. It focuses in particular on a number of legal, technical and institutional aspects related to the future roles and tasks of the EDPS under the AI Act. The recommendations in this Opinion aim to ensure that persons impacted by the use of AI systems enjoy both an appropriate level of protection and legal certainty.

First, the EDPS recalls the crucial importance of the ‘red lines’ provided in the EDPS-EDPB Joint Opinion on the AI Act (‘the Joint Opinion’), notably on AI systems to be prohibited due to their unacceptable risks and on high-risk AI systems.

Second, the EDPS reiterates that AI systems already in use at the date of applicability of the AI Act, including AI systems which are components of EU large-scale IT systems, should not be exempted from the scope of the AI Act. Instead, they should comply with the AI Act requirements from its date of applicability.

Third, the EDPS welcomes that the Proposal designates the EDPS as notified body, market surveillance authority and competent authority for the supervision of the development, provision or use of AI systems by EU institutions, bodies, offices and agencies (‘EUIs’). In this regard, the EDPS provides specific recommendations as to how those tasks, duties and powers should be further clarified, taking into account the specificities of the legal framework applicable to EUIs. In particular, the EDPS highlights the need to clearly define when the EDPS would act as notified body and market surveillance authority, with the powers attributed to these authorities by the AI Act. The EDPS also calls for the allocation of adequate human and financial resources to fulfil these important tasks.

Fourth, the EDPS supports the inclusion in the AI Act of the right of persons affected by the use of the AI systems to lodge a complaint before a competent authority and to an effective judicial remedy against a decision of the authority before which a complaint has been brought. In addition, the EDPS recommends that the competence of the EDPS as supervisory authority to which a complaint may be brought is explicitly included and specified in the AI Act. Furthermore, the EDPS reiterates the call, already made in the Joint Opinion, for the designation of data protection authorities as the national supervisory authorities under the AI Act.

Fifth, the EDPS considers that a truly European approach to enforcement, in particular in crossborder cases with significant impact on persons concerned by the use of the AI systems, is needed. The EDPS welcomes the establishment of the European Artificial Intelligence Office (‘AI Office’), proposed by the European Parliament as a good starting point towards this European approach.

The EDPS considers that the establishment of an EU-level body would help avoid fragmentation having regard to the enforcement of the AI Act, thus ensuring its effective and harmonised implementation. The EDPS remarks, however, that as a member of the European AI Office with (only) observer status he cannot effectively cooperate on an ‘equal footing with national supervisory authorities participating in the decision-making that might nonetheless impact on its supervisory activities. Therefore, the EDPS calls upon the co-legislator to expressly attribute the voting rights in the AI Office to the EDPS as full member of the management board of the AI Office. 2

The EDPS welcomes that the AI Office would be able to coordinate joint investigations carried out by national supervisory authorities. In this regard, the EDPS recommends ensuring that the EDPS is also able to cooperate and participate in these activities on equal footing with national supervisory authorities, as well as in the other activities of the AI Office.

Sixth, the EDPS proposes that the Secretariat of the AI Office is provided by the EDPS. The EDPS is in a position to build from its experience and to harness collaborative advantages, which will be essential for setting up of the Secretariat of the AI Office. In doing this, the EDPS, similarly to his experience as provider of the EDPB Secretariat, would be able to enact a clear differentiation between his support role and tasks with regard to the Secretariat of the AI Office and his role as member of the management board of the AI Office. Moreover, the EDPS has experience in building internal structures such as the Data Protection support pool of experts, a structure that is also envisaged under the AI Office.

3

THE EUROPEAN DATA PROTECTION SUPERVISOR,

Having regard to the Treaty on the Functioning of the European Union,

Having regard to Regulation (EU) No 2018/1725 of the European Parliament and of the Council of 23 October 2018 on the protection of individuals with regard to the processing of personal data by the Community institutions and bodies and on the free movement of such data (‘EUDPR’) , and in particular Article 58(3)(c) thereof,

HAS ADOPTED THE FOLLOWING OPINION:

1. Introduction

1. On 21 April 2021, the European Commission issued a proposal for a Regulation of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act, hence ‘the AI Act’) and amending certain Union legislative Acts (‘the Proposal’) .

2. On 18 June 2021, the European Data Protection Board (‘EDPB’) and the European Data Protection Supervisor (‘EDPS’) issued a Joint Opinion on the Proposal (‘the Joint Opinion’) .

3. The present Opinion of the EDPS is an own-initiative EDPS Opinion, pursuant to Article 58(3)(c) EUDPR. This Opinion focusses on a number of institutional, legal and technical aspects related to the role and tasks of the EDPS in the AI Act in the light of legislative developments, notably the Council mandate for negotiations with the European Parliament (‘general approach’) and the European Parliament negotiating position (‘EP negotiating mandate’) . 5

2. General remarks

5. The use of AI with its specific characteristics (e.g. opacity, complexity, dependency on data, autonomy) can adversely affect a number of fundamental rights enshrined in the EU Charter of Fundamental Rights (‘the Charter’) . The AI Act aims to enhance and promote the protection of the rights protected by the Charter . The EDPS supports the approach consisting in establishing a legal framework for AI systems based on the EU values as enshrined in the Charter, as well as in the European Convention on Human Rights (‘the ECHR’).

6. The EDPS underlines that respect of the fundamental rights to privacy and to the protection 11 12 of personal data enshrined in the ECHR and in the Charter constitute an essential prerequisite for the exercise of other fundamental rights, such as the rights to freedom of expression and freedom of assembly . The fundamental rights to privacy and protection of personal data, and are also closely related to human dignity and to the right to the integrity of the person, including the right to respect for his or her physical and mental integrity .

7. In their Joint Opinion, the EDPS and EDPB have identified several uses of AI that should be prohibited because they pose unacceptable risks to fundamental rights. In particular, the AI Act should explicitly prohibit: - any use of AI to carry out any type of ‘social scoring’ (and not only when performed ‘over a certain period of time’ or ‘by public authorities or on their behalf’) ; - any use of AI for automated recognition of human features in publicly accessible spaces - such as of faces but also of gait, fingerprints, DNA, voice, keystrokes and other biometric or behavioural signals ; - the use of AI to infer emotions of a natural person except for certain well-specified use-cases, namely for health or research purposes (e.g., patients where emotion recognition is important) with appropriate safeguards in place ; - any use of AI systems categorising individuals from biometrics (for instance, from face recognition) into clusters according to ethnicity, gender, as well as political or sexual orientation, or other grounds for discrimination prohibited under Article 21 of the Charter ; 6 - AI systems intended to be used by law enforcement authorities as polygraphs and similar tools whose scientific validity is not proven or which are in direct conflict with essential values of the EU ; - any use of systems intended to be used by law enforcement authorities for making individual risk assessments of natural persons in order to assess the risk of a natural person for criminal offending or reoffending (cf. Annex III, 6. (a)), or for predicting the occurrence or reoccurrence of an actual or potential criminal offence based on profiling of a natural person or on assessing personality traits and characteristics or past criminal behaviour (cf. Annex III, 6.(e)).

8. These uses of AI are so intrusive and affecting human dignity that they should be prohibited . It is not sufficient for any of these uses to be classified as ‘‘high risk’’ in Annex III of the Proposal. The use of certain AI systems that the EDPS considers that should be prohibited are against EU fundamental values ; would not meet the necessity and proportionality requirements ; or would be in direct conflict with essential values of the 24 25 EU and affect human dignity .

9. The EDPS welcomes that the Proposal designates the EDPS as notified body, as market surveillance authority as well as competent authority for the supervision of the development, provision or use of AI systems by EU institutions, bodies, offices and agencies (‘EUIs’). As the EU’s independent data protection authority responsible for supervising the processing of personal data by EUIs, the EDPS has long-standing experience in monitoring the impact of new technologies on data protection (as inextricably linked to compliance with fundamental rights and freedoms), including cooperating with other data protection authorities, as well as with other competent authorities, to ensure the consistent enforcement of the applicable legal frameworks on data protection.

10. The EDPS is committed to leverage this expertise, including technical expertise, also building on the supervisory experience in the context of AI systems in sensitive domains such as migration and asylum and the prevention, investigation, detection or prosecution of criminal offences.

11. With this Opinion, the EDPS provides specific advice as to how these tasks, duties and powers could be further clarified in the AI Act, taking into account the specificities entailed by the legal framework applicable to EUIs. The EDPS also provides further recommendations on how to foster an effective enforcement of the AI Act through a true ‘European approach’, in particular in cross-border cases with significant impact on persons concerned by the use of the AI systems. 7

3. Scope of the AI Act

3.1. AI systems already on the market or in use

12. The EDPS notes that pursuant to Article 83(2) of the Proposal, the AI Act would not apply to operators of high-risk AI systems that were already on the market or in use before the date of applicability of the AI Act, except in cases when these systems are subject to significant changes in their design or purpose or, according to the EP negotiating mandate, in case of ‘‘substantial modifications’’ .

13. The EDPS considers that such provisions lack the necessary clarity as to which criteria should be applied to determine what constitutes a ‘significant change’ or ‘substantial modification’. As a consequence, there may be considerable legal uncertainty as to when the AI Act effectively applies to AI systems already on the market or in use . More importantly, the EDPS is concerned that such an approach may lead to situations where certain AI systems, which pose high risk of harm or adverse impact on fundamental rights as per Article 7 of the Proposal, may never fall within the scope of the AI Act because they continue to operate without any ‘significant changes’ or ‘substantial modifications’.

14. The EDPS therefore recommends removing the exemption from the scope of the AI Act of the existing high-risk AI systems provided for in Article 83(2) of the Proposal, and instead providing for the applicability of the AI Act to existing high-risk AI systems on the date of applicability of the AI Act.

3.2. Providers of AI systems

15. Most of the legal requirements laid down in the Proposal would apply to the ‘providers’ of AI systems. The definition of ‘provider’ in the Proposal refers to the ‘development’ of an AI system. However, the term ‘development’ or ‘to develop’ is not defined in the Proposal.

16. Data processing for training is a fundamental part of the development of machine learningbased AI systems. Therefore, anyone training an AI system could be considered as a provider of the AI system. However, AI systems can be trained more than once in their lifecycle. Retraining can be necessary for many reasons, notably due to a lack of large training data or due to a degradation of the system performance. In some cases, pre- 8 trained AI systems are retrained with the aim of being applied to a similar task in a different domain (transfer learning). Finally, some AI systems apply continuous learning (which means that the training takes place on an ongoing basis).

17. The Proposal, as well as the general approach and the EP negotiating mandate, do not clearly define the role of AI operators who retrain pre-trained AI systems or use continuous learning AI systems. In other words, the Proposal does not clarify if the retraining or continuous training activities should be considered as part of the ‘development’ of the AI system and, therefore, these operators should be considered as ‘providers’ within the meaning of Article 3(2) of the Proposal.

18. This issue is particularly relevant having regard to foundation models. The emergence of AI system training techniques such as Parameter Efficient Fine-Tuning (PEFT) have significantly decreased the cost, and thus made broadly available, the retraining of foundation models.

19. Given the impact that the retraining of AI systems can have on their performance, purpose and their compliance with legal requirements, the EDPS considers necessary to clarify the role of AI operators retraining AI systems, specifying whether the ‘re-trainers’ are providers. The EDPS understands that AI operators retraining pre-trained AI systems should be qualified as ‘providers’ in order to guarantee that compliance the requirements of the AI Act remains ensured. The EDPS recommends the co-legislators to clarify this aspect in the AI Act beyond any doubt.

3.3. EU large-scale IT systems and international law enforcement cooperation

20. It is necessary to ensure that the AI Act covers all relevant areas where AI systems would be deployed, including - as it is the case in the Proposal - AI systems to be used in the law enforcement and migration areas. The EDPS notes that certain AI components of the systems for migration management and border control are currently being procured, for instance, by eu-LISA, the EU Agency responsible for the operational management of the EU large-scale database systems in the Justice and Home Affairs (JHA) area . The EDPS also recalls that other types of AI systems have been tested but not deployed due, among other things, to their impact on human rights . These AI systems should be properly regulated, 9 including via express prohibition, in case of unacceptable risks, already at the design and development stages .

21. As already highlighted in the Joint Opinion , the EDPS is concerned about the exclusion from the scope of application of the AI Act of AI systems which are components of the EU large-scale IT systems established by the legal acts listed in Annex IX of the Proposal. These large-scale information systems involve the automated processing of vast amounts of personal data, including sensitive data, automated risk assessment systems and the use of technology for biometric identification, which could be supported by AI applications and therefore should be subject to the safeguards established in the AI Act. The EDPS notes with concern that neither the EP negotiating mandate or general approach would ensure a sufficient level of protection. He therefore strongly recommends removing the exemption from the scope of the AI Act as regards AI systems which are components of the EU largescale IT systems established by the legal acts listed in Annex IX of the Proposal and to ensure that the requirements of the AI Act become applicable to these systems on the date of applicability of the AI Act.

22. Given that there will be a considerable amount of time between the entering into force and applicability, the EDPS considers that the operators of AI systems which are components of large-scale information systems would have sufficient time to take the necessary steps to comply with the requirements laid down in the AI Act from its date of applicability.

23. In the same vein, the exclusion from the scope of the AI Act of AI systems to be used in the context of international law enforcement cooperation, pursuant to Article 2(4) of the Proposal, raises serious concerns , since it would deprive the persons affected by the use of these systems of the legal protections provided by the AI Act. Again the EDPS notes with concern that neither the EP negotiating mandate or general approach would remedy this issue. The EDPS therefore recommends removing the exclusion from the scope of the AI Act of AI systems to be used in the context of international law enforcement cooperation. 10

4. High-risk AI systems

24. The EDPS notes that the general approach, in Article 6(3), proposes to add that AI systems referred to in Annex III shall be considered high-risk ‘unless the output of the system is purely accessory in respect of the relevant action or decision to be taken’.

25. The EDPS considers that this exception raises concerns since it could jeopardise the safeguards applicable to AI systems. This is because that exception would leave an unduly broad margin of discretion for the provider of the AI system to assess whether the output of the system is ‘purely accessory’ to the decision-making, in particular considering that in the general approach these terms are not defined. Moreover, the provider is not necessarily in a position to assess whether the AI system will be used in an ‘accessory way’ by the AI user. The EDPS therefore recommends not to include this exception in the AI Act.

26. The EDPS also notes that the EP negotiating mandate provides for the possibility for the provider not to consider the AI system in Annex III as ‘high risk’ if they do not pose a significant risk of harm to the health, safety or fundamental rights of natural persons . The EDPS considers that these amendments would significantly jeopardise the implementation of the safeguards provided for high risk systems and, ultimately, decrease legal certainty and foreseeability. This is due to the fact that the assessment of the ‘significant risk’ for health, safety or fundamental rights of natural persons or to the environment would confer an excessively broad leeway to the provider not to apply the requirements for high-risk AI systems. Moreover, it can be noted that the assessment of the risk by the AI provider might not correspond to the risk-scenario at the moment of deployment of the AI system, and therefore such risk assessment might not be accurate. The EDPS therefore recommends not including these provisions in the final text of the AI Act.

27. The EDPS underlines that insofar as high-risk AI systems require the processing of personal data or process personal data to fulfil their task, the certification of these AI systems in the context of the CE marking should expressly include a verification of compliance, among others, with the data protection by design and by default , given the possible high level of interference of the high-risk AI systems with the fundamental rights to privacy and to the protection of personal data, as well as the need to ensure a high level of trust in the AI system. The EDPS recalls that the assessment of compliance of the high-risk AI system 11 with legal obligations arising from Union legislation, including on the personal data protection, should be a precondition for obtaining the CE marking of the AI system .

28. As recommended in the Joint Opinion , the EDPS also calls to adapt the conformity assessment procedure under Article 43 of the Proposal to the effect that an ex ante third party assessment must generally be carried out for these high-risk AI systems. The EDPS also notes that, taking into account the sectoral legislation applicable to the activity in the context of which the AI system is used, the third party assessment of the high risk AI system, to ensure the trustworthiness of the AI, requires the involvement of the oversight authority having specific expertise in the field (e.g., the health authority, in case of medical AI triage system) .

5. EDPS roles and tasks

5.1. Overview of EDPS roles and tasks

29. Pursuant to the Proposal, the EDPS would act as: a) in the context of pre-market control (conformity assessment) : - notified body, that is as conformity assessment body, when the high-risk AI system is intended to be put into service by Union institutions, bodies, offices and agencies (‘EUIs’), (e.g., in case of a high-risk AI system to be used by the European Commission), or when a high-risk AI system is developed by an EUIs, (e.g. by the Joint Research Centre acting as provider) (Article 43(1)); b) in the context of post-market control (post-market monitoring, information sharing, market surveillance): 12 c) as competent supervisory authority: - competent authority for the supervision of compliance of EUIs with the AI Act (Article 59(8)): the EDPS would be the competent authority for the supervision (‘competent authority’) of EUIs falling within the scope of the AI Act. This role involves monitoring and enforcing the compliance of EUIs with the AI Act and establishing AI regulatory sandboxes .

30. The EDPS welcomes that the Proposal designates the EDPS as competent authority, as notified body and as well as market surveillance authority for the supervision of the development, provision or use of AI systems by EUIs. The EDPS already supervises AI systems developed, procured or used by EUIs, exercising the powers granted to the EDPS by the EUDPR or by other sectorial legislation insofar as the processing of personal data is concerned. In this context, the EDPS currently focuses on sectors and data processing activities that pose higher risks to the fundamental rights of persons impacted by the use of these systems (e.g. law enforcement and migration).

31. Nevertheless, the role and tasks of the EDPS in the AI Act, including the powers that would be available to the EDPS for the oversight of AI systems, must be further clarified in order to make the relevant provisions operational, notably having regard to the role of the EDPS as notified body and as market surveillance authority.

5.2. EDPS as notified body

32. The EDPS notes that ‘notified body’ is a conformity assessment body designated in accordance with the AI Act and other relevant Union harmonisation legislation by an EU Member State to assess the conformity of AI systems with the AI Act before being placed on the market . These bodies, having regard to Regulation (EU) 2019/1020 (‘EU Market Surveillance Regulation’) , carry out tasks related to conformity assessment procedures to ensure that products are compliant with Union harmonisation legislation in order to guarantee their free movement within the Union.

33. Article 43(1) of the Proposal provides that for the purpose of the conformity assessment procedure, the provider may choose any of the notified bodies. However, ‘when the system is intended to be put into service by law enforcement, immigration or asylum authorities as well as EU institutions, bodies or agencies, the market surveillance authority referred to in Article 63(5) or (6), as applicable, shall act as a notified body.’ (emphasis added)

34. The EDPS considers that the competence of the EDPS as notified body under the AI Act should be further clarified. For instance, issues related to the identification of the competent notified body might arise in case of AI system intended to be put into service by law enforcement authorities in different Member States as well as by EUIs (such as Europol or Eurojust). The EDPS therefore recommends clarifying the rules for the identification of the competent notified body, by explicitly providing that the EDPS would be the notified body [only] when the provider - within the meaning of Article 3(2) of the Proposal - is an EU institution, body, office or agency.

35. In this context, the EDPS also recalls his previous recommendation to unambiguously define the notion of ‘provider’ (see section 3.3. of the present Opinion) and recalls that, both in specific case of retraining of foundation model (referred to in section 3.3. of the present Opinion) and in case of substantial modifications of the AI system (pursuant to Article 28(1)(c) of the Proposal), the EUI as provider would fall under the oversight of the EDPS as notified body. The EDPS indeed points out that, according to Article 28(1)(c) of the Proposal , when any distributor, importer, user or other third-party makes a substantial modification to the high-risk AI system, it will be considered a provider, subject to the obligations of the provider under Article 16 of the Proposal.

5.3. EDPS as market surveillance authority

36. Having regard to the EDPS role as market surveillance authority, the EDPS highlights that a market surveillance authority is an authority designated by a Member State under the EU Market Surveillance Regulation as responsible for carrying out market surveillance in the territory of that Member State .

37. The EDPS highlights that the market in which it would expect to exercise its market surveillance authority role is not defined by geography, but by the nature of the providers and users involved (i.e., Union institutions, agencies and bodies falling within the scope of the Proposal). Nonetheless, most of the tasks and duties of competent authorities as defined in the EU Market Surveillance Regulation apply taking into account as ‘relevant market’ the territory of Member State(s) .

38. According to Article 65(3) of the Proposal, where the market surveillance authority considers that non-compliance is not restricted to its national territory, the market surveillance authority must inform the Commission and the other Member States of the results of the evaluation and of the actions prescribed to the AI provider or user. In this regard, the EDPS notes that the term ‘national territory’, while being clear having regard to national market surveillance authorities, is unclear having regard to the possible scope of action of the EDPS. 14

39. The EDPS recommends therefore clarifying in the AI Act that ‘national territory’ for the purpose of market surveillance for the EDPS means: ‘where the AI system is provided or deployed by Union institutions, bodies, offices and agencies’.

40. As laid down in Article 65(5) of the Proposal, if the operator of an AI system does not take adequate corrective action, the market surveillance authority must take all appropriate provisional measures to prohibit or restrict the availability of the AI system on the national market, to withdraw it from the market or recall it. Also in this case, it is unclear what would be the relevant (national) market in case of prohibition of an AI system by the EDPS (acting as market surveillance authority). Hence, also in this case, a definition of ‘relevant market’ as ‘where the AI system is provided or deployed by Union institutions, bodies, offices and agencies’ would be essential to avoid legal uncertainties.

41. The EDPS observes that market surveillance authorities referred to in the AI Act would have the powers provided to them pursuant to the EU Market Surveillance Regulation . However, the EDPS recommends explicitly specifying in the enacting terms of the AI Act that also the EDPS should have such powers as market surveillance authorityfor EUIs.

42. In the light of the above, the EDPS recommends including the following provisions in the enacting terms of the AI Act: In Title I, General provisions: ‘This Regulation is without prejudice to the competences, tasks, powers and independence of relevant national public authorities or bodies which supervise the application of Union law protecting fundamental rights, including equality bodies and data protection authorities’; In Title I, General provisions: ‘The EDPS exercises its powers pursuant to the AI Act, including the powers as market surveillance authority under the EU Market Surveillance Regulation, without prejudice to its tasks and powers under the EUDPR or other applicable Union law, such as Regulation (EU) 2016/794’; In Title VIII, Post-market monitoring, information sharing, market surveillance: ‘The EDPS shall act as market surveillance authority having regard to the development, deployment and use of AI systems developed by or used by Union institutions, bodies, offices or agencies’; In Title VIII, Post-market monitoring, information sharing, market surveillance: ‘‘National territory’ for the purpose of market surveillance for the EDPS means: ‘where the AI system is provided or deployed by Union institutions, bodies, offices and agencies’.

43. Finally, the EDPS is very concerned by the amendments introduced by both the EP Compromise Amendments and by the general approach to Article 64 (access to data and documentation) of the Proposal. The general approach deletes Articles 64(1) and 64(2) of the Proposal that would have granted market surveillance authorities access to the training, validation and testing data and to the ‘source code’ of the AI system.

44. The EDPS welcomes that the EP negotiating mandate still includes these provisions. Nonetheless, the EDPS notes that the access to data, documentation and models seems to be limited. The access would only be possible ‘upon reasoned request’ to data ‘strictly necessary for the purpose of the request’. In the same vein, access to the trained model and relevant parameters would only be granted ‘after all other reasonable ways to verify conformity including paragraph 1 have been exhausted and have proven to be insufficient’. In this regard, the EDPS notes that the requirement for market surveillance authorities to prove (to AI providers or deployers) that all other ways to verify conformity have been exhausted and were insufficient is unwarranted. Such requirement would hinder the verification activities. Moreover, the terms ‘strictly necessary’, as well as such ‘exhaustion of the reasonable ways to verify conformity’ are unclear and could therefore undermine the investigation powers of market surveillance authorities.

45. The EDPS considers that the AI Act should provide for clear rules about the investigative powers of market surveillance authorities. These rules should include the authority's power to access training, validation, and testing data. Furthermore, said authorities should be able to review the source code of AI systems. In addition, the AI Act should empower authorities to carry out unannounced on-site and remote inspections of high-risk AI systems, as provided in Art. 63(3a)(a) of the EP negotiation mandate. In the absence of these powers, authorities will not be able to check the compliance of AI providers with the requirements set out in the AI Act. Indeed, without access to training, validation, and testing data, as well as to the source code, oversight authorities would not be able to check for instance that training, validation and testing data are ‘relevant, representative, free of errors and complete’ (as prescribed under Article 10(3) of the Proposal). Moreover, without access to the source code of AI systems the capacities of oversight authorities to check AI systems’ ‘robustness and cybersecurity’ (as prescribed under Article 15 of the Proposal) would be very limited.

46. The AI Act should also allow these authorities to access any necessary data, documentation, and software for their oversight activities. Importantly, while the exercise of the enforcement powers by market surveillance authorities should of course be subject to appropriate safeguards, starting with strict confidentiality requirements and include effective judicial remedies and guarantees of due process, set out in Union law, these powers should not be made subject to restrictions by the legislators that would hamper their supervision and enforcement tasks.

5.4. EDPS as competent authority for the supervision of EUIs 5.4.1. In relation to complaints lodged by the persons affected by the use of AI systems

47. The EDPS welcomes that the general approach lays down in Article 63(11) that any natural or legal person having ground to consider that there has been an infringement of the AI Act has the right to submit a complaint to the competent market surveillance authority. 16

48. The EP negotiating mandate , further developing the provisions proposed in the general approach, provides to persons affected by the use of the AI systems the right to lodge a complaint with a national supervisory authority, in particular in the Member State of his or her residence, in case of infringement of the AI Act relating to him or her. Furthermore, the right to an effective judicial remedy against a decision of the supervisory authority is also set out .

49. In this regard, the EDPS considers that the competent supervisory authority (rather than market surveillance authorities) would be better placed to address complaints from persons impacted by the use of the AI system, due to the market-focus of market surveillance authorities, on the one hand, and the possibly broader approach of competent authorities, on the other hand. This is particularly relevant considering that in most cases the development and the use of high-risk AI systems can have a significant impact on the fundamental rights and freedoms of persons affected by the use of the AI systems .

50. The EDPS also considers that the competence of the EDPS as supervisory authority to which a complaint may be brought, as well as the right of the person concerned to an effective judicial remedy against an EDPS decision, should be explicitly provided for in the AI Act.

51. The EDPS also welcomes the inclusion of the definition of ‘affected person’ as ‘any natural person or group of persons who are subject to or otherwise affected by an AI system’ , entitled to the right to judicial remedy. Concerning rights and remedies that should be available to individuals subject to AI systems , the EDPS points out that, in order to facilitate the exercise of the right to lodge a complaint, affected persons should have the right to be always informed in particular: about if/when they are subject to an AI system; about whether and how the use of the AI system has a possible impact on their fundamental rights; as well as about the venues for administrative (and judicial) redress towards providers and deployers of these AI systems.

52. In this regard, the EDPS notes that Article 52 as it stands in the Proposal, and in the general approach, would provide for transparency obligations for providers and deployers of AI systems intended to ‘interact’ with natural persons. The EDPS considers that these transparency requirements should be complemented by a broader transparency obligation for providers and deployers of AI systems to inform the persons affected by the use of the AI system, notably when these persons are subject to decisions made via the AI systems or to recommendations provided by the AI system that may impact their life, economic or legal situation, or social status (which is the case for some high-risk AI systems in Annex III of the Proposal). Hence, the EDPS recommends that persons affected by the use of AI systems as a rule are informed about such use (not only in the cases indicated in Article 52 of the Proposal).

53. Moreover, the EDPS recommends including in the AI Act, as right for the person affected by the use of the AI system, notably in case of use of high-risk AI systems, the right to obtain human intervention from the user of the AI system having regard to the decision- 17 making affecting him or her and to contest the output of the decision-making, as well as a right to explanation of individual decision-making. The EDPS welcomes the reference in the EP negotiating mandate to the right to explanations from the deployer of the AI system of the decision-making significantly affecting the AI subject, notably having regard to the role of the AI system in the decision-making, the main parameters of the decision taken, and the related input data. This right would complement and be without prejudice to, the rights established by the GDPR (notably, Article 22 GDPR), as well those rights established by the legislation applicable to the activity in the context of which the AI system is deployed (consumer credit, insurance services, employment, etc.).

54. The EDPS also welcomes the introduction of provisions on the protection of ‘whistle- 63 64 blowers’ pursuant to Directive (EU) 2019/1937.

5.4.2. As authority establishing the AI regulatory sandboxes

55. According to the Proposal, AI regulatory sandboxes, to be established by one or more Member States’ competent authority or by the EDPS, would provide a controlled environment for development, testing and validation of innovative AI systems for a limited time before their placement on the market or putting into service pursuant to a specific plan. This shall take place under the direct supervision and guidance by the competent authorities with a view to ensuring compliance with the requirements of the AI Act and, where relevant, other Union and Member States legislation supervised within the sandbox .

56. In this regard, the EDPS supports the specification introduced in the EP negotiating mandate, adding the possible establishment of regulatory sandboxes by the Commission and by the EDPS, on their own, jointly or in collaboration with one or more Member States, at Union level .

57. The EDPS also welcomes that the EP negotiating mandate specifies, as objectives of the regulatory sandboxes: the increase of the understanding of technical developments by the authorities establishing the sandbox; the improvement of their supervisory methods; offering guidance to providers of AI systems on regulatory compliance with the AI Act, other applicable Union and Member States legislation, as well as with the Charter .The EDPS welcomes that the EP also specifies the legal effect of the sandboxing exercise, namely a presumption of conformity (limited to the tested regulatory requirements) for providers 18 of AI systems exiting the sandboxing, since this would increase legal certainty and further incentivise the establishment of regulatory sandboxes.

58. Finally, the EDPS welcomes and supports the important reference - also included in the EP negotiating mandate - to the role of data protection authorities and of the EDPS, ensuring compliance with data protection law in the context of the sandboxing exercise .

5.5. Allocation of adequate human and financial resources

59. The various functions that the EDPS would need to perform as notified body, as market surveillance authority as well as competent supervisory authority for the supervision of the development, provision or use of AI systems by EUIs will require specific expertise and resources.

60. In particular, the Proposal indicates that the notified bodies must have sufficient internal competences to be able to effectively evaluate the tasks conducted by external parties on their behalf. The notified body must have permanent availability of sufficient administrative, technical, legal and scientific personnel who possess experience and knowledge relating to the relevant AI technologies, data and data computing and to the requirements set out in Chapter 2 of this Title .

61. The EDPS notes that the certification process to be performed by notified bodies is not a ‘one off’ exercise. According to the Proposal, certificates issued by notified bodies will be valid for a period not exceeding five years and high-risk AI systems must undergo a new conformity assessment procedure whenever they are substantially modified. Moreover, the EDPS role as notified body would entail the need to carry out periodic audits on AI providers to ensure that the latter maintains and applies the requirements in Chapter 2 of Title III of the AI Act, notably the quality management system . These obligations would imply continuous monitoring that would increase over time with each AI system the EDPS would need to assess. Consequently, the certification process would require a significant allocation of resources in order for the EDPS to fulfil this role .

62. It also has to be noted that, according to Article 33(8) of the Proposal, notified bodies must take out appropriate liability insurance for their conformity assessment activities. Also in this regard, the EU (budgetary authorities) should ensure that the EDPS is provided with adequate financial resources. 19

63. Moreover, also having regard to the handling of complaints, the EDPS highlights the need for additional resources in order to effectively fulfil this task.

64. Finally, concerning the possible deployment of regulatory sandboxing by the Commission and the EDPS, it is necessary that the AI Act ensures that the EDPS is equipped with adequate technical, financial and human resources.

65. To effectively accomplish these new tasks stemming from the AI Act, the EDPS underscores the necessity for a substantial allocation of both human and financial resources to the EDPS. To this end, the EDPS recommends adding the following wording in Article 59(4) of the AI Act: ‘The obligation to ensure adequate resources shall also apply to the EU budgetary authorities with regard to the EDPS as notified body, market surveillance authority, competent supervisory authority and for any other tasks and duties attributed under this Act’.

5.6. Data protection authorities as national supervisory authorities

66. The EDPS reiterates the recommendation made in the Joint Opinion, calling for the designation of data protection authorities as national supervisory authorities pursuant to Article 59 of the Proposal . This recommendation is based in particular on the following reasons: a) data protection authorities are enforcing the GDPR, the EUDPR and the LED and other sectorial legislation when AI systems involve the processing of personal data in order to ensure the fundamental rights to privacy and to the protection of personal data, often inextricably linked to the protection of other fundamental rights, such as the right to dignity or to an effective judicial remedy and a fair trial; b) data protection authorities, in addition to have expertise in assessing risks to fundamental rights posed by new technologies like AI (e.g. AI using biometric data) are also, due to their complete independence, the authorities that can provide effective independent supervision of AI systems that can impact on fundamental rights and freedoms ; c) the designation of data protection authorities as national supervisory authorities would contribute to an interpretation and implementation of the AI Act consistent with the applicable data protection laws.

67. At the same time, the EDPS underlines that it is essential that data protection authorities as national supervisory authority for AI systems closely cooperate with other oversight authorities who have expertise in the specific field of deployment of the AI to ensure that any given AI system is trustworthy, safe and in compliance with any other Union legislation 20 applicable to the activity in the context of which the AI system is used (e.g., strictly cooperating with the competent oversight authority on health and safety at work, in case of AI system for work management; or with the heath oversight authority, in case of AI system for urgency medical care triage) . Hence, the EDPS recommends specifying in Article 59 of the Proposal that data protection authorities as national supervisory authorities will closely cooperate with the oversight authorities who have expertise and responsibilities in the specific field of deployment of the AI system to ensure its trustworthiness.

5.7. AI Governance - the proposed European AI Office 5.7.1. The AI Office as an independent Union body

68. The EDPS welcomes the proposed establishment, under the EP negotiating mandate , of the European Artificial Intelligence Office (‘AI Office’) as an independent body of the Union having legal personality, responsible for a number of advisory and coordination tasks, including issuing opinions, recommendations, advice or guidance on matters related to the implementation of the AI Act. The EDPS also welcomes that the amendments expressly provide that the AI Office must be adequately funded and staffed for the purpose of performing its tasks .

69. The AI Office should be an independent body of the Union having the required know-how from both a legal and technical perspective. An independent entity tasked with avoiding fragmentation when enforcing the AI Act would be best placed to ensure its effective and harmonised implementation. The EDPS believes that the independence of the AI Office is instrumental for achieving a development of the AI with high level of protection of fundamental rights, democracy and the rule of law across the Union with regard to AI systems. For the development of AI it will be important indeed to ensure an environment in which enforcement will not follow the political directions of the moment but rather will be ensured in full impartiality. The AI Office as an independent body of the Union is likely to both strengthen enforcement in a way that would mitigate the risk of possible 'forum shopping' by providers of AI systems, and also ensure a true level-playing field where AI systems can flourish without undermining fundamental rights and essential European values.

70. In order to preserve the homogeneous interpretation of the new AI rules across European and national public administrations, the AI Office and the EDPS, in its role of AI supervisor, should work in alignment. The EDPS notes with regret that, according to the EP negotiating mandate , the EDPS does not have voting rights in the management board of the AI Office. As a consequence, the EDPS would not be able to effectively cooperate on an ‘equal footing’ with national supervisory authorities (who do have voting rights and could make decisions that can also directly affect the supervisory activities of the EDPS). The EDPS welcomes that the AI Office would be able in particular to coordinate joint investigations carried out 21 by national supervisory authorities . In this regard, the EDPS strongly recommends including in the AI Act a clear and express specification that the EDPS shall be considered equivalent to national supervisory authorities for the purposes of the AI Act, so that it is able to participate in joint investigations on equal footing with other national supervisory authorities, as well as in the other activities of the AI Office.

71. The EDPS wishes to emphasise the need to have a European approach to enforcement. The AI Office is a good starting point towards this European approach, although it should receive enhanced powers for enforcement. While a European approach would be beneficial to enforce rules on any AI systems, it appears even more necessary in cases of types of AI systems that for their nature and for its cross-border diffusion have a significant impact, such as foundation models. The EDPS supports the amendments of the EP negotiating mandate adding a number of tasks to the AI Office, including: to serve as a mediator in discussions about serious disagreements that may arise between competent authorities regarding the application of the AI Act ; to facilitate the creation and maintenance of a Union pool of experts , and ensuring their vetting; to assist authorities in the establishment and development of regulatory sandboxes and to facilitate cooperation among regulatory sandboxes .

72. The EDPS welcomes that according to the EP negotiating mandate , the AI Office would be tasked with issuing opinions, recommendations or written contributions including with regard to technical specifications or existing standards, Commission’s guidelines, codes of conducts, etc. In such cases, it is essential that data protection considerations are duly taken into account. This is because, as referred to in the Joint Opinion , most of the systems in the scope of the AI Act are or will be based on the processing of personal data. Therefore, the EDPS stands ready to work together with the AI Office to achieve this goal.

73. The EDPS stresses the need to be able to cooperate as a fully-fledged supervisory authority, given its role in supervising EUIs' activities in particularly sensitive domains, such as police and justice, migration, where Europol, Eurojust, the European Border and Coast Guard Agency, the European Asylum Support Office are called to action, as well as in the healthdomain, where EU agencies such as the European Medicines Agency and the European Centre for Disease Prevention and Control play a key role, or the economic and monetary policy aspects managed by the European Central Bank. In all these areas, AI systems, regulated under the AI Act, might be provided and deployed by EUIs. The EDPS therefore requests the co-legislator to specifically designate the EDPS as a full member of the AI Office with voting rights.

74. Finally, the EDPS welcomes the task attributed to the AI Office, due to the global dimension of the regulation of AI systems, of contributing to the effective cooperation with the competent authorities of third countries and with international organisations . Also in this regard, based on its experience at the international level, the EDPS has an important role 22 to play to provide the highest possible level of protection of many fundamental rights, of which the rights to privacy and to the protection of personal data are key enablers.

5.7.2. Secretariat of the AI Office

75. The Secretariat of the AI Office would provide analytical, administrative and logistical support to the AI Office , in a similar way that the current EDPB Secretariat provides assistance to its members.

76. The EDPS stands ready to offer support in this regard, providing the Secretariat of the AI Office. The EDPS providing the Secretariat of the AI Office would allow the most effective use of resources, because it allows making use of existing administrative structures, avoiding the additional cost that would be linked to the set-up of a new entity. This would allow budgetary savings since the human resources and the budgetary aspects would be dealt with by the EDPS, as an already established EU institution, and thus contribute to sound management of EU finances.

77. At the same time, such savings can be realised by inscribing administratively the AI Office within an independent authority such as the EDPS, which would offer guarantees of impartiality to all stakeholders.

78. The EDPS has already long-standing experience in the establishment of a new administrative structure for an independent European body, the European Data Protection Board, in full respect of its independence in the performance of its tasks. The EDPS is in a position to build on its experience and to harness collaborative advantages, and explore synergies, thus facilitating the setting up of the Secretariat of the AI Office. In this regard, the EDPS notes that its role as member of the AI Office and the coordination with this body should be kept separate from its role as provider of the Secretariat, as it happens already when it comes to its two roles played in the context of the EDPB. As a result, there would be a clear differentiation between, on the one hand, the administrative role of the EDPS as provider of the Secretariat and, on the other hand, its role as a member of the AI Office. Moreover, it can also be noted that the EDPS has experience in building internal structures such as the Data Protection support pool of experts, a structure that is also envisaged under the AI Office. As a consequence, the EDPS proposes that the AI Act mandate the EDPS to provide the Secretariat of the AI Office.

6. Conclusions

79. In light of the above, the EDPS recommends: (1) including in the AI Act explicit prohibitions of: a. any use of AI to carry out any type of ‘social scoring’ (not only when performed ‘over a certain period of time’ or ‘by public authorities or on their behalf’); b. any use of AI for automated recognition of human features in publicly accessible spaces - such as of faces but also of gait, fingerprints, DNA, voice, keystrokes and other biometric or behavioural signals; c. the use of AI to infer emotions of a natural person except for certain wellspecified use-cases, namely for health or research purposes (e.g., patients where emotion recognition is important) with appropriate safeguards in place; d. any use of AI systems categorizing individuals from biometrics into clusters according to ethnicity, gender, as well as political or sexual orientation, or other grounds for discrimination prohibited under Article 21 of the Charter; e. AI systems intended to be used by law enforcement authorities as polygraphs and similar tools whose scientific validity is not proven or which are in direct conflict with essential values of the EU; f. any use of systems intended to be used by law enforcement authorities for making individual risk assessments of natural persons in order to assess the risk of a natural person offending or reoffending, or for predicting the occurrence or reoccurrence of an actual or potential criminal offence based on profiling of a natural person or on assessing personality traits and characteristics or past criminal behaviour; (2) removing the exemption from the scope of the AI Act of the existing high-risk AI systems provided for in Article 83(2) of the Proposal, thus ensuring the application to such system of the AI Act from its date of applicability. The same recommendation is also made as regards AI systems which are components of the EU large-scale IT systems established by the legal acts listed in Annex IX of the Proposal; (3) removing the exclusion from the scope of the AI Act AI systems to be used in the context of international law enforcement cooperation; (4) specifying in the AI Act that AI operators retraining pre-trained AI systems should be qualified as ‘providers’; (5) having regard to high-risk AI systems, not including in the final text of the AI Act the provisions under Article 6(3) of the general approach and under Article 6(2a) of the EP negotiating mandate allowing AI systems in Annex III not to be considered high-risk; (6) specifying that the certification of these AI systems in the context of the CE marking should expressly include a verification of compliance, among others, with the data protection by design and by default; (7) adapting the conformity assessment procedure under Article 43 of the Proposal to the effect that an ex ante third party assessment must generally be carried out for these high-risk AI systems; (8) having regard to the role of the EDPS as notified body, clarifying the rules for the identification of the competent notified body, by explicitly providing that the EDPS would be the notified body [only] when the provider - within the meaning of Article 3(2) of the Proposal - is an EU institution, body, office or agency; 24 (9) clarifying in the AI Act that ‘national territory’ for the purpose of market surveillance for the EDPS means: ‘where the AI system is provided or deployed by Union institutions, bodies, offices and agencies’; (10) clearly specifying in the AI Act that the EDPS, as market surveillance authority, would have the powers conferred on these authorities; (11) including the following provisions in the enacting terms of the AI Act: a. In Title I, General provisions: ‘This Regulation is without prejudice to the competences, tasks, powers and independence of relevant national public authorities or bodies which supervise the application of Union law protecting fundamental rights, including equality bodies and data protection authorities’; b. In Title I, General provisions: ‘The EDPS exercises its powers under the AI Act, including the powers as market surveillance authority under the EU Market Surveillance Regulation, without prejudice to its tasks and powers under the EUDPR or other applicable Union law, such as Regulation (EU) 2016/794’; c. In Title VIII, Post-market monitoring, information sharing, market surveillance: ‘The EDPS shall act as market surveillance authority having regard to the development, deployment and use of AI systems developed by or used by Union institutions, bodies, offices or agencies’; d. In Title VIII, Post-market monitoring, information sharing, market surveillance: ‘National territory’ for the purpose of market surveillance for the EDPS means: ‘where the AI system is provided or deployed by Union institutions, bodies, offices and agencies’. (12) providing for clear rules about the investigative powers of market surveillance authorities. These rules should include the authority's power to access training, validation, and testing data; to review the source code of AI systems; to carry out unannounced on-site and remote inspections; to access any necessary data, documentation and software; (13) introducing in the AI Act the right of the person affected by the use of the AI system to lodge a complaint before the competent supervisory authority and to an effective judicial remedy against a decision of the authority before which a complaint has been brought; (14) adding and specifying the competence of the EDPS as supervisory authority to which a complaint may be brought, as well as the right of the person concerned to an effective judicial remedy against an EDPS decision; (15) in order to facilitate the right to lodge a complaint, providing in the AI Act the right for persons affected by the use of the AI system to be always informed about when they are subject to the AI system; about whether and how the use of the AI system has a possible impact on their fundamental rights; as well as about the venues for administrative (and judicial) redress; (16) providing in the AI Act, notably in case of use of high-risk AI systems, the right to obtain human intervention from the user of the AI system having regard to the decision-making affecting him or her and to contest the output of the decision-making, as well as a right to explanations from the deployer of the AI system of the decision-making significantly affecting him or her; 25 (17) introducing specifications on the objectives and legal effects of the regulatory sandboxes, as well as on the role of data protection authorities and of the EDPS in the context of the regulatory sandboxes; (18) providing for a substantial allocation of both human and financial resources to the EDPS to effectively accomplish the new tasks stemming from the AI Act; (19) adding the following wording in Article 59(4) of the AI Act: ‘The obligation to ensure adequate resources shall also apply to the EU budgetary authorities with regard to the EDPS as notified body, market surveillance authority, competent supervisory authority and for any other tasks and duties attributed under this Act’; (20) designating data protection authorities as national supervisory authorities pursuant to Article 59 of the Proposal, specifying that data protection authorities as national supervisory authorities will closely cooperate with the oversight authorities who have expertise and responsibilities in the specific field of deployment of the AI system to ensure its trustworthiness; (21) establishing the European AI Office as an independent body of the Union and strengthening its enforcement role in a way that it would mitigate the risk of possible 'forum shopping' by providers of AI systems; (22) providing for joint investigations, to be coordinated at the European level by the AI Office; (23) including in the AI Act a clear and express specification that the EDPS is considered tequivalent to national supervisory authorities for the purposes of the AI Act, so that it is able to participate in joint investigations on equal footing with other national supervisory authorities, as well as in the other activities of the AI Office; (24) supporting the amendments of the EP negotiating mandate adding a number of tasks to the AI Office, including: to serve as a mediator in discussions about serious disagreements that may arise between competent authorities regarding the application of the AI Act; to facilitate the creation and maintenance of a Union pool of experts, and ensuring their vetting; to assist authorities in the establishment and development of regulatory sandboxes and to facilitate cooperation among regulatory sandboxes; (25) specifically designating in the AI Act the EDPS as a full member of the AI Office with voting rights; (26) inserting a provision in the AI Act that mandates the EDPS to provide the Secretariat of the AI Office. Brussels, 23 October 2023 (e-signed) Wojciech Rafał WIEWIÓROWSKI 26