Artificial intelligence, human rights, democracy and the rule of law

13 October 2022

Opinion 20/2022

on the Recommendation for a Council Decision authorising the opening of negotiations on behalf of the European Union for a Council of Europe convention on artificial intelligence, human rights, democracy and the rule of law

0

The European Data Protection Supervisor (EDPS) is an independent institution of the EU, responsible under Article 52(2) of Regulation 2018/1725 ‘With respect to the processing of personal data… for ensuring that the fundamental rights and freedoms of natural persons, and in particular their right to data protection, are respected by Union institutions and bodies’, and under Article 52(3)‘…for advising Union institutions and bodies and data subjects on all matters concerning the processing of personal data’.

Wojciech Rafał Wiewiórowski was appointed as Supervisor on 5 December 2019 for a term of five years. Under Article 42(1) of Regulation 2018/1725, the Commission shall ‘following the adoption of proposals for a legislative act, of recommendations or of proposals to the Council pursuant to Article 218 TFEU or when preparing delegated acts or implementing acts, consult the EDPS where there is an impact on the protection of individuals’ rights and freedoms with regard to the processing of personal data’.

This Opinion relates to the Recommendation for a Council Decision authorising the opening of negotiations on behalf of the European Union for a Council of Europe convention on artificial intelligence, human rights, democracy and the rule of law. This Opinion does not preclude any future additional comments or recommendations by the EDPS, in particular if further issues are identified or new information becomes available. Furthermore, this Opinion is without prejudice to any future action that may be taken by the EDPS in the exercise of his powers pursuant to Regulation (EU) 2018/1725. This Opinion is limited to the provisions of the draft Proposal that are relevant from a data protection perspective.

1

Executive Summary

On 18 August 2022, the European Commission issued a Recommendation for a Council Decision authorising the opening of negotiations on behalf of the European Union for a Council of Europe convention on artificial intelligence (AI), human rights, democracy and the rule of law (‘the convention’), pursuant to Article 218 TFEU.

Having regard to the ‘trans-border’ nature of artificial intelligence, the EDPS welcomes the general objective, pursued by the Council of Europe, of elaborating the first legally binding international instrument on artificial intelligence, based on the Council of Europe’s standards on human rights, democracy and the rule of law. Accordingly, the EDPS supports the opening of negotiations on behalf of the Union for the convention, and welcomes the Union’s role in promoting trustworthy AI that is consistent with the Union’s values.

The EDPS takes note of the fact that the subject matter of the convention would be regulated in the EU by the proposed AI Act, and acknowledges the Commission’s aim to ensure that the convention is compatible with the proposed AI Act, taking into account future developments in the legislative process. However, the EDPS considers that the convention represents an important opportunity to complement the proposed AI Act by strengthening the protection of fundamental rights of all persons affected by AI systems and therefore advocates that the convention provides clear and strong safeguards for the persons affected by the use of AI systems.

In the light of the above, the EDPS makes four main recommendations on the negotiating directives:

- the general objectives for the negotiation of the convention should give more prominence to the safeguards and rights to be provided to the individuals - and groups of individuals - subject to AI systems, in line with the primary focus and objectives of the Council of Europe;

- an explicit reference to compliance of the convention with the existing EU legal framework on data protection should be included in a specific directive;

2

- in line with the risk-based approach, the objective of imposing a prohibition on AI systems posing unacceptable risks should be introduced;

- the convention should promote the adoption of a data protection by design and by default approach at every step of AI systems’ life-cycle.

Additionally, the Opinion offers further recommendations on inclusion in the convention of minimum procedural safeguards, as well as minimum requirements for transparency, explainability and auditability, compliance and control mechanisms, on cross-border cooperation between competent authorities to be designated by the parties to the convention for the supervision of the safeguards and rights to be provided in accordance with the convention.

3

THE EUROPEAN DATA PROTECTION SUPERVISOR,

Having regard to the Treaty on the Functioning of the European Union,

Having regard to Regulation (EU) No 2018/1725 of the European Parliament and of the Council of 23 October 2018 on the protection of individuals with regard to the processing of personal data by Union institutions, bodies, offices and agencies and on the free movement of such data (‘EUDPR’) , and in particular Article 42(1) thereof,

HAS ADOPTED THE FOLLOWING OPINION:

1. Introduction

1. On 18 August 2022, the European Commission issued a Recommendation for a Council Decision authorising the opening of negotiations on behalf of the European Union for a Council of Europe convention on artificial intelligence (AI), human rights, democracy and the rule of law , pursuant to Article 218 TFEU (‘the Recommendation’).

2. The objective of the Recommendation is to authorise the opening of negotiations on behalf of the Union for a future Council of Europe convention on AI, human rights, democracy and the rule of law (‘the convention’), to adopt negotiating directives and to nominate the Commission as the Union negotiator .

3. In the explanatory memorandum , the Commission stresses that negotiations for the convention relate to matters falling under the exclusive Union competence, also due to the significant overlap between the ‘zero draft’ of the convention circulated by the Committee on Artificial Intelligence (CAI) of the Council of Europe, on the one hand, and of the Commission proposal for a regulation on AI (‘the proposed AI Act’) , on the other hand, in terms of their scope and content .

4. The explanatory memorandum to the Recommendation highlights that the ‘zero draft’ proposes to include the following provisions:  purpose and scope of the (framework) convention; 5  definitions for an AI system, lifecycle, provider, user and ‘AI subject’;  certain fundamental principles, including procedural safeguards and rights for AI subjects that would apply to all AI systems, irrespective of their level of risk;  additional measures for the public sector as well as AI systems posing ‘unacceptable’ and ‘significant’ levels of risk identified on the basis of a risk and impact assessment methodology (to be set out later in an annex to the convention);  follow-up and cooperation mechanism between the parties;  final provisions, including a possibility for EU Member States to apply EU law in their mutual relations for matters covered by the convention and a possibility for the Union to accede to the convention.

5. The Recommendation, under recitals (6) and (7), highlights that the conclusion of the convention may affect existing and foreseeable Union rules. To protect the integrity of Union law and ensure consistency between the rules of international law and Union law, the Commission should be authorised to negotiate the convention on behalf of the Union.

6. The present Opinion of the EDPS is issued in response to a consultation by the Commission of 18 August 2022, pursuant to Article 42(1) of EUDPR. The EDPS welcomes the reference to this consultation in Recital 8 of the Recommendation.

2. General remarks

7. The EDPS welcomes the general objective, pursued by the Council of Europe, of elaborating a “legally binding instrument of a transversal nature on artificial intelligence, based on the Council of Europe’s standards on human rights, democracy and the rule of law” . Indeed, the ‘trans-border’ nature of artificial intelligence development, deployment and use “may affect all humanity” . The convention will be open to participation not only by the 46 Member States of the Council of Europe, but also by non-Member States of the organisation.

8. Accordingly, the EDPS supports the opening of negotiations on behalf of the Union for a future convention on AI, and welcomes the Union’s role in promoting trustworthy AI that is consistent with the Union’s values, through the first legally binding international instrument on AI, based on shared values and principles , notably on human dignity, democracy and the rule of law . 6

9. However, the EDPS notes that the first directive (directive (5)) regarding the general objectives for the negotiations, starts with a reference to the compatibility of the future convention ‘with EU single market law’, and not to the fundamental rights. Similarly, the first directive on the substance of the negotiations (directive (11)) lays down, as aim of the negotiations, that “the provisions of the convention are fully compatible with EU single market”.

10. The EDPS notes that this market-centric approach is aligned with one of the main objectives of the proposed AI Act , the single market dimension of the regulation of AI systems. In this respect, the EDPS recalls the recommendations put forward in the EDPS-EDPB Joint Opinion 5/2021 (‘the joint Opinion’) . At the same time, the remit of the Council of Europe is much broader. Indeed, the Committee on Artificial Intelligence (CAI), set up by the Committee of Ministers of the Council of Europe for the period 2022-2024, has been instructed to “establish an international negotiation process and conduct work to elaborate an appropriate legal framework on the development, design and application of artificial

intelligence, based on Council of Europe’s standards of human rights, democracy and

the rule of law, and conducive to innovation.” [emphasis added] .

11. Against this background, the EDPS considers that the convention represents an important opportunity to complement the proposed AI Act by strengthening the protection of fundamental rights of all persons affected by AI systems. Accordingly, and in line with the Joint Opinion on the AI Act, the EDPS considers that safeguarding the rights of individuals and groups of individuals subject to the use of AI systems should be given greater prominence among the general objectives for the negotiation of the convention.

12. The EDPS underlines that AI systems can be used in a variety of context in which EU and national law provide specific substantive and procedural guarantees aiming at safeguarding fundamental rights and freedoms other than privacy and personal data protection. These include the presumption of innocence and the right to fair trial , or the principle of equal treatment in employment and occupation .

13. The EDPS recalls that those fundamental rights and relevant instruments of EU law must also be taken into account when ensuring consistency of the convention with EU law . 7

3. Relationship with other instruments

3.1 Interplay with EU law, including the Charter

14. Negotiating directive (5) lays down the objective of compatibility of the convention with EU law, stating that the Union should aim to achieve that the convention “is compatible with EU single market law and other areas of EU law, including its general principles of EU law and the fundamental rights and freedoms as enshrined in the EU Charter of Fundamental Rights and implemented through secondary EU legislation”.

15. The EDPS recommends deleting the word “including” to better reflect the interplay between general principles and fundamental rights, on the one hand, and secondary law (EU single market law and other areas of law), on the other hand. The same recommendation applies to directive (11).

16. In this regard, given that data, including personal data, “are in many cases the key premise for autonomous decisions which will inevitably affect individuals’ lives at various levels” the EDPS underlines the importance for the future convention to fully respect the EU acquis in the area of personal data protection.

17. In addition, the EDPS welcomes directive (13), which specifies that the convention should “in no way” undermine the level of protection of fundamental rights and freedoms and the guarantees provided in Union law, including the independence of authorities supervising fundamental rights insofar as required under EU law and recalls the need to interpret such freedoms and guarantees broadly, as set out in Section 2 above. 3.2 Interplay of the convention with the proposed AI Act

18. The EDPS takes note of the overlap of the scope of the convention with the scope of the future AI Act and supports the Commission’s aim to ensure that the convention is consistent with the proposed AI Act, taking into account future developments in the legislative process, as laid down in Directives (6) and (12). At the same time, the EDPS recalls that “a lot of work remains to be done until the Proposal (of the AI Act) can give birth to a well-functioning legal framework, efficiently supplementing the GDPR in protecting basic human rights while fostering innovation” . Against this background, the EDPS would welcome the inclusion in the convention of provisions aiming at reinforcing the rights of persons impacted by the use of AI systems that would complement the future AI Act.

19. In particular, the EDPS suggests that the Commission should aim at including in the convention a methodology for assessing the risks posed by AI systems on fundamental 8 rights, enshrined in the European Convention on Human Rights (‘ECHR’), providing for clear, concrete and objective criteria for such human rights impact assessment (HRIA). 3.3 Interplay of the convention with existing data protection legal framework

20. Apart from a general reference to Convention 108+, the mandate is missing a clear directive referring to the relation of the convention to data protection law. In this regard, in the Joint Opinion the EDPS and EDPB observed that data, including personal data, “are in many cases the key premise for autonomous decisions which will inevitably affect individuals’ lives at various levels” .

21. Hence, the EDPS recommends inserting a directive explicitly referring to consistency of the convention with the existing legal framework on data protection. Indeed, consistency with data protection principles and laws should be a prerequisite on which the convention should build upon.

22. The EDPS considers in particular that directive (16), stating that the convention should

[emphasis added] “avoid overlaps and provide meaningful added value compared to

other relevant international or regional convention, in particular in the area of data protection”, does not provide sufficient clarity in this respect.

4. Scope of the convention

23. The EDPS welcomes the proposed scope of the convention, which should cover both public and private providers and users of AI systems , in accordance with its ‘transversal nature’ . In this regard, the EDPS notes that the proposed AI Act would also have a horizontal nature, since it would be applicable to providers and users of AI systems as public or private entities.

24. The EDPS considers that, unless justified or required under EU primary or secondary law, the same approach should be taken having regard to the use of AI systems, regardless of whether providers and users of AI systems are public or private entities. This would allow a more coherent implementation of the risk-based approach . 9

25. According to the explanatory memorandum , the convention should include “additional measures for the public sector”. The EDPS welcomes this approach, insofar as these measures complement the safeguards for persons impacted by the use of AI, to take into account the role, tasks, responsibilities and rules to which bodies entrusted with missions of public interest are subject. In this regard, the EDPS notes that these additional measures should also apply to private entities when providing public or essential services.

26. Moreover, the EDPS recalls that “the use of AI in the area of police and law enforcement requires area-specific, precise, foreseeable and proportionate rules that need to consider the interests of the persons concerned and the effects on the functioning of a democratic society.” Accordingly, the EDPS recommends taking into consideration not only the interests of the law enforcement and judicial authorities, as stated in directive (20) of the Annex, but also the specific risks associated with the use of AI in the area of criminal justice and law enforcement. To this end, the EDPS recommends adding a directive recalling the necessity to strike the right balance between the public interest and the interests of the persons subject to AI systems, and to ensure full compliance with the fundamental rights to privacy and to the protection of personal data, as well as with the other fundamental rights, notably the right to presumption of innocence and to a fair trial. These rights are often inextricably linked with the fundamental rights to privacy and to the protection of personal data .

27. The EDPS also welcomes the inclusion in the convention, among the definitions to be provided, of the notion of “AI subject” and, in connection with this definition, he inclusion of procedural safeguards and rights for “AI subjects” (namely, persons affected by the use of AI systems, e.g. workers affected by the use of AI work management systems; natural persons applying for a loan affected by the use of AI creditworthiness systems; migrants and asylum seekers affected by the use of AI for border and migration control, etc.).

28. In the Joint Opinion, the EDPS and EDPB deplored the absence of any reference in the proposed AI Act to the individual affected by AI systems, and considered such an absence as a ‘blind spot’ in the proposed AI Act. The EDPS also notes that in the draft report on the AI Act , the co-rapporteurs of the European Parliament (LIBE Committee) have proposed the insertion of new articles on the right to lodge a complaint against the providers or users of AI systems and the right to an effective judicial remedy against a national supervisory authority for the persons affected by the use of AI systems, both as individuals and as group of persons concerned. 10

29. Accordingly, the EDPS recommends specifying in a directive that the convention should provide procedural certain minimum safeguards for the persons affected by the use of the AI systems. Such rights would complement but be without prejudice to the rights established under EU primary law, existing secondary EU legislation or national law .

30. These should include requirements for transparency, explainability and auditability of AI systems.

5. Risk-based approach and AI systems posing unacceptable risks

31. The EDPS welcomes the reference in directive (14) to the risk-based approach, according to which the convention should lay down rules that are proportionate, effective and clear across the AI value-chain. Such an approach is the one underpinning the proposed AI Act, too.

32. In the Joint Opinion, the EDPS and EDPB also appreciated that the proposal “would apply to any AI systems, including those which do not involve the processing of personal data but can still have an impact on interests or fundamental rights and freedoms” .

33. Also having regard to this possible scenario related to the use of AI systems, the EDPS recommends that directive (14) specify that societal/group risks posed by AI systems (the risks for groups of individuals or the society as a whole, e.g. collective effects with a particular relevance, like group discrimination or expression of political opinions in public spaces) must also be assessed and mitigated .

34. Moreover, the EDPS notes that, although the explanatory memorandum refers to ‘AI systems posing ‘unacceptable’ risks’ , this key issue is not reflected in the directives. Therefore, the EDPS strongly recommends including in the negotiating directives that certain AI systems, posing unacceptable risks, should be prohibited, as well as to provide an indication of such AI systems.

35. In addition to the narrow prohibitions already set out in the proposed AI Act, the EDPS recalls that the following AI systems should also be prohibited: - social scoring, by public authorities or on their behalf, as well as by private companies ; 11 - biometric identification of individuals in publicly accessible spaces ; more specifically, the negotiating directive should include that “the convention prohibits any use of AI for automated recognition of human features in publicly accessible spaces - such as of faces but also of gait, fingerprints, DNA, voice, keystrokes and other biometric or behavioral signals - in any context”; - AI systems categorizing individuals from biometrics (for instance, from face or voice recognition) into clusters according to ethnicity, gender, as well as political or sexual orientation, or other grounds for discrimination prohibited under Article 21 of the Charter (biometric categorization systems) ; - AI systems whose scientific validity is not proven or which are in direct conflict with essential values of the EU (e.g., the polygraph) ; - AI systems intended to be used by law enforcement authorities for making individual risk assessments of natural persons in order to assess the risk of a natural person for offending of reoffending criminal offences , or for predicting the occurrence or reoccurrence of an actual or potential criminal offence based on profiling of a natural person or on assessing personality traits and characteristics or past criminal behavior ; - AI systems inferring ‘emotions’ of natural persons (so-called emotion categorization systems), except for well-specified use-cases, namely for health or research purposes (e.g., a patient where emotion recognition is important) with appropriate safeguards in place and subject to all data protection conditions and limits including purpose limitation .

36. According to the EDPS and the EDPB, these types of practice would not meet the necessity and proportionality requirements, or may even affect the essence of the right to human dignity. Thus, they could be considered unacceptable interferences with 12 fundamental rights by the CJEU and the European Court of Human Rights. The Union should therefore aim to achieve that these AI systems are prohibited .

6. Design and development of AI systems

37. The EDPS, in line with the recommendation made in the Joint Opinion , recommends including a negotiating directive according to which the convention should promote the adoption of a data protection by design and by default approach at every step of AI systems’ lifecycle, allowing the effective implementation of data protection principles by means of state-of-the-art technologies.

38. The EDPS notes that directive (17) concerns different aspects related to the implementation of the rules applicable to the design, development and application of AI systems. The EDPS is aware that the negotiating directives by their nature cannot be too prescriptive. However, the EDPS considers that directive (17) should be more specific.

39. Directive (17) refers to appropriate ex ante and ex post compliance and control mechanisms. In this regard, the Joint Opinion welcomed that AI systems posing a highrisk must be subject to a prior conformity assessment before they can be placed on the market or otherwise put into operation in the EU. However, the EDPS and EDPB also advocated adapting the conformity assessment to the effect that an ex ante third-party conformity assessment must be carried out for high-risk AI . Similarly, the EDPS recommends a clear inclusion in the directives of this requirement (namely, third-party assessment, as opposed to self-assessment by the provider of the AI system), taking into account the high risks for the persons affected by the use of AI systems (the ‘AI subjects’). 13

40. It should also be clear in the mandate that the high-risk AI systems should be subject to a new conformity assessment procedure whenever a significant change occurs .

41. Directive (17) also refers to certification mechanisms. Their role should be recognised by the convention. However, the object and legal effects of these certifications should be better specified. In particular, in order to ensure that the certification is implemented in a way that is compatible with the fundamental rights and freedoms as enshrined in the ECHR and the Charter as implemented through secondary EU legislation , such certifications should be consistent with the requirements under the applicable EU and Member States laws.

42. Concerning the role of standards referred to in directive (17), the EDPS recommends specifying that technical standards, on the one hand, can have a positive impact on harmonization of products and services; on the other hand, their role is to provide technical specifications of rules (clear and legally binding obligations for the design, development and application of AI systems) already established by law .

43. Technical standards should indeed be used for the specification of requirements (for instance safety and quality requirements, with regard to reliability, robustness, performance and functional safety) established by the corresponding Union legislation. The EDPS considers that the mandate for negotiation should acknowledge the role as well as the conditions and limits of technical standardisation of AI systems, referring to the possible adoption of technical standards to allow harmonised implementation of the requirements already set out at legislative level.

44. This remark is particularly important in light of complex systems, such as AI systems, whose design, development and application concern different services, related to areas which are already subject to specific EU secondary legislation . 14

7. Supervision of AI systems

45. The EDPS welcomes that directive (17) and, more specifically, directive (21), refer to effective supervision by competent authorities. Directive (21) specifies that the convention should provide for cooperation mechanisms that allow effective enforcement.

46. Indeed, due to the heterogeneity of areas to which the AI systems refer (ranging from work and employment to financial services, education and health-care, administration of justice, fraud prevention, etc.), there is a need for structured and institutionalised cooperation between different competent authorities (in particular, between the data protection authorities and the competent sectoral authorities).

47. Moreover, the EDPS recommends including a directive according to which the convention should provide that competent supervisory authorities must be granted adequate investigatory and enforcement powers. These authorities should be empowered in particular to have access to any relevant documents, information and data necessary to open and conduct investigations and to monitor compliance, as well as to require access to, and explanation relating to, databases, algorithms and source codes.

48. As discussed in Section 2 above, the development, deployment and use of AI systems often has a cross-border nature. Thus, the EDPS recommends adding a negotiating directive aiming at ensuring that the convention facilitates and encourages cross-border cooperation between competent authorities.

8. Conclusions

49. In light of the above, the EDPS makes the following recommendations: (1) to give more prominence to the objective of “ensuring a high level of protection of human rights and preservation of European values”, in line with the nature and mandate of the Council of Europe. (2) to delete the word “including” after “the EU single market law and other areas of law” in directives (5) and (11), to better reflect interplay between general principles and fundamental rights, on the one hand, and secondary law (EU single market law and other areas of law), on the other hand. (3) to add a specific directive recalling the necessity to strike the right balance between the public interest and the interests of the persons subject to AI systems, to ensure full compliance with the rights to privacy and to the protection of personal data, as well as with other fundamental rights at stake, notably the right to presumption of innocence and to a fair trial, the right to good administration and the principle of non-discrimination. 15 (4) to specify in a directive that the convention should provide certain minimum procedural safeguards and rights for the persons affected by the use of the AI systems. (5) to specify in a directive that the convention should provide for minimum requirements on transparency, explainability and auditability of AI systems. (6) to include in directive (14) the specification that societal/group risks posed by AI systems must also be assessed and mitigated. (7) to specify in the negotiating directives that certain AI systems, posing unacceptable risks, should be prohibited, as well as to provide an indicative list of such AI systems. (8) to include a negotiating directive according to which the convention should promote the adoption of a data protection by design and by default approach at every step of AI systems’ lifecycle. (9) to specify the content of directive (17) as follows: - an ex ante third-party conformity assessment must be carried out for high-risk AI; - the high-risk AI systems should be subject to a new conformity assessment procedure whenever a significant change occurs; - specify the object and the legal effect of certifications; - specify that technical standards, on the one hand, can have a positive impact on harmonization of products and services; on the other hand, their role is to provide technical specifications of rules already established by law. (10) to include a directive according to which the convention should provide that competent supervisory authorities must be granted adequate investigatory and enforcement powers. (11) to add a negotiating directive aiming at ensuring that the convention facilitates and encourages cross-border cooperation between competent authorities. Brussels, 13 October 2022 (e-signed) Wojciech Rafał WIEWIÓROWSKI 16