Cybersecurity Challenges in the Uptake of Artificial Intelligence in Autonomous Driving

EUR 30568 EN

ABOUT ENISA

The European Union Agency for Cybersecurity, ENISA, is the Union’s agency dedicated to achieving a high common level of cybersecurity across Europe. Established in 2004 and strengthened by the EU Cybersecurity Act, the European Union Agency for Cybersecurity contributes to EU cyber policy, enhances the trustworthiness of ICT products, services and processes with cybersecurity certifi cation schemes, cooperates with Member States and EU bodies, and helps Europe prepare for the cyber challenges of tomorrow. Through knowledge sharing, capacity building and awareness raising, the Agency works together with its key stakeholders to strengthen trust in the connected economy, to boost resilience of the Union’s infrastructure, and, ultimately, to keep Europe’s society and citizens digitally secure. For more information, visit www.enisa.europa.eu .

ABOUT JRC

The Joint Research Centre is the European Commission’s science and knowledge service. The JRC is a Directorate- General of the European Commission under the responsibility of Mariya Gabriel, Commissioner for Innovation, Research, Culture, Education and Youth. Our researchers provide EU and national authorities with solid facts and independent support to help tackle the big challenges facing our societies today. Our headquarters are in Brussels and we have research sites in fi ve Member States: Geel (Belgium), Ispra (Italy), Karlsruhe (Germany), Petten (the Netherlands) and Seville (Spain). Our work is largely funded by the EU’s budget for Research and Innovation. We create, manage and make sense of knowledge, delivering the best scientifi c evidence and innovative tools for the policies that matter to citizens, businesses and governments.

For more information, visit https://ec.europa.eu/jrc.

CONTACT

For contacting the authors, please use resilience@enisa.europa.eu or https://ec.europa.eu/jrc/en/contact/form For media enquiries about this paper, please use press@enisa.europa.eu or jrc-press@ec.europa.eu

AUTHORS

ENISA JRC

Georgia Dede Ronan Hamon Rossen Naydenov Henrik Junklewitz Apostolos Malatras Ignacio Sanchez

ACKNOWLEDGEMENTS

We would like to acknowledge the following experts who have reviewed the report (in alphabetical order):

Christian Berghoff (BSI) Ernesto Damiani (Universita degli Studi di Milano) David Fernandez Llorca (JRC) Tijink Jasja (Kapsch) Victor Marginean (Continental Automotive GmbH) Gerhard Menzel (JRC) Isabel Praça (Instituto Superior de Engenharia do Porto) Alexandru Vasinca (Cognizant Softvision)

LEGAL NOTICE

The scientific output expressed does not imply a policy position of the European Commission. Neither the European Commission nor any person acting on behalf of the Commission is responsible for the use that might be made of this publication. For information on the methodology and quality underlying the data used in this publication for which the source is neither Eurostat nor other Commission services, users should contact the referenced source. The designations employed and the presentation of material on the maps do not imply the expression of any opinion whatsoever on the part of the European Union concerning the legal status of any country, territory, city or area or of its authorities, or concerning the delimitation of its frontiers or boundaries.

COPYRIGHT

JRC122440 EUR 30568 EN PDF ISBN 978-92-76-28646-2 ISSN 1831-9424 doi:10.2760/551271

Luxembourg, Publications Office of the European Union, 2021

© European Union, 2021

The reuse policy of the European Commission is implemented by the Commission Decision 2011/833/EU of 12 December 2011 on the reuse of Commission documents (OJ L 330, 14.12.2011, p. 39). Except otherwise noted, the reuse of this document is authorised under the Creative Commons Attribution 4.0 International (CC BY 4.0) licence (https://creativecommons.org/licenses/by/4.0/). This means that reuse is allowed provided appropriate credit is given and any changes are indicated. For any use or reproduction of photos or other material that is not owned by the EU, permission must be sought directly from the copyright holders.

All content © European Union, 2021, except cover ©metamorworks - https://www.shutterstock.com/

How to cite this report: Dede, G., Hamon, R., Junklewitz, H., Naydenov, R., Malatras, A. and Sanchez, I., Cybersecurity challenges in the uptake of artificial intelligence in autonomous driving, EUR 30568 EN, Publications Office of the European Union, Luxembourg, 2021, ISBN 978-92-76-28646-2, doi:10.2760/551271, JRC122440.

TABLE OF CONTENTS

EXECUTIVE SUMMARY 6 1. INTRODUCTION 9

1.1 Definitions 9 1.2 Scope 11 1.3 Target audience 11 1.4 EU and international policy context 11

2. AI TECHNIQUES IN AUTOMOTIVE FUNCTIONS 14

2.1 AI in autonomous vehicles 14 2.1.1 High-level automotive functions 15 2.2 Hardware and sensors 15 2.2.1 LIDARs and cameras for computer vision 17 2.3 AI techniques 18 2.3.1 Machine learning: paradigms and methodologies 19 2.3.2 Relevant application fields in autonomous driving 20 2.4 AI software in automotive systems 23 2.4.1 Perception 24 2.4.2. Planning 26 2.4.3 Control 27 2.4.4 Infotainment and vehicle interior monitoring 28 2.4.5 Current trends in AI research for autonomous driving 28 2.5 Mapping between automotive functionalities, hardware and software components and AI techniques 29

3. CYBERSECURITY OF AI TECHNIQUES IN AUTONOMOUS DRIVING CONTEXTS 31

3.1 Vulnerabilities of AI for autonomous driving 31 3.1.1 Adversarial machine learning 31 3.1.2 Adversarial examples in computer vision 32 3.1.3 AI-based physical attacks against autonomous vehicles 35 3.2 Attack scenarios related to AI in autonomous driving 35 3.2.1 Attack scenarios 36 3.2.2 Illustration: Fooling a traffic sign recognition system 41

4. AI CYBERSECURITY CHALLENGES AND RECOMMENDATIONS FOR AUTONOMOUS DRIVING 43

4.1 Systematic security validation of AI models and data 43 4.2 Supply chain challenges related to AI cybersecurity 44 4.3 End-to-end holistic approach for integrating AI cybersecurity with traditional cybersecurity principles 44 4.4 Incident handling and vulnerability discovery related to AI and lessons learned 45 4.5 Limited capacity and expertise on AI cybersecurity in the automotive industry 46

References 48

List of figures

Figure 1. Vehicles automation levels as defined in SAE J3016. 9

Figure 2. Typical elements of autonomous driving systems. Inputs from the environment are obtained from the sensors of the vehicle or external mapping information. They are used to perceive and understand the environment, plan the trajectory of the vehicle, and act on the vehicle’s commands. 14

Figure 5. Localization of the sensors on the vehicle and their main uses. 16

Figure 4. Examples of images from an online set of Italian traffic signs [63] captured by a camera in three different environmental conditions (top) daytime (middle) fog (bottom) night-time. 18

Figure 5. Superposition of outputs from cameras (RGB images) and from LIDARs (range maps) (adapted from the Waymo Open Dataset [64]). For the range, the colour is coded from yellow (close) to purple (far). 18

Figure 6. Example of a typical CNN architecture used for classification. Convolutional layers are filters applied on portions of the images. At each layer, the number of intermediate images increases, while their dimensions is reduced. Only a small proportion of links between layers is displayed. The final layer condenses the values to return scores for each class, the highest score being the predicted class. 21

Figure 8. Illustration of an adversarial example using the Basic Iterative Method [184]. The classifier used is Inceptionv3 [71]. The image comes from the validation set of the ImageNet dataset [185]. (Left) Original image, correctly classified as a school bus. (Middle) Perturbation added to the image, with a 10x amplification. (Right) Adversarial example, wrongly classified with high confidence. 32

Figure 9. Visualization of the overflow on a TSR model. (Top) Normal output of the TSR system: the two signs are detected as expected. (Middle) Amplified intensity of the perturbation added to each pixel of the image. (Bottom) Output of the TSR system on the adversarial image: more than 100 signs are detected with high confidence. 42

Figure 10. Visualization of the class spoofing attack on a TSR model. (Top) Normal behaviour of the model: the “keep right” sign is correctly detected. (Bottom) Output of the TSR system on the adversarial image: the sticker-like perturbation on the sign makes the model incorrectly classify the sign as “priority road”. 42

EXECUTIVE SUMMARY

New generations of cars are making use of advanc- This report aims to provide insights on the cyberes in the field of Artificial Intelligence (AI) to provide security challenges specifically connected to the semi-autonomous and autonomous driving capabili- uptake of AI techniques in autonomous vehicles. It ties, achieving a technological breakthrough that will starts by describing the dynamic policy context with strongly impact existing behaviours and practices. which this initiative is aligned, at both the Europe- Beyond the undeniable benefits of autonomous driv- an and international levels. Institutional and private ing for many aspects of our societies, the question of actors have been very active to outline the high-levthe safety and security of this technology, which by el principles and standards that should govern the definition is intended to operate with limited human development of AV, either explicitly, with dedicated supervision, has emerged. The answers provided by automotive guidelines, or through the definition of regulatory bodies on these issues are likely to play sets of practices driving the expansion of AI and cyan important role for the adoption of autonomous bersecurity. In this respect, the European institutions vehicles (AVs) in society. This is all the more impor- have conducted various initiatives for developing tant given that machine learning (ML) techniques, at trustworthy AI, where cybersecurity and intelligent the core of the AI components developed to mimic transportation play a significant role. human cognitive capabilities, have been proven to be highly vulnerable to a wide range of attacks that Subsequently, this report delves into the technical could compromise the proper functioning of autono- aspects of AI in the automotive sector, with the aims mous vehicles, and pose serious threats to the safe- to better comprehend the technological concerns of ty of persons, both inside and outside of a vehicle. In AI, as well as to get a sense of the level of integrathis context, understanding the AI techniques used tion of AI in AV. This includes an extended descripfor autonomous driving and their vulnerabilities in tion of the areas in which AI plays a role, to ensure the cybersecurity threat landscape is essential to al- the proper implementation of cognitive capabilities leviate the risks and ensure that benefits will not be inside automotive systems. Autonomous driving recounterbalanced by stronger safety risks. quires addressing a host of smaller subtasks (recognizing traffic signs or roads, detecting vehicles, esti- Cybersecurity of AVs is classically approached mating their speed, planning the path of the vehicle, through the angle of the security of digital systems. etc.), each of them trivially performed by humans, This is all the more relevant as modern vehicles are but requiring carefully engineered AI systems to aufully controlled by electronic components, vulnera- tomatically address them. AI software components ble to physical and remote attacks exploiting clas- in an AV do not form a monolithic system, but rather sical cybersecurity vulnerabilities. With this report rely on a complex combination of large and varied however, the objective is to raise awareness about collections of data, themselves obtained by several the potential risks connected to the AI components types of sensors, and a rich set of AI methodoloin charge of replicating tasks previously addressed gies, based on scientific works from statistics, mathby human drivers, such as making sense of the en- ematics, computing, and robotics. Starting from the vironment or taking decisions on the behaviours of high-level functions, an extended description of the the vehicle. By their nature, those AI components do landscape combining AI techniques, sensors, data not obey the same rules as traditional software: ML types, and cognitive tasks highlights the sheer abuntechniques are indeed relying on implicit rules that dance of approaches and ideas that have made AV are grounded on the statistical analysis of large col- a reality. We claim that the understanding of these lections of data. While this enables automation to technical elements in the automotive context is esreach unprecedented cognitive capabilities, it opens sential to put into perspective their cybersecurity imat the same time new opportunities for malicious ac- plications of these AI-based components. A mapping tors, who can exploit the high complexity of AI sys- of automotive functions to AI techniques is provided tems to their own advantage. Securing such systems to highlight the connections between automotive requires to consider these AI specific issues on top of and scientific concepts, making direct links between the traditional cybersecurity risks connected to digi- automotive functionalities, intermediate subtasks, tal systems, in the context of the full supply chain in- and ML techniques. volved in their development and of their integration with other automotive systems.

After this technical presentation, a state-of-the-art This requires that the security and robustness asliterature survey on security of AI in the automotive sessments of AI components do not just take place context discusses the main concepts behind the cy- at a given point in time during their development, bersecurity of AI for autonomous cars. Security of but instead are systematically performed through- AI in general lies outside the scope of this report, out their lifecycle. and the interested reader is referred to the recently published ENISA AI Threat Landscape [1] to get This systematic validation of both AI models and the full picture on this matter. Instead, a focus is data is essential to ensure the right behaviour of the specifically made on adversarial machine learning vehicle when faced with either unexpected situations that regroups a set of techniques that are current- or malicious actions such as attacks based on the ly the main approaches susceptible to compromise alteration of inputs, including poisoning and evasion AI components of AVs. They allow a malicious ac- attacks. This implies developing and maintaining tor to design specific attacks that could deceive AI strict continuous processes to make sure that data systems while staying undetectable by human su- that are used at the development and production pervisors. Concretely, carefully crafted patterns can stages have not been altered with a malicious intent, be disseminated in the environment to alter the and that models do not contain vulnerabilities that decision-making process and induce unexpected could be exploited. It also means that links between behaviour of the vehicle. Typical examples include industrial actors and research centres have to be readding paint on the road to misguide the navigation, inforced to address the challenges associated with or stickers on a stop sign to prevent its recognition. the implementation of this systematic validation. Despite the complexity to undertake these kinds of attacks, and in particular to make them undetectable by human eyes, the dire consequences in terms of AI supply chain security in the automotive safety should encourage car manufacturers to im- industry plement defence mechanisms to mitigate these type of AI risks. The description of these attacks, which The security of the software and hardware supply may not necessarily require access to the internal chain is of paramount importance in cybersecurity. system of the vehicle, is accompanied by real-world The increased uptake of AI technologies has further cases involving autonomous or semi-autonomous amplified this issue with the addition of complex and cars fooled by attackers. This is subsequently illus- opaque ML algorithms, dedicated AI modules and trated, both theoretically and experimentally, by re- third party pre-trained models that now become part alistic attack scenarios against the AI components of of the supply chain. The particularities of the supply vehicles, extending the discussion to other types of chains in the automotive sector, with large and comvulnerabilities of AI. plex dependencies on both hardware and software, add to this complexity. In conclusion of this report, a set of challenges and recommendations is provided to improve AI security Proper AI security policies should be established in AVs and mitigate potential threats and risks. This is across the supply chain, including third-party promotivated by the importance of relying on the pillars viders, ensuring a proper governance and developthat have been at the core of cybersecurity meth- ing an AI security culture across the supply chain. odologies developed along the years for traditional Continuous risk assessment processes supported by software, while at the same time taking into account threat intelligence could enable the relevant actors the particularities of AI systems. In light of the con- to promptly identify and monitor potential AI risks nections between AI and AVs brought forward in this and emerging threats related to the update of AI in report and their consequences in terms of security, autonomous driving. Compliance with specific regthe following recommendations are put forward. ulations in the automotive sector (such as UNECE R155 [2]) could be considered to ensure the security of the supply chain.

Systematic security validation of AI models and data Cybersecurity processes and controls of AI

Data and AI models play an important role in the techniques in autonomous driving implementation of autonomous capabilities in AVs. These components are dynamic in nature and can The uptake of AI in autonomous driving brings about change their behaviour overtime as they learn from important cybersecurity concerns. The increased dignew data, are updated by manufacturers, or encoun- italization of vehicles and the inclusion of AI functer unexpected or intentionally manipulated data. tionalities result in a larger attack surface and might

significantly increase the incentives for attackers to omous driving vehicles will further amplify this trend target AVs. Cyberattacks against AVs do not only placing cybersecurity as one of the critical requireconcern the particularities related to AI, but also in- ments to ensure safety and promote trust. clude the security of the underlying digital infrastructure and related digital systems. It is thus crucial to In this respect, the lack of sufficient security knowlevolve existing security processes and practices to edge and expertise among developers and system consider this increased uptake of AI technologies designers on AI cybersecurity is a major barrier that and digitalization in vehicles, particularly in the con- hampers the integration of security in the automotext of autonomous driving. tive sector. The proper application of the security by design principle requires that all actors involved in The automotive industry should embrace a security the lifecycle of the product are sufficiently proficient by design approach for the development and deploy- on cybersecurity and work systematically together ment of AI functionalities. This could include the us- towards the common goal of building a secure prodage of standardised approaches and homogeneous uct. AI cybersecurity cannot just be an afterthought interoperable AI solutions for automotive systems. where security controls are implemented as add-ons It is important to promote a culture of cybersecu- and defence strategies are of reactive nature. rity (particularly on AI enabled vehicles) across the automotive ecosystem, developing best practices, Particularly in the automotive sector, cybersecurity promoting R&D and innovation and progressively in- is a multidisciplinary endeavour. This is especialtegrating cybersecurity controls and assessments in ly true for AI systems that are usually designed by the current industry processes connected to the life- computer scientists and further implemented and cycle of autonomous driving AI products and services. integrated by engineers. AI systems should be designed, implemented and deployed by teams where the automotive domain expert, the ML expert and

Increase preparedness and incident response the cybersecurity expert collaborate. capabilities

The current cybersecurity landscape connected to the uptake of AI in AVs is limited to theoretical analysis and experimental use case studies carried out in laboratories and controlled environments. However, the expected increase in the deployment of higher levels of automation in road vehicles could quickly change this picture.

It is important that the automotive sector increases its level of preparedness and reinforces its incident response capabilities to handle emerging cybersecurity issues connected to AI. This includes the establishment of cybersecurity incident handling and response plans based on standards, including vulnerability management processes and patch deployment strategies. Cyber exercises in the form of simulations can also be of help to better understand potential impacts of newly discovered vulnerabilities, raise awareness in the organisations, train the several actors, and evaluate existing plans and procedures.

Increase capacity and expertise on AI cybersecurity for automotive systems

The digital transformation experienced by the automotive sector in the last decade with the growth of the adoption of digital components in vehicles has driven the industry to increasingly face cybersecurity challenges. The uptake of AI as an enabler for auton-

1. INTRODUCTION

Advances in Artificial Intelligence (AI) have opened translate into serious safety problems. The automoa whole new realm of opportunities in many areas tive sector constitutes a high-risk domain, which is of our connected digital society. The possibility to au- directly affected by the risks associated with these tomate large parts of our daily activities, considered cybersecurity issues. In fact, automotive security is until now as out of the reach of computing machines, tightly linked to safety: cyberattacks can cause safety offers new perspectives to address the many chal- problems and harms in the physical world, potentially lenges humans are facing. In the transportation sec- at large scale. All of this constitutes a reason for fotor, AI is playing a key role in the development of new cusing efforts on risk alleviation in this sector. generations of cars that will provide autonomous and semi-autonomous driving services to passengers and This report aims to provide insights on these cyberenable high levels of automation, with tangible ben- security challenges, specifically connected to the upefits in terms of road mortality, traffic congestion, or take of AI techniques in autonomous driving vehicles. mobility opportunities.

In this respect, AI is utilised as a means to enhance 1.1 Definitions service provisioning and offer more secure and safe driving conditions. However, at the same time, there The SAE J3016 standard [3] defines six levels of drivare security implications of AI to the entire eco- ing automation for on-road vehicles, ranging from system of digital products and services. AI enables level 0 with no driving automation at all to level 5 new use-cases where cyber impacts cross the bar- with full driving automation and no need for a driver, rier between the digital and physical world and can as shown in Figure 1.

This study focuses on semi-autonomous and au- vehicles to communicate with road infrastructure tonomous cars, which are also referred to as Auto- and vice versa to support variety of traffic manmated Driving System-Dedicated Vehicle (ADS-DV) agement applications and services. in SAE J3016 standard, defined as follows: • Vehicle-to-Person (V2P) technology enables • Semi-autonomous cars (level 4 of automa- vehicle’s connection to smartphones and wearation): refers to highly automated cars that are ble devices, so that pedestrians or any other vulequipped with a multitude of sensors in order to be nerable road user (e.g. cyclists, e-scooter users, able to autonomously (i.e. without any human driv- etc.) can share data with cars. This may be used er intervention) perform all driving functions under to share location information and coordinate the certain conditions (e.g. on a given type of roads). operation of the vehicle with pedestrian’s behaviour (e.g. alerting drivers if, for instance, they need • Autonomous cars (level 5 of automation): more time to cross the road refers to fully automated cars that are equipped with a multitude of sensors in order to be able to • Vehicle-to-Device (V2D) and Vehicle-toautonomously perform all driving functions under Everything (V2X) technologies enable the connecall conditions (i.e. at any time and on any road). tion of vehicles with any surrounded device, object, This type of car may not even include a steering and infrastructure connected to the Internet. wheel or accelerator/brake pedals. The combination of the two trends (toward connect- Various national and international bodies have ed and AVs) will eventually result in the full developadopted the definition of the SAE standard for au- ment of Cooperative, connected and automated motonomous cars, among them the Australian National bility (CCAM) [12], in which Connected Autonomous Transport Commission [4], the Government of Japan Vehicles (CAVs) are expected to improve significantly [5], the Government of Singapore [6], the UK’s De- road safety, traffic efficiency and comfort of driving, partment for Transport (DfT) [7], the US National by helping the driver to take the right decisions and Highway Traffic Safety Administration (NHTSA) [8], adapt in real-time to the traffic situation. the Government of Ontario, Canada [9] and the European Road Transport Research Advisory Council Modern vehicles currently available on the market (ERTRAC) [10]. already include Advanced Driver Assistance Systems (ADAS) as safety features that assist drivers in spe- Besides automating driving, another innovation cific circumstances, such as keeping the car from trend in the car industry consists in providing unprec- drifting out of the lane or helping the driver stop in edented levels of connectivity. Connectivity supports time to avoid a crash or reduce its severity. Advanced the communication of vehicles with all sorts of infra- functionalities required in such autonomous and constructures and devices, and may serve (i) increasing nected systems heavily rely on methods for data acfunctionalities and services for drivers, (ii) integrat- quisition, communication, processing and understanding information needed to enact autonomous driving ing, which empower the vehicle to sense the inner and and (iii) enable new driving patterns such as vehicle outer environment and make decisions. AI and its platooning [11]. Commonly available connection mo- subfield of Machine Learning (ML) are the core enadalities comprise: bling technologies of such functionalities. It is easy to foresee that the uptake of these new technologies will • Vehicle-to-Network (V2N) connects the vehicle introduce new vulnerabilities. Outlining the range of to Internet and/or the cloud, to enable exchange vulnerabilities that vehicles featuring these systems real-time information about traffic, routes, and may exhibit is an important goal of this report. To this road situation. This connection is at the base of end, an overview of the functional architecture uninfotainment systems and an option available in derlying CCAMs and ADAS may prove useful to frame most of the current vehicles. how and where AI and ML get involved. • Vehicle-to-Vehicle (V2V) connects vehicles to exchange information comprising their respec- AVs have been under development for a relatively tive location, direction, speed, braking status, and long time, and numerous approaches and proofs-ofsteering wheel position. Since V2V technology en- concept have been developed to provide solutions to ables sensor outreach of neighbour cars, it may be the tasks discussed above, however, without reaching an enabler of autonomous driving integrating the a sufficient level of maturity to be implemented as an on-board sensing of the environment. end-user product. The rise of AI and ML techniques based on Deep Learning (DL) has been a game-chang- • Vehicle-to-Infrastructure (V2I) and Infraer, with major breakthroughs in computer vision and

structure-to-Vehicle (I2V) technologies allow

behaviour modelling that galvanized the industry.

Many companies have developed prototypes that are 1.3 Target audience

based on these techniques, and cars with high level of autonomy are already commercialized. This study focuses on the cybersecurity of AI components and systems for AVs. A set of challenges Today, AI and ML are the keystone of highly-ac- and recommendations is also provided to improve AI cessorised smart cars and will be the key for the security in this field and mitigate potential threats next-generation cars, whose driving experience will and risks. Hence, the target audience of this rebe more and more ameliorated with user services port comprises the following profiles: and automated assistance towards more secure and • Policy makers will be informed about the AI parsafe driving. Not by chance, many car manufactures ticularities in autonomous driving in order to esare currently running media campaigns advertising tablish proper AI security policies across the autothe use of AI in their products [13]. There is a growmotive supply chain. ing tendency in the automotive sector towards prioritizing security in both hardware and software, of- • Regulatory bodies will better understand the AI ten in collaboration with academic research centres. security needs in the automotive industry in order The ultimate goal of this escalating use of AI is fully to invest on efforts for the development of cyberautonomous driving on automation level 5 and the security regulations incorporating the AI special delivery of AVs for both public and private usage. characteristics.

• Standardisation bodies will be informed about the AI particularities in autonomous driving in

1.2 Scope

order to strive for standardised AI components/ solutions and standards that incorporate AI in AVs The scope of this report focuses on the analysis of aiming to ensure properly secure vehicles. the cybersecurity challenges specifically connected to the uptake of AI techniques in autonomous driv- • National authorities will be further informed ing, considering the AI specific cybersecurity issues about the AI cybersecurity in autonomous driving that surface on top of the more general cyber risks in order to agree on cybersecurity policies that capconnected to digital systems. ture the particularities of AI in automotive systems.

• Original Equipment Manufacturers (OEMs) Cybersecurity of digital systems in general, including will better understand the AI particularities as those supporting AI, lies outside the scope of this a first step to incorporate secure AI components work. The interested reader is referred to related while they design new cars and handle the assemwork on the topic of securing AI, and in particular bly of the various car components. the recently published ENISA AI Threat Landscape [1]. The threat landscape serves as a baseline for • Tier 1 and Tier 2 car components suppliers the identification of relevant assets and threats in will better understand the special characteristics the AI ecosystem and was developed collaboratively of AI and the need for secure AI solutions in auwith the ENISA ad hoc Working Group on Artificial tomotive industry, in order to provide car compo- Intelligence Security [14], in which JRC is actively nents that incorporate secure AI solutions. participating. • AI developers will be informed about the special • The main contributions of this report are summa- needs of security in automotive industry in order rized below: to develop AI components and systems having security in mind. • State-of-the-art literature survey on AI in the context of AVs.

• Mapping of AVs’ functions to their respective AI

1.4 EU and international policy

techniques. • Analysis of cybersecurity vulnerabilities of AI in context

the context of autonomous driving. Various attacks in the automotive context [15]–[20], either against AI or not, which were publicly reported • Presentation and illustration (theoretical and exover the last three years, led to a relatively quick perimental) of possible attack scenarios against awareness of policy makers, regulatory bodies and the AI components of vehicles. the automotive industry for the security needs and • Presentation of challenges and corresponding rec- the development of several cybersecurity regulaommendations to enhance security of AI in auton- tions and initiatives [21], aiming to ensure properly omous driving. secure vehicles, as presented below.

At the European level: and pre-deployment activities for CCAM. In 2020,

to successfully implement the pilot on common • Early 2014, the European Commission's Directo- EU-wide cybersecurity infrastructures and processrate-General for Mobility and Transport (DG MOVE) es, a sub-group on C-ITS under the Commission set up a Cooperative Intelligent Transport Systems Expert Group on Intelligent Transport Systems [12] (C-ITS) deployment platform. This latter was conwas set up. The sub-group's task shall be to assist ceived as a cooperative framework including nathe Commission in working on the implementation tional authorities, stakeholders and the European of the aforementioned pilot and to foster exchange Commission, with the objective to identify and of experience and good practice in the field. agree on how to ensure interoperability of C-ITS across borders and along the whole value chain, • In September 2020, the European Commission

as well as to identify the most likely and suitable published a report by an independent group of

deployment scenario(s). experts on Ethics of Connected and Automated

Vehicles [28]. The report includes 20 recommen- • In 2016, the European Commission adopted a Eurodations covering dilemma situations, the creation pean Strategy on Cooperative Intelligent Transport of a culture of responsibility, and the promotion Systems, a milestone initiative towards cooperative, of data, algorithm and AI literacy through public connected and automated mobility [22]. The objecparticipation. tive of the C-ITS Strategy is to facilitate the convergence of investments and regulatory frameworks • The protection of road users’ privacy and person-

across the EU, in order to see deployment of mature al information is also addressed by the recent EU

C-ITS services in 2019 and beyond [23]. General Data Protection Regulation (GDPR) [29],

which officially went into effect in May 2018. • In 2016, the Member States and the European Commission launched the C-Roads Platform to • The Network and Information Security directive

link C-ITS deployment activities, jointly develop (NIS) [30] also addresses AVs’ cybersecurity issues

and share technical specifications and to verify as it intends to provide generic security measures

interoperability through cross-site testing. Initially in order to enhance cybersecurity across EU.

created for C-ITS deployment initiatives co-funded

International Context

by the EU, C-Roads is open to all deployment activ-

ities for interoperability testing. • Several international cybersecurity standards and

recommendation documents are also under devel- • In 2017, the European Commission’s Directoopment. In particular, the United Nations Economic rate-General for Internal Market, Industry, Entrepre- Commission for Europe (UNECE) has issued a regneurship and Small and Medium-sized Enterprises ulation on cybersecurity [2] which defines a set (SMEs) (DG GROW) launched an initiative on safety of requirements that shall be fulfilled by vehicle regulations with the aim to contribute to a further manufacturers, suppliers and service providers, decrease of the number of road fatalities and injucovering the entire vehicle lifecycle (i.e. from the ries considering amendments to the General Safety vehicle development to its decommissioning). Regulation and the Pedestrian Safety Regulation.

• Transport Canada released in 2020 “Canada’s Ve- • In 2018, the European Commission published the hicle Cyber Security Guidance” [31], which provides EU Strategy for mobility of the future [24]. This guiding principles to help ensure vehicles are cystrategy sets out a specific action to implement ber-safe. This Cyber Guidance aims to support ina pilot on common EU-wide cybersecurity infradustry stakeholders by providing technology neutral structures and processes that are needed for seand non-prescriptive guiding principles to strengthcure and trustful communication between vehicles en cyber security throughout the vehicle lifecycle. and infrastructure for road safety and traffic man-

agement. Since 2018 the European Commission • The European OEMs published a set of cyberse-

is implementing the EU C-ITS Security Credential curity principles, through the ACEA Principles of

Management System (EU CCMS) based on the Eu- Automobile Cybersecurity [32], which are already

ropean C-ITS Security Policy (SP) and C-ITS Certif- implemented by OEM companies.

icate Policy (CP) published on the website of the • The National Highway Traffic Safety Administra- C-ITS Point of Contact (CPOC) [25]. tion (NHTSA) from the U.S. government issued in

• In 2019, the European Commission has set up late 2016 a document introducing several cyber-

a Commission Expert group on cooperative, con- security best practices for smart cars [33].

nected, automated and autonomous mobility, • The Singapore Standards Council released in 2019 named “CCAM” [26], [27], to provide advice and a set of guidelines for the deployment of AVs support to the Commission in the field of testing

called Technical Reference 68 [6], whose Part 3 is defines common requirements for security to be related to define cybersecurity principles and an implemented in hardware for ground vehicles. assessment framework. In 2019, ENISA performed, with the involvement of • China’s National Development and Reform Corporathe JRC, a study on “Good practices for security of tion (NDRC) updated in February 2020 its “Strategy smart cars” focused on semi-autonomous and aufor Development of Intelligent Vehicles” [34], which tonomous cars [46]. Moreover, in 2016, ENISA has establishes five key missions, including the estabperformed a study on smart cars security issues, lishment of a “comprehensive cybersecurity system”. which resulted in a document entitled “Cyber Secu- • The US Automotive Information Sharing and Anal- rity and Resilience of smart cars” [47]. In 2020, JRC ysis Center (Auto-ISAC) has been maintaining since published a report on the future of road transport 2016 a series of Automotive Cybersecurity Best [234]. In the same year, ENISA established the Con- Practices [35], which provide guidance on the imple- nected and Automated Mobility Security (CAMSec) mentation of automotive cybersecurity principles. experts Group, to address the cybersecurity threats, challenges and solutions of Intelligent Transport Standards Systems (ITS) and CAM Transport. The members of CAMSec are vehicle manufacturers with focus on cy- • The British Standards Institution (BSI) Group pubbersecurity, suppliers and developers of embedded lished in December 2018 two Publicly Available hardware/software for smart cars, associations and Specifications (PAS), namely PAS 1885 [36] and non-profit organisations involved in vehicle security, PAS 11281 [37]. The former, which is entitled “The road authorities and academia, as well as standardfundamental principles of automotive cyber secuisation bodies and policy makers. In previous years, rity”, provides high-level guidance to provide and ENISA has also established the Cars and Roads maintain cybersecurity. As regards to PAS 11281, SECurity (CaRSEC) working group which addresses entitled “Connected automotive ecosystems – Imsmart cars cybersecurity threats, challenges and pact of security on safety – Code of practice”, it solutions to protect road users’ safety. JRC contribprovides recommendations for managing security utes to these security expert groups. risks in a connected automotive ecosystem. • The European Telecommunications Standards In- In parallel, the use of AI techniques for decision-makstitute (ETSI) has been developing a set of techni- ing systems in high-risk domains [48], including aucal specifications [38]–[41] to define an Intelligent tonomous driving, has led to a growing awareness Transport System (ITS) security architecture along of the shortcomings of current AI systems and has with services specification to ensure information raised concerns in society about the compliance of AI confidentiality and prevent unauthorized access to systems with respect to a certain number of require- ITS services. They also address the trust and pri- ments, including explainability, fairness, reliability or vacy management for ITS communications. These transparency. These recent years, many proposals standards are integral foundation of the European have been published by public and private actors to C-ITS Certificate and Security policies [42], which establish principles which AI systems should follow are the governing policy documents enforcing to ensure they will respect fundamental rights, and some of the ETSI standards as baseline for inter- act in a safe and secure manner. The European Comoperable and secure deployment of C-ITS in the EU. mission is particularly active on this topic, with the

establishment of multiple initiatives to ensure trust-

• The standard of Society of Automotive Engineers worthy AI [49] at the service of the citizens. SAE J3061 [43], officially published in January 2016, is considered as the first standard addressing automotive cybersecurity. It provides a set of high-level cybersecurity principles and guidance for cyber-physical vehicle systems. • The International Organization for Standardization (ISO) and SAE collaborated to supersede the SAE J3061 recommended practice and proposed the ISO/SAE 21434 [44]. This standard focuses on automotive cybersecurity engineering by specifying requirements and providing recommendations for cybersecurity risk management for cars (including their components, software and interfaces) all along their entire lifecycle. Finally, SAE J3101 [45]

2. AI TECHNIQUES IN AUTOMOTIVE FUNCTIONS

2.1 AI in autonomous vehicles group of tasks [50]. The pipeline considers input data, generally from sensors, and returns commands to The last decade has seen an increase of efforts to- the actuators of the vehicle. The main components wards the development of AVs. An AV is a driving of a driving-assistant as well as of an AV are broadsystem that observes and understands its environ- ly grouped into hardware and software components. ment, makes decisions to safely, smoothly reach The hardware component includes sensors, V2X faa desired location, and takes actions based on these cilities, and actuators for control. The software part decisions to control the vehicle. A key enabler of this comprises methods to implement the vehicle perrace towards fully AVs are the recent advances in AI, ception, planning, decision and control capability. and in particular in ML. Designing an AV is a chal- Figure 3 displays typical elements of this pipeline. lenging problem that requires tackling a wide range They are implemented by decomposing each probof environmental conditions (lightning, weather, etc.) lem into smaller tasks, and developing independent and multiple complex tasks such as: models, usually using ML, for each of these tasks. • Road following This chapter is structured as follows: First, a brief • Obstacle avoidance introduction to the main high-level automotive functions where AI plays an important role is giv- • Abiding with the legislation en, as well as a presentation of the main hard- • Smooth driving style ware sensors that can be found on vehicles, and that generate the data that are processed by AI • Manoeuvre coordination with other elements of software components. After these two sections, the ecosystem (e.g. vehicles, scooters, bikes, pea description of the main AI techniques commondestrians, etc.) ly used is given, followed by a discussion on how • Control of the commands of the vehicle these techniques are leveraged to implement the high-level functions in AVs. Finally, a summary of Usually, autonomous driving is described as a se- the chapter is presented in the form of three tables, quential perception-planning-control pipeline, each highlighting the links between functions, hardware of the stages being designed to solve one specific and software components, and techniques.

• Blind spot / cross traffic / lane change assis-

2.1.1 High-level automotive functions

tance consists in the detection of vehicles and pe-

destrians located on the side, behind and in front Currently, fully autonomous driving solutions are beof the vehicle, e.g. when the vehicle turns in an ing mostly experimented with prototypes. Nonetheintersection or when it changes lanes. Detection is less, vehicles with levels of automation up to level usually performed using sensors located in differ- 3 are already on the road, with driving assistance ent points of the car [54], [55]. functionalities relying on AI and ML. Technology-enhanced functionalities featured by commercialised • Collision avoidance (or forward collision vehicles that leverage the use of AI and ML are, for warning) systems, consist in detecting potential instance, braking assistance, smart parking, or vocal forward collisions, and monitoring the speed to interactions with the infotainment system. avoid them. These systems typically estimate the location and the speed of forward vehicles, pedes- Features of AVs can be decomposed into several trians, or objects blocking a road, and react proachigh-level automotive functions that are typically tively to situations where a collision might happen. used by car manufacturers to advertise the auton- • Automated lane keeping systems (ALKS) omous capabilities of their products. As of today, consist in keeping the vehicle centred in its traffic the technical specifications of such functions are not lane, through steering. This includes the detection uniformly defined and vary between manufacturers. of lane markings, the estimation of the trajectory In the following, we provide a non-exhaustive list of of the lane in possible challenging conditions, and the most common automotive functions that are the generation of actions to steer the vehicle [56]. deemed as essential to achieve autonomous driving [10]. It is worth noting that most functions have • Traffic sign recognition consists in recognizing been primarily designed to assist drivers rather than the traffic signs put on the road and more generreplace them (in vehicles with a level of autonomy ally all traffic markings giving driving instructions, from 1 to 3), by providing warnings, or taking con- such as traffic lights, road markings or signs. This trol of the vehicles in limited situations. With fully implies to detect from camera sensors various developed AVs, these functions are part of the driv- indicators based on shape, colours, symbols, and ing process and, essentially, contribute to replacing texts [57]. the driver1. At the end of this chapter, the following • Environmental sound detection: consists in functions are considered and are mapped to specific the detection and interpretation of environmental AI tasks: sounds that are relevant in a driving context, such • Adaptive cruise control (ACC) consists in ad- as horn honking or sirens. This requires performing justing the speed of the vehicle in order to main- sound event detection in noisy situations. tain an optimal distance from vehicles ahead. In what follows, we first analyse the standard blocks ACC estimates the distance between vehicles and of hardware and sensor components. We then give accelerate or decelerate to preserve the right disa brief overview over the most important AI techtance [51]. niques and their software realization used for de- • Automatic Parking (or parking assistance) sys- signing AVs. The chapter concludes by mapping tems consist in moving the vehicle from a traffic automotive functions to AI functions in order to falane into a car park. This includes taking into ac- cilitate the identification of relevant vulnerabilities count the markings on the road, the surroundings and cybersecurity threats in autonomous driving. By vehicles, and the space available, and generate narrowing down the AI techniques that are actually a sequence of commands to perform the manoeu- used in AVs, one scopes down the problem of idenvre [52]. tifying pertinent cybersecurity threats related to the use of AI in autonomous driving. • Automotive navigation consists in finding directions to reach the desired destination, using position data provided by GNSS devices and the

2.2 Hardware and sensors

position of the vehicle in the perceived environment [53]. Humans drive cars by taking actions with hands and feet, based on decisions made considering the input received from our senses, mainly sight and hearing. 1 To that end, we do not consider these functions as warning systems, nor did we include functions that are necessarily acting on the behaviour Similarly, AVs rely on a variety of sensors to observe of the driver (such as driver drowsiness detection). Likewise, functions the surroundings and provide data to the AI systems that increase safety but are not directly linked to cognitive capabilities, such as anti-lock braking system (ABS) or tire pressure monitoring, are not of the vehicle, and on actuators to control the momentioned. tion of the vehicle. The hardware components allow

the vehicle to sense the outside surroundings as well clude the set of analogue measurements that are

as the inside environment via specific sensors, to act encoded in digital form indicating values such as

via the actuators that regulate the car movement, the engine’s revolution per minute (RPM), speed of

and to communicate with other agents/devices via the car (as measured by wheel’s rotation), direc-

the V2X technology. tion of steering wheel, etc.

• Other sensors are those sending the information Sensors, as the primary source of information for AI that the vehicle might receive from its digital comsystems, are a critical element of AVs. All sensors munication with other vehicles, V2V communicacan be broadly classified in three distinct groups [58]: tions or V2I. They mainly concern the connected

• Exteroceptive sensors are those sensors that are infrastructure of vehicles, and therefore they will

designed to perceive the environment that sur- not be discussed in the rest of the report.

rounds the vehicle. They are relatively new sen- The integration of sensors in vehicles varies acsors present in cars, and are the eyes and ears of cording to carmakers [59], [60] and depends on the the car. Cameras and Light Detection and Ranging software strategy chosen to process the streams of system (LIDARs) are the main vectors of informadata. Very often, the inputs from multiple sensors tion for driving purposes. Other sensors, such as are combined in a process called data fusion [61] to Global Navigation Satellite Systems (GNSSs), Ineralign all data streams before processing, as sensors tial Measurement Unit (IMU), radars and ultrasonic are usually providing images from different natures sensors, are also used to probe the environment, (2D images, 3D point clouds, etc.) with different tembut tend to be limited to specific tasks (e.g. close poral and spatial resolution. detections, sound listening) or to add redundancy,

increasing the reliability of results in the case of Table 1 presents the main characteristics of the malfunction of a sensor. most common sensors found on autonomous cars, in

• Proprioceptive sensors, on the other hand, are addition to the LIDARs and cameras. The localization

those that take measurements within the vehicle of these sensors on the vehicle and their main uses

itself. They have been present in cars for decades, are illustrated in Figure 3.

and are mostly used for control purposes. They in-

2.2.1 LIDARs and cameras for computer nal received allows the generation of a depth map of the scene (see Figure 4). The depth map is further

vision

processed to recreate 3D maps of the environment Cameras and LiDARs are the most widespread sen- [54] considering missing values in the acquired 3D sors in autonomous cars, used to reproduce and en- data points, unexpected reflections due to wrong perhance human vision. Digital video cameras are able ception of surfaces, and many other issues that may to obtain a 2D representation of the 3-dimensional appear during the acquisition in real world scenarios. world. They provide a stream (video feed, as a sequence of images) of 2D maps of points (pixels) Compared to LiDARs, cameras have the advantage encoding colour information. Computer stereo vision that they distinguish colours, allowing the recognition techniques can be applied using multiple cameras of elements such as road signs, traffic lights, vehicle and/or considering the different images in relation lights or text warnings. However, cameras also exhibit to the known movement of the vehicle. Examples of certain limitations compared to LiDAR: camera vision images from cameras are depicted in Figure 4. could be impaired by certain weather conditions such as rain, fog or sudden light changes such as when A LiDAR illuminates the environments with lasers and a vehicle gets out of a tunnel, while these conditions collects the reflected light. The analysis of the sig- would affect to a lesser degree a LiDAR system.

2.3 AI techniques These competencies are usually considered as being natural to humans but are difficult to translate ex- AI is generally defined as a collection of methods plicitly into algorithms. Nowadays, from a scientific capable of rational and autonomous reasoning, ac- perspective, AI is actually a heterogeneous field that tion or decision making, adaptation to complex envi- regroups different subfields with diverse views on ronments and/or to previously unseen circumstances how to address these problems. [65]. AI was initially born as an academic discipline in the second half of the twentieth century and led Research on AVs was historically pioneered in the since then to significant advances in the automa- field of robotics, with several cars in the 1980s, and tion of some human level tasks, nonetheless with- later on, able to drive autonomously in controlled out much impact beyond academic circles for a long environments. Nonetheless, the complexity of retime [66]. It is deeply rooted in the fields of computer al-world environments, and the necessary reliability science, discrete mathematics and statistics, with an that are required for such vehicles, has curbed their eventful history before gaining the popularity that development until the significant recent progress makes it nowadays a key domain of the current made in ML. Since the last decade, tremendous miledigital revolution, thanks to the tremendous perfor- stones have been reached in computer vision, natumances achieved by modern systems. ral language processing or game reasoning, pushing autonomous driving a leap forward. Although some Typical problems related to AI require the develop- functions are still solved using traditional methods, ment of programs able to demonstrate some forms ML is increasingly used, relying on the huge quanof reasoning, knowledge representation, planning, tity of data that are collected by companies, with learning, and, more generally, cognitive capabilities. millions of kilometres travelled by autonomous cars

under human supervision in real-world conditions or 2.3.1.1 Paradigms of machine learning using simulated environments. Three different paradigms are commonly consid- In the following, a short introduction to the relevant ered in ML: fields of ML for autonomous driving is provided, Be- • Supervised learning makes use of large and repsides giving the technical basis that will support the resentative set of labelled data to train the model. discussion in the rest of the report, the goal of this The underlying problem consists then to return the section is also to highlight the diversity of techniques right label for the input data. Supervised learning that are being employed for the different tasks, and includes classification, when the label is discrete the complexity of the full processing chain. (e.g. the make of a car), and regression, when the label is continuous (e.g. the speed of the car). The availability of labelled data is a limiting factor for

2.3.1 Machine learning: paradigms and

supervised learning, as labelling can be, in some methodologies contexts, expensive and time-consuming.

ML is the scientific field dedicated to the study of • Unsupervised learning (or self-learning) conmodels that are able to improve automatically sists in extracting meaningful patterns from the through experience [67]. This acquisition of experi- data without labels by reducing the natural variaence can take different forms, and is usually achieved bility of the data, while preserving the similarity or by extracting relevant patterns from large collection absence of similarity between examples. Unsuperof data. Machine learning algorithms are therefore vised learning is used for various purposes, such able to achieve high performance for a variety of as clustering the samples (e.g. grouping individuals complex tasks, hard to solve using conventional pro- based on their habits) or anomaly detection (e.g. gramming techniques, without being explicitly in- detecting a vehicle with an unusual behaviour). structed how to perform them. Prominent examples • Reinforcement learning regroups a set of techinclude recognizing faces in a picture, identifying niques to make models learn sequences of actions objects in video streams, predicting the price of an in a possibly uncontrolled and/or unknown enviasset quoted in a financial market, grouping users on ronment. Contrary to the supervised learning setan online platform based on their activities, recogting, in which the ground-truth is given as labels, nizing the emotion of a person, or teaching a robot learning is guided by indications on how good an to move in an unknown environment. action is, given the state of the environment. Consequently, the learning process is dynamic with The central element of ML systems is the model that respect to the feedback it gets from the environtakes as inputs a set of pre-processed data, and rement, in a trial-and-error approach. turns a prediction. This model is usually described as a mathematical function, with a collection of parameters that have a direct influence on the mapping 2.3.1.2 Classical machine learning between the inputs and the predictions. To adapt the model to the desired task, a training stage is per- A wide range of techniques have laid out the founformed, and consists in running an algorithm that will dation of the field of ML coming from statistics and update these parameters to fit a training dataset, i.e. expert systems, such as linear regression, support a list of samples serving as examples to guide the vector machine (SVM), k-nearest neighbour (kNN) model towards the expected function. The capability classifiers, or decision trees. The common point to perform well on data outside the training data, of these methods is that they usually operate on called the generalization, is a desirable property of handcrafted features, whose quality can drasticalthe resulting model, which is often measured by ly change the performances of the model. Although metrics such as accuracy or mean squared deviation these techniques show limitations in complex probon previously unseen data. Training a model implies lems such as the ones encountered in computer viapplying a host of ad-hoc procedures to increase sion or natural language processing, they are still the generalization capabilities of systems. Popular very popular to solve a large range of problems, in techniques include data augmentation that consists particular when the volume of data is small, when in artificially increasing the amount of training data the time available for model training is limited, or if by applying random transformations on the training the context domain is well understood. data, and hyperparameter optimization search that tests various settings for the training procedure. The full pipeline includes several additional steps during training and testing that are not detailed here.

2.3.1.3 Deep learning cessing chain, from the acquisition of images to the processing and analysis of the image, to the rep- Although the ideas behind neural networks are as resentation of knowledge as numerical or symbolic old as the field of ML, DL techniques have disrupt- information. To date, computer vision is the most ed the ML landscape these last years, and exported relevant field of ML with existing applications in AVs. the whole field of AI outside academic circles, thanks As such, the most significant and well-known vulnerto simultaneous progress in computing capabilities, abilities and possible attack scenarios on AI moddata acquisition and storage, and ML algorithms. els employed in AVs are involving computer vision The advances in hardware and the digitalisation of techniques. A more detailed focus is then given with the society have permitted to train models on high respect to other application fields of ML. performance computing infrastructure and to collect huge datasets to do so, in addition of progress made Images can take several forms, depending on the to speed up training algorithms. type of hardware sensors that have been used to obtain them. Computer vision has been historically One strength of DL is its ability to learn from raw interested in the handling of standard RGB images, data the most adapted representation for the con- that represent a large proportion of applications in sidered problem, removing the need to handcraft robotics and image processing, but has also gained features. DL techniques employ neural networks in more and more interest in the analysis of other layered architectures, denoted deep neural networks forms of images such as 3D point clouds, hyper- (DNN), allowing for flexible designs able to represent spectral images, acoustic images, to name but a few. relationships between inputs and outputs. Each lay- This trend has been significantly fostered with the er is composed of a number of units called neurons recent availability of large data sets. In addition to that perform simple linear combinations between the mode of acquisition, other variables such as the the outputs of the previous layer. These stacked size, the resolution, the quality obtained, the enviarchitectures exhibits a specialization of groups of ronment of acquisition, etc. have led to specialized neurons in the deepest layers, able to extract more sub-domains adapted to specific tasks. and more complex patterns. The advent of highly performant convolutional neu- Although powerful, DL is not a silver bullet as it suf- ral networks (CNN) has been a major breakthrough fers from several limitations that make it impracti- that has drastically changed the technical landscape cal in some situations. First, the training of neural in computer vision. CNNs are an evolution of DNNs network models needs substantial amounts of good specifically designed to take into account the spaquality data and of computational power to be effi- tial structure of images [68] by grouping the weights cient. Secondly, the development of such models, in that are locally close. They compensate for one of particular during the training phase, lies on strong the drawbacks of fully-connected networks by sigengineering practices with limited theoretical guar- nificantly reducing the number of parameters to antees on the overall performances. This severely learn, making learning on high-dimensional data, hinders the understanding of the behaviour of DL which is typically the case of images that are commodels, and is a reason of their vulnerabilities. Third- posed of millions of pixels, a more tractable probly, DNNs are notoriously known to provide accurate lem. Convolutional layers act as a series of filters results but with an inherent lack of interpretability, that are applied on a small portion of the image to making them acting as black boxes. The robustness detect a specific pattern such as edges, a specific of such systems with respect to unusual inputs or shape, a dark area, etc., the particularity being that malicious actions is also under scrutiny by the re- these filters are learned from the data. The size of search community. the filters determines the complexity patterns, and has to be calibrated according the characteristics of the image. CNNs have been successfully used to 2.3.2 Relevant application fields in extract, directly from the raw inputs, efficient representations that are adapted to the problem, and

autonomous driving

that take into account natural invariances that often appear in images, such as symmetries. Figure 4 il- 2.3.2.1 Computer vision lustrates a typical CNN architecture used for classification, and the basic mechanisms at play during Computer vision is an interdisciplinary field, at the the processing. intersection of ML, robotics, and signal processing, concerned with extracting information from digital Today, the overwhelming majority of computer viimages and videos. This covers all stages of the pro- sion techniques are relying in one way or another on

CNNs, and more and more sophisticated approaches design detectors only employ one single network arare considered to address always more complicated chitecture to classify directly pixels or regions. The problems. In the following, the most relevant prob- “You Only Look Once” (YOLO) [76] architecture is lems for autonomous driving are briefly summarized. an example of a successful single-stage detector, largely used in many recognition systems.

Object recognition, in its most common form, in-

cludes two tasks: detecting and classifying objects in Object recognition in 3D representations, such as an image. Localization is usually achieved by assign- point clouds, is becoming more popular these last ing bounding boxes to regions of the image, while years, especially with advances in autonomous drivclassification assigns to these regions a label from ing using LiDAR and radar. Albeit less mature than its a list of pre-defined categories. counterpart in 2D, some first standard architectures have already emerged. Among those are techniques To achieve high performance in the classification to project the 3D point clouds into a 2D space to task, several architectures have been built by the ML use one of the known 2D detectors, for example the community during the last decade, employing vari- YOLO3D architecture [77], or, those techniques that ous innovations aimed at increasing the expressive directly develop algorithms working on the 3D data, power of models, while limiting at the same time the such as VoxelNet [78] or PointNet [79]. cost of training. Among them, we can cite: AlexNet [69] is widely considered as the first breakthrough Segmentation is an extension of object recognition using CNNs; VGG [70] introduces the use of numer- that does not consider bounding boxes to delimit obous layers, with different size of filters; GoogLeN- jects, but pixel-wise regions acting as masks over et [71], [72] makes use of the so-called Inception the image. A label is then assigned to each region module, including at the same level various filters of to classify them into prescribed categories. Severdifferent sizes; ResNet [73] uses shortcut connec- al types of segmentation problems have been distions between layers to limit the tendency of large cussed, among them, instance segmentation detects models to memorize the training data (also called pixels of each object and assigns an identifier for overfitting); SqueezeNet [74] is designed to be em- each object; as for semantic segmentation, non-obbedded in systems with low capabilities by reducing ject characteristics, such as sky, water, horizon or the number of parameters. textures are part of the elements to segment. In the latter case, the full image is completely segmented, On top of classification architectures, object detec- resulting in a label assigned to each individual pixels, tion is implemented in two competing designs: single forming continuous regions. As for object recognition staged and double staged. Double staged approach- models, segmentation makes an intensive use of es split the detection procedure into two stages, CNNs. State-of-the-art approaches include SegNet region proposal and bounding box search. By far [80], EnET [81], PSPNet [82], DeepLab [83], or Mask the most widely accepted state-of-the-art double R-CNN [84]. stage design is the family of Fast Region-based NN (R-CNN) [75] architectures. Conversely, single stage

Vehicle localization, often called Visual Odometry high relevance for autonomous driving. Sequential (VO), is a technique to estimate using a sequence of data encompass data sets that result from dynamiimages captured over time by the camera mounted cal or ordered process introducing a clear sequence on the vehicle the pose of the camera, i.e. its position and correlation between instances of the data set. and its orientation. A common approach consists in Examples include time series modelling, prediction tracking key-point features of clear landmarks and of trajectories, speech and language processing. reconstructing the complete pose from these fea- Sequence modelling plays a significant role in pretures. Classical VO techniques still dominate the diction and planning tasks and is used in robotics field for AV, although there have been increasingly and signal processing in applications concerned with promising results based on DNNs [85]. DL has also interpreting continuous flows of environmental data. been built on top of classical algorithms implement- Classically, the field is dominated by Markov models, ing outlier rejection scheme in order to discriminate autoregressive modelling and dynamical linear filter ephemeral from static parts in images [86]. While systems [92], which are built around the assumption most techniques focus on 2D images, various pro- of a certain correlation length between successive posals have been published to tackle the 3D pose elements of the series paired with a probabilistic estimation using DL [87]. process to model the next element.

Tracking of objects is used to determine the dy- Recently, DL and unsupervised representation learnnamics of moving objects. This can be seen as an ing have introduced major advances into the field, additional layer on top of image recognition sys- mostly in form of specialized network structures, tems, providing for each frame of a video stream such as recurrent neural networks (RNN) [93] and the objects present in a scene, the temporal connec- modern variants such as the Long Short-Term Memtions between these objects, and a prediction of their ory networks (LSTM) [94], able to deal with the sefuture positions. Tracking systems, also referred as quential nature of data. Key advantages from DL MOT (for Multiple Object Tracking) provide this func- based systems are their capability to easily discover tionality by estimating the heading and velocity of long and short term correlations and to automatiobjects, and applying a motion model to predict the cally learn representations of dynamical processes, trajectory. Tracking is a very active field of research even in complex contexts. in the computer vision community, and has been studied for a wide range of applications and contexts [88]. Techniques vary according to parameters such 2.3.2.3 Automated planning as the quality of the detection of objects, the type of data considered, the frame rate, or the nature of the Automated planning [95] is a rich field connected motions involved. to ML at the intersection of other fields such as robotics, complex infrastructure management, deci- Typically, tracking is solved by assigning an identifier sion theory, and probabilistic modelling. It is mainly to objects and trying to keep this identifier consistent concerned with the search of optimal strategies, ofthrough successive image frames. This consistence ten described as a sequence of actions that should is implemented either by using measures of similar- be followed by agents evolving in complex enviity applied on handcrafted features based on image ronments, and how to perform them. There exists characteristics, such as colours or gradients [89], or a wide variety of methods to find the optimal stratby using CNNs [90]. The modelling of the dynamics egy in the specific context of problems that this field is then done using sequential modelling tools to take aims to address. In the following, we provide a brief into account the temporal dependencies between description of methods that have been used in an frames. The trend towards models that jointly ad- automotive context. dress multiple tasks concerns also the tracking and the prediction of objects, that is often coupled with Graph-based planning is used when systems can object recognition systems. For example, the “Fast be represented as networks, including a wide range and Furious” [91] architecture considers simulta- of applications as diverse as social relationships, neously 3D detection of objects, their tracking over transportation or telecommunication networks [96]. time, and the forecasting of their motions. In its simplest form, a graph is composed of nodes, that represent the entities, and of edges, that represent a link between the entities. For planning purpos- 2.3.2.2 Sequence modelling es, algorithmic approaches have been used to find optimal trajectories along the edges of the graph to Modelling sequential data and training predictive go from one node to another one, using pre-defined systems is a very important subfield of ML with constraints. Classical algorithms coming from graph

constraints the vehicle has to respect along the theory, such as the Dijkstra, Bellman-Ford, or Floyd entire path. These constraints include the design algorithms[97], are popular approaches that do not of a safe and smooth route taking into account all require ML techniques to provide satisfactory results. possible obstacles (still objects, moving vehicles,

etc.) and the compliance with driving rules, but

Deep Reinforcement learning provides a range

also require to take into consideration behaviourof methods well suited for planning tasks [98]. It al aspects due to the presence of humans in the involves learning mapping situations to actions so environment. as to maximize a numerical reward signal. In an es-

sential way, they are closed-loop problems where • The control module is responsible to execute the

the learner is not told which actions to take, as in sequences of actions planned by the system by

many forms of ML, but instead discovers which ac- acting on the actuators (speed, steering angle,

tions yield the most reward by trying them out. In lights, etc.) to ensure that the trajectory is correct-

the most interesting and challenging cases, actions ly performed.

may affect not only the immediate reward but also

the state of the environment and, through that, all The decomposition of the general driving activity

subsequent rewards. Many approaches have been into subtasks is a standard approach to address

developed to find the policy that is optimal, i.e. the complex problems that enable constructors to select

policy that returns in average the highest future re- the right methodology for each of the subcompo-

ward. The use of DL architectures [99] to model the nents. ML has been mostly used in the tasks related

environment has enabled significant advances, with to perception, to sense the surroundings and provide

techniques such as deep Q-learning or actor critic a useful representation of the environment. This was

learning capable of scaling to previously unsolvable spurred by recent advances in computer vision that

problems. created industrial opportunities in this sector. Conse-

quently, companies have started to invest in vehicles

to collect recordings of driving situations, build up 2.4 AI software in automotive infrastructures to collect, label and process those

large datasets, and hire computer vision high-level

systems

engineering teams to develop model to address the

Driving in real world environments with other hu- related tasks. Although solving these problems is still

man-operated vehicles is far from an easy task, which an active area of development, commercial products

requires complex socio-ethical and decision capabili- and services are already in use in modern cars to at-

ties able to cope with unexpected and dangerous sit- tain low to medium levels of automation. Converse-

uations. AI software components embedded in AVs ly, the use of ML for planning and control purposes

are in charge of reproducing these capabilities, by is still in its infancy, while improvement is going fast,

processing data gathered via the sensors and inter- supported by large investments from tech compa-

pret them in order to decide the action to undertake nies. In this context, behaviour modelling techniques

(e.g. move, stop, slow down, etc.). Three main types of are leveraged to learn relevant driving policies that

data processing capabilities are involved: will determine the action to undertake to achieve the

trip according to the environment encountered by • The perception module is responsible to collect the vehicle. These problems are different from permultiple streams of data obtained from the senception problems, and are still considered as frontier sors, and extract from them relevant information research in the academic community. about the environment. This includes the contex-

tual understanding of the scene: detection and Current systems are already achieving tremendous tracking over time of vehicles (cars, trucks, bikes, performances in a wide range of conditions, but their etc.), pedestrians, and objects, and their tracking capacity to generalize is limited by the extreme comover time, recognition of traffic signs, traffic lights, plexity and diversity of the world. A crucial challenge marking, lanes, and more generally any element still unsolved for ML systems is the right general of interest for the driving. The perception module handling of edge cases, where an unknown situation keeps also track of the localization of the vehicle outside of the training data distribution is encounin this environment, detecting its position and oritered. It is very challenging to guarantee that an AI entation with respect to the road and other agents system will output the right results in unusual coninvolved in the scene. ditions, leading to potentially hazardous situations,

• The planning module is in charge of the calcula- for instance ignoring a stop sign partially covered by

tion of the trajectory that the vehicle will under- snow, or stopping in front of a bush slightly over-

take, considering the route between the start loca- hanging the side of the road.

tion and the desired destination, as well as all the

In the rest of the section, an overview of the different • Variability of appearance: elements present in the tasks, and the main techniques that are employed environment can have a wide diversity of aspects: to address them, is presented. This overview is pri- objects, actors and surfaces can have various marily based on works published by the research shapes, colours, texture, orientation, brightness, etc.; community and may not reflect accurately the tech- • Variability of environments: the environment itself nology embedded in actual semi-autonomous cars, varies according to various factors, including the as only limited information is released by car manutime of the day, the season, the weather, but also facturers on this matter. In addition, perception tasks societal factors (maintenance works, strikes, beare prevalently discussed over others, as they conhaviours of agents, etc.); stitute currently the main sources of vulnerabilities of AI systems. • Variability of meaning: objects can have different meanings in different contexts or at different times (e.g. traffic signs with time or space constraints), or wrong appearances, for instance in case of re-

2.4.1 Perception

flections. The perception system refers to the ability of an AV to make sense of the raw information coming in Identification of roads and lanes This requires through its sensors. It aims at the creation of an in- distinguishing drivable areas (roads, driveway, trail, termediate level representation of the environmen- etc.) from non-drivable areas (pavement, roundatal state around the vehicle, and at the tracking of bouts, etc.), the different types of surface, and the the evolution of this state over time. This includes, various lanes present on the road indicating the diamongst others, the capability to detect, classify rection of the traffic flow. This task has been widely and identify everything that an AV potentially could investigated over decades, and has been integrated encounter or has to interact with, such as the in- in vehicles through functions such as lane keeping frastructure (roads, signs, traffic lights, etc.), agents assistance or lane change assistance. Therefore, (cars, cyclists, pedestrians, etc.), or obstacles. It also such technologies generally do not rely on recent consists in the construction of an internal map of the trends in ML, but rather on a wide collection of techenvironment, allowing the vehicle to localize itself niques based on handcrafted features [102] that and other objects of agents in space and time. have proven to be very efficient and reliable. This is done on 2D and 3D images, and is often comple- The most relevant perception tasks for AVs can be mented by real street maps. Despite this, traditional ordered along the terms of scene understanding, methods tend to be limited to understand the chalscene flow estimation and scene representation and lenging semantics conveyed by lanes on the road, localization. Nowadays, much of the field is domi- and this task is getting more and more integrated nated by DL techniques, albeit strong influences are in end-to-end DL systems. This is for instance the coming from robotics, especially for localization and case of the Tesla’s Autopilot [103], whose the ADAS mapping techniques [100], and classical time series to detect stop lines along the road is implemented pattern recognition filters [101]. using a DL architecture.

Detection of moving agents and obstacles The

2.4.1.1 Scene understanding detection of moving agents (pedestrians, cyclist, vehicles, etc.) and obstacles (plants, objects, etc.) is Scene understanding encompasses all tasks that mainly addressed using object detection, segmenaim to provide a current picture of the immediate tation, and tracking techniques. Detection and clasenvironment of the AV. Typical tasks include the de- sification of objects from 2D or 3D images such as tection and recognition of all elements present in the camera data can be successfully tackled with comenvironment. Most approaches that fall under scene puter vision architectures, providing that the trainunderstanding make use of data streams from var- ing dataset is rich enough to characterize fully the ious sensors and employ very successful computer diversity of environments. Recognition and segmenvision based architectures. However, understand- tation techniques are used to detect drivable areas, ing objects in a realistic traffic environment in real objects, pedestrian paths or buildings. time poses a number of additional complexities to the theoretically often extremely accurate computer Traffic signs and markings recognition The devision systems. This requires indeed taking into ac- tection and recognition of the given sign and/or the count the variability of the scene: written indications is crucial to ensure a safe driving. Driving instructions are typically provided according to several vectors:

• Traffic signs, usually using a symbolic rep- techniques require nonetheless adjustments to resentation, are the most common driving indica- adapt to the relatively small size of signs and marktors. While signs vary depending on the shape, the ings compared to bigger objects such as vehicles, as colour, and the pictogram drawn on it, internation- CNNs are typically compressing the image, resultal conventions have helped to achieve a degree ing to small objects, such as signs, to be overlooked. of uniformity, despite small local variations. While Recognition is also greatly affected by unusual enviimage-processing approaches have been mainly ronmental conditions, with degraded performances taking advantage of the well-defined shape and when the symbols are partially occluded by obstacolours of signs, DL has greatly improved the rate cles or stickers, or hard to distinguish due too direct of detection [104]. Models based on CNNs are now sunlight or heavy precipitation. regularly employed [105], taking advantage of various datasets released for this task [106]. Sound event classification Recognizing environmental sounds is an important aspect for the under- • Traffic lights are using a position (usually top, standing of the driving scene: many elements, such middle, bottom) and colour (usually red, orange, as tire screeching, honking, or even engine throbbing, red) code to indicate if the vehicle should stop, convey information about the vicinity of the vehicle, prepare to stop, or is allowed to pass. Particular and can help anticipate hazardous situations. This cases should also be taken into account, for inis particularly true for sirens of emergency vehicles stance when a light is blinking. Research works that indicate a situation where driving rules have to are currently mostly limited by the scarcity of repbe adapted. Albeit visual lights are usually present, resentative public datasets, but are nonetheless they may not be visible to the vehicle, for instance led to the development of DL systems including in the case of a busy intersection, and ignoring the special processing in the colour space [107], [108]. sound alarm could have dramatic consequences. Industrial actors have nonetheless already includ- Sounds also complement the other sensors in low ed traffic light recognition in their vehicles as an visibility situations. assistance feature [109]. • Textual indications are also an important way As for images, sound processing largely makes use to convey information, in particular in situations of DL techniques [115], either on raw data or on where unanticipated rules should apply (e.g. de- spectrograms (frequency representation of sound tours, accidents, traffic jam, etc.). Text can be signals), even if traditional approaches are still widefound on traffic signs, painted on the road, or on ly used due to the long-standing work on handcraftvariable-message signs. Understanding textual ed features relying on physical and cognitive prinindications commonly requires three steps: 1) the ciples. Related to AVs, few works [116] have been detection of the text in the image, 2) the recogni- proposed, partly because of the small interest of the tion of the characters, 3) the understanding of the AV community on these topics compared to vision, meaning. The last step is all the more important and a lack of availability of dedicated datasets. Desince numerous text signs without connection with tecting relevant sounds in urban areas, especially in traffic indications can be found alongside roads, noisy situations, is nonetheless an open challenge like advertisements or touristic information. Cur- that will play an important role in the capacity of AVs rently, this task is mostly done in an ad-hoc man- to achieve human-like performances. ner, without learning involved. Text detection and recognition in natural scenes have nonetheless been an important area of research, either on stat- 2.4.1.2 Scene flow estimation ic images [110] or videos [111], taking into account artefacts, such as distortions or out-of-focus texts. Scene flow estimation collects those perception State-of-the-art approaches have combined con- tasks, which are concerned with the dynamical bevolutional and recurrent neural networks [112], to haviour of the scene, mostly the movement of the achieve text recognition and understanding. The objects and vehicles. availability of datasets [113], [114], although limited, is expected to foster the scientific community Tracking and prediction of moving agents and to advance research on this ongoing topic that will obstacles also benefit autonomous driving. The most important task in scene flow understand- Previously based on handcrafted features describ- ing is to track objects and vehicles to predict their ing images based for example on the colour or the individual motion and may require modelling the shape, object recognition and semantic segmenta- behaviour of other traffic participants, which is very tion techniques are now systematically used. These relevant for planning tasks. The two main challenges

are the tracking of self-motion or stationary objects have also been used, e.g. for subsequent classificaand the prediction of object motion and behaviour. tion of likely objects or the road type [120], [121].

Tracking and predicting the motion of objects located in the immediate environment of the AVs rely ev- 2.4.2. Planning idently first on the ability to detect those objects, but pose a number of additional challenges, which led to Planning tasks comprise all the calculations needed the development of a class of tracking and prediction to perform vehicle actions autonomously, from route methods [117]. The basic object detection problem is planning to the implementation of an immediate moextended through a time axis dimension, where indi- tion trajectory in a given driving situation. They are vidual objects need to be tracked frame by frame, or, confronted with the difficulty to evaluate correctly conversely their likely trajectory is to be extrapolat- the system predictions: Contrary to perception tasks, ed into the future. The employed techniques usually where the ground-truth information is usually known rely on elements from sequence modelling, such as and can be compared with predictions, assessing the probabilistic Markov models or, increasingly, deep re- performances of plann ing systems requires current neural network architectures. Both, tracking a real world testing in controlled environments, or and prediction – as object detection itself – are con- an evaluation stage in a simulator. Even under these ducted using either 2D camera image data, 3D point challenging settings, AVs are able to handle most cloud (mostly LIDAR) data, or both. situations, but may fail to take the right decision in scenarios that have not been considered in the data, the model, or the simulations. The reasoning func- 2.4.1.3 Scene representation tionalities mainly rely on advanced AI methods for autonomous agents and robotics [122], [123]. Scene representation tasks involve the simultaneous mapping of the environment and continuous localization of the AV itself within the environment. 2.4.2.1 Route planning

Localization It consists in estimating the position Route planning (or routing), also called global planand orientation of the AV with respect to the sur- ning, consists in finding the best route between the rounding environment. These techniques actually current position of the vehicle and the destination belong to the wider set of methods from the area that is requested by the user. It relies on GNSS coorof Simultaneous Localization and Mapping (SLAM), dinates and offline maps that are embedded in the which has been addressing the same range of prob- vehicle. The road network is classically represented lems since decades for mobile robots. SLAM algo- as a directed graph: nodes of the graph are way rithms traditionally do not require a priori informa- points, usually referring to intersections between tion about the environment, which allows them to roads, while edges correspond to the road segments, be used anywhere, but the challenging environment and are weighted to reflect the cost of traversing in which vehicles evolve, especially in urbanized ar- along this road, the cost being computed through eas, makes the use of maps [118] a crucial element a metric considering the distance of the segment to achieve a high level of accuracy. Localization is and/or the time of travel. The problem is then to find achieved by matching maps with sensory informa- the shortest path between two nodes of the graph. tion, such as GNSS or cameras and LIDARs outputs. The output of route planning is then a sequence of The fundamental technique being used in this con- way points that are used to generate the trajectory text is visual odometry. Various proposals have also of the vehicle in the environment. been published to tackle the 3D pose estimation [85]. Several approaches have been developed to address Occupancy Maps or Occupancy Grids An occu- this problem. Routing algorithms are usually relying pancy grid is a type of probabilistic mask that re- on specialized heuristics [124] based on graph theoturns for each cell of a gridded map of the environ- ry algorithms that have been developed to take into ment the probability that the cell is occupied. This is account the size of such graphs (usually several milanother popular technique from robotics [119] that lions of edges) and circumvent the intractability of is used for localization and mapping in autonomous standard shortest-path algorithms. The efficiency of driving. It can be inferred from the camera and LI- algorithmic graph solutions makes the use of AI tech- DAR data. Techniques for calculating the occupancy niques less relevant, albeit ML could be leveraged to grid vary, and have been mainly based on ad-hoc adapt in real-time the topology of the graph with methods, even if currently DL based approaches external information [125] or provide personalized routes that includes for example touristic sites [126].

2.4.2.2 Behavioural planning coordinates of the perceived environment. This trajectory has to take into account several constraints, Behaviour planning implies to select what is the most such as being feasible by the vehicle (taking into appropriate driving behaviour to adopt for the vehi- account for example the current speed), being safe, cle, based on the representation of the environment lawful and respectful to other participants present in and on the route to follow. Such a behaviour can be the environment, as well as ensuring a smooth drivdescribed as a sequence of high-level actions. As an ing for the passenger. example, if the route imposes to turn left at the following intersection, an appropriate behaviour could Traditional approaches have been developed in the consist in a sequences of actions such as “Stop the robotics community. Typically, the environment is divehicle before the intersection”, “Observe the behav- vided into a dynamic grid, where each cell has temiour of vehicles that are coming on the opposite lane poral attributes that are informed by the perception and in the crossing lane”, “Detect any potential pe- module. The objective is then to find a trajectory destrian that are about to cross the road”, and finally between two given cells under multiple constraints, “Wait till the path is clear, and then turn left”. This relying on techniques based on graph search, samdecision-making process can be modelled by a finite pling, or curve interpolation [117]. Recently, ML techstate machine, where states are the different be- niques have been employed for local planning, with haviours of the vehicle, and the transitions between promising results, in particular in their capacity to states are governed by the perceived driving context. avoid erratic trajectories and achieve human-like motions. Generally, these approaches are address- One of the key tasks for behavioural modelling is the ing perception and planning at the same time, either detection of the driving style of other agents. Driving through segmented image data including path prostyle designates the various behaviours drivers can posals [136], or by extracting features from LIDAR adopt while driving, classified with qualifying terms point clouds [137]. DL has also been used solely for such as aggressive, sporty, calm, moderate, low-skill, planning, using RNNs to model sequences of wayor overcautious, to name but a few [127]. Recognis- points of trajectories based on a dataset of human ing the behaviour adopted by a human-driven vehicle motions [138], or reinforcement learning in simulatis crucial to understand its dynamics, and is in this ed environments to learn a driving policy that can be respect closely related to the task of tracking the extended to real-life situations [139]. other moving agents. Furthermore, planning systems have to adopt themselves a driving style, possibly ML has achieved tremendous progress in local plangiving to the human user a choice between different ning, but has not yet reached a level of maturity sufpresets, and find the right trade-off between a con- ficient to be implemented in commercial cars. A maservative driving that could lead to longer journey jor limitation is indeed the difficulty to make sure and aggressive driving that could be unsafe and/or that safety measures are properly learnt, as they uncomfortable for the passengers. The learning of cannot be hard-coded in the planning systems as for driving style is an important yet unexplored topic of traditional systems. Nonetheless, their flexibility and research, mostly taking advantage of unsupervised they capacity to adapt to unknown situations, proapproaches [128]–[130] to circumvent the absence vided the context is similar to the one in which the of labelled datasets. The use DL has considerably model has been trained, are a strong argument in extended the range of modelling capabilities [131], favour of future deployment of ML based planners. using the vast amount of driving activities recorded by companies to provide simulating environments in which planning models learn to react to different 2.4.3 Control driving scenarios, either by imitating human drivers [132], [133], or through deep reinforcement learning The control system is responsible for the execution to perform safe and efficient driving [134], [135]. of the trajectory that has been calculated by the planning system by applying commands for the various actuators of the vehicle at the hardware level. 2.4.2.3 Motion planning Broadly speaking, a vehicle has two types of motions: lateral, controlled by the steering of the vehicle, and Motion planning (or local planning) is responsible of longitudinal, controlled by the gas and brake pedals. finding the best trajectory of the vehicle in its perceived environment in accordance to the route that Control techniques regroup a set of methods to monhas been calculated and the behaviour that has been itor the dynamics of a system, in order to achieve selected. This consists in the translation of high-level a given action, while satisfying a set of constraints. actions into a sequence of way points referring in the Such systems act in a closed loop manner, with an

objective value (e.g. a desired speed) prescribed to Human machine interface (HMI) It provides pasthe controller, which has the ability to actuate on sengers the ability to interact with the car, either the systems (e.g. through braking and acceleration), to give commands to the driving or entertainment while getting feedback through monitoring to ensure systems for instance, or to receive information, such an optimal and stable trajectory of the dynamics of as the current itinerary. DL is used to provide novel the system. Two popular approaches are Proportion- communication vectors, such as speech or gestures. al-Integral-Derivative (PID) control [140] and Model Speech recognition embedded in vehicles are able predictive control (MPC) [141]. The former consists in to understand spoken commands that follows the continuously calculating the difference between the syntax of normal spoken conversation. Gesture recdesired and the measured values of the controlled ognition systems are able to interpret common hand variable by tuning the relative importance between gestures, so that gesture based controls become three different terms, describing corrections to apply applicable to interactive displays. Recommendation to get an accurate and smooth trajectory. The latter systems to anticipate the choice of users can also be relies on predictions of the changes of the controlled included as part of HMI systems. value, based on a model of the system. Compared to PID controller, it is costlier in terms of complexity, Vehicle interior monitoring It consists in monitorbut allows considering situations where the dynam- ing the interior of the vehicle through sensors (e.g. ics has a higher variability, or the delays between cameras, microphones, temperature sensors, etc.) to actions and feedbacks are higher. ensure the general comfort of passengers. This function has been originally designed to monitor drivers’ In the case of AVs, the main difficulty lies in the high fatigue, through the monitoring of driver behaviour. complexity of the relationships between controlled Therefore, real-time analysis of biometric factors variables (such as speed or steering angle) and actu- (e.g. heart rate, respiratory rate, eye blinking, etc.) al commands to actuators. Human drivers, with their could trigger a warning alarm to alert the driver. For experience and their understanding of the physical fully AVs, this function could be used to control the world, are constantly monitoring the movement of level of comfort, by automatically adjusting sounds, the vehicle, and the different indicators, such as the lights, or any additional factors based on predictive speedometer, to correct and make sure the behav- models of the well-being of human passengers. iour of the vehicle is compliant with their intentions. By doing that, they integrate implicitly parameters as complex as the total weight of the vehicle, the 2.4.5 Current trends in AI research for friction forces of the tires on the road, or the wind

autonomous driving

intensity, through their sensory perception and their modelling of the environment. While it is straightfor- End-to-end approaches: A general trend in ML is ward to formalize an accurate model between the the use of an end-to-end, holistic approach to tackle actuators and the actual behaviour of the vehicle at several problems at the same time. The rationale below speed, additional factors linked to the environ- hind this approach is that developing separate modment strongly increase the complexity of this model ules tend to be inefficient in terms of computational at high speed. Nonlinear control or model predictive power, but also may lead to poorer performance, as control have to be used to take into account this uncertainties appearing in the upstream section of complexity. To this end, ML techniques have already the driving pipeline are propagated and amplified shown great potential to improve the predictive pow- along all modules. Several variants of this approach er of control models, albeit as of now they are not exist: a popular one is to consider jointly all tasks of deployed in commercial vehicles. perception or planning, or even all tasks from both modules altogether. Nonetheless, the approach relies on a wide diversity of techniques, ranging from 2.4.4 Infotainment and vehicle interior the prediction of driving paths from camera images [136], point clouds, GNSS measurements, and or ex-

monitoring

ternal information [137], to the prediction of steer- AI is not confined to driving functions, and has also ing commands from the same kind of inputs [144]. been proven useful in infotainment systems and ve- Behaviours can also be predicted in an end-to-end hicle interior monitoring. These features are start- fashion from raw pixels [145], [146]. ing to be increasingly integrated in modern vehicles [142], [143], offering embedded hardware dedicated Simulation: The cost of data acquisition has led to voice recognition, or personal assistant controlla- to the development of many simulators, in order to ble via vocal control and facial expressions. address large quantities of data. These simulation environments, many of them released as open-

source software, also lowered the upfront invest- 2.5 Mapping between automotive

ment necessary to do research on AVs, and have

functionalities, hardware and

been the basis for numerous research works, some of them described in this report. Popular simulators software components and AI

include TORCS [147], CARLA [148] and AirSim [149],

techniques

which take advantage of graphics engine used in video games to offer a realistic representation of As a conclusion of this section, the most important the world. These simulators, as well as others [150] key findings are summarized in the form of three taalso include tools to customize sensors (e.g. camer- bles. First, a correspondence between the high-level as or LIDARs [151]), offer typical driving scenarios functionalities and the intermediate tasks is given. to play, and provide easy integration of ML tools Then these tasks are mapped respectively with the for quick development. Other initiatives have been hardware and software components that have been launched to promote autonomous driving research identified, and finally with the AI techniques. towards students and tech enthusiasts, such as DeepTraffic [152].

3. CYBERSECURITY OF AI TECHNIQUES IN AUTONOMOUS DRIVING CONTEXTS

3.1 Vulnerabilities of AI for fer from several issues. This includes unfairness of the decision made due to the propagation of biases

autonomous driving

from data to models and outcomes, opacity of the The development of increasingly autonomous and decision process due to complex model structures connected vehicles inevitably requires a higher lev- and mathematical operations that escape from an el of computational functionality and connectivity, ease straightforward interpretation, unsafety due to which, in turn, widen the attack surface and the like- critical scenarios badly represented or outside the lihood of physical and cyber-attacks. Cybersecurity training data fed to the model during the developrisks in autonomous driving vehicles can have a di- ment phases, or challenging reproducibility and verirect impact for the safety of passengers, pedestrians, fication that can convey a mismatch among real and other vehicles and related infrastructures. It is there- expected results of ML methods and cause issues fore essential to investigate potential vulnerabilities when reproducing and investigating the decisional introduced by the usage of AI. This section focuses process. These issues affect also the reliability of the on the general vulnerabilities and security challenges methods when used in practice. in autonomous driving posed by AI, with a particular focus on ML. It also includes an analysis for specific The present report focuses on the exploitation of AI-related vulnerabilities in autonomous cars. AI vulnerabilities to compromise the integrity and availability of AVs, which belongs to the category Following consolidated threat modelling practice of intentional threats. In particular, adversarial ML [153], threats related to AI can be divided into two is discussed, as a prominent field of research linked groups: intentional and unintentional. Intentional to cybersecurity of AI, and as an immediate threat threats include those coming from a malevolent ex- for AVs. It is nonetheless worth mentioning the same ploitation of the limitations and vulnerabilities pres- technical challenges underpin both intentional and ent in AI and ML methods to cause intended offence unintentional challenges, and improving all aspects and harm. Intentional misuse of AI leads to change described above benefits security and safety of AI of the current cybersecurity landscape by introducing systems as a whole. Research in ML is gathering a new class of vulnerabilities and raising the ceiling a lot of interest and aggregating substantial comof potential impacts. The growing use of AI to au- munity effort, providing advances to increase the tomate decision-making in a diversity of sectors ex- robustness and reliability of AI methods in both norposes digital systems to cyberattacks that can take mal and adversarial settings. A larger overview of advantage of the flaws and vulnerabilities of AI and AI cybersecurity is discussed in the ENISA AI Threat ML methods. Since AI systems tend to be involved in Landscape [1], its relevance in the larger context of high-stake decisions, successful cyberattacks against digital transformation is outlined in a JRC report on them can have serious impacts. AI can also act as an cybersecurity [235]. enabler for cybercriminals: Cybercriminals can use AI to automate aspects of their attacks, enabling them to launch attacks more quickly, at a greater scale and 3.1.1 Adversarial machine learning a lower cost and with higher precision. Adversarial ML emerged in 2004 dealing with the ro- Unintentional threats come as side effects of be- bustness of antispam filters [154] and has since then nevolent usages, due to open issues inherent in the evolved investigating how to challenge and guarantrustworthiness, robustness, limitations and safety tee the security of ML methods and systems [155]. of current AI and ML methods. Unintentional threats Since then, a large amount of work has been done, comprise unpredictable malfunctioning, failures or suggesting that ML-based systems could introduce negative aftermaths caused by shortcomings, poor further vulnerabilities easily exploitable by skilled design and/or inner peculiarities of AI and ML. Ex- attackers. Paradigmatic cases of attacks against ML perimental research and real-settings operations systems used for pattern recognition in cybersecurity have demonstrated that these methods may suf- are: submitting a fake biometric trait to a biometric

authentication system (spoofing attack) [156], [157]; this area of research has gained much attraction in modifying network packets belonging to intrusive the last years [174]–[178] . Recently, reinforcement traffic to evade intrusion detection systems [158], learning models have been probed with respect to [159]; manipulating the content of spam emails to vulnerabilities [179]–[181], as a consequence of make them escape spam filters (e.g. by misspell- their reliance on DL models [182]. Other attacks are ing common spam words to avoid their detection) continuously devised by the research community, for [160], [161]; manipulating of malware to evade ML- instance against real-time video classification sysbased malware detection; deceiving face recognition tems [183], or against RNNs [177]. systems [162], [163]; taking control of a voice interface system, by way of hidden voice commands, unintelligible to a human listener [164] or the deceit 3.1.2 Adversarial examples in computer of reading comprehension systems [165]. The possi-

vision

bility to subvert otherwise-reliable ML systems has received considerable attention since 2014, when it Adversarial examples are the result of an evasion was shown that CNNs for object recognition could be attack, and consist in tiny perturbations of the input tricked by passing them slightly perturbed images that cannot be detected by human but are leading to [166], [167]. Much effort has been devoted to the a misclassification with high confidence by ML modtopic since then, establishing the subfield of adver- els. Albeit adversarial examples can be found for any sarial ML as the most active area of research focus- kind of inputs, such attacks have been particularly exing on the security and robustness of ML systems plored for computer vision models. As shown in Figure to adversarial input, especially those relying on DL. 8, adversarial examples are typically created by adding a small amount of carefully calculated noise to The most common attacks on AI systems can be dis- a natural image. This kind of attack can fool state-oftinguished between evasion and poisoning attacks. the-art, highly performant image-recognition models The first type of attacks aims to manipulate what is whilst being often imperceptible to humans. fed into the AI system in order to induce a system output that serves the attacker’s goal. On the other The research on adversarial examples has gradually hand, poisoning attacks corrupt the training, so that become a hotspot in the computer vision community, the resulting system malfunctions in a way desired and researchers have constantly proposed new adby the attacker. A big share concentrates on attacks versarial attack methods. A commonly referred setto supervised learning models, including attacks ting for adversarial crafting considers that the adagainst regression methods [168], [169], SVM [162], versary’s goal is to define a perturbation that, applied and ensembles of classifiers [170]. Vulnerabilities of to an input image, makes the model misclassify the unsupervised learning models have also been ex- resulting perturbed image [186]. The ways of genplored, examining possible attacks against cluster- eration of adversarial perturbation depends on the ing methods [171]–[173]. With the greater integra- adversary’s knowledge of the system. A distinction tion of DL techniques in many critical applications, is commonly made between white-box and black-

box attacks [187]: In white-box attacks, the attacker process), or improving the algorithm used to minihas a full knowledge of the model that typically in- mize it (i.e. finding the iterative steps that will concludes the parameters of the model and sometimes verge to a good solution). Fast Gradient Sign Meththe data used for the training. Conversely, black-box od (FGSM) [191] relies on a single optimization step attacks consider that the attacker has only access to that includes only the signs of the gradients. Basic a limited set of pairs of inputs-outputs or is only able Iterative Method (BIM) [184] extends this approach to submit its own inputs to the model and gets the by applying FGSM iteratively, increasing the chance corresponding outputs. An intermediate situation, of successes of attack by crafting more complicatoften referred as grey-box setting, considers the ed perturbations [192], [193]. Other approaches, same situation as for black-box attacks except the like Jacobian Saliency Map (JSMA) [194], rely on the adversary has a limited knowledge about the model, localization of the salient pixels of the images and e.g. its training set or the family of models that is focus on these specific areas to craft the perturbaemployed [188]. The generation of adversarial ex- tions. C&W attacks [195] add multiple refinements amples for ML models is most commonly studied to previously mentioned techniques to increase the from the standpoint of a white-box attack. Black- chance of success, and in particular to bypass severbox attacks have nonetheless been demonstrated in al defensive mechanisms that have been suggested many contexts, and are usually built on top of white- to counter adversarial attacks. box attacks using substitute models [187], relying on the transferability of adversarial examples from A second line of research that is complementary and a model to another one, i.e. their capacity to work on usually considered simultaneously in the design of a range of close yet different architectures. the optimization problems concerns the choice of the constraints to apply on the perturbations. Most The field of adversarial ML has been mainly built on techniques are typically trying to find the minimal top of computer vision techniques, and in particular perturbation possible, in order to make it less perof classification models. This report reflects this situa- ceptible by a human auditor. This is often expressed tion, which is also especially relevant for AVs that are as the average over all pixels of the intensity of the composed of multiple classification systems working perturbation, the maximal intensity, or the number on images. It is nonetheless worth mentioning the of pixels that are perturbed. Depending on the type techniques introduced can be generalized to other of constraints, the optimization algorithms will be types of problems (detection, regression, behaviour chosen accordingly to achieve a good convergence, modelling, etc.) and data (text, sound, tabular data, i.e. make sure that the whole process returns a peretc.), extending the scope of adversarial attacks. turbation that is indeed a good minimum for the considered loss function. As an example, the DeepFool attack [175] computes a minimal norm adversarial 3.1.2.1 Overview of adversarial example attacks perturbation for a given image in an iterative manner, in order to find the decision boundary closest Following the seminal work of Szegedy et al. [167], to the clean input image and find the minimal admost techniques proposed to perform adversarial at- versarial sample. Some attacks are also considering tacks are exploiting the gradients of the model, com- very specific setting, like the One Pixel attack [196] puted in order to maximise the effect of adversarial that constraints the perturbation to affect only one perturbations in altering the output of the model. As single pixel. Other approaches exist to address diffor the training phase that exploits the same idea to ferent contexts and applications. The Houdini attack update the model parameters, the minimization of [197] works on non-differentiable loss functions, and a loss function is used to describe the adversarial has been proven useful in domains such as natural problem [189]. The complexity of the problem usu- language processing, while the Zeroth Order Optimially makes the resort to an optimization algorithm zation (ZOO) attack [198] has been used in blacknecessary to get a satisfying solution, which is most- box settings to estimate gradients based on outputs. ly based on a classical technique called Projected A current trend is the use of generative models Gradient Descent (PGD) [190]. [199], [200] to synthesize adversarial examples entirely from scratch. While these approaches do not A first line of research consists in improving the opti- allow modifying existing images, their capacity to mization process at play during the attack to make it generate realistic samples wrongly classified is getpractical for large dataset and allows for better solu- ting more and more problematic, and is closely relattions. This is done either by finding better terms to ed to the growing use of deepfakes [201]. include in the loss function (i.e. that are both a good approximation of the objective one want to achieve Similar approaches have been followed to generate and have properties that improves the minimization adversarial examples for other types of data, not

coming from cameras but from other sensors that sive and defensive adversarial ML, which is for now can be found on autonomous vehicles. Examples in- taking place in the ML research community, but is clude attacks on 3D points clouds outputted by LI- expected to develop into a more typical cybersecu- DARs [18], [202]–[204], on radars [205] or ultrason- rity situation between adversaries and defenders in ics sensors [206]. The paucity of standard datasets the future. A short overview of the main techniques makes this area of research less fruitful compared developed in the ML community is given in the folto traditional RGB images, but the democratization lowing. Adversarial attacks may be counteracted by of devices and the development of methods follow- approaches to make the ML model more robust and ing the same concepts as for traditional imagery resilient to adversarial examples. The solutions pre- (e.g. transferability of adversarial examples between sented in the literature, especially in the last years 3D point cloud models [207]) will increase the effi- in relation to DL models, may be categorised in two ciency of adversarial attacks. strategies: acting on the data or acting on the model.

Acting on data can take place at training of inference 3.1.2.2 Physical adversarial examples time: Adversarial training [213] consists in introducing adversarial examples generated using classical Adversarial attacks described above mainly refer to attacks into the training dataset to improve the roa setting where attackers have the ability to update bustness of the model [154]: the model integrates the inputs with the only constraints that perturba- the variability added by the adversarial perturbations tions were imperceptible for a human supervisor. in the model. The resulting model is less subject to This setting has some limitations when data are pro- adversarial attacks, and is able to correctly handle cessed after acquisition, as it often happens in com- adversarial inputs generated using substitute modputer vision: a model may be trained to recognize els [214]. Once the model is trained, a second action a certain type of objects using a large collection of to mitigate the risk of adversarial attacks consists in digital images, but the end application would involve performing data sanitization, which is used to detect a camera acquiring and processing on-the-fly imag- and reject samples that are too far from the training es. In the last couple of years, it has been extensive- data distribution [215]. For computer vision DL modly demonstrated that adversarial examples could be els, the use of generative networks has also been transferred to the physical world [208], [209]. This is proposed to identify and reject adversarial exammainly done through the alteration or the creation of ples prior submitting them to the model [216], [217]. objects with specific features that are misclassified These two approaches techniques have nonetheless by a DL model trained to recognise them after acqui- strong limitations, adversarial training protects the sition by a camera. model only against a limited set of attacks, and has a significant impact on the training performances, The generation of successful physical adversarial both in terms of accuracy and training time. As for attack is particularly challenging due to the loss of data sanitization, the full check can also be compusensitivity of adversarial perturbations when they tationally expensive, with limited effectiveness deare subject to minor transformations [210] either pending on the kind of attacks. happening in the physical environment, such as lighting variations, change of angles, motion blur- Actions on the model mainly aim at increasing the ring, etc. or in the acquisition phase, such as filter- robustness of the model by making changes in the ing or resizing. An interesting workaround consists in training algorithm. In this frame, a classic technique including these transformations in the optimization called regularization appears to improve the generalscheme, either in the iterative process like for the ization capacity of the model by adding penalty terms expectation-over-transformation (EOT) attack [211] to the cost function forcing the parameters of the that alternates between gradient steps and random models to exhibit desirable properties, like smooth transformations, or adding terms in the loss function decision boundaries, and to increase the resistance of that constrain the perturbations to be physically re- the model to attacks on unknown data [218], [219]. alisable [20]. Defensive distillation [220] is also used to smooth the outputs of the model, and avoid hard decision boundaries that are exploited by many adversarial attacks. 3.1.2.3 Countermeasures to adversarial attacks These defences are however heuristic, with no formal guarantees on convergence or robustness properties. In parallel of the development of adversarial at- Formal verification approaches [221], [222] are being tacks, defensive measures to make these systems more and more popular in the research community, less vulnerable have been developed [212]. This has and have demonstrated convincing results. Their use led to a cat-and-mouse situation between offen- in AI systems for AV is nonetheless premature due

to the low scalability of these techniques. Finally, ML in the perception systems of the vehicle. They were ensembles have also been exploited to improve se- able to trigger the auto wipers by projecting noise on curity against evasion attempts, e.g. by implementing an electronic display placed in front of the vehicle, rejection-based mechanisms or secure fusion rules thus fooling the visual sensor of the system. They [170], [223], [224], here again with an additional also investigated the lane detection system: it was computational cost. demonstrated that after application of aggressive blur to a traffic lane the perception system might not detect it, and that fake lanes might be produced by 3.1.3 AI-based physical attacks against placing certain stickers on the road (the latter was not demonstrated yet in real driving conditions). In

autonomous vehicles

this situation, a human driver would have probably Several examples of physical adversarial attacks on noticed the perturbation, but would have relied on AI components of semi-autonomous cars were re- common sense to react properly. ported in recent years. DARTS [225] (Deceiving Autonomous caRs with Toxic Signs) targets the traffic-sign A recent work has also demonstrated that the steerrecognition functionalities of autonomous cars. The ing angle predicting systems of an autonomous car method includes a pipeline for upscaling adversarial is vulnerable to adversarial evasion attacks at opperturbation in such a way it becomes printable, and eration time [230]. The authors have adapted the has been evaluated on real-size printed signs to fool Carlini & Wagner attack to change the predicted a classifier getting images from a front-facing cam- steering direction. era in a real vehicle. The adversarial creation pipeline proposed for DARTS has been extended [226] to deceive a commercial car perception system in re- 3.2 Attack scenarios related to AI al-world driving conditions, with improved random

in autonomous driving

augmentation techniques and the ability to create perturbations that are tailored to speed limit traffic Considerable research effort is being invested in signs and, therefore, less perceptible to a human identifying AI security issues and vulnerabilities for viewer. The pipeline allows for robust production and AVs, recommending potential mitigation techniques, evaluation of printing-size adversarial signs in black- as well as highlighting the potential impacts on the box models. Spoofed and clean signs were positioned vehicle itself and related infrastructures becoming around the track, and were perceived by the traffic compromised. Various threats associated with the sign recognition system of the car driving around the different sensors, controls, and connection mechatrack. Results showed that the altered signs were not nisms have been identified. In addition to the vulonly misclassified, but also caused some unexpected nerabilities specific to ML systems discussed in the behaviours of the vehicle. previous section, AI-related security issues are taking advantage of the more classical hardware and Another example of an attack consisted in deceiving software vulnerabilities present in digital systems, Tesla cars into accelerating well past a speed limit extending standard attack vectors. More precisely, [227]. By slightly elongating using black tape the mid- some of these security issues and vulnerabilities dle line in the "3" on a 35-mph (around 56 km/h) speed usually mentioned include: sign, the system predicted a speed limit of 85 mph • Sensor jamming, spoofing and blinding/saturation: (around 137 km/h). In another work [228] focusing on sensors may be blinded or jammed. In this way, the popular external ADAS Mobileye, the researchers the attacker may manipulate the AI model, feed injected spoofed traffic signs to assess the influence the algorithm with erroneous data or intentionally of environmental changes (e.g. changes in colour, provide scarce data and thus diminishing the efshape, projection speed, diameter and ambient light) fectiveness of automated Decision-making. Stemon the outcome of an attack. To conduct this experiming from the first attempts [17], recent works ment in a realistic scenario, they used a drone to carry have demonstrated for instance the possibility to a portable projector, which projected the spoofed trafsaturate [18] or spoof [202] LiDAR sensors and its fic sign. Their experiments show that it is possible to underlying ML method for data interpretation. fool Mobileye so that it interprets the drone projected spoofed traffic sign as a real traffic sign. • DoS/DDoS attacks: disrupting the communication channels available to an AV makes it essentially A research group from Tencent Keen security lab blind to the outside world. It has a direct impact performed a comprehensive study of reverse en- on its availability and hinders operations needed gineering for finding security flaws in a Tesla car for autonomous driving. The objective of DDoS at- [229]. Among other things, they found weaknesses tacks is to disrupt such communication channels.

• Manipulating vehicle communications: hijacking tential adversaries to gain access to this type of and manipulating communication channels have information and cause a data breach. a severe effect on autonomous driving operations, allowing an adversary to modify transmitted sensor readings or falsely interpret messages coming 3.2.1 Attack scenarios from road infrastructure. Five hypothetical scenarios are presented in this sec- • Information disclosure: given the abundance of tion, to illustrate the exploitation of AI vulnerabilities (personal and sensitive) information stored and in an automotive context using both classical cyberutilized by vehicles for the purpose of autonomous security and AI-specific vulnerabilities. driving, including critical data on the AI components , a particular motivation emerges for po-

3.2.2 Illustration: Fooling a traffic sign stalled a malicious piece of software running on the internal computer of the car. Here, we consider that

recognition system

the attacker is looking to apply the minimal pertur- A short experiment is provided to illustrate the im- bation possible to make the adversarial image looks plementation of an attack performed in the context similar for a human user. described in attack scenario 1. We implemented two practical adversarial attacks of an AI model for traffic Figure 9 shows the result of the attack on a driving sign recognition (TSR), responsible for the detection scene extracted from the GTSDB dataset: more than and the identification of traffic signs along the road. 100 signs are detected all over the image with high confidence, in contrast with the two signs present in In this illustration, we develop a custom detection the scene. This attack impairs the availability of the system based on the DL based architecture YOLO TSR system that can either affect the responsiveness [76] to perform traffic sign recognition, using an im- of the autonomous system if not handled correctly, plementation of the YOLOv5 framework [231]. For or make the vehicle ignore actual signs present on our application, the model is trained on the German the road, leading to wrong behaviours of the vehicle. Traffic Sign Detection Database (GTSDB) [106]. The model is tested to make sure the performances of the model were good on previously unseen images. 3.2.2.2 Class spoofing attack in the physical We implemented two attacks to fool the outputs of context the TSR system. As described in Section 3.1, a distinction is made between adversarial attacks oper- Class spoofing consists in making the system outating at the level of digital systems (referred in the putting a different category for the signs that are rest of the section as “digital context”) and those ap- detected, the localization of the sign remaining idenplied in the physical environment (referred as “phys- tical. In the physical context, the attacker can only ical context”). alter the environment in which the car is evolving, for example adding stickers, projecting light, or physical altering signs. The goal is to cause the TSR system 3.2.2.1 Overflow attack in the classical context to produce invalid, yet plausible output, and thus to induce a wrong behaviour of the vehicle. The sys- The overflow attack consists in making the system tem cannot detect the attack, reducing its integrity. outputting a large number of detections of signs in The car may adopt a behaviour prescribed by the the current frame. It is performed in the digital con- attacker. In the physical context, the attack can only text, where adversaries have access to the system in alter the environment in which the vehicle evolves. which the AI component is evolving, and aim to up- For practical reasons, this is translated in this experidate the numerical values that are being inputted to ment as a constraint that the attack can only modify the detection systems. This happens if the attacker a few pixels of the image on a sign, the same way gained access to the internal system, either remote- a sticker would do. ly (e.g. using network access) or physically (e.g. using a vulnerability in the infotainment system), and in-

Figure 10 shows the result of the attack on a driving scene extracted from the GTSDB dataset: adding a sticker-like perturbation located on the sign fools

model. (Top) Normal output of the TSR system: the TSR system that correctly predicts the localithe two signs are detected as expected. (Middle) zation of the sign but misclassifies the sign. This Amplified intensity of the perturbation added to attack impairs the integrity of the system, with poeach pixel of the image. (Bottom) Output of the tential high impacts on the behaviour of the vehicle

4. AI CYBERSECURITY CHALLENGES AND RECOMMENDATIONS FOR AUTONOMOUS DRIVING

their quality and reliability in relation to data de- 4.1 Systematic security validation pendencies, model complexity, reproducibility, testing, and changes in the external world. This also in-

of AI models and data

cludes the data used by the ML models, which may Data plays an important role when building and val- eventually contain unexpected patterns, not presentidating AI systems, at the core of the learning pro- ed in the training datasets, unintentional (change of cess of ML models. AVs have multiple sensors col- environments, etc.) or be intentionally altered to conlecting each second millions of values describing the duct a cyberattack. environment according to various modalities. These large sets of data are feeding complex AI models that are dynamic in nature. In this context, system- RECOMMENDATIONS atic data validation is of paramount importance in • Establish monitoring and maintenance processes order to prevent unexpected behaviour due the wider for the AI models either proactive or reactive. variety of situations that vehicles may encounter in the real world, including attacks based on the alter- The proactive monitoring works to identify how to ation of inputs such as poisoning and evasion at- improve continuous learning to provide software tacks. Companies and research groups are not only updates. The reactive approach entails detecting relying on real-world static data sets, but also make a wrong output and identifying its causes to unuse of simulation environments to get large volumes derstand how the method or the outputs can be of realistic yet simplified data, adding another layer rectified. In this context, the same situation can be of concern in terms of security. More globally, the checked before or after software updates to check definition of data governance adapted to the par- if there is a faster way to take decisions. ticularity of data used in autonomous driving should • Conduct systematic risk assessments considering be implemented to understand, among others, who specifically the AI components throughout their liowns the data, who has access, or the appropriate fecycle. usage of the data. • Adopt resilience mechanisms preparing alterna- A particularity of AI models is that they can change tive plans and incident response activities in case their behaviour overtime, implying that security and of incidents. robustness assessments do not just take place at • Establish feedback loops of testing vehicle operaa given point in time during their development, but tions as a continuous monitoring process and lesinstead should be systematically performed throughsons learned activities. out the AI model lifecycle. This is of particular importance when considering the proliferation and use of • Establish audit processes to support forensic analpre-trained models from third parties, and the fact ysis after incidents and address relevant concerns that AI models are constantly learning from newly for the future. acquired sets of data. The systematic validation of AI models is a challenging issue for the cybersecurity For example, keep audit trails to later check how of AVs, ensuring security in the AI systems in auton- a decision was made and perform post incident omous cars to make sure model updates do not add analysis, keep logs of serving data since data vulnerabilities that may be exploited by attackers. might delay things and lead to accidents. This is related both to information security incidents and In this context, it is important to ensure that the se- traffics accidents. curity and robustness of model updates is system- • Introduce additional validation checkpoints to limit atically assessed and tested, as part of a broader the impact of erroneous data. systematic validation and testing process to ensure

For example, potential solutions of additional val- use secure embedded components to perform the idation mechanisms for continuous validation of most critical AI functions, similarly to the usage of data. hardware security module for cryptography. Having unauthorized access to an unsecured element could lead to threats for the whole automotive ecosystem.

4.2 Supply chain challenges

Cybersecurity is a shared responsibility between all

related to AI cybersecurity

stakeholders, including OEM, Tier 1 and Tier 2 enti- The security of the software and hardware supply ties, who should address security concerns to mitchain is of paramount importance in cybersecuri- igate the different risks and ensure people’s safety. The supply chain should be strong, capturing all ty. The current draft of the UNECE regulation [2] the involved parts in order to ensure security of the specifies that OEMs, suppliers and service providers software. Supply chain management is a well-known should consider cybersecurity concerns and implechallenge acknowledged by the majority of involved ment appropriate security controls. However, the actors and stakeholders. The absence of proper se- security regulations and good practices should take curity policies and sufficient strategies across the into account the specific features and the relevant supply chain of AI components results in a lack of impact of the involved AI models. resilience and the presence of potential security breaches in systems. Ensuring proper governance of security policy across the supply chain requires in- RECOMMENDATIONS volving stakeholders as diverse as developers, man- • Establish a proper AI security policy across the ufacturers, providers, vendors, aftermarket support supply chain, including third-party providers. operators, end users, or third-party providers of online services. • Ensure governance of AI security policy across the supply chain. Recently, the situation has become even more com- • Identify and monitor potential risks and threats replicated as AI systems are becoming increasingly lated to AI in autonomous driving. involved in autonomous vehicles, and have an additional impact on the supply chain and its complexity. • Develop an AI security culture across the supply Security aspects in all the phases of the AI lifecycle chain, involving all the stakeholders. introduce new security risks in the automotive sup- • Request compliance with regulations in the autoply chain. For example, checking for security issues motive sector across the supply chain. (intentional such as backdoors and non-intentional) in pre-trained models is challenging considering the complexity and the opaqueness of AI models. Besides, the distinct open-source culture in ML limits 4.3 End-to-end holistic approach the tracing of such assets, pre-trained models be-

for integrating AI cybersecurity

ing available online and widely used in ML systems, without guarantee on their origin. with traditional cybersecurity

principles

Another particularity of the supply chain security issue in AI for autonomous driving is connected to the The push to implement AI security solutions in autospecific way in which the automotive industry works motive systems responds to rapidly evolving threats with respect to the digital components of the vehicle. and raises the need to secure AI systems in relation Whilst there are new players (e.g. Tesla) that inte- with the other components and services of the augrate themselves the electronic control units (ECUs), tonomous car. In particular, AI cybersecurity should most manufacturers rely on ECUs from third parties, be integrated with traditional cybersecurity principles. resulting in a vehicle having dozens of ECUs from several manufacturers [232]. Increasing dependence on AI for critical functions and services in AVs will not only create greater in- Accordingly, security processes of the supply chain centives for attackers to target those algorithms, but should capture the specific AI features and become will also step up the potential for each successful dynamic and flexible in every potential change. De- attack to have more severe consequences [233]. pending on the architecture of autonomous cars, Best practices for secure systems often ignore that highly accessible components and lack of robust an AV is a multidimensional environment with differ- AI models may entail significant concerns with re- ent components that may themselves include one spect to cybersecurity. It could be very beneficial to or several AI models of different natures. In light of

this, ensuring cybersecurity in AVs requires an end- • Ensure proper governance of AI cybersecurity polto-end holistic approach taking into account all the icy in the organizations defining specific roles and different components, the diversity of AI systems, responsibilities. and their interactions. Building security as an inte- • Create an AI cybersecurity culture across the autograted procedure that involves various systems and motive ecosystem. takes into account all the phases of the AI systems is vital for the resilience of systems to potential secu- • Promote innovation and R&D activities for incorpority breaches. Applying defence-in-depth strategies rating AI cybersecurity in the organizations. plays a significant role in measuring and enforcing • Promote dialogues between industrial actors to security requirements. Integrating AI cybersecurity, ensure interoperability in the development of AI for all the steps of the AI lifecycle, with traditional solutions. security principles is very important, since a missing vulnerability may jeopardize the security of the • Promote security patterns for the design and imwhole autonomous vehicle. plementation of the AI-based components. • Promote research projects on the security of AI Even if a system is designed and developed with components for autonomous driving. security in mind, systems change over time with • Implement solutions that can detect if not prevent additional equipment, software and functionalities. the potential jamming of sensors. This situation imposes the need for an integrated approach that needs to be maintained and updated, capturing all the systems, the AI models and their interactions. Towards this end, companies and organ- 4.4 Incident handling and izations involved in the automotive sector need to

vulnerability discovery related to

update their cybersecurity policies accordingly.

AI and lessons learned

There is a crucial need for a holistic approach integrating AI with traditional cybersecurity principles In many organisations, although cybersecurity teams as well as a thorough documentation of AI systems know the main threats to which many components in automotive context. Contrary to classical secure and systems in AVs are exposed, often people only software development, for which “prepared state- become truly aware of the importance of security ments”, to avoid for example SQL injection attacks, when they suffer an incident or discover a vulnerare readily available, security patterns for the design ability. Despite the vast publicity regarding security and implementation of AI-based components are vulnerabilities, the related awareness and commitmissing. This is nonetheless needed for the AI design ment to security remains significantly low, especially and development: Securing AI pipelines throughout with regards to vulnerabilities of AI systems. the whole AI lifecycle requires tamper-resistant implementations of each stage, mutual authentication It is worth highlighting that, in many cases, vulnerbetween all the stages and confidentiality/integrity ability discovery may influence the security pracat the interfaces between the different stages. In the tices more than the a-priori information about the automotive sector, this translates into recommend- existence of potential high risks. Optimistic bias is ing tamper-resistant sensor, strongly authenticated the main reason for this situation, stemming from components of the on-board network, adversarially the belief of many people that they are less likely trained on-board models, limited plasticity after de- to experience a negative event, because either they ployment, etc. do not have the sufficient knowledge on actual risks or they are motivated to underestimate the risks. In this context, the optimistic bias may have a robust RECOMMENDATIONS negative impact on the perception of AI security risks in the automotive. • Establish security processes in the organizations integrating AI particularities. The absence of AI security awareness and an inad- • Promote security by design principles when it equate AI security training also aggravate and percomes with deployment and development of AI in petuate this bias. In this context, given the lack of automotive context. information about the expected outcomes of a potential breach, the case studies and the first-hand • Promote the use of standardised components and accounts of security incidents and other security homogeneous AI solutions in automotive context. shortcomings will require significant amount of time to be properly resolved using current practices. It is

therefore essential to apply real-world training in or- • Promote the use of mandatory standards for AI der to deal with the negative AI security impacts of security incidents reporting. optimistic bias. • Organise disaster drills, involving high management, so that they understand the potential im- A clear and established cybersecurity incident hanpact in case a vulnerability is discovered. dling and response plan should be considered, taking into account the increased number of digital compo- • Develop simulated incidents for raising awareness nents in the vehicle and in particular, the ones based and knowledge in this sector. on AI. An AI incident could be considered as an incident in which the behaviour of the vehicle as dictated by the planning module of the AV system is sus- 4.5 Limited capacity and expertise ceptible to cause harm, either due to an intentional

on AI cybersecurity in the

malicious attack or due to the failure of an element in the ML pipeline. This may include potential viola- automotive industry tions of privacy and security, such as an external attacker attempting to manipulate the model or steal The absence of sufficient security knowledge and data encoded in the model, or incorrect predictions expertise among developers and system designers that can cause dangerous situations in which a traf- on AI cybersecurity is a major barrier that hampers fic accident may happen. the integration of security in the automotive sector. Many organisations associate security directly to the A well-structured and domain-specific plan of ac- extent to which developers use security practices. tion that immediately acts following an AI security The lack of AI knowledge by developers is then the breach or failure is essential in order to reduce the source of several issues that may allow attackers to incident costs and damages to the organizations and easily target AI components of AVs. First, the develthe end users of the autonomous cars. There should opment processes do not include security tests and also be a way to supervise AI systems and detect code analysis specific to AI components. Second, AI bad predictions (e.g. by comparing it against some system designers tend to have a limited expertise ground truth such as maps and/or V2X messages on the domain of application, leading to poor design from external sources). Frameworks further incorpo- decisions. rating AI weaknesses (e.g. red-teaming) or penetration testing considering AI specific issues should also As a result of these security shortcomings, AI secube considered. rity is often an afterthought, and AI security controls implemented as add-ons without full integration The apparition of new actors with no previous expe- may arise, leading to complex, expensive, and hard rience in security incidents raises the need to build to maintain architectures. This situation is fertile a cybersecurity culture to be able to comprehend the grounds for security vulnerabilities. Even if an AI potential vulnerabilities and the underlying threats system is designed and developed with security in inherent to their systems, in order to know the cor- mind, the volatility of AI systems, that need to be rect steps to secure systems and to prioritize actions constantly updated with additional equipment, softin case of incident. All stakeholders involved in the ware and functionalities, imposes the need for AI automotive supply chain should then stay aware of security-aware security teams in order to authorize the growing AI threat landscape, in order to be able and track the changes as well as to evaluate poto map the risks and attacks to business operations. tential AI security issues as part of a configuration Lastly, processes for lessons learned when experi- management process. encing an AI security incident can also stimulate the creation of a security program across the whole sup- Most people are not trained either properly or not ply chain. at all in order to be able to recognize the security implications of AI software requirements. They do not know the security implications of the way that RECOMMENDATIONS AI software is modelled, architected, designed, implemented, tested and prepared for distribution and • Adapt incident response plan to include AI particudeployment. Under these circumstances, AI software larities. may not only deviate from its predefined security re- • Establish a culture of learning from AI security in- quirements but also these requirements may have cidents. been inadequate in the first place for its use in AVs. Without such knowledge, developers may not even • Promote knowledge sharing. recognize the security implications of certain design

and implementation choices, and that their mistakes • Launch security education and training focused and omissions in the development phase can lead on AI systems cybersecurity and their integration to exploitable vulnerabilities in the software when it across the automotive ecosystem. becomes operational. • Deploy tools inside the continuous integration toolchain system that allows automated security The lack of project management to support and entesting of each pull request. This allows for an imcourage the developers and designers to become AI proved response and a better vulnerability remesecurity-aware through education and training is an diation as they appear. aspect of paramount importance. Companies may also lack the resources to offer AI security training, • Bring industry expertise to academic curriculum by developers may remain focused mainly on their pri- welcoming lead people in the field to guest lectures mary functional task and security ignoring the AI or by defining special courses that tackle this topic. cybersecurity features. Even if developers want to bear security in mind, sometimes there are budget and time related limitations posed by the top management and other stakeholders. It is then essential that project managers obtain an appropriate level of AI cybersecurity education and training in order to be less likely to make decisions that undermine the security of AVs.

Security awareness should not stop with the developers and managers, since it is important for all members associated with an AI software to receive security training to ensure that AI cybersecurity is a core concern in AVs. AI cybersecurity training of all involved parties will also help to make cybersecurity more prominent during discussion, planning and board meetings towards a secure automotive sector. In order to avoid damages that may deteriorate the reputation of the companies, the entire organisations must be aware of the importance of the implementation of AI security in AVs, and the consequences of not considering it as a priority objective. People should be trained and understand that cybersecurity of AI is not only about countering ML adversarial attacks, but also includes (together with adversarial mitigation measures) aspects from traditional cybersecurity, e.g. forensics, incident response, etc. In the automotive industry the AI systems should be designed by teams where automotive domain experts, ML experts and cybersecurity experts can collaborate.

RECOMMENDATIONS

• Integrate AI cybersecurity particularities in the whole organization policy. • Create diverse teams consisted of experts from ML related fields, cybersecurity and the automotive sector. • Involve mentors assisting the adoption of AI security practices in the organizations.

REFERENCES

[1] ‘Artificial Intelligence Cybersecurity Challenges - Threat Landscape for Artificial Intelligence’, ENISA, 2020. [2] United Nations - Economic Commission for Europe, UN Regulation on uniform provisions concerning the approval of vehicles with regards to cyber security and cyber security management system. . [3] ‘SAE J3016B: Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles’. 2018. [4] Y. Sun, D. Olaru, B. Smith, S. Greaves, and A. Collins, ‘Road to autonomous vehicles in Australia: an exploratory literature review’, Road Transp. Res. J. Aust. N. Z. Res. Pract., vol. 26, no. 1, p. 34, 2017. [5] T. Imai, ‘Legal regulation of autonomous driving technology: Current conditions and issues in Japan’, IATSS Res., vol. 43, no. 4, pp. 263–267, 2019. [6] Singapore Standards Council, ‘Technical Reference 68 - Autonomous vehicles’. 2019. [7] B. Clark, G. Parkhurst, and M. Ricci, ‘Understanding the socioeconomic adoption scenarios for autonomous vehicles: A literature review’, University of the West of England, Project Report, 2016. [8] National Highway Traffic Safety Administration (NHTSA), ‘Automated Driving Systems 2.0: A Vision for Safety’, U.S. Department of Transportation, 2017. [9] D. Ticoll, ‘Automated Vehicles in Toronto’, Innovation Policy Lab , Munk School of Global Affairs, University of Toronto, Discussion Paper. [10] ERTRAC Working Group ‘Connectivity and Automated Driving’, ‘Connected Automated Driving Roadmap’, ERTRAC, 2019. [11] C. Krupitzer, V. Lesch, M. Pfannemüller, C. Becker, and M. Segata, ‘A Modular Simulation Framework for Analyzing Platooning Coordination’, in Proceedings of the 1st ACM MobiHoc Workshop on Technologies, mOdels, and Protocols for Cooperative Connected Cars, New York, NY, USA, 2019, pp. 25–30. [12] ‘Cooperative, connected and automated mobility (CCAM)’, Mobility and Transport - European Commission, 2016. [Online]. Available: https://ec.europa.eu/transport/themes/its/c-its_en. [Accessed: 01-Dec-2020]. [13] LSH Auto UK, Mercedes-Benz MBUX Artificial Intelligence | LSH Auto UK. 2019. [14] ‘Ad-Hoc Working Group on Artificial Intelligence Cybersecurity’. [Online]. Available: https://www.enisa.europa. eu/topics/iot-and-smart-infrastructures/artificial_intelligence/adhoc_wg_calls. [Accessed: 01-Dec-2020]. [15] Andy Greenberg, ‘Hackers Remotely Kill a Jeep on the Highway—With Me in It’, Wired, 2015. [16] K. (Curtis) Zeng et al., ‘All Your {GPS} Are Belong To Us: Towards Stealthy Manipulation of Road Navigation Systems’, in Proceedings of the 27th {USENIX} Security Symposium, 2018, pp. 1527–1544. [17] J. Petit, B. Stottelaar, and M. Feiri, ‘Remote Attacks on Automated Vehicles Sensors : Experiments on Camera and LiDAR’, presented at the Black Hat Europe, 2015. [18] H. Shin, D. Kim, Y. Kwon, and Y. Kim, ‘Illusion and Dazzle: Adversarial Optical Channel Exploits Against Lidars for Automotive Applications’, in Proceedings of the International Conference on Cryptographic Hardware and Embedded Systems, 2017, pp. 445–467. [19] I. Foster, A. Prudhomme, K. Koscher, and S. Savage, ‘Fast and vulnerable: A story of telematic failures’, in Proceedings of the 9th USENIX Workshop On Offensive Technologies (WOOT 15), Washington, D.C., 2015. [20] K. Eykholt et al., ‘Robust Physical-World Attacks on Deep Learning Visual Classification’, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 1625–1634. [21] KPMG International, ‘2020 Autonomous Vehicles Readiness Index’, 2020. [22] European Commission, ‘A European strategy on Cooperative Intelligent Transport Systems, a milestone towards cooperative, connected and automated mobility’, COM/2016/0766 final, 2016. [23] ‘Platform: C-Roads’. [Online]. Available: https://www.c-roads.eu/platform.html. [Accessed: 07-Dec-2020]. [24] European Commission, ‘On the road to automated mobility: An EU strategy for mobility of the future’, COM(2018) 283 final, 2018. [25] ‘C-ITS Point of Contact’. [Online]. Available: https://cpoc.jrc.ec.europa.eu/index.html. [Accessed: 01-Dec-2020]. [26] ‘European Commission Launches CCAM Single Platform - Connected Automated Driving Europe’, 2019. [Online]. Available: https://connectedautomateddriving.eu/mediaroom/european-commission-launches-ccam-single-platform/. [Accessed: 01-Dec-2020]. [27] ‘Expert group on cooperative, connected, automated and autonomous mobility (E03657)’, 2019. [Online]. Available: https://ec.europa.eu/transparency/regexpert/index.cfm?do=groupDetail.groupDetail&groupID=3657. [Accessed: 01-Dec-2020].

[28] Horizon 2020 Commission Expert Group to advise on specific ethical issues raised by driverless mobility (E03659), ‘Ethics of Connected and Automated Vehicles: recommendations on road safety, privacy, fairness, explainability and responsibility’, European Commission - Directorate-General for Research and Innovation, 2020. [29] European Parliament and Council of the European Union, Regulation (EU) 2016/679 of the European Parliament and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regulation). 2016. [30] European Parliament and Council of the European Union, Directive (EU) 2016/ 1148 of the European Parliament and of the Council of 6 July 2016 concerning measures for a high common level of security of network and information systems across the Union. . [31] ‘Canada’s vehicle cyber security guidance’, Transport Canada, 2020. [32] ACEA, ‘Principles of Automobile Cybersecurity’, 2017. [33] National Highway Traffic Safety Administration (NHTSA), ‘Cybersecurity Best Practices for Modern Vehicles’, U.S. Department of Transportation, 2016. [34] M. Schaub and A. Zhao, ‘China Releases Big Plan for Autonomous Vehicles’, China Law Insight, 2020. [35] Auto-ISAC, ‘Best Practices - Executive Summary’, 2016. [Online]. Available: https://automotiveisac.com/ best-practices/. [Accessed: 01-Dec-2020]. [36] BSI, ‘PAS 1885:2018 The fundamental principles of automotive cyber security. Specification’. 2018. [37] BSI, ‘PAS 11281:2018 Connected automotive ecosystems. Impact of security on safety. Code of practice’. 2018. [38] ETSI, ‘ETSI TS 102 940 - V1.3.1 - Intelligent Tranport Systems (ITS); Security; ITS communications security architecture and security management’. 2018. [39] ETSI, ‘TS 102 941 - V1.2.1 - Intelligent Transport Systems (ITS); Security; Trust and Privacy Management’. 2018. [40] ETSI, ‘TS 102 942 - V1.1.1 - Intelligent Transport Systems (ITS); Security; Access Control’. 2012. [41] ETSI, ‘TS 102 943 - V1.1.1 - Intelligent Transport Systems (ITS); Security; Confidentiality services’. 2012. [42] ‘ECTL - Documentation’. [Online]. Available: https://cpoc.jrc.ec.europa.eu/Documentation.html. [43] SAE, ‘SAE J3061: Cybersecurity Guidebook for Cyber-Physical Vehicle Systems’. 2016. [44] ‘ISO/SAE DIS 21434 - Road vehicles — Cybersecurity engineering’. 2020. [45] ‘SAE J3101: Hardware Protected Security for Ground Vehicles’. . [46] ENISA, ‘Good Practices For Security Of Smart Cars’, ENISA, 2019. [47] ‘Cyber Security and Resilience of smart cars - Good practices and recommendations’, ENISA, 2016. [48] M. Craglia et al., ‘Artificial Intelligence–a European perspective’, European Commission - Joint Research Centre, Science for policy, 2018. [49] European Commission High Level Expert Group on Artificial Intelligence, ‘Ethics Guidelines for Trustworthy AI’. European Commission, 2019. [50] S. D. Pendleton et al., ‘Perception, Planning, Control, and Coordination for Autonomous Vehicles’, Machines, vol. 5, no. 1, p. 6, Mar. 2017. [51] S. Edelstein, ‘What is adaptive cruise control?’, Digital Trends, 2020. [52] Y. Song and C. Liao, ‘Analysis and review of state-of-the-art automatic parking assist system’, in 2016 IEEE International Conference on Vehicular Electronics and Safety (ICVES), 2016, pp. 1–6. [53] J. Zhao, B. Liang, and Q. Chen, ‘The key technology toward the self-driving car’, Int. J. Intell. Unmanned Syst., vol. 6, no. 1, pp. 2–20, Jan. 2018. [54] C. Jensen, ‘Are Blind Spots a Myth?’, Wheels Blog, 2009. . [55] ‘Lane Change Assistance System : a lateral alert system’, Valeo. [Online]. Available: https://www.valeo. com/en/lane-change-assistance-system/. [Accessed: 01-Dec-2020]. [56] A. Goodwin, ‘Lane-keeping assist is preparing you for self-driving cars’, Roadshow, 2017. [57] K. Lim, Y. Hong, Y. Choi, and H. Byun, ‘Real-time traffic sign recognition based on a general purpose GPU and deep-learning’, PLOS ONE, vol. 12, no. 3, p. e0173317, 2017. [58] Z. Wang, Y. Wu, and Q. Niu, ‘Multi-Sensor Fusion in Automated Driving: A Survey’, IEEE Access, vol. 8, pp. 2847–2868, 2020. [59] G. Sharabok, ‘Why Tesla Won’t Use LIDAR’, Medium, 2020. [Online]. Available: https://towardsdatascience.com/why-tesla-wont-use-lidar-57c325ae2ed5. [Accessed: 01-Dec-2020]. [60] A. J. Hawkins, ‘Waymo will sell LIDAR to customers who won’t compete with its robot taxi business’, The Verge, 2019.

[61] V. De Silva, J. Roche, and A. Kondoz, ‘Fusion of LiDAR and camera sensor data for environment sensing in driverless vehicles’, Jan. 2018. [62] S. Campbell et al., ‘Sensor Technology in Autonomous Vehicles : A review’, in 2018 29th Irish Signals and Systems Conference (ISSC), Belfast, 2018, pp. 1–4. [63] ‘DITS - Data set of Italian Traffic Signs’, 2016. [Online]. Available: http://users.diag.uniroma1.it/bloisi/ds/ dits.html. [64] waymo-research/waymo-open-dataset. Waymo Research, 2019. [65] S. J. Russell and P. Norvig, Artificial intelligence: a modern approach, 3rd ed. Pearson, 2009. [66] J. Moor, ‘The Dartmouth College Artificial Intelligence Conference: The Next Fifty Years’, AI Mag., vol. 27, no. 4, p. 87, Dec. 2006. [67] S. Shalev-Shwartz and S. Ben-David, Understanding machine learning: From theory to algorithms. Cambridge University Press, 2014. [68] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, ‘Gradient-based learning applied to document recognition’, Proc. IEEE, vol. 86, no. 11, pp. 2278–2324, 1998. [69] A. Krizhevsky, I. Sutskever, and G. E. Hinton, ‘ImageNet classification with deep convolutional neural networks’, Commun. ACM, vol. 60, no. 6, pp. 84–90, 2017. [70] K. Simonyan and A. Zisserman, ‘Very Deep Convolutional Networks for Large-Scale Image Recognition’, ArXiv14091556 Cs, Apr. 2015. [71] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, ‘Rethinking the Inception architecture for computer vision’, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2818–2826. [72] C. Szegedy et al., ‘Going Deeper with Convolutions’, preprint arxiv: 1409.4842, 2014. [73] K. He, X. Zhang, S. Ren, and J. Sun, ‘Deep Residual Learning for Image Recognition’, in 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778. [74] F. N. Iandola, S. Han, M. W. Moskewicz, K. Ashraf, W. J. Dally, and K. Keutzer, ‘SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size’, 2016. [75] S. Ren, K. He, R. Girshick, and J. Sun, ‘Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks’, IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 6, pp. 1137–1149, 2017. [76] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, ‘You Only Look Once: Unified, Real-Time Object Detection’, 2016. [77] W. Ali, S. Abdelkarim, M. Zahran, M. Zidan, and A. E. Sallab, ‘YOLO3D: End-to-end real-time 3D Oriented Object Bounding Box Detection from LiDAR Point Cloud’, in Proceedings of the European Conference on Computer Vision Workshop, 2018. [78] Y. Zhou and O. Tuzel, ‘VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection’, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 4490–4499. [79] C. R. Qi, H. Su, M. Kaichun, and L. J. Guibas, ‘PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation’, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Honolulu, HI, 2017, pp. 77–85. [80] V. Badrinarayanan, A. Kendall, and R. Cipolla, ‘SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation’, in Proceedings of the IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, vol. 39, pp. 2481–2495. [81] A. Paszke, A. Chaurasia, S. Kim, and E. Culurciello, ‘ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation’, ArXiv160602147 Cs, Jun. 2016. [82] H. Zhao, J. Shi, X. Qi, X. Wang, and J. Jia, ‘Pyramid Scene Parsing Network’, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2881–2890. [83] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, ‘Rethinking Atrous Convolution for Semantic Image Segmentation’, 2017. [84] K. He, G. Gkioxari, P. Dollar, and R. Girshick, ‘Mask R-CNN’, in Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 2961–2969. [85] S. Grigorescu, B. Trasnea, T. Cocias, and G. Macesanu, ‘A Survey of Deep Learning Techniques for Autonomous Driving’, J. Field Robot., 2019. [86] D. Barnes, W. Maddern, G. Pascoe, and I. Posner, ‘Driven to Distraction: Self-Supervised Distractor Learning for Robust Monocular Visual Odometry in Urban Environments’, in Proceedings of the IEEE International Conference on Robotics and Automation, 2018, pp. 1894–1900. [87] I. A. Barsan, S. Wang, A. Pokrovsky, and R. Urtasun, ‘Learning to Localize Using a LiDAR Intensity Map’, in Conference on Robot Learning, 2018, pp. 605–616. [88] W. Luo et al., ‘Multiple Object Tracking: A Literature Review’, 2017.

[89] S. Agarwal, J. O. D. Terrail, and F. Jurie, ‘Recent Advances in Object Detection in the Age of Deep Convolutional Neural Networks’, ArXiv180903193 Cs, 2019. [90] A. K. Ushani and R. M. Eustice, ‘Feature Learning for Scene Flow Estimation from LIDAR’, presented at the Conference on Robot Learning, 2018, pp. 283–292. [91] W. Luo, B. Yang, and R. Urtasun, ‘Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with a Single Convolutional Net’, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, 2018, pp. 3569–3577. [92] C. M. Bishop, Pattern recognition and machine learning. Springer, 2006. [93] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, ‘Learning representations by back-propagating errors’, Nature, vol. 323, no. 6088, pp. 533–536, 1986. [94] S. Hochreiter and J. Schmidhuber, ‘Long short-term memory’, Neural Comput., vol. 9, no. 8, pp. 1735– 1780, 1997. [95] M. Ghallab, D. Nau, and P. Traverso, Automated Planning and Acting. Cambridge University Press, 2016. [96] M. Newman, Networks: An Introduction. New York, NY, USA: Oxford University Press, Inc., 2010. [97] J. Clark and D. A. Holton, A first look at graph theory. World Scientific, 1991. [98] R. S. Sutton and A. G. Barto, Reinforcement Learning: An Introduction. . [99] V. Francois-Lavet, P. Henderson, R. Islam, M. G. Bellemare, and J. Pineau, An Introduction to Deep Reinforcement Learning. 2018. [100] G. Bresson, Z. Alsayed, L. Yu, and S. Glaser, ‘Simultaneous Localization and Mapping: A Survey of Current Trends in Autonomous Driving’, IEEE Trans. Intell. Veh., vol. 2, no. 3, 2017. [101] F. Leon and M. Gavrilescu, ‘A Review of Tracking, Prediction and Decision Making Methods for Autonomous Driving’, preprint arXiv: 1909.07707, 2019. [102] A. Bar Hillel, R. Lerner, D. Levi, and G. Raz, ‘Recent progress in road and lane detection: a survey’, Mach. Vis. Appl., vol. 25, no. 3, pp. 727–745, 2014. [103] [CVPR’20 Workshop on Scalability in Autonomous Driving] Keynote - Andrej Karpathy. 2020. [104] Y. Saadna and A. Behloul, ‘An overview of traffic sign detection and classification methods’, Int. J. Multimed. Inf. Retr., vol. 6, no. 3, pp. 193–210, 2017. [105] Z. Zhu, D. Liang, S. Zhang, X. Huang, B. Li, and S. Hu, ‘Traffic-Sign Detection and Classification in the Wild’, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2110–2118. [106] S. Houben, J. Stallkamp, J. Salmen, M. Schlipsing, and C. Igel, ‘Detection of traffic signs in real-world images: The German traffic sign detection benchmark’, in The 2013 International Joint Conference on Neural Networks (IJCNN), 2013, pp. 1–8. [107] H.-K. Kim, J. H. Park, and H.-Y. Jung, ‘An Efficient Color Space for Deep-Learning Based Traffic Light Recognition’, J. Adv. Transp., vol. 2018, pp. 1–12, Dec. 2018. [108] H.-K. Kim, K.-Y. Yoo, J. H. Park, and H.-Y. Jung, ‘Traffic Light Recognition Based on Binary Semantic Segmentation Network’, Sensors, vol. 19, no. 7, 2019. [109] S. Alvarez, ‘Tesla Autopilot’s stop sign, traffic light recognition and response is operating in “Shadow Mode”’, TESLARATI, 2019. [110] Kai Wang, B. Babenko, and S. Belongie, ‘End-to-end scene text recognition’, in 2011 International Conference on Computer Vision, 2011, pp. 1457–1464. [111] Phuc Xuan Nguyen, K. Wang, and S. Belongie, ‘Video text detection and recognition: Dataset and benchmark’, in Proceedings of the IEEE Winter Conference on Applications of Computer Vision, 2014, pp. 776–783. [112] B. Shi, X. Bai, and C. Yao, ‘An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition’, IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 11, pp. 2298–2304, 2017. [113] D. Karatzas et al., ‘ICDAR 2013 Robust Reading Competition’, in Proceedings of the 12th International Conference on Document Analysis and Recognition, 2013, pp. 1484–1493. [114] S. Reddy, M. Mathew, L. Gomez, M. Rusinol, D. Karatzas., and C. V. Jawahar, ‘RoadText-1K: Text Detection & Recognition Dataset for Driving Videos’, in Proceedings of the International Conference on Robotics and Automation, 2020. [115] Q. Kong, Y. Cao, T. Iqbal, Y. Xu, W. Wang, and M. D. Plumbley, ‘Cross-task learning for audio tagging, sound event detection and spatial localization: DCASE 2019 baseline systems’, preprint arXiv: 1904.03476, 2019. [116] M. K. Nandwana and T. Hasan, ‘Towards Smart-Cars That Can Listen: Abnormal Acoustic Event Detection on the Road’, presented at the Interspeech 2016, 2016, pp. 2968–2971.

[117] B. Paden, M. Čáp, S. Z. Yong, D. Yershov, and E. Frazzoli, ‘A Survey of Motion Planning and Control Techniques for Self-Driving Urban Vehicles’, IEEE Trans. Intell. Veh., vol. 1, no. 1, pp. 33–55, Mar. 2016. [118] J. Levinson, M. Montemerlo, and S. Thrun, ‘Map-Based Precision Vehicle Localization in Urban Environments’, Robot. Sci. Syst., p. 8, 2007. [119] S. Thrun, ‘Probabilistic robotics’, Commun. ACM, vol. 45, no. 3, Mar. 2002. [120] L. A. Marina, B. Trasnea, T. Cocias, A. Vasilcoi, F. Moldoveanu, and S. M. Grigorescu, ‘Deep Grid Net (DGN): A Deep Learning System for Real-Time Driving Context Understanding’, in Proceedings of the 3rd IEEE International Conference on Robotic Computing, 2019, pp. 399–402. [121] C. Seeger, A. Muller, L. Schwarz, and M. Manz, ‘Towards Road Type Classification with Occupancy Grids’, IEEE Intell. Veh. Symp., p. 4, 2016. [122] W. Schwarting, J. Alonso-Mora, and D. Rus, ‘Planning and Decision-Making for Autonomous Vehicles’, Annu. Rev. Control Robot. Auton. Syst., vol. 1, no. 1, pp. 187–210, 2018. [123] E. Yurtsever, J. Lambert, A. Carballo, and K. Takeda, ‘A Survey of Autonomous Driving: Common Practices and Emerging Technologies’, IEEE Access, vol. 8, pp. 58443–58469, 2020. [124] H. Bast et al., ‘Route Planning in Transportation Networks’, in Algorithm Engineering: Selected Results and Surveys, L. Kliemann and P. Sanders, Eds. Cham: Springer International Publishing, 2016, pp. 19–80. [125] Z. Zhuang, J. Wang, Q. Qi, H. Sun, and J. Liao, ‘Toward Greater Intelligence in Route Planning: A Graph- Aware Deep Learning Approach’, IEEE Syst. J., vol. 14, no. 2, pp. 1658–1669, 2020. [126] X. Zhou, M. Su, Z. Liu, Y. Hu, B. Sun, and G. Feng, ‘Smart Tour Route Planning Algorithm Based on Naïve Bayes Interest Data Mining Machine Learning’, ISPRS Int. J. Geo-Inf., vol. 9, no. 2, p. 112, Feb. 2020. [127] M. Kuderer, S. Gulati, and W. Burgard, ‘Learning driving styles for autonomous vehicles from demonstration’, in 2015 IEEE International Conference on Robotics and Automation, 2015, pp. 2641–2646. [128] C. Miyajima and K. Takeda, ‘Driver-Behavior Modeling Using On-Road Driving Data: A new application for behavior signal processing’, IEEE Signal Process. Mag., vol. 33, no. 6, pp. 14–21, 2016. [129] E. Yurtsever, C. Miyajima, S. Selpi, and K. Takeda, ‘Driving signature extraction’, in Proceedings of the 3rd International Symposium on Future Active Safety Technology Toward zero traffic accidents, 2015. [130] E. Yurtsever, C. Miyajima, and K. Takeda, ‘A Traffic Flow Simulation Framework for Learning Driver Heterogeneity from Naturalistic Driving Data using Autoencoders’, Int. J. Automot. Eng., vol. 10, no. 1, pp. 86–93, 2019. [131] B. R. Kiran et al., ‘Deep Reinforcement Learning for Autonomous Driving: A Survey’, preprint arXiv: 2002.00444, 2020. [132] L. Sun, C. Peng, W. Zhan, and M. Tomizuka, ‘A Fast Integrated Planning and Control Framework for Autonomous Driving via Imitation Learning’, presented at the ASME 2018 Dynamic Systems and Control Conference, 2018. [133] M. Bansal, A. Krizhevsky, and A. Ogale, ‘ChauffeurNet: Learning to Drive by Imitating the Best and Synthesizing the Worst’, 2018. [134] A. Kendall et al., ‘Learning to Drive in a Day’, in Proceedings of the International Conference on Robotics and Automation (ICRA), 2019, pp. 8248–8254. [135] X. Pan, Y. You, Z. Wang, and C. Lu, ‘Virtual to Real Reinforcement Learning for Autonomous Driving’, preprint arXiv: 1704.03952, 2017. [136] D. Barnes, W. Maddern, and I. Posner, ‘Find your own way: Weakly-supervised segmentation of path proposals for urban autonomy’, in Proceedings of the IEEE International Conference on Robotics and Automation, 2017, pp. 203–210. [137] L. Caltagirone, M. Bellone, L. Svensson, and M. Wahde, ‘LIDAR-based driving path generation using fully convolutional neural networks’, in Proceedings of the 20th IEEE International Conference on Intelligent Transportation Systems (ITSC), 2017, pp. 1–6. [138] S. H. Park, B. Kim, C. M. Kang, C. C. Chung, and J. W. Choi, ‘Sequence-to-Sequence Prediction of Vehicle Trajectory via LSTM Encoder-Decoder Architecture’, in Proceedings of the IEEE Intelligent Vehicles Symposium (IV), 2018, pp. 1672–1678. [139] D. Isele, R. Rahimi, A. Cosgun, K. Subramanian, and K. Fujimura, ‘Navigating Occluded Intersections with Autonomous Vehicles Using Deep Reinforcement Learning’, in Proceedings of the IEEE International Conference on Robotics and Automation, 2018, pp. 2034–2039. [140] K. K. Tan, Q.-G. Wang, and C. C. Hang, Advances in PID Control. Springer Science & Business Media, 2012. [141] M. Morari, C. E. Garcia, and D. M. Prett, ‘Model predictive control: Theory and practice’, IFAC Proc. Vol., vol. 21, no. 4, pp. 1–12, 1988. [142] J. R. Quain and 2015, ‘BMW 7 Series Tested: Here’s The Tech You Get for $100K’, Tom’s Guide. [Online]. Available: https://www.tomsguide.com/us/bmw-7-series-tested,review-3040.html. [Accessed: 01-Dec-2020].

[143] Toyota Motor Corporation, ‘Toyota’s New “LQ” Wants to Build an Emotional Bond with Its Driver’. 2019. [144] M. Bojarski et al., ‘End to End Learning for Self-Driving Cars’, 2016. [145] T. P. Lillicrap et al., ‘Continuous control with deep reinforcement learning’, 2019. [146] H. Xu, Y. Gao, F. Yu, and T. Darrell, ‘End-To-End Learning of Driving Models From Large-Scale Video Datasets’, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2174–2182. [147] B. Wymann, C. Dimitrakakis, A. Sumner, E. Espie, and C. Guionneau, ‘TORCS: The open racing car simulator’, p. 5, 2015. [148] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, ‘CARLA: An Open Urban Driving Simulator’, preprint arXiv: 1711.03938, 2017. [149] S. Shah, D. Dey, C. Lovett, and A. Kapoor, ‘Aerial Informatics and Robotics Platform’, Technical Report. [150] G. Rong et al., ‘LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving’, preprint arxiv: 2005.03778, 2020. [151] S. Manivasagam et al., ‘LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World’, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020. [152] L. Fridman, J. Terwilliger, and B. Jenik, ‘DeepTraffic: Crowdsourced Hyperparameter Tuning of Deep Reinforcement Learning Systems for Multi-Agent Dense Traffic Navigation’, presented at the Neural Information Processing Systems (NIPS 2018) Deep Reinforcement Learning Workshop, 2019. [153] Ram Shankar Siva Kumar, David O’Brien, Jeffrey Snover, Kendra Albert, and Salome Viljoen, ‘Failure Modes in Machine Learning - Security documentation’. [Online]. Available: https://docs.microsoft.com/ en-us/security/engineering/failure-modes-in-machine-learning. [Accessed: 26-May-2020]. [154] N. Dalvi, P. Domingos, S. Sanghai, D. Verma, and others, ‘Adversarial classification’, in Proceedings of the 10th ACM international conference on Knowledge discovery and data mining, 2004, pp. 99–108. [155] L. Huang, A. D. Joseph, B. Nelson, B. I. Rubinstein, and J. Tygar, ‘Adversarial machine learning’, in Proceedings of the 4th ACM workshop on Security and artificial intelligence, 2011, pp. 43–58. [156] P. A. Johnson, B. Tan, and S. Schuckers, ‘Multimodal fusion vulnerability to non-zero effort (spoof) imposters’, in Proceedings of the IEEE International Workshop on Information Forensics and Security, 2010, pp. 1–5. [157] A. Adler, ‘Vulnerabilities in Biometric Encryption Systems’, in Proceedings of the International Conference on Audio- and Video-Based Biometric Person Authentication, Berlin, Heidelberg, 2005, pp. 1100–1109. [158] P. Fogla, M. Sharif, R. Perdisci, O. Kolesnikov, and W. Lee, ‘Polymorphic Blending Attacks’, presented at the {USENIX} Security Symposium, 2006, p. 16. [159] C.-H. Huang, T.-H. Lee, L. Chang, J.-R. Lin, and G. Horng, ‘Adversarial Attacks on SDN-Based Deep Learning IDS System’, in Proceedings of the International Conference on Mobile and Wireless Technology (ICMWT), Singapore, 2019, pp. 181–191. [160] D. Lowd, ‘Good word attacks on statistical spam filters’, in Proceedings of the 2nd Conference on Email and Anti-Spam, 2005. [161] A. Kołcz and C. H. Teo, ‘Feature Weighting for Improved Classifier Robustness’, in Proceedings of the 6th Conference on Email and Anti-Spam, 2009, p. 8. [162] B. Biggio, B. Nelson, and P. Laskov, ‘Poisoning attacks against support vector machines’, in Proceedings of the 29th International Conference on International Conference on Machine Learning, 2012, pp. 1467–1474. [163] M. Sharif, S. Bhagavatula, L. Bauer, and M. K. Reiter, ‘A General Framework for Adversarial Examples with Objectives’, ACM Trans. Priv. Secur., vol. 22, no. 3, pp. 16:1–16:30, 2019. [164] N. Carlini et al., ‘Hidden Voice Commands’, in Proceedings of the 25th {USENIX} Security Symposium, 2016, pp. 513–530. [165] R. Jia and P. Liang, ‘Adversarial Examples for Evaluating Reading Comprehension Systems’, in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017. [166] I. J. Goodfellow, J. Shlens, and C. Szegedy, ‘Explaining and harnessing adversarial examples’, in Proceedings of the International conference on learning representations, 2015. [167] C. Szegedy et al., ‘Intriguing properties of neural networks’, in Proceedings of the International Conference on Learning Representations, 2014. [168] M. Großhans, C. Sawade, M. Brückner, and T. Scheffer, ‘Bayesian Games for Adversarial Regression Problems’, in Proceedings of the International Conference on Machine Learning, 2013, pp. 55--63. [169] C. Liu, B. Li, Y. Vorobeychik, and A. Oprea, ‘Robust Linear Regression Against Training Data Poisoning’, in Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, New York, NY, USA, 2017, pp. 91–102.

[170] B. Biggio, G. Fumera, and F. Roli, ‘Multiple Classifier Systems under Attack’, in Proceedings of the International Workshop on Multiple Classifier Systems, Berlin, Heidelberg, 2010, pp. 74–83. [171] B. Biggio, I. Pillai, S. Rota Bulò, D. Ariu, M. Pelillo, and F. Roli, ‘Is data clustering in adversarial settings secure?’, in Proceedings of the ACM workshop on Artificial intelligence and security, New York, NY, USA, 2013, pp. 87–98. [172] B. Biggio et al., ‘Poisoning Complete-Linkage Hierarchical Clustering’, in Proceedings of the Workshop on Syntactic Pattern Recognition, Berlin, Heidelberg, 2014, pp. 42–52. [173] J. G. Dutrisac and D. B. Skillicorn, ‘Hiding clusters in adversarial settings’, in Proceedings of the IEEE International Conference on Intelligence and Security Informatics, 2008, pp. 185–187. [174] N. Narodytska and S. Kasiviswanathan, ‘Simple Black-Box Adversarial Attacks on Deep Neural Networks’, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017, pp. 1310–1318. [175] S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, ‘DeepFool: a simple and accurate method to fool deep neural networks’, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2574–2582. [176] L. Muñoz-González et al., ‘Towards Poisoning of Deep Learning Algorithms with Back-gradient Optimization’, in Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, New York, NY, USA, 2017, pp. 27–38. [177] N. Papernot, P. McDaniel, A. Swami, and R. Harang, ‘Crafting adversarial input sequences for recurrent neural networks’, in Proceedings of the IEEE Military Communications Conference, 2016, pp. 49–54. [178] T. Gu, B. Dolan-Gavitt, and S. Garg, ‘BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain’, preprint arXiv: 1708.06733, 2019. [179] W. Uther and M. Veloso, ‘Adversarial Reinforcement Learning’, preprint, 1997. [180] Y.-C. Lin, Z.-W. Hong, Y.-H. Liao, M.-L. Shih, M.-Y. Liu, and M. Sun, ‘Tactics of adversarial attack on deep reinforcement learning agents’, in Proceedings of the 26th International Joint Conference on Artificial Intelligence, Melbourne, Australia, 2017, pp. 3756–3762. [181] C. Xiao et al., ‘Characterizing Attacks on Deep Reinforcement Learning’, preprint arxiv: 1907.09470, 2019. [182] V. Behzadan and A. Munir, ‘Vulnerability of Deep Reinforcement Learning to Policy Induction Attacks’, in Proceedings of the International Conference on Machine Learning and Data Mining in Pattern Recognition, 2017, pp. 262–275. [183] S. Li et al., ‘Stealthy Adversarial Perturbations Against Real-Time Video Classification Systems’, in Proceedings of the Network and Distributed Systems Security Symposium, 2019, p. 15. [184] A. Kurakin, I. J. Goodfellow, and S. Bengio, ‘Adversarial machine learning at scale’, in Proceedings of the International Conference on Learning Representations, 2016. [185] O. Russakovsky et al., ‘ImageNet Large Scale Visual Recognition Challenge’, Int. J. Comput. Vis. IJCV, vol. 115, no. 3, pp. 211–252, 2015. [186] S. Qiu, Q. Liu, S. Zhou, and C. Wu, ‘Review of Artificial Intelligence Adversarial Attack and Defense Technologies’, Appl. Sci., vol. 9, p. 909, 2019. [187] E. Tabassi, K. J. Burns, M. Hadjimichael, A. D. Molina-Markham, and J. T. Sexton, ‘A taxonomy and terminology of adversarial machine learning’, Draft NISTIR 8269, 2019. [188] B. Biggio and F. Roli, ‘Wild patterns: Ten years after the rise of adversarial machine learning’, Pattern Recognit., vol. 84, pp. 317–331, 2018. [189] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, ‘Towards deep learning models resistant to adversarial attacks’, in International conference on learning representations, 2018. [190] S. Bubeck, ‘Convex Optimization: Algorithms and Complexity’, Found. Trends® Mach. Learn., vol. 8, no. 3–4, pp. 231–357, 2015. [191] I. J. Goodfellow, J. Shlens, and C. Szegedy, ‘Explaining and harnessing adversarial examples’, in Proceedings of the International conference on learning representations, 2015. [192] Y. Dong et al., ‘Boosting Adversarial Attacks With Momentum’, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 9185–9193. [193] T. Miyato, S.-I. Maeda, M. Koyama, and S. Ishii, ‘Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning’, IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 8, pp. 1979–1993, 2019. [194] N. Papernot, P. McDaniel, S. Jha, M. Fredrikson, Z. B. Celik, and A. Swami, ‘The limitations of deep learning in adversarial settings’, in Proceedings of IEEE European Symposium on Security and Privacy, 2016, pp. 372–387.

[195] N. Carlini and D. Wagner, ‘Towards evaluating the robustness of neural networks’, in Proceedings of the IEEE Symposium on Security and Privacy, 2017. [196] J. Su, D. V. Vargas, and K. Sakurai, ‘One pixel attack for fooling deep neural networks’, IEEE Trans. Evol. Comput., 2019. [197] M. M. Cisse, Y. Adi, N. Neverova, and J. Keshet, ‘Houdini: Fooling Deep Structured Visual and Speech Recognition Models with Adversarial Examples’, in Proceedings of the Advances in Neural Information Processing Systems, 2017, vol. 30, pp. 6977–6987. [198] P.-Y. Chen, H. Zhang, Y. Sharma, J. Yi, and C.-J. Hsieh, ‘ZOO: Zeroth Order Optimization Based Black-box Attacks to Deep Neural Networks without Training Substitute Models’, in Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, New York, NY, USA, 2017, pp. 15–26. [199] Z. Liu, P. Luo, X. Wang, and X. Tang, ‘Deep Learning Face Attributes in the Wild’, in Proceedings of the IEEE International Conference on Computer Vision, 2015, pp. 3730–3738. [200] Y. Song, R. Shu, N. Kushman, and S. Ermon, ‘Constructing Unrestricted Adversarial Examples with Generative Models’, in Proceedings of the Advances in Neural Information Processing Systems, 2018, vol. 31, pp. 8312–8323. [201] T. Hwang, ‘Deepfakes - A Grounded Threat’, CSET, 2020. [202] Y. Cao et al., ‘Adversarial Sensor Attack on LiDAR-based Perception in Autonomous Driving’, in Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, New York, NY, USA, 2019, pp. 2267–2281. [203] Y. Cao et al., ‘Adversarial Objects Against LiDAR-Based Autonomous Driving Systems’, preprint arXiv: 1907.05418, 2019. [204] J. Sun, Y. Cao, Q. A. Chen, and Z. M. Mao, ‘Towards Robust LiDAR-based Perception in Autonomous Driving: General Black-box Adversarial Sensor Attack and Countermeasures’, presented at the 29th {USE- NIX} Security Symposium ({USENIX} Security 20), 2020, pp. 877–894. [205] C. Yan, W. Xu, and J. Liu, ‘Can You Trust Autonomous Vehicles: Contactless Attacks against Sensors of Self-driving Vehicle’, presented at the DEF CON, 2016, vol. 24. [206] W. Xu, C. Yan, W. Jia, X. Ji, and J. Liu, ‘Analyzing and Enhancing the Security of Ultrasonic Sensors for Autonomous Vehicles’, IEEE Internet Things J., vol. 5, no. 6, pp. 5015–5029, 2018. [207] A. Hamdi, S. Rojas, A. Thabet, and B. Ghanem, ‘AdvPC: Transferable Adversarial Perturbations on 3D Point Clouds’, in Proceedings of the European Conference on Computer Vision, 2020, pp. 241–257. [208] M. Sharif, S. Bhagavatula, L. Bauer, and M. K. Reiter, ‘A General Framework for Adversarial Examples with Objectives’, ACM Trans. Priv. Secur., vol. 22, no. 3, pp. 16:1–16:30, 2019. [209] K. Eykholt et al., ‘Robust Physical-World Attacks on Deep Learning Visual Classification’, in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 1625–1634. [210] Y. Luo, X. Boix, G. Roig, T. Poggio, and Q. Zhao, ‘Foveation-based Mechanisms Alleviate Adversarial Examples’, preprint arXiv: 1511.06292, 2016. [211] A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok, ‘Synthesizing robust adversarial examples’, in Proceedings of the International Conference on Machine Learning, 2018, pp. 284–293. [212] C. Berghoff, M. Neu, and A. von Twickel, ‘Vulnerabilities of Connectionist AI Applications: Evaluation and Defense’, Front. Big Data, vol. 3, 2020. [213] R. Huang, B. Xu, D. Schuurmans, and C. Szepesvari, ‘Learning with a Strong Adversary’, preprint arXiv: 1511.03034, 2016. [214] H. Hosseini, Y. Chen, S. Kannan, B. Zhang, and R. Poovendran, ‘Blocking Transferability of Adversarial Examples in Black-Box Learning Systems’, preprint arXiv: 1703.04318, 2017. [215] B. Biggio et al., ‘One-and-a-Half-Class Multiple Classifier Systems for Secure Learning Against Evasion Attacks at Test Time’, in Proceedings of the International Workshop on Multiple Classifier Systems, 2015, pp. 168–180. [216] D. Meng and H. Chen, ‘MagNet: A Two-Pronged Defense against Adversarial Examples’, in Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security, New York, NY, USA, 2017, pp. 135–147. [217] P. Samangouei, M. Kabkab, and R. Chellappa, ‘Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models’, in Proceedings of the International Conference on Learning Representations, 2018. [218] C. Lyu, K. Huang, and H.-N. Liang, ‘A Unified Gradient Regularization Family for Adversarial Examples’, in Proceedings of the IEEE International Conference on Data Mining, 2015, pp. 301–309. [219] Q. Zhao and L. D. Griffin, ‘Suppressing the Unusual: towards Robust CNNs using Symmetric Activation Functions’, preprint arXiv: 1603.05145, 2016.

[220] N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami, ‘Distillation as a defense to adversarial perturbations against deep neural networks’, in 2016 IEEE Symposium on Security and Privacy, 2016, pp. 582–597. [221] C. Qin et al., ‘Verification of non-linear specifications for neural networks’, in Proceedings of the international conference on learning representations (ICLR), 2019. [222] J. M. Cohen, E. Rosenfeld, and J. Z. Kolter, ‘Certified Adversarial Robustness via Randomized Smoothing’, ArXiv190202918 Cs Stat, 2019. [223] P. Wild, P. Radu, L. Chen, and J. Ferryman, ‘Robust multimodal face and fingerprint fusion in the presence of spoofing attacks’, Pattern Recognit., vol. 50, pp. 17–25, 2016. [224] B. Biggio, G. Fumera, G. L. Marcialis, and F. Roli, ‘Statistical Meta-Analysis of Presentation Attacks for Secure Multibiometric Systems’, IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 3, pp. 561–575, 2017. [225] C. Sitawarin, A. Nitin Bhagoji, A. Mosenia, M. Chiang, and P. Mittal, ‘DARTS: Deceiving Autonomous Cars with Toxic Signs’, preprint arxiv: 1802.06430, 2018. [226] N. Morgulis, A. Kreines, S. Mendelowitz, and Y. Weisglass, ‘Fooling a Real Car with Adversarial Traffic Signs’, preprint arxiv: 1907.00374, 2019. [227] S. Povolny and S. Trivedi, ‘Model Hacking ADAS to Pave Safer Roads for Autonomous Vehicles’, McAfee Blogs, 2020. [228] D. Nassi, R. Ben-Netanel, Y. Elovici, and B. Nassi, ‘MobilBye: Attacking ADAS with Camera Spoofing’, preprint arxiv: 1906.09765, 2019. [229] Tencent Keen Security Lab, ‘Experimental security research of Tesla autopilot’, 2019. [230] A. Chernikova, A. Oprea, C. Nita-Rotaru, and B. Kim, ‘Are Self-Driving Cars Secure? Evasion Attacks Against Deep Neural Networks for Steering Angle Prediction’, in Proceedings of the IEEE Security and Privacy Workshops, 2019, pp. 132–137. [231] G. Jocher et al., ultralytics/yolov5: v3.1 - Bug Fixes and Performance Improvements. Ultralytics LLC, 2020. [232] H. Kume, ‘Tesla teardown finds electronics 6 years ahead of Toyota and VW’, Nikkei Asia, 2020. [233] J. Wolff, ‘How to improve cybersecurity for artificial intelligence’, 2020. [234] M. Alonso Raposo et al., ‘The future of road transport’, EUR 29748 EN, Publications Office of the European Union, Luxembourg, ISBN 978-92-76-14318-5, doi:10.2760/668964, 2019. [235] I. Nai Fovino et al., ‘Cybersecurity, our digital anchor’, EUR 30276 EN, Publications Office of the European Union, Luxembourg, ISBN 978-92-76-19957-1, doi:10.2760/352218, 2020.

KJ-NA-30568-EN-N ABOUT ENISA ABOUT JRC enisa.europa.eu ec.europa.eu/jrc