Artificial Intelligence Cybersecurity Challenges

Fotnoter

1. 2 Evidently white box models are also susceptible to cyber attacks because adversaries have widely accessible information to tailor attacks.
2. 3 This refers to both physical attacks on AI systems as well as robustness of AI systems against naturally occurring variations and events. 4 See EC White Paper on Ai under consultation at: https://ec.europa.eu/info/files/white-paper-artificial-intelligenceeuropean-approach-excellence-and-trust_en
3. 5 In doing so, ENISA will take stock of existing initiatives and studies that are ongoing in the area of AI cybersecurity such as the results of EU projects in this area (H2020) and will avoid duplication of efforts, rather focus on provide harmonized view of ongoing works. 6 See https://ec.europa.eu/digital-single-market/en/news/coordinated-plan-artificial-intelligence 7 See https://ec.europa.eu/info/sites/info/files/communication-eu-security-union-strategy.pdf 8 See https://ec.europa.eu/digital-single-market/en/high-level-expert-group-artificial-intelligence See http://ec.europa.eu/digital-single-market/en/artificial-intelligence 10 See https://ec.europa.eu/digital-single-market/en/news/policy-and-investment-recommendations-trustworthy-artificialintelligence 11 See https://ec.europa.eu/futurium/en/ai-alliance-consultation/guidelines 12 See https://ec.europa.eu/digital-single-market/en/news/assessment-list-trustworthy-artificial-intelligence-altai-selfassessment 13 See https://www.eda.europa.eu/info-hub/press-centre/latest-news/2020/08/25/artificial-intelligence-joint-quest-for-futuredefence-applications 14 See https://www.etsi.org/technologies/securing-artificial-intelligence 15 See https://ec.europa.eu/jrc/en/publication/ai-watch-defining-artificial-intelligence
4. 16 This applies both at the time of the determination of the means for processing and at the time of the processing itself. 17 Lawrence Gordon e Martin Loeb, The Economics of Information Security Investment, in ACM Transactions on Information and System Security, vol. 5, n. 4, November 2002, pp. 438–457, and Lawrence Gordon e Martin Loeb, You May Be Fighting the Wrong Security Battles, in the Wall Street Journal, 26 September 2011 18 Such incidents may lead to severe risks that can compromise and impact data protection and individuals’ fundamental rights and freedoms.
5. 19 See https://ec.europa.eu/futurium/en/ai-alliance-consultation/guidelines, April 2019 20 Due consideration to existing and ongoing pieces of work on cybersecurity for AI such as the ones by ETSI, ISC/IEC JTC 1/SC42, etc. in an effort to avoid duplication and base the work on already established knowledge sources.
6. 21 See https://www.iso.org/standard/75281.html 22 The the identification and analysis of assets and cyber threats are based on work conducted by ENISA and the ENISA Ad hoc Working Group on AI (ahWGAI) based on the study of specifications, white papers and literature, without attempting any interpretation/evaluation of the assumptions stated in these reports. 23 See https://www.enisa.europa.eu/topics/iot-and-smart-infrastructures/artificial_intelligence/adhoc_wg_calls 24 See https://ec.europa.eu/digital-single-market/en/news/definition-artificial-intelligence-main-capabilities-and-scientificdisciplines 25 See https://publications.jrc.ec.europa.eu/repository/bitstream/JRC118163/jrc118163_ai_watch._defining_artificial_intelligence_1 .pdf and https://publications.jrc.ec.europa.eu/repository/bitstream/JRC113826/ai-flagship-report-online.pdf 26 See https://www.eda.europa.eu/info-hub/press-centre/latest-news/2020/08/25/artificial-intelligence-joint-quest-for-futuredefence-applications 27 See https://csrc.nist.gov/publications/detail/nistir/8269/draft 28 See https://www.etsi.org/images/files/ETSIWhitePapers/etsi_wp34_Artificial_Intellignce_and_future_directions_for_ETSI.pdf 29 See https://www.stiftung-nv.de/de/publikation/securing-artificial-intelligence 30 See https://github.com/mitre/advmlthreatmatrix 31 See ISO/IEC JTC 1/SC 42: https://www.iso.org/committee/6794475/x/catalogue/p/0/u/1/w/0/d/0
7. 32 In this document we consider the data sources for AI have been protected and are considered to be secure. In our approach, the AI application life cycle (in short, AI Lifecycle) is considered as a generic model for the foundation of assets and threats identification, and not intended as a statement. Feedback loops presented are not exhaustive as different use cases might follow different pipelines. and omit some of the phases of the generic life cyle.. Mind maps were included as a first step towards a complete reference model. 33 See https://ec.europa.eu/digital-single-market/en/news/assessment-list-trustworthy-artificial-intelligence-altai-selfassessment 34 The threat landscape assumes basic understanding of AI terminology and concepts. For further details and to gain deeper understanding the interested reader is referred to AI tectbook material. 35 Including already referenced work from EC JRC, EC AI HLEG, EDA, ETSI ISG SAI, NIST, Stiftung Neue Verantwortung, Mcrosoft (https://docs.microsoft.com/en-us/security/engineering/threat-modeling-aiml), Berryville Institute of Machine Learning (https://berryvilleiml.com /) and BSI (https://doi.org/10.3389/fdata.2020.00023 ).
8. 36 In terms of data categories and the provenance of the data we distinguish between the following. Self-reported data, voluntarily provided by a “trustful” operator (e.g. AIS for a ship or ADS-B for an aircraft, cooperative and governmental data). Observed data collected by active or passive “secure” systems (e.g. IDSs, sensors, RFIDs, cameras, IoTs in general, radars), the integrity of the data depends upon a variety of parameters (resolution, range, refresh, latency, environmental conditions, size, orientation, electromagnetic characteristics). Information registers and databases: they contain information linking data (aircrafts or ship IDs, human IDs from civil legacy systems, smart objects IDs from industries) with details on its structure, construction, appearance, history and interactions, activity, Social Media from Free and Open Internet Sources (e.g Twitter, Youtube, Facebook, WhatsApp, Media, Open DB) are also included in this category.
9. 37 The reference model detals the typical, different phases of the AI lifecycle. A noteworthy reference needs to be made to automated machine learning solutions (offered by several vendors) that encompass the vast majority of the AI lifecycle stages to facilitate product developers. Despite numerous research and commercial initiatives for developing efficient automated machine learning mechanisms and tools, many challenges have been identified including transparency issues (black-box operation), limited reproducibility, etc. 38 Please note that in the case of personal data the role of data owners is equivalent to that of data controllers. 39 Evidently if such cases occur, then there is a clear lack of compliance to GDPR provisions and further legal analysis (outside the scope of this work) is highly recommended.
10. 40 The discussion mainly refers to nunerical, tabular data. It needs nevertheless to be mentioned that AI systems may also use other types of data, e.g. speech, images. These are also numerical, but sanity checks have an advanced degree of complexity, for which no data exploration as described here is performed. 41 Multimedia data are complex data that are very relevant in the context of deep learning. 42 Re-scaling is used to make sure all variables are expressed on the same scale, as some methods may overlook variables with lower intensity. Standardization is used to change the mean of a distribution of values to 0, while normalization maps data to a compact representation interval (e.g., the interval (0, 1), by dividing all values by the maximum). Labelling (done by human experts or by other AI applications) associates each data item to a category or a prediction.
11. 43 Machine Learning Models are algorithms trained with historical data that discover patterns and relations, and construct mathematical models using these discoveries. 44 It is noteworthy that this is not always the case. In particular, in recent deep learning approaches that consider end-toend deep learning approaches, where no feature processing is done. 45 Stuart J. Russell and Peter Norvig, “Artificial Intelligence: A Modern Approach”, Prentice Hall Press. ISBN:978-0-13- 604259-4 46 By composition of methods we refer to model ensembling that consists in combining the outputs of multiple models to take advantage of the advantages of different approaches, at the cost of a greater complexity.
12. 47 In deep learning where possibly highly complex loss functions are designed, and are a key element of the training process.
13. 48 Re-tuning of hyper-paratameters is often a challenging task given that the space of hyper-parameters is usually immense and the process requires a large amount of time and computing resources. Moreover, it needs to be noted that this type of tuning requires frequent re-training of the model. 49 It should be noted that this process is very expensive computationally, and tends to be limited, especially in deep learning applications where training may take days or weeks.
14. 50 See https://stiftung-nv.de/ml-supplychain
15. 51 See https://www.idgconnect.com/news/1506124/deepfakes-ai-deceives , September 2020 52 See https://thenewstack.io/camouflaged-graffiti-road-signs-can-fool-machine-learning-models/, September 2017 53 See https://www.media.mit.edu/publications/adversarial-attacks-on-medical-machine-learning/, March 2019 54 Given the broad nature of AI systems and their deployment in diverse sectors, the listing is generic and does not imply a ranking of the likelihood of a threat actor to attack AI systems. 55 With AI-as-a-service gaining traction, such systems will be increasingly available to non-technical savvy actors .
16. 56 See, Sailio, M.; Latvala, O.-M.; Szanto, A. Cyber Threat Actors for the Factory of the Future. Appl. Sci. 2020, 10, 4334 https://www.mdpi.com/2076-3417/10/12/4334/htm 57 See Shostack, Adam (2014). "Threat Modeling: Designing for Security". John Wiley & Sons Inc: Indianapolis. 58 See https://docs.microsoft.com/en-us/previous-versions/commerce-server/ee823878(v=cs.20). STRIDE is an acronym that stands for 6 threat categories, namely Spoofing, Tampering with data, Repudiation, Information disclosure, Denial of Service and Elevation of privilege. 59 Vulnerability identification has not been extensively explored in this report given that specific use cases need to be considered in order to perform this step. Since the report aims to present the AI threat landscape in a domain agnostic manner, vulnerability identification will be pursued in more detail in further works. 60 The AI specific security properties have been based on the work of the EC AI HLEG on assessment list for trustworthy AI: https://ec.europa.eu/digital-single-market/en/news/assessment-list-trustworthy-artificial-intelligence-altai-selfassessment
17. 61 See https://link.springer.com/article/10.1186/s40887-018-0025-1 62 See https://www.enisa.europa.eu/topics/threat-risk-management/threats-and-trends/enisa-threat-landscape/threattaxonomy/view
18. 63 See for example https://www.enisa.europa.eu/publications/cloud-computing-risk-assessment, https://www.enisa.europa.eu/publications/enisa-threat-landscape-for-5g-networks and https://www.enisa.europa.eu/publications/baseline-security-recommendations-for-iot
19. 64 Moreover, the impact of AI cannot be totally anticipated, which put us in the position of, maybe, rewriting all the rules that we have used so far. 65 See . https://www.europol.europa.eu/newsroom/news/new-report-finds-criminals-leverage-ai-for-malicious-use- %E2%80%93-and-it%E2%80%99s-not-just-deep-fakes
20. An augmented data set is a (usually labeled) data set which has been augmented by adding data produced by transformations or by generative ML models. Augmentation significantly increases labeled data sets’ - Data pre- Augmented Data Set diversity (which is supposed to prevent overfitting) in view processing of using augmented datasets for training other ML models. In image recognition, data augmentation techniques include cropping, padding, and horizontal flipping.
21. The evaluation data is used to evaluate the predictive quality of the trained model. The ML system evaluates Evaluation Data predictive performance by comparing predictions on the - Model Tuning evaluation data set with true values (known as ground truth) using a variety of metrics.
22. The term “Labeled Data” refers to a set of scalar or multidimensional data items that have been tagged with one or - Data pre-
23. Labeled Data Set
24. more informative labels, usually for the purpose of training processing a supervised ML model.
25. The sorts of numbers we collect when we measure - Feature Metric Data Set something. Metric data can be ratio scale, interval scale, Selection integer scale and cardinal numbers.
26. Pre-processed Data The data is pre-processed before feeding it into our ML - Data pre- Set model. processing
27. Public data set is information that can be freely used, reused and redistributed by anyone with no existing local, - Data Exploration
28. Public Data set
29. national or international legal restrictions on access or - Data Ingestion usage.
30. Raw data refers to any type of information gathered for AI Raw Data analysis purposes, possibly after cleaning but before it is - Data Ingestion transformed or analyzed in any way.
31. It is a dataset used to provide an unbiased evaluation of a final ML model fitted on the training dataset. We use testing data to test the model. If the data in the test Testing Data - Model Training dataset has never been used in training (e.g. in crossvalidation), the test dataset is also called a holdout dataset.
32. - Model Selection / Building Training data refers to the initial data that is used to Training Data develop a ML model, from which the model adapts its - Model Training internal parameters to refine its rules. - Transfer Learning
33. Validation data sets are labelled data sets, which differ from ordinary labelled data sets only in their usage and, usually, in their collection circumstances. Validation data sets are mostly used to perform an evaluation of a ML Validation Data Set - Model Tuning model in-training, for example by stopping the ML model’s training (early stopping) when the error on the validation dataset increases too much, as this is considered a sign of overfitting the model to the training dataset.
34. ML algorithms are programs (math and logic) that adjust themselves to perform better as they are exposed to more data. The “learning” part of ML means that those programs change how they process data over time, much - Model Training Algorithms as humans change how they process data by learning. So a ML algorithm is a program with a specific way to adjusting its own parameters, given feedback on its previous performance in making predictions about a dataset.
35. The data pre-processing employs techniques to clean, integrate and transform the data, resulting in an improved data quality that will improve performance and efficiency Data Pre-Processing by saving time during the analytic models’ training phase - Data pre- Algorithms and by promoting a better quality of results. Specifically, processing the term data cleaning designates techniques to correct inconsistencies, remove noise and anonymize/pseudonymize data.
36. Hyper-parameters define high-level concepts about ML models, such as the frequency of the adjustment of the Hyper-parameters internal parameters on the part of the training algorithm. - Model Tuning They cannot be learnt from input data but need to be set by trial-and-error using model space search techniques.
37. Training algorithms are procedures for adjusting the Models parameters of ML models. In supervised training, the correct output for each input vector of a training set is - Model Selection presented to the model, and multiple iterations through / Building
38. Training Algorithms
39. the training data may be required to adjust the parameters. In unsupervised training, the model parameters are adjusted without specifying the correct output for any of the input vectors.
40. - Model Training
41. - Model Tuning
42. - Model Selection The term ML model designates computer algorithms / Building Model implementing parametric mathematical models that improve through experience. - Model Deployment
43. - Model Maintenance
44. A model parameter is a configuration variable that is Model parameters internal to the model and whose value can be estimated - Model Training from the given data.
45. The ML model performance is the accuracy and speed of the model’s computation that receives inputs from the - Model Training
46. Model performance
47. production-ready environment and outputs the model’ - Model Tuning classifications, predictions or decisions.
48. Feature selection algorithms are techniques that select a subset of relevant features from an original feature set, in Subspace (Feature) order to increase the performance of ML models trained - Feature selection Algorithm on the subset. Some feature selection methods used for Selection classification problems are supervised and use class labels as a guide.
49. A trained ML model is a model whose internal parameters - Model Training have been adjusted by training to reach a minimum of the Trained models - Transfer error function that defines the distance between the actual and expected outputs. Learning
50. ML model training parameters are quantities adjusted by - Model Selection the learning process by applying training algorithms based / Building Training parameters on training data. Training parameters values determine the actual classification, prediction or detection function - Model training computed by the ML model.
51. A tuned ML model is a model where the hyper-parameters affecting the training algorithm operation have been set to Tuned Model - Model Tuning maximize the convergence and speed of the training algorithm.
52. Cloud providers are third parties that offer computational platforms, and even in some cases tend to offer some - Data Ingestion data analyses capabilities or “Machine learning as Cloud Provider service” (for this please check Model Provider threats - Model training below). Cloud provider adds to AI the same attack vectors - Model tuning as to other domains: data breaches, insufficient authentication and authorization, insecure interfaces, etc.
53. - Data Ingestion
54. - Data Exploration
55. - Data pre- Data Engineers are professionals that prepare the processing computational infrastructure and mainly focus on the design, management and optimization of the flow of data. - Feature They’re usually more intervenient in the first stages of AI Selection Lifecycle: extraction and assembling of data from different - Model Selection Data Engineers sources, transformation, cleaning and loading it in a Actors / Building standardized format and in an adequate repository. Data Engineers must have specialized skills in creating - Model Training software solutions around data: software engineering, - Model Tuning distributed systems, open frameworks, SQL, Cloud platforms, data modelling. - Model Deployment
56. - Model Maintenance
57. - Business Goal Data owner can be a data broker or provider, as Definition Data Owner described before, or the business owner, who asks for the - Data Ingestion AI study. In this section, the focus is on the latter. - Data Exploration
58. Data Provider/Data
59. Third parties’ providing data for the AI process. - Data Ingestion
60. - Business Goal Definition
61. - Data Ingestion
62. - Data Exploration
63. - Data preprocessing
64. - Feature Professionals that apply statistics, Machine Learning and Selection analytic approaches to analyse different datasets of
65. Data Scientists / AI
66. different sizes and shapes and solve complex and critical - Model Selection
67. problems. Skills in computer science fundamentals and / Building
68. programming, including experience with languages and - Model Training database (big/small) technologies are essential. - Model Tuning
69. - Transfer Learning
70. - Model Deployment
71. - Model Maintenance
72. - Business Goal Definition
73. - Data Ingestion
74. Those inside an organization that use and benefit from the - Data Exploration
75. End Users
76. results provided by the AI/ML system/service. - Model Maintenance
77. - Business Understanding
78. In the context of transfer and/or federated learning, third parties that provide models (called as “Teacher” models), previously trained and fine-tuned with large datasets that - Transfer
79. Model provider
80. are useful to learn from small datasets and/or by Learning organizations without access to high computational clusters, with GPU.
81. - Model Service consumers / AI/ML users that rely on pre-trained models, or consume Maintenance Model users them through available services. - Business Understanding
82. Techniques used to increase the amount of data by adding slightly modified copies of already existing data or newly created synthetic data from existing data. It helps - Data prereduce overfitting when training a machine learning. Data processing
83. Data augmentation
84. augmentation usually consists in applying transformations which are known to be label-preserving, i.e. the model - Data Exploration
85. should not change its prediction when presented with the transformed data items.
86. It is the process of gathering and measuring information Data Collection on specific variables (needed for the AI system) from - Data Ingestion countless different sources.
87. Data - Data Exploration Exploration/Pre- Understanding, preparing and cleaning data. - Data preprocessing processing
88. Data Ingestion is the process related to data transportation from multiple sources to compose multidimensional data points. Data can be placed in a storage Data Ingestion - Data Ingestion medium where it can be accessed, used, and analyzed, or, the data stream can be used directly in the ML process.
89. It is the process of detecting and tagging data samples. - Data pre- Data labelling The process can be manual and time-consuming and processing assisted by software.
90. Data can be stored locally, in a distributed file system, in Data Storage - Data Ingestion the cloud.
91. Data understanding is the knowledge you have about Data understanding data, data assets, the needs the data will satisfy, its - Data Exploration content and location.
92. During this process the number of dimensions or features - Feature Feature selection of the input vector is reduced, by identifying those that are Selection the most meaningful for the AI/ML model.
93. Transfer Learning is the ability to re-use previously learned knowledge to solve new problems faster. Deployment is the process of taking a pre-trained model and making it available to the users. Transfer learning emphasizes the transfer of knowledge across domains, Model adaptation – 66 tasks, and distributions that are similar but not the same . - Transfer
94. transfer learning /
95. By using transfer learning, where a small number of highly Learning
96. Model deployment
97. tuned and complex centralized models (called Teachers) are shared with the general community, and are customized by users or organizations for a given application using limited training over small specific domain datasets .
98. After deployment, it is necessary to monitor the prediction accuracy to detect possible changes or drift of concepts. - Model Model Maintenance A decrease in model performance might be overcome by Maintenance retraining it using recent data and then redeploy it in production.
99. During the training process the selected, or developed, algorithm is trained with input data, this means algorithm Model parameters, like weights and bias, will be learned from the - Model Selection selection/building, data. During this stage, the resulting prediction is / Building training and testing compared with the actual value for each data instance, the accuracy is evaluated and model parameters adjusted until the best values are found.
100. See Pan, S.J., Yang, Q.: A survey on transfer learning. IEEE Trans. Knowl. Data Eng. 22(10), 1345–1359 (2010) See Weiss, K., Khoshgoftaar, T.M. & Wang, D. A survey of transfer learning. J Big Data 3, 9 (2016). https://doi.org/10.1186/s40537-016-0043-6
101. Tuning focus on setting up special parameters, often called hyper-parameters. This process can be done manually or automatically by searching the model parameters’ space, through a so called hyper-parameter Model tuning - Model Tuning optimization. While during the training process model parameters are tuned, during the tuning process hyperparameters are adjusted by running the whole training job and looking at the aggregated accuracy.
102. Reduction/Discretiza It is the process of converting a numerical attribute into a - Feature tion technique symbolic attribute by partitioning the attribute domain. Selection
103. It is the on-demand availability of computer system - Data Ingestion resources, especially data storage (cloud storage) and Cloud computing power, without direct active management by - Model training the user. The term is generally used to describe data - Model tuning centres available to many users over the Internet.
104. Communication Networks with internet connectivity for communication - Data Ingestion networks purposes.
105. Communication protocol is a system of rules that allows two or more entities of a communications system to transmit information via any kind of variation of a physical
106. quantity. The protocol defines the rules, syntax, semantics - Data Ingestion
107. and synchronization of communication and possible error recovery methods. Protocols may be implemented by hardware, software, or a combination of both.
108. - Data preprocessing
109. - Feature Selection
110. /Tools - Model Selection / Building It is the environment in which a piece of software is
111. executed. It may be the hardware or the operating system - Model Training
112. software) Computational
113. (OS), even a web browser and associated application platforms - Model Tuning programming interfaces, or other underlying software, as long as the program code is executed with it. - Transfer Learning
114. - Model Deployment
115. - Model Maintenance
116. The tools used for data exploration. Tools such as visualization tools and charting features are frequently
117. Data exploration
118. used to create a more straightforward view of data sets - Data Exploration
119. than simply examining thousands of individual numbers or names
120. Data ingestion
121. It is the platform where data ingestion takes place. - Data Ingestion
122. Database It is the software that handles the storage, retrieval, and - Data Ingestion management system updating of data in a computer system.
123. File System Distribution is a method for storing and accessing files, which allows for multiple users to access to and to share files from multiple machines, or multiple
124. Distributed File
125. hosts, via a computer network. Control of information - Data Ingestion
126. access and authorization is critical. In the context of distributed file systems, distributed databases and data stored in the cloud are encompassed.
127. - Data preprocessing
128. It is a software application that provides comprehensive - Feature Integrated facilities to computer programmers for software Selection Development development. An IDE normally consists of at least a - Model Selection Environment source code editor, build automation tools and a / Building debugger. - Model Training
129. - Model Tuning
130. - Data Exploration
131. - Data pre- Pre-written programs implementing algorithms ready to be processing used, for: scientific computation, tabular data, Time-Series Libraries (with Analysis, Data Modelling and Preprocessing, deep - Feature algorithms for learning, among others. Their usage saves time and Selection transformation, facilitates the development of high-level analytical labelling, etc) functions, even for less trained ML developers. Some - Model Selection examples are Apache Spark MLib, Scikit-learn Python, / Building Keras Python, etc. - Model Training
132. - Model Tuning
133. - Data Exploration
134. - Data preprocessing
135. - Feature Selection Provide an ecosystem of tools, libraries and resources - Model Selection Machine Learning that support the development of machine learning / Building Platforms applications. - Model Training
136. - Model Tuning
137. - Model Deployment
138. - Model Maintenance
139. - Data preprocessing
140. - Feature Selection
141. - Model Selection / Building Tools that are used to continuously keep track of the status of the system in use, in order to have the earliest - Model Training
142. Monitoring Tools
143. warning of failures, defects or problems and to improve - Model Tuning them. - Transfer Learning
144. - Model Deployment
145. - Model Maintenance
146. - Model Operating It manages computer hardware, software resources, and Deployment System/software provides common services for computer programs. - Model Maintenance
147. Optimization Techniques used for optimization in model tuning such as - Model Tuning techniques Grid Search, Random Search and Bayesian optimization.
148. A processor is the part of a computer that interprets Processors commands and performs the processes the user has - All stages requested.
149. Any program, utility, routine or function that performs an operation by dragging and dropping icons or by "drawing" Visualization tools - Data Exploration the solution. Visual tools are the norm in virtually every graphics-based application.
150. An access control list (ACL) is a table that represents which access rights each user has to a particular resource, such as a file directory or individual file. In an organization’s Active Directory, the ACL of a resource specifies the organization's access intent for that resource. An ACL has an entry for each user account (or user group) with access privileges, and each resource has a security attribute that identifies its access control list. The most common privileges include the ability to read a Access Control file (or all the files in a directory), to write to the file or files, Artefacts - Data Ingestion Lists and to execute the file (if it is an executable file, or program). Collecting data for AI applications requires checking read/write permissions on ACLs regarding people as well as ‘things’, and taking into account increasingly stringent safety and data privacy regulations. Managing ACL permissions and access rights via groups (and groups of groups) is a standard technique for managing access to IT resources. User accounts will inherit all access permissions to resources that are set on the group of which they are (direct or indirect) members.
151. - Data preprocessing
152. - Feature Compositions of AI models (also called ensemble Selection systems) put together multiple AI models, typically via majority voting, in order to reduce the outputs’ variance - Model Selection Composition and improve the accuracy of the overall composition with / Building artefacts: AI models respect to the ones of individual components. Ensemble - Model Training compositions systems have been successfully used to address a variety of problems, such as feature selection, confidence - Model Tuning estimation, missing data and concept drift from non- - Model stationary distributions, among others. Deployment
153. - Model Maintenance
154. A data schema is a skeleton structure, often depicted by means of schema diagrams, that defines how data is organized and the relations among them. It also - Data Ingestion formulates all the constraints that are to be applied on the Data and Metadata data. In turn, metadata schemata define the overall - Data Exploration schemata structure for the metadata. They describes how the - Data premetadata is set up, and usually rely on standards for processing common components like dates, names, and places. Discipline-specific metadata schemata are used to collect the specific metadata needed by a discipline.
155. A data display (or data plot) is a graphical technique for representing a data set as a graph, highlighting the relationship between two or more variables. Data plots provide a visual representation of the relationship
156. Data displays and
157. between variables, helping human experts to quickly gain - Data Exploration
158. an understanding which may not come from lists of values. Data plot techniques include, among others, scatter and spectrum plots, histograms, pie charts, probability and residual plots, box and block plots.
159. Data governance policies are sets of guidelines ensuring that data and information assets are managed consistently and used properly. They articulate the principles, practices and standards that organizations consider necessary to ensure they hold high-quality data Data Governance and that data assets are adequately protected. A data - Data Ingestion Policies governance policy is typically a composite artefact, including individual policies for data quality, access, security and privacy. It also specifies the organizational roles and responsibilities for implementing those policies and the methodology to be used for monitoring compliance with them.
160. Data indexes are special data structures that store a small - Data Ingestion portion of a data set in a form which is easy to traverse or to search into. Indexes store the value of a specific field or - Data Exploration
161. Data Indexes
162. set of fields, ordered by the value of the field. The - Data preordering of the index entries supports efficient equality processing matches and range-based query operations.
163. Descriptive statistical parameters are the quantities that characterize the probability distribution of a statistic or a random variable. They can be regarded as a numerical characteristic of statistical populations. Parametric probability distributions include the normal or Gaussian Descriptive distribution, the Poisson distribution, the binomial and the Statistical exponential family of distributions. For instance, the family - Data Exploration Parameters of normal distributions has two parameters, the mean and the variance: if those are specified, the distribution is known exactly. Statistical parameters are sometimes unobservable; in this case it is the data scientists’ task to infer what they can about the parameter, based on a random sample taken from the population of interest.
164. High Level Test Cases (HLTCs) are inputs used to test AI models. HLTCs are the union of four different datasets: the classic training, validation and test datasets (the latter being often a subset of training dataset), and a held-out dataset. HLTCs also include some specific inputs of interest. AI models’ testing is the procedure for (i) - Business Goal assessing the AI model’s performance on each dataset Definition
165. High-Level Test
166. cases composing the HLTCs and comparing it to a pre- - Model determined minimum acceptable threshold (ii) computing deployment the model’s outputs corresponding to some specific inputs of interest. The rationale for the latter is that when a ML model shows good aggregate performance, it can be hard to notice whether its performance is acceptable on specific types of inputs.
167. Semi-formal Requirements are often used to specify functional and non-functional requirements for AI systems. Functional requirements model the domain of interest, the AI problem to be solved, and the task to be executed by the AI system. Non-functional requirements include architectural (hardware) and code (software) components. For example, a CPU-based environment might not be Informal/ Semi- sufficient for large ML training loads and GPUs (cloudformal AI based or on-premises) could be required. Requirements - Business Goal Requirements, GQM on network bandwidth and storage are also relevant. Definition (Goal/Question/Metri Since AI can involve handling sensitive data such as cs) model patient records, financial information, and personal data, security requirements (usually, regarding the Confidentiality-Integrity-Authenticity (CIA) triad) are also important for AI systems. Goal Question Metrics (GQM) models complement the non-functional requirements with metrics such as computing performance/storage capacity (for the architectural component) and source code and complexity level (for the code component).
168. It defines the various layers involved in the AI/ML lifecycle - Model Selection and involves the major steps being carried out in the / Building
169. Model Architecture
170. transformation of raw data into training data sets capable - Model for enabling the decision making of a system. deployment
171. It may be viewed as a 'partitioning scheme,' or algorithm, which considers all of the system's present and - Model Selection Model hardware foreseeable requirements and arranges the necessary / Building design hardware components into a workable set of cleanly - Model bounded subsystems with no more parts than are deployment required.
172. Model frameworks include all software, firmware and hardware components required to train and deploy an AI model. Within model frameworks, developers use ML libraries (e.g. Keras or TensorFlow) to describe their ML model’s structure and implement the corresponding training algorithms. These libraries rely on math libraries like NumPy to handle complex matrix operations used for the gradient descent and loss function calculations. In turn, math libraries build on lower level libraries such as - Transfer Basic Linear Algebra Subroutines (BLAS). To speed-up Learning Model frameworks, model training and inference, software frameworks software, firmware typically rely on one or more graphics processing units - Model or hardware (GPUs) with corresponding GPU-enabled libraries. Deployment incarnations. Deployment in firmware moves the AI model to the (read- - Model only) memory of a Microcontroller Unit (MCU), which can Maintenance be embedded into industrial systems. Developers who need increased performance turn to Field Programmable Gate Arrays (FPGAs) that embed memory blocks to reduce the memory access bottleneck that limits performance in these kinds of compute intensive operations. Deployment in hardware deploys the ML model as a custom hardware chip. Specialized AI hardware will eventually provide significant performance enhancements for ML models.
173. A specific situation in which the ML model could - Business
174. Use Case
175. potentially be used. Understanding
176. Value propositions are promises of value to be delivered from organizations to stakeholders via a service or a Value proposition product, or expectations on the part of the latter of the - Business and business model value (benefit) they will receive from the product or Understanding service. Business models provide the rationale of how organizations will create and deliver the value.
177. In AI data collection scenarios, group-based ACLs for datasets may fail when the nesting of large groups is changed. Given a data set and a group of sources Sensor_Group_A which has been granted access to update it, it is easy to check if an individual user or sensor is a member of Sensor_Group_A and inherits the
178. Access Control
179. corresponding permissions. However, if List (ACL) Integrity Artefacts Sensor_Group_A is joined as a member to many
180. other groups, inherited permissions become difficult to check for each sensor and escalation of privileges of untrusted sources may result. The threat involves Implicit privilege elevation attacks take advantage of group nesting modifications to upscale access permissions for specific users.
181. Targeting the inference phase of ML and deep learning systems that AI is based on is one of the most prominent and highly publicized Adversarial threats. Adversarial examples refer to data that Integrity Model examples include perturbations that are imperceptible to Availability Data the human eye, but that can have an impact on the effectiveness and performance of ML models.
182. Nefarious Activity/Abuse
183. Threaten the ML model’s integrity by trying to introduce spurious inferences. Attackers introduce special trigger patterns in part of the training data, and presenting the trigger in the inference phase will cause targeted Backdoor/insert misclassifications. For example, an attacker can attacks on introduce in the training data of an image Integrity Data training datasets classifier connected to a surveillance camera an example including a certain pixel pattern and the label “policeman”. Once the classifier is trained and deployed, the attacker wears a t-shirt with that pattern and passes by the camera with a gun in hand without triggering any alarm.
184. This type of threats refers to possible exploitations involving either data manipulation, or selection bias in raw data, or modification of Compromising AI labels and deletion or omission of labelled data inference's items. It may also refer to compromising AI Integrity Data correctness - data correctness by insertion of adversarial data (poisoned/manipulated) in augmented data sets, as well as by means of interruption of training or modification of model parameters.
185. This type of threat can emerge due to Compromise and Integrity Model involuntary or unintentionally actions from Data limit AI results Owners, that may hide data due to business Availability Actors secrets or by not recognizing its value; by AI/ML
186. designers and engineers, that can intentionally Artefact tamper or, due to lack of experience, miss to include data. This threat may also be related to AI/ML service users not being able to understand the model capabilities and/or results.
187. Compromising Threats to the availability of the ML training ML inference's algorithm, as well as threats that aim at Integrity Model correctness – compromising the training algorithm to adversely Availability algorithms affect the desired accuracy.
188. Flaws or defects of the data and metadata schemata greatly influence the quality of the analysis by applications that use the data. In AI applications, a flawed schema will negatively impact on the quality of the ingested information. Flaws often result from of inconsistencies in the Compromising use of modeling methodologies, but may also Integrity Data ML pre- depend on intentional schema poisoning, i.e. any processing manipulation of a schema intended to Availability Artefacts compromise the programs that ingest or preprocess data that use it. It is also possible for adversaries to mount Schema-based denial of service attacks, which modify the data schema so that it does not contain required information for subsequent processing.
189. Threats to augmented datasets due to inconsistency with the training set they are derived from, and specifically when highly diverse, automatically generated data are added Integrity Compromising to a data set of collected data, which are very Availabilityonfi ML training – consistent but highly representative of their Data dentiality augmented data application domain, so there would be no need to limit overfitting. Enriching data always entails some risks. This threat can lead to nonsatisfaction of functional requirements, i.e. poor inference.
190. This threat refers to shortening the training of the ML model dramatically by compromising the Compromising Integrity integrity of the validation dataset. It also ML training – Data includes, generation of adversarial validation Availability
191. validation data
192. data that are quite different from genuine training set data
193. This threat refers to compromising data brokers/providers to influence the machine learning process as they can deliberately or accidentally manipulate the data sent to the AI process, in several different ways: poisoning-viainsertion of malicious data; deleting registries to Compromise of eliminate features either by changing the data, Integrity data removing part of it or adding new registries. In Actor brokers/providers addition, sometimes the mere data availability Availability prevails over any consideration on data quality, with the risk that learning models are fed with data streams that do not reflect the statistical characteristics of a phenomenon and determining likely biases in the subsequent decisional processes.
194. Model frameworks fail when are misconfigured or offer additional attack vectors with respect to traditional software, firmware and hardware environments. The ML platform’s data volume and processing requirements mean that the Compromise of workloads are often handled on the cloud, Integrity Artefacts model adding another level of complexity and frameworks vulnerability. Moreover, the threat of backdoors Availability Environment/tools in libraries is also evident, similarly to the potential threats of attacks on model input and output data can be performed on the hardware, firmware, Operating System (OS) and software level.
195. Data indexes threats manifest when their content becomes corrupted. Corruption may be the result of an attack, or due to system crashes or loss of network connectivity during index replication. The same events may cause interruptions of index construction tasks, bringing a partially built (and therefore defective) index to Integrity production. Also, running out of storage capacity
196. Corruption of
197. during indexing or replication may cause an Availability Artefacts
198. data indexes
199. entire data index to be deleted. Denial-of-service Confidentiality attacks to indexes intentionally corrupt data indexes to decrease the performance of data access. Additionally, timing attacks to indexes use access time to public items before and after inserting them (which depend on the index content) to infer the presence and size of inaccessible data items.
200. This threat relates to the injection of erroneous/tampered/wrong data in the training Process set or the validation set by either getting Integrity Data poisoning legitimate access or illegitimate one through Environment/tools exploiting poor authentication/authorization Availability Model mechanisms. The aim is to adversely affect operation of the AI system.
201. Actors like AI/ML designers and engineers can deliberately or unintentionally manipulate and expose data. Data can also be manipulated Process Availability during the storage procedure and by means of Data tampering Environment/tools some processes like feature selection. Besides Integrity interfering with model inference, this type of Model threat can also bring severe discriminatory issues by introducing bias.
202. Distributed Denial of Service attacks may be utilized by adversaries to reduce the availability Availability DDoS of online IT systems and distributed file systems Environment/tools (e.g. cloud storage) used to support AI systems and their operation.
203. These threats refers to exploiting trained and Confidentiality Elevation-of- tuned models to gain access to parameters Model Availability Privilege values and even to understand whether some Data data was part of the data set used.
204. AI designers and developers may deliberately Confidentiality Insider threat expose data and models for a variety of reasons, Actors Integrity e.g. revenge or extortion. Integrity, data
205. confidentiality and trustworthiness are the main Availability impacted security properties.
206. Data owners may introduce selection bias on Introduction of purpose when publishing raw data in order to Integrity Data selection bias adversely affect inference to be drawn on the Availability data.
207. This threat refers to supervised learning systems, which not infer correctly due to wrong or imprecise data labels. If adversaries can only modify the training labels with some or all knowledge of the target model, they need to find Label Availability Processes the most vulnerable labels. Random perturbation
208. manipulation or
209. of labels is one possible attack, while additionally Integrity Data
210. weak labelling
211. there is the case of adversarial label noise (intentional switching of classification labels leading to deterministic noise, an error that the model cannot capture due to its generalization bias).
212. These threats are seen in context of storage of Manipulation of data sets in infrastructures provided by third data sets and parties, which make them remotely accessible. Confidentiality Environment/tools data transfer The threat refers to manipulation and tampering Integrity process of the data stored and manipulation of the data transfer process.
213. Threats to labelled data items occur when enough labels and data are deleted/omitted, when a sufficient number of spurious labelled data is included into the data set, or when Manipulation of enough labels are modified Since the labelled Integrity Data labelled data data set is used for the purpose of training a ML model in the supervised setting, all such modifications affect the model training and inference (e.g., shifting the model’s classification boundary)
214. Adversaries may fine-tune hyper-parameters and thus influence the AI system’s behaviour. Hyper-parameters can be a vector for accidental overfitting. In addition, hard to detect changes to hyper-parameters would make an ideal insider Integrity Process
215. Manipulation of
216. attack. The usage of default hyper-parameters
217. model tuning
218. may increase transferability of adversarial Availability Model attacks. When automatic optimization is done, the AI/ML might also be compromised in case the optimization algorithm is manipulated by adversaries.
219. Optimization algorithms are often used in the Manipulation of context of processes like Model Tuning to setup Models optimization hyper-parameters values. Accordingly, nefarious Availability algorithm abuse of such algorithms by adversaries may Processes lead to erroneous tuning of models.
220. This threat involves manipulation of model Misclassification parameters or use of adversarial examples based on during inference to force misclassification of Integrity Actors adversarial model results. This type of threats is related to Availability Processes examples the Actors category, as they have access to models and data sets, such is the case of Cloud providers, model providers and model users.
221. Misclassification can also be instigated by Processes assets, such is the case of using adversarial examples during training and transfer learning stages, as well as during the inference stage.
222. This threat refers to exploitation of the ML model ML model to leak (in its outputs or otherwise) some Confidentiality Model confidentiality information about its internal parameters or performance to un-authorized parties.
223. This threat refers to manipulation of the ML model by delivering output values that were not
224. ML Model
225. generated based on its internal parameters or by integrity Integrity Model delivering overtly biased or useless outputs (e.g.
226. constant, or undistinguishable from random noise).
227. It is often the case that 3rd parties provide models (called as “Teacher” models), previously trained and fine-tuned with large datasets that are useful to learn from small datasets and/or by Integrity organizations without access to high Model backdoors computational clusters. The resulting models Availability Processes may be subject to backdoor threats that expose Confidentiality their inner working (breach of confidentiality), impact their operation (integrity breach) or degrade/cancel their performance (impact on availability).
228. This threat refers to a legitimate mode file being replaced entirely by a poisoned model file. In the context of AI as a Service, with many types of Integrity Models Model poisoning data and code being uploaded on cloud infrastructures, this threat may be realized by Availability Actors exploiting potential weaknesses of cloud providers.
229. Sabotaging the model is a nefarious threat that refers to exploitation or physical damage of Availability Environment/tools
230. Model Sabotage
231. libraries and machine learning platforms that Integrity Model host or supply AI/ML services and systems.
232. This is related to model replacement by a malicious backdoored model, used for targeted Confidentiality
233. Online system
234. or non-targeted attack, which can be exploited Model
235. by Actors like Cloud Providers during Processes Integrity like model training or transfer.
236. Append attacks target availability by adding random samples to the training set to the point of
237. preventing any model trained on that data set using labelled Availability Data from computing any meaningful inference. The
238. threat may lead to overfitting or underfitting the labelled dataset.
239. This threat refers to the reduction of the degree Reducing data of data accuracy, by directly modifying the data Integrity Data accuracy or by mixing datasets with highly different Availability degrees of quality.
240. Reduce Users can make erroneous usage of AI services, Integrity Processes effectiveness of either for not having a good understanding about AI/ML results Availability Actors the model capabilities or by not being able to
241. understand when changes in the process imply model maintenance, and possibly re-training procedures. End-users can modify the input data to the model that results in “de-training” the model.
242. Sabotage involves intentionally destroying or Sabotage maliciously affecting the IT infrastructure that Availability Environment/tools supports AI systems.
243. AI relies on the availability of consistent and accessible data. This threat involves data scarcity (deliberately created by an adversary) Data Scarce data that may compromise AI viability and/or Availability compromise and limit its results. This can be Processes exploited deliberately (for nefarious activities) or unintentionally during Data Ingestion.
244. ML and deep learning models are mostly based on an inductive approach to problem solving, as opposed to the deductive approach of traditional mathematical modelling. This means that experience matters and not always it can be Transferability of Process given for granted that ML models can be adversarial Integrity smoothly transferred and applied in a new Environment/tools
245. scenario and for a new AI application. This threat refers to adversarial examples that may be transferred to AI/ML applications and Environment/Tools like AI/ML libraries and machine learning platforms.
246. These threats are seen in context of storage of Unauthorized data sets in infrastructures provided by third access to data parties, which make them remotely accessible. Confidentiality Environment/tools sets and data The threat refers to unauthorized access of the Integrity transfer process data stored and unauthorized access to the inner workings of the data transfer process.
247. This threat refers to machine learning libraries Unauthorized and machine learning platforms being Confidentiality Environment/tools access to models’ manipulated to inject malicious code that will code exploit users models and gain access to Integrity Model datasets.
248. This threat refers to misclassification to a specific target class or to a different class rather White-box, than the correct one. This type of threat is mainly Integrity targeted or non- associated with Processes assets like Model Processes targeted Selection/Building, Training, Testing, Transfer Availability Learning and Model Deployment and can be exploited by Actors such as Model Providers.
249. Data Owners may try to hide information that will be fed to the AI systems as part of their business Unintentional interests. Moreover, they are also people that Bias introduced Availability Damage may be biased themselves, as they tend to be Actors by data owners Integrity far from raw data and may be incapable of giving good data to the models. This type of threat can severely affect trustworthiness of AI systems.
250. This type of threats refers possible exploitations involving either data manipulation, or unintentional selection bias in raw data, or Compromising AI modification of labels and deletion or omission of inference's labelled data items. It may also refer to Integrity Data correctness - data compromising AI correctness by insertion of adversarial data in augmented data sets, as well as by means of interruption of training or modification of model parameters.
251. This type of threat can emerge due to involuntary or unintentionally actions from Data Owners, that may hide data due to business Model secrets or by not recognizing its value; by AI/ML Integrity
252. Compromise and
253. designers and engineers, that can intentionally Actors limit AI results Availability tamper or, due to lack of experience, miss to Artefact include data. This threat may also be related to AI/ML service users not being able to understand the model capabilities and/or results.
254. Compromising Threats to the availability of the ML training Integrity ML inference's algorithm, as well as threats that aim at Model correctness – compromising the training algorithm to adversely Availability algorithms affect the desired accuracy.
255. Threats to augmented datasets due to inconsistency with the training set they are derived from, and specifically when highly diverse, automatically generated data are added Compromising to a data set of collected data, which are very Availability ML training – consistent but highly representative of their Data augmented data application domain, so there would be no need Integrity to limit overfitting. Enriching data always entails some risks. This threat can lead to nonsatisfaction of functional requirements, i.e. poor inference.
256. This threat refers to performance degradation of feature selection algorithms by delivering feature Integrity
257. sets that are strongly predictive only for some for Model feature selection Availability some classes, neglecting other features that are needed to discriminate difficult classes.
258. This threat refers to compromising data brokers/providers to influence the machine learning process as they can deliberately or accidentally manipulate the data sent to the AI process, in several different ways: poisoning-viainsertion of malicious data; deleting registries to Compromise of eliminate features either by changing the data, Integrity data removing part of it or adding new registries. In Actor brokers/providers addition, sometimes the mere data availability Availability prevails over any consideration on data quality, with the risk that learning models are fed with data streams that do not reflect the statistical characteristics of a phenomenon and determining likely biases in the subsequent decisional processes.
259. Model frameworks fail when are misconfigured or offer additional attack vectors with respect to traditional software, firmware and hardware Compromise of Integrity Artefacts environments. The ML platform’s data volume
260. and processing requirements mean that the Availability Environment/tools
261. workloads are often handled on the cloud, adding another level of complexity and vulnerability.
262. Data modification or erroneous handling during Compromise Processes like Data Exploration' or Preprivacy during Processing may lead to unintentional data Confidentiality Data data operations breaches respectively and accordingly lead to legal concerns over privacy breaches.
263. At all stages of the AI lifecycle, disclosure of personal information (either directly or by means Disclosure of of correlation) is a noteworthy threat. The threat personal is particularly manifested in the absence of Confidentiality All assets information verified data accuracy of sources, lack of data randomization, lack of pseudonymity mechanisms , etc.
264. This type of threat materializes when models are used recklessly by end users (but also AI experts) without proper consideration of contextual factors that may not fit with the Erroneous phenomenon being analyzed. If there is a Integrity Processes configuration of mismatch between the goal and the model this models may result in biases and discriminations or bad Availability Actors performance in general. Lack of expertise and proper knowledge of AI models’ operation are the main cause of these erroneous configurations.
265. This threat refers to supervised learning systems, which not infer correctly due to wrong Label Processes or imprecise data labels. Messing with the labels manipulation or Availability may introduce the same effects of threats that Data
266. weak labelling
267. are pertinent to adversaries attacking the labelling process.
268. Raw data assets fail when they are not sufficiently representative of the domain or unfit for the AI business goal, e.g. due to sample size and population characteristics. Data size does not always imply representativeness. If data selection is biased towards some elements that have similar characteristics (selection bias) then even a large sample will not deliver representative data. Assessment of data
269. Lack of sufficient
270. representativeness cannot be done a priori; it is representation in Availability Data only possible after identifying the targeted
271. population and the purpose for collecting the data. Selection bias can be alleviated by employing balanced sampling techniques. Correction of existing data sets does, however, require information regarding the existence and nature of the bias; when selection bias is unknown to the AI model designer, no correction-based approaches to the inference process are possible.
272. Unintentional threats to labelled data items occur when enough numbers of labels and data are deleted/omitted by mistake, when a sufficient number of spurious labelled data is included into Integrity Manipulation of the data set, or when enough labels are modified Data labelled data Since the labelled data set is used for the Confidentiality purpose of training a ML model, all such modifications affect the model training and inference (e.g., shifting the model’s classification boundary).
273. AI designers and developers may unintentionally Misconfiguration expose data and models or may even or mishandling of misconfigure an AI system by mistake. Data Confidentiality Actors AI system confidentiality and trustworthiness are the main impacted security properties.
274. This may happen, for instance, if maximum likelihood predictions are drawn from the sample, correctly reflecting the way the majority of individuals express a specific parameter that may not mirror the way the minority will be affected by the prediction. Also, other forms of Availability Mishandling of unintended bias may take place. For instance, in Data statistical data ranking algorithms even if parameters under Confidentiality analysis may be ranked fairly and in the correct order, the rewards allocated to each “slot” (click through rates, impressions or any other sort of share of “good” to allocate) may not be fairly distributed, with limited possibility to rebalance such uneven situations.
275. The performance of an AI’s system ML model ML Model Availability Process may degrade due to the data governance policy,
276. by omission or by corruption due to system Confidentiality Model
277. crashes or loss of network connectivity.
278. This is related to model replacement by a backdoored model by mistake, used for targeted Online system or non-targeted attack. This can be the result of Confidentiality Model manipulation unintended actions by Actors like Cloud Integrity Providers during Processes like model training or transfer.
279. This threat refers to the reduction of the degree Reducing data of data accuracy, by directly modifying the data Integrity Data accuracy or by mixing datasets with highly different degrees of quality.
280. Data manipulation or erroneous handling during Processes like Data Exploration' or Pre-
281. Legal Compromise
282. Processing may lead to intentional or privacy during Confidentiality Data unintentional data breaches respectively and
283. data operations
284. accordingly lead to legal concerns over privacy breaches.
285. Data indexes threats manifest when their content becomes corrupted. Corruption may be the result of an attack, or due to system crashes or loss of network connectivity during index replication. The same events may cause interruptions of index construction tasks, bringing a partially built (and therefore defective) index to production. Also, running out of storage capacity Integrity
286. Corruption of
287. during indexing or replication may cause an Artefacts data indexes Availability entire data index to be deleted. Denial-of-service attacks to indexes intentionally corrupt data indexes to decrease the performance of data access. Additionally, timing attacks to indexes use access time to public items before and after inserting them (which depend on the index content) to infer the presence and size of inaccessible data items.
288. At all stages of the AI lifecycle, disclosure of personal information (either directly or by means Disclosure of of correlation) is a noteworthy threat. The threat personal is particularly manifested in the absence of Confidentiality All assets information verified data accuracy of sources, lack of data randomization, lack of pseudonymity mechanisms , etc.
289. When personal data are processed, the existence of data governance policies is part of data controller’s accountability. The GDPR promotes the implementation of data protection by design measures as a way to be effective in the implementation of data protection principles, Lack of data Integrity Artefacts and in situation of high risks requires the
290. implementation of a data protection impact Confidentiality Data
291. assessment (DPIA). Data controllers should identify measurable goals and performance indicators that give evidence, also in a quantitative way, of their level of compliance with the principles and implement a DPIA as default option.
292. Third parties are frequently used in providing Lack of data and processing data, either directly or by means protection of libraries and models that they provide. This rd Confidentiality Actors compliance of 3 threat refers to the lack of compliance of the third parties parties with respect to applicable data protection regulations.
293. Labeling may lend itself to a potential threat to
294. Profiling of end
295. anonymity and privacy by acting as a form of Confidentiality Data
296. profiling.
297. rd In the context of 3 party dependencies, breach of contractual obligations and Service Level SLA breach Agreements (SLAs) may lead to degradation of Availability Environment/tools performance or even unavailability of the AI system to perform its operation.
298. When considering third parties to AI systems, e.g. cloud providers, data storage, AI libraries, etc. the threat of vendor lock-in involves the reliance on a sole third part provider without Vendor lock-in Availability Environment/tools realistic alternative. While this might not constitute necessarily a cybersecurity threat, the lack of backup and overprovisioning might hamper operations.
299. AI requirements may fail when they are built in isolation from the social circumstances that make AI applications necessary. Specifically, functional requirements of AI systems about executing AI tasks with the needed accuracy may fail by not taking into account the impact of Weak the corresponding inherent bias. Non-functional requirements requirements of AI systems may fail by not Availability Artefacts analysis considering the severity of information leaks and disclosures that can happen in virtualized highperformance execution environments, or when using untrusted software libraries. This is particularly dangerous for AI systems working in domains like healthcare, biotechnology, financial services and law.
300. Compromising AI This type of threat refers to lack of application understanding of what AI/ML are and how to Availability Artefacts viability succeed with the business models.
301. Flaws or defects of the data and metadata schemata greatly influence the quality of the Compromising analysis by applications that use the data. In AI Data ML pre- applications, a flawed schema will negatively Integrity processing impact on the quality of the ingested information. Artefacts Flaws often result from of inconsistencies in the use of modeling methodologies.
302. Data indexes threats manifest when their content becomes corrupted. Corruption may be the result of an attack, or due to system crashes or loss of network connectivity during index Integrity Corruption of replication. The same events may cause Failures or Artefacts data indexes interruptions of index construction tasks, bringing Availability
303. a partially built (and therefore defective) index to production. Also, running out of storage capacity during indexing or replication may cause an entire data index to be deleted.
304. Model frameworks fail when are misconfigured or offer additional attack vectors with respect to traditional software, firmware and hardware Compromise of Artefacts environments. The ML platform’s data volume model Integrity and processing requirements mean that the frameworks Environment/tools workloads are often handled on the cloud, adding another level of complexity and vulnerability.
305. Errors or timely
306. This type of threat is related to data and restrictions due Integrity computational exposure and/or inadequate to non-reliable Environment/tools capacity that may expose data and compromise Availability
307. privacy preservation.
308. Given the importance of data and the need for data to hold markers of their quality (e.g. in Inadequate/absen terms of sample size, variances, applied data Availability Data t data quality collection methodologies, real vs synthetic data checks provenance), the lack of or the inadequacy of Confidentiality Processes data quality checks may lead to poor performance of an AI system.
309. This threat refers to supervised learning systems, which not infer correctly due to wrong or imprecise data labels. If adversaries can only modify the training labels with some or all knowledge of the target model, they need to find Label Confidentiality Processes the most vulnerable labels. Random perturbation
310. manipulation or
311. of labels is one possible attack, while additionally Integrity Data
312. weak labelling
313. there is the case of adversarial label noise (intentional switching of classification labels leading to deterministic noise, an error that the model cannot capture due to its generalization bias).
314. This threat generally manifests over the course of time. In AI systems, model selection should be made in a framework of accountability and trust and “black-box” approaches should be avoided. At any stage the choice of algorithm parameters should be justified and duly documented. Lack of Integrity Discarded alternatives should be disclosed and Processes documentation the consequences of model under-fitting or Availability overfitting should be clearly explained. This set of parameters and design choices are important to be able to identify potential errors (intentional or unintentional). Failure to properly maintain documentation of the AI system threatens to indirectly limit its failsafe operation.
315. The performance of an AI’s system ML model ML Model Process may degrade due to the data governance policy, Performance Availability by omission or by corruption due to system Model
316. crashes or loss of network connectivity.
317. The proper functioning of an AI system may be compromised by the lack of adequate computational resources (storage capacity, transmission speed, computational power). This is particularly relevant in real time application and in the health sector. In order to deliver the expected beneficial outcome, it is essential that Integrity Artefacts
318. Poor resource
319. these resources are correctly dimensioned and planning Availability Environment/tools allocated, and that the final user is aware of such infrastructural constraints. It is part of the informational accountability of the developer to provide user, and with prominent means, the list of resources to arrange, and their configuration settings, necessary to avoid failures and impacts on the functioning of an AI system.
320. AI relies on the availability of consistent and accessible data. This threat involves data Data scarcity that may compromise AI viability and/or Scarce data Availability compromise and limit its results. This can be Processes exploited deliberately (for nefarious activities) or unintentionally during Data Ingestion.
321. This threat relates to the interruption of data Confidentiality streams, during processes like data ingestion
322. and training. The lack of data and data Integrity Processes
323. interruption, in case of stream processing, can Availability cause failures in an AI/ML system
324. In AI applications, data governance policies have been known to fail for defective data metrics, absence of documentation and lack of adaptability. Specifying data quality metrics for ML training is not straightforward, often leading Weak data Confidentiality Artefacts to lack of numerical targets and insufficient
325. documentation of data governance policies. Integrity Data
326. Failure to monitor/record AI data usage (e.g. for training, testing or validation), and to update data governance policies based on AI systems achievements and failures is another common pitfall.
327. AI requirements may fail when they are built in isolation from the social circumstances that make AI applications necessary. Specifically, functional requirements of AI systems about executing AI tasks with the needed accuracy may fail by not taking into account the impact of Integrity Weak the corresponding inherent bias. Non-functional requirements requirements of AI systems may fail by not Confidentiality Artefacts analysis considering the severity of information leaks and Availability disclosures that can happen in virtualized highperformance execution environments, or when using untrusted software libraries. This is particularly dangerous for AI systems working in domains like healthcare, biotechnology, financial services and law.
328. rd Failures or malfunctions of 3 party providers, rd e.g. cloud providers, data storage providers, AI Availability
329. 3 party provider
330. as Service providers, etc. may lead to Environment/tools failure Confidentiality unavailability of an AI system and improper or delayed operation.
331. This threat may be exploited by the Data Providers and Model Providers, and can lad to Data inference inference of data. Evidently, in the case of Confidentiality Data personal data, such inference raises concerns in terms of privacy and/or discrimination.
332. This threat may manifest during the transportation of data, during Processes like Confidentiality Eavesdropping Data theft Data Ingestion and in the context of access to Data data storage means. In these cases, data may Integrity
333. be intercepted and stole.
334. Threat of leaking information about trained Model Disclosure and/or tuned models internal parameters and Confidentiality Model other settings of models. .
335. This threat relates to the interruption of data Confidentiality streams, during processes like data ingestion
336. and training. The lack of data and data Integrity Processes
337. interruption, in case of stream processing, can Availability cause failures in an AI/ML system
338. In the context of AI, this threat is related to assets of the Processes category, and refers to Data potential eavesdropping of data or hijacking of Confidentiality Weak encryption communications in the case of data Processes transfers/storage/processing. The threat when Integrity Environment/tools materialized may expose data sets and even personal and sensitive information.
339. Tampering of communication networks may lead to their unavailability and thus is a major threat that may be exploited by adversaries. The Communication corresponding outages may lead to delays in Confidentiality networks decision-making, delays in the processing of Environment/tools tampering data streams and entire AI systems being placed Availability offline. Moreover, side-channel attacks may expose private and sensitive information that traverses communication networks.
340. Errors or timely
341. This type of threat is related to data and restrictions due Integrity computational exposure and/or inadequate to non-reliable Environment/tools capacity that may expose data and compromise data Availability privacy preservation.
342. infrastructures Physical
343. attacks Physical attacks against infrastructure (IT and corporate services) that supports AI systems’ operation and maintenance are a potential
344. threat. Manifestation of this threat leads to tem physical Availability Environment/tools degraded performance and even unavailability.
345. The threat manifests by physically attacking, dismantling or even destroying the actual physical infrastructure.
346. Sabotaging the model is a nefarious threat that refers to exploitation or physical damage to Environment/tools Model Sabotage hardware hosting libraries and machine learning Availability platforms that host or supply AI/ML services and Model
347. systems.
348. Sabotage involves intentionally destroying or Sabotage maliciously affecting the IT infrastructure that Availability Environment/tools supports AI systems.
349. Outages to communication networks may
350. adversely influence the performance and
351. networks outages
352. operation of AI systems. Such outages may lead Availability Environment/tools to delays in decision-making, delays in the processing of data streams and entire AI Outages systems being placed offline.
353. Outages to infrastructure (IT and corporate services) that supports AI systems’ operation
354. and maintenance. Manifestation of this threat Availability Environment/tools
355. tem outages
356. leads to degraded performance and even unavailability.
357. Environmental phenomena may adversely influence the operation of IT infrastructure and hardware systems that support AI systems. Environmental Climate change in particular has been phenomena consistently highlighted in ENISA reports on Availability Environment/tools (heating, cooling, telecom incident reporting as the main cause of Disasters climate change) telecom outages. Such outages may lead to delays in decision-making, delays in the processing of data streams and entire AI systems being placed offline.
358. Natural disasters may lead to unavailability or
359. Natural disasters
360. destruction of the IT infrastructures and (earthquake, Availability Environment/tools hardware that enables the operation,
361. flood, fire, etc.)
362. deployment and maintenance of AI systems.