AMC & GM to Part-ORA — Amendment 5

Annex II to ED Decision 2018/001/R

Annex II to ED Decision 2018/001/R

‘Acceptable Means of Compliance (AMCs) and Guidance Material (GM) to Part-ORA Amendment 5’

The text of the amendment is arranged to show deleted text, new or amended text as shown below: (a) deleted text is struck through; (b) new or amended text is highlighted in grey; (c) an ellipsis ‘[…]’ indicates that the rest of the text is unchanged.

1. AMC1 ORA.ATO.230(a) ‘Training manual and operations manual’ is amended as follows:

‘AMC1 ORA.ATO.230(a) Training manual and operations manual TRAINING MANUAL Training manuals for use at an ATO conducting to conduct integrated or modular flight training courses should include the following: (a) The training plan: […] (9) Assessments, tests and examinations (i) Flying: (A) progress checks; (B) skill tests. (ii) Theoretical knowledge: (A) progress tests; (B) theoretical knowledge examinations; (C) Area 100 KSA assessments. (iii) Authorisation for test; (iv) Rules concerning refresher training before retest; (v) Test and assessment reports and records; (vi) Procedures for examination paper preparation, type of question and assessment, standard required for ‘pass’; (vii) Procedure for question analysis and review and for raising replacement papers; (viii) Examination resit procedures.

[…] (d) Theoretical knowledge instruction […]

Annex II to ED Decision 2018/001/R

(6) Review procedure […] (7) Appendices (i) Examples of Area 100 KSA summative assessments; (ii) Area 100 KSA mental maths test example.’

2. New AMC2 to ORA.ATO.230(a) ‘Training manual and operations manual’ is inserted:

‘AMC2 ORA.ATO.230(a) Training manual and operations manual THEORETICAL KNOWLEDGE COURSE DESIGN REQUIREMENTS An ATO that delivers theoretical knowledge instruction for professional pilot licences should ensure that: (a) the courses are designed and developed using the instructional systems design (ISD) methodology, which is supported by a robust and effective management system; (b) the courses include a standardised and dynamic assessment and testing system; (c) instructors that deliver KSA instruction have received appropriate training covering at least learning styles, teaching methods, facilitation techniques, threat and error management (TEM), the applicable competencies, and the content of the subject(s) and exercises that they are to deliver; (d) the recurrent training of instructors is conducted at least annually; (e) the instructors that are responsible for assessing Area 100 KSA have received appropriate training regarding the assessment(s) that they are to conduct, and are to be standardised to ensure that the assessment grades awarded are consistent across the ATO; this standardisation should include at least familiarisation with the performance indicators, the ATO’s word pictures for grading, and the ATO’s debriefing system; and (f) recurrent standardisation training is conducted at least annually to ensure continued inter-rater reliability.’

3. New AMC3 to ORA.ATO.230(a) ‘Training manual and operations manual’ is inserted:

‘AMC3 ORA.ATO.230(a) Training manual and operations manual AREA KSA 100 02 AND 100 03 LEARNING OBJECTIVES, ASSESSMENTS AND RECORDS (a) An ATO that delivers theoretical knowledge instruction for professional pilot licences should ensure that for the learning objectives (LOs) in topics 100 02 and 100 03 of Area 100 KSA there are at least two summative assessments and at least one formative assessment. The summative assessments are to be documented in the student’s training records. Both the summative assessments and the formative assessment(s) should be debriefed. (b) The formative assessment(s) should: (1) be designed such that the student has the opportunity to ask questions and develop competencies in most of the LOs in 100 02 and 100 03 of Area 100 KSA; (2) be conducted during the training; the ATO may in addition conduct a formative evaluation (continuous assessment) over a specified phase of the course; and (3) be conducted by an instructor that is trained to deliver the formative assessment. (c) The summative assessments should:

Annex II to ED Decision 2018/001/R

(1) be designed so that they collectively give the student the opportunity to demonstrate competency in all LOs in 100 02 and 100 03 of Area 100 KSA; each individual summative assessment may address some of the LOs in 100 02 and 100 03 of Area 100 KSA; (2) be satisfactorily completed before the student is recommended by the ATO for their first attempt to take the final theoretical knowledge examination paper, and the outcome of the assessments should be included in the student’s training record; (3) require that for a student to be considered that they have achieved a ‘Satisfactory’ standard, they: (i) meet at least 35 % (which defines the term ‘some’ used in the word pictures) of the indicators relevant to the assessment exercise, in each competency; (ii) have an overall positive effect on the outcome or completion of the exercise without any external input from the instructor, or where the assessment requires the instructor to facilitate the exercise, without the instructor providing any knowledge or corrective input to assist in the completion of the exercise; and (4) be conducted by an instructor that is trained to deliver the summative assessments. (d) The training manual should include the following elements regarding the theoretical knowledge training and assessment of the LOs in topics 100 02 and 100 03 of Area 100 KSA: (1) the positions, or range of positions, of the formative assessment exercise(s) and summative assessment exercises in the training programme; (2) a description of the summative assessments, including a matrix that shows which Area 100 KSA LOs are covered in each exercise; (3) the grading system of the Area 100 KSA summative assessment and a description of the ATO’s minimum required standard; (4) the template for the information about Area 100 KSA to be included in the student’s training record, which should include at least the dates and result (‘Pass’ or ‘Fail’) of the summative assessments and the date and score of the mental maths test; (5) the method of assessment debrief for each summative and formative assessment; (6) for a student who performs below the satisfactory standard in a summative assessment(s), the method to further develop the student’s competencies and how to conduct the reassessment. (e) Access to the information on Area 100 KSA kept in the student’s training records should be restricted to the student and authorised ATO personnel, and should not be disclosed outside the ATO. The information on the record should first be de-identified before it is used to support course design improvements.’

4. New AMC4 to ORA.ATO.230(a) ‘Training manual and operations manual’ is inserted:

‘AMC4 ORA.ATO.230(a) Training manual and operations manual AREA 100 04 LEARNING OBJECTIVES: MENTAL MATHS TEST AND RECORDS (a) An ATO that delivers theoretical knowledge instruction for professional pilot licences should ensure that at least one KSA mental maths test is conducted and the outcome(s) documented in the student’s training records. (b) The mental maths test(s) may be written or oral in format and should, where possible, be scenariobased, with at least two questions per LO in topic 100 04 of Area 100 KSA.

Annex II to ED Decision 2018/001/R

(c) The minimum score to pass the Area 100 KSA mental maths test(s) should be 75 % of the marks allocated to a test. However, a higher pass mark may be defined by the ATO. (d) The mental maths test(s) should be satisfactorily completed before the student is recommended by the ATO for their first attempt to take their final theoretical knowledge examination paper.’

5. New GM1 ORA.ATO.230(a) ‘Training manual and operations manual’ is inserted:

‘GM1 ORA.ATO.230(a) Training manual and operations manual ASSESSMENT OF STUDENTS IN AREA 100 KSA (a) The Area 100 KSA formative assessment(s) and summative assessments may include but not be limited to: written planning exercises combining multiple subjects; practical exercises using training devices (if available); scenario-based oral board (viva voce); scenario-based communications exercises; written assignments or project work; and preparation and delivery of group or individual presentations. (b) The format of formative and summative assessment debriefs should be effective, highlighting the student’s strengths and weaknesses and enabling future improvement.’

6. New GM2 ORA.ATO.230(a) ‘Training manual and operations manual’ is inserted:

‘GM2 ORA.ATO.230(a) Training manual and operations manual AREA 100 KSA WORD PICTURES (a) ‘Word pictures’ are a proven assessment tool that standardises pilot core competencies, and can be used to assess student’s competency in the Area 100 KSA LOs in topics 100 02 and 100 03. Word pictures describe the student’s performance. Each word picture is associated with a numerical grade; within the range of grades, the minimum acceptable standard is defined. Additionally, a word picture describing performance that falls below the minimum satisfactory standard should be included in the range, as well as additional word pictures that relate to grades which exceed this minimum satisfactory standard. Word pictures enable the standardisation of the assessment performance and facilitate inter-rater reliability within an ATO. (b) This GM provides two examples of word pictures. (c) The most commonly used word pictures are shown in Section A below. They are based on performance indicators, which explain what the student should demonstrate in order to attain the specific Area 100 KSA LOs that are addressed by the assessment exercise. Word pictures are formed of elements that contain the following: (1) HOW MANY of the performance indicators were observed and, where relevant, HOW OFTEN; (2) HOW WELL the competency was demonstrated in the assessment exercise to have an overall positive effect on the outcome or completion of the assessment exercise; (3) the level of success in the OUTCOME of the assessment exercise. (d) An ATO could establish its own set of word picture descriptions as long as they are comparable in the grading of each competency, similar to the ‘Communication’ and ‘Application of knowledge, UPRT and resilience’ word pictures example in Section B below. (e) The advantage of word pictures is that they provide meaningful and standard data to enable identification of individual, crew, class, instructor and ATO trends, which can be analysed in order to provide feedback for further improvement or development.

Annex II to ED Decision 2018/001/R

(f) An ATO should ensure that the detailed information obtained through its grading in Area 100 KSA is deidentified before using it to support course improvement. SECTION A — EXAMPLE 1 AREA 100 KSA WORD PICTURE GRADE LEVELS (USING INDICATORS) (g) The example shown below in this Section contains the most commonly used word pictures, which are formed of elements that contain the following: (1) HOW MANY of the performance indicators in the table further below relevant to that summative assessment were observed in that competency (as a percentage); (2) HOW WELL the competency was demonstrated in the assessment; and (3) the level of success in the OUTCOME of the summative assessment. (h) In order to satisfactorily complete an Area KSA 100 summative assessment, the student should reach at least the minimum satisfactory level in each competency covered by that assessment. In case the student fails to reach the minimum satisfactory level in each competency, the student should repeat the summative assessment or another summative assessment that covers the competency(ies) where performance was previously assessed as unsatisfactory.

Annex II to ED Decision 2018/001/R

Table 1: Example generic competency framework that can be applied for assessing the student’s level of performance

Competency Level 1 Level 2 Level 3 Level 4 Level 5 Unsatisfactory Satisfactory Good Very good Excellent

General The student’s The student’s The student’s The student’s The student’s description of performance in this performance in this performance in this performance in this performance in this each competency competency was competency was competency was competency was competency was level. ineffective or satisfactory, which had effective, which in the highly effective, exemplary, which in inadequate, which in a slightly positive effect case of an exercise which in the case of the case of an relation to this on the satisfactory where the student is an exercise where exercise where the To be applied to competency had a outcome of the the only participant, the student is the student is the only each individual neutral or negative exercise, and in group significantly only participant, participant, had an competency in effect on others or on situations had a slightly contributed to a good significantly outstanding effect LOs 100 02 and the outcome of the positive effect on outcome. enhanced the very on the excellent 100 03 of Area exercise. others. In group situations, good outcome. outcome of the 100 KSA. the student’s In group situations, exercise. In group The student showed The student showed at contribution had a the student’s situations, the none or few of the least some* of the good effect on others contribution had a student’s relevant performance relevant performance and significantly very good effect on contribution had an indicators in this indicators in this contributed to the others and excellent effect on competency. competency. overall outcome of the significantly others and had an exercise. enhanced the overall outstanding effect outcome of the on the overall The student showed exercise. outcome of the most of the relevant exercise. performance The student showed indicators to a good most or all of the The student showed standard. relevant all of the relevant performance performance indicators to a very indicators to an good standard. excellent standard.

Annex II to ED Decision 2018/001/R

AREA 100 KSA ASSESSMENT PERFORMANCE INDICATORS (i) The performance indicators that relate to the LOs in topics 100 02 and 100 03 of Area 100 KSA can be used to form a word picture.

Table 2: Performance indicators relevant to the LOs in topics 100 02 to 100 03 of Area 100 KSA

Competency Competency description Indicators

Communication Demonstrates effective — Ensures the recipient is ready and prepared to oral, non-verbal and receive the information. written communication — Selects appropriately what, when, how and with skills in classroom exercise whom to communicate. and assessment situations. — Conveys messages clearly, accurately and concisely. — Confirms that the recipient correctly understands important information. — Listens actively and demonstrates understanding of the information they receive. — Asks relevant and effective questions. — Adheres to standard radio-telephony phraseology. — Accurately reads, interprets, constructs and responds to given documentation in English. — Correctly interprets non-verbal communication. — Uses eye contact, body language and gestures that are consistent with and support verbal messages. Leadership and Displays effective — Creates an atmosphere of open communication and teamwork leadership and teamwork. encourages team participation. — Uses initiative and gives directions when required. — Admits mistakes and takes responsibility. — Anticipates and responds appropriately to others’ needs. — Carries out instructions when directed. — Communicates relevant concerns and intentions. — Gives and receives feedback constructively. — Demonstrates empathy and shows respect and tolerance for others. — Engages others in planning and allocates activities fairly and appropriately according to abilities. — Addresses and resolves conflicts and disagreements in a constructive manner. — Projects self-control.

Annex II to ED Decision 2018/001/R

Problem-solving Accurately identifies risks — Seeks accurate and adequate information from and decision- and resolves problems. appropriate sources. making Uses the appropriate — Identifies and verifies what and why things have decision-making processes. gone wrong. — Employs proper problem-solving strategies. — Perseveres in working through problems. — Uses appropriate decision-making processes in a timely manner. — Sets priorities appropriately. — Identifies and considers options effectively. — Monitors, reviews, and adapts decisions as required. — Identifies and manages risks effectively. Situation Perceives and — Identifies and assesses accurately the general awareness comprehends all the environment as it may affect the operation. relevant information — Identifies and manages threats, errors, and available, anticipates what undesirable aircraft states. could happen that could affect the exercise or situations discussed in the classroom, and gives effective solutions to resolve the situation. Workload Manages available — Maintains self-control. management resources or time to — Plans, prioritises and schedules tasks effectively. efficiently prioritise and — Manages time efficiently when carrying out tasks. complete or perform tasks — Offers and accepts assistance, delegates when in a timely manner. necessary, and asks for help early. — Manages and recovers from interruptions, distractions, variations, and failures effectively. Application of Demonstrates correct and — Correctly completes pre-flight planning in the knowledge, deep understanding of the practical exercise. UPRT and subject(s), and is able to — Demonstrates KSA and TEM relating to phases of resilience effectively relate this flight in the ground training environment. knowledge between — Correctly and effectively applies knowledge to subjects and apply the identify and manage threats and errors that could knowledge for effective lead to a potential upset in scenario situations. threat and error — Recognises potential upset ‘threats’ and suggests management (TEM). effective ‘threat management’ in scenario situations. — Recognises potential upset ‘errors’ and suggests effective ‘error management’ in scenario situations.

Annex II to ED Decision 2018/001/R

— Identifies the causes of and contributing factors to upsets in aircraft accident and incident reviews and in reported recovered situations or scenarios. — Is resilient, i.e. recognises and adapts to disruptions during scenarios and other exercises. — Identifies the signs of stress and discusses the effects of stress, fatigue and aviation lifestyle on situation awareness, including how to cope with these in order to maintain situation awareness.

Annex II to ED Decision 2018/001/R

SECTION B — EXAMPLE 2

AREA 100 KSA WORD PICTURES (USING DESCRIPTIONS)

(j) An ATO may devise its own word pictures for each of the competencies, to be used alongside or combined with those given in EXAMPLE 1.

A descriptive word picture typically includes descriptive examples that the ATO’s instructors could readily identify and then equate to given

competency levels. Below are two examples (for ‘Communication’ and ‘Application of knowledge, UPRT and resilience’).

Table 3: Example word pictures using descriptions

Competency Level 1 Level 2 Level 3 Level 4 Level 5 Unsatisfactory Satisfactory Good Very good Excellent

Communication The student’s The student’s written or The student’s The student’s The student’s

performance in oral communication was written or oral written or oral communication skills

communication had a sufficient to convey the communication was communication was were exemplary.

neutral or negative intent of the exercise. good. consistently very At all times the

effect on the exercise good. student observed In oral communication Explanations, or situation. others and ensured the listener may have discussions, All communication that their own The student may not rarely needed to ask for directions and was clear, concise communication was have contributed to clarification which the comments were and well-structured, extremely effective. the exercise or the student then positively well-structured and which ensured a very

communication was and clearly provided. clear. effective outcome. In group situations,

unclear or insufficient. the student’s The student listened to The student listened In group situations, communication The student may have instructions but may have to others actively the student’s ability enabled all members occasionally occasionally been reticent and when unsure to interpret others’ to contribute to their interrupted others, and hesitant to ask asked appropriate body language, and greatest ability not listened, or questions or make questions to seek the use of body whilst also ensuring showed frustration or comments. clarification. language to ensure a an excellent inappropriate non- positive outcome, The student may have The student showed outcome of the objective was very effective. rarely shown appropriate exercise. communication. underconfidence or confidence and open At all times the In group situations, The student may have passiveness during the body language. student was calm, when appropriate, asked unrelated or exercise. However, engaged and the student

Annex II to ED Decision 2018/001/R

unclear questions, or overall their confident. proactively and provided insufficient communication was subtly managed the clarity in directions or sufficient to ensure a group’s mood or comments for others positive outcome of the motivated the group to understand. exercise. members appropriately. The student may have written without structure or clarity.

Competency Level 1 Level 2 Level 3 Level 4 Level 5 Unsatisfactory Satisfactory Good Very good Excellent

Application of The student’s The student had the The student The student The student had an knowledge, knowledge was at minimum acceptable demonstrated a demonstrated a very excellent level of UPRT and times insufficient or level of knowledge to good level of good level of understanding which resilience incorrect, which had complete the exercise to knowledge with the knowledge, and they immediately an adverse effect on a satisfactory standard. ability to relate this correctly and readily and appropriately the exercise. The student occasionally knowledge related this applied across demonstrated the ability effectively between knowledge across subjects and to the The student displayed to relate knowledge subjects and in subjects and in exercise or scenario limited ability to relate between subjects. scenario exercises or scenario situations. situation. knowledge between situations. subjects or to apply The student could The student The student knowledge to identify some threats or The student identified most identified all actual scenarios, exercises or errors, and when identified many threats and errors, threats and errors in in answers to presented with a threat threats and errors, and immediately scenario situations, questions. or error could in most and when presented used their anticipated some situations suggest at with threats or knowledge to possible threats and least one possible errors used their manage them errors (what if’s), effective method of knowledge effectively. and used their mitigation. effectively to suggest knowledge to appropriate manage them mitigations and efficiently and very actions. effectively.’

Annex II to ED Decision 2018/001/R

7. New GM3 ORA.ATO.230(a) ‘Training manual and operations manual’ is inserted:

‘GM3 ORA.ATO.230(a) Training manual and operations manual AREA 100 KSA EXERCISES AND ASSESSMENTS Exercises and assessments are to be interwoven into the theoretical knowledge training, utilising a range of learning styles; they should address subject or cross-subject topics, with the application of threat and error management (TEM) and, where possible, be scenario-based. The exercises and assessments do not need to be confined to a classroom. (a) Area 100 KSA exercises may be of short duration within a lesson, and the student’s performance in the exercises does not need to be recorded, although the main subject and KSA learning points are likely to be discussed (or for distance learning, reviewed) within a post exercise debrief or lesson summary. To allow for flexibility and development, the exercises do not need to be specified in the training plan. (b) When a single formative assessment is specified in the training plan, it is likely to be extensive as it will cover many of the LOs in Area KSA 100 02 and 100 03. Alternatively, an ATO may specify a number of shorter-duration formative assessments that each covers a narrower range of LOs, and these may build in terms of content difficulty. (c) The exercises and formative and summative assessments may include but not be limited to: scenario planning exercises combining multiple subjects; practical exercises using training devices (where available); oral communication exercises; written assignments and/or project work; discussions; the preparation and delivery of group or individual presentations and discussions; and enable scenariobased content and individual, pair or group situation(s). (d) The type of assessment and the environment should be recorded in the ATO’s training plan.’

8. New GM4 ORA.ATO.230(a) ‘Training manual and operations manual’ is inserted:

‘GM4 ORA.ATO.230(a) Training manual and operations manual

AREA 100 KSA INSTRUCTION AND ASSESSMENT TRAINING (a) The following material has been developed to provide additional guidance to organisations to help them develop an effective KSA 100 instruction and assessment training programme that satisfies the provisions in AMC2 ORA.ATO.230 (c) to (f). (b) An ATO should ensure that an instructor who conducts the Area 100 KSA formative assessment(s) has received adequate training to be familiar with the: (1) relevant competencies and performance indicators; (2) Area 100 KSA Learning Objectives (LOs); (3) formative assessment(s) that they will conduct including: the applicable LOs, purpose and content of the assessment(s) and position(s) in the training plan, assessment resources, assessment environment; (4) Area 100 KSA grading system, including familiarisation with the performance indicators and the ATO’s word pictures; and (5) student debrief methods and procedure. (c) An ATO should ensure that an instructor who conducts the Area 100 KSA summative assessments has received adequate training to be familiar with:

Annex II to ED Decision 2018/001/R

(1) the summative assessments that they will conduct including: the applicable LOs, purpose and content of the exercise(s) and position(s) in the training plan, assessment resources, assessment environment, and the minimum acceptable level; (2) the assessment feedback, evaluation and development process; and (3) KSA candidate appeal procedure. (d) An Area 100 KSA instruction and assessment course should include practical training on the conduct of an assessment, including grading to achieve inter-rater reliability, and the debrief under supervision.’

9. New GM5 ORA.ATO.230(a) ‘Training manual and operations manual’ is inserted:

‘GM5 ORA.ATO.230(a) Training manual and operations manual INSTRUCTIONAL SYSTEMS DESIGN (a) The instructional systems design (ISD) provides a systematic and iterative process for course design based on educational best practices. There are several effective ISD models in use today, with the analyse, design, develop, implement and evaluate (ADDIE) framework being generic to all. The purpose of using ISD to design training courses is to facilitate the students’ efficient and effective acquisition of knowledge and skills based on current training needs. (b) To provide evidence of the effective use of the ISD methodology in the design and continued development of their course(s), an ATO may use documentation and records that relate to the ISD phases. (d) ADDIE model example. The ‘analysis’, ‘design’, ‘development’, ‘implementation’ and ‘evaluation’ phases of the ADDIE model are shown below with brief phase descriptions.

Annex II to ED Decision 2018/001/R

Analyse the training needs

Determine the requirements to be fulfilled: required trainee standard at course start and how this is achieved (selection, or screening and where necessary pre-training).

Evaluate the course Design the training course,

Analyse results, reports, tests and assessment plan feedback and data to Includes establishing determine how well the objectives and performance course met its objectives. indicators (including ATO Determine whether assessment environment(s), training needs have been standards and procedures). modified by using external Design the course syllabus inputs/data. and the training and Iteratively improve and assessment plan. further develop the course. .

Implement the course Develop the training and

Delivery of the course. assessment materials Collection of feedback Includes the selection of from trainees, training methods and instructors and tools, the production of assessors during and training, testing and after the course. assessment materials, and the training of instructors and assessors.’

10. AMC1 ORA.ATO.300 ‘General’ is amended as follows:

‘AMC1 ORA.ATO.300 General DISTANCE LEARNING […] (c) Where an assessment (e.g. planning, written, scenario or practical exercise, or other assessment) is conducted outside the classroom via distance learning, the ATO should have a procedure or process in place to establish that the student themselves have completed the assessment and that the assessment method(s) for that particular exercise has (have) been effective.’