BEREC Guidelines detailing Quality of Service Parameters

Final

BEREC Guidelines detailing Quality of Service Parameters

6 MARCH 2020 1. Introduction

1. According to Article 4 of the BEREC Regulation , BEREC shall issue guidelines on the implementation of the European Union regulatory framework for electronic communications, as referred to in Directive (EU) 2018/1972 (hereinafter referred to as “the EECC”), on, among other things, relevant quality of service (“QoS”) parameters which National Regulatory Authorities (“NRAs”), in coordination with other competent authorities, should take utmost account of. Annex 1 to the Guidelines sets out the wording of Article 104 and Annex X of the EECC as well as the related EECC recitals. For the avoidance of doubt, definitions as set out in Article 2 of the EECC shall be used in this document unless otherwise stated (see Annex 2 to the Guidelines). 2. In accordance with Article 104(2) of the EECC, the reason for issuing the present guidelines (hereinafter also referred to as the “Guidelines”) is to provide guidance to NRAs in respect to Article 104 of the EECC and to contribute to the consistent application of Article 104(2) and Annex X, with the aim of defining: a) the relevant QoS parameters, including the parameters relevant for end-users with disabilities; b) the applicable measurement methods for these QoS parameters, including, 3 4 where appropriate, the ETSI and ITU standards set out in Annex X of the EECC in relation to interpersonal communications services (“ICS”) and Internet access services (“IAS”), respectively; c) the content and format of publication of the QoS information, and d) the quality certification mechanisms. 3. Annex 3 to the Guidelines sets out the input received from NRAs in respect to any specified QoS parameters, measurement methods, and the content, form and manner of the information published, under the relevant provisions of Article 22 of the Universal Service Directive . Annex 4 to the Guidelines sets out other benchmarking undertaken in respect to QoS indicators across Member States. 4. Article 104 of the EECC contains a specific reference to Regulation (EU) 2015/2120 : “the measures to ensure quality of service shall comply with Regulation (EU) 2015/2120”, whose provisions in terms of QoS have been analysed in a number of BEREC documents. The Guidelines focus on providing definitions and measurement methods for the IAS QoS parameters related to the network 2 performance listed in Annex X of the EECC, i.e. latency, jitter and packet loss, taking into account previous BEREC reports, especially BEREC Net Neutrality Regulatory Assessment Methodology BoR (17) 178 , in order to ensure the consistency of BEREC’s documents. 5. Additional guidance can be found in the BEREC reports: Monitoring quality of Internet access services in the context of net neutrality BoR (14) 117 ; Net neutrality 9 10 measurement tool specification BoR (17) 179 and BoR (18) 32 Annex 1 . These Guidelines shall be applied to information published on QoS parameters, as specified by NRAs, following transposition of the EECC into national law.

2. Policy principle, legal basis and scope of the BEREC Guidelines

2.1 Policy principle

6. In the ever-connected, globalized, digital environment that is developing faster and faster, electronic communication services play a key role in citizens’ everyday activities. As set out in the QoS regulation manual ITU 2017 pgs. 5 & 110 – “The profusion of ever-evolving technologies, networks, services and devices with different QoS capabilities further adds to the complexity of regulation in this area. Quality can be impacted by many factors at the network level and along the value chain. In this regard, a common approach to regulating QoS can enable greater quality prospects irrespective of the locations of the consumer and service provider”. Indeed, the correlation between quality and pricing of services suggests: “If the demand for services that require high QoS is very low (compared to demand for services not requiring QoS), then the willingness to pay for high QoS will be also very low. In such a case, telecommunication operators (which are in fact the ISPs nowadays) will have lower interest in QoS. When the demand for services that require high QoS is comparable with demand for services not requiring QoS, then the willingness to pay for QoS is higher ”. 7. The QoS, as perceived by the end-user, is a crucial factor for both customers and service providers and, with the profusion of ever evolving technologies, networks and services with different levels of QoS, it is becoming increasingly more complex to manage, measure and regulate it. Indeed, quality can be impacted by many factors at the network level and along the value chain, including the device, hardware, infrastructure, service and applications . 3

8. Regulatory development in the European electronic communications sector is intended to help improve the end-user experience, to lead to greater competition and investment, and to benefit all the different players in the digital ecosystem. This development has resulted in the EECC and BEREC undertaking complex work aimed, inter alia, at achieving one of the very clear objectives of the EECC, i.e., empowering and protecting end-users.

9. The European harmonisation of QoS parameters for data collection and publication practices would result in substantive benefits, such as enabling comparability among Member States and providing better information on the European electronic communications market, while at the same time promoting the consistent application of regulatory obligations and improving transparency for end-users and public authorities in relation to QoS.

2.2 Legal basis

10. The rationale for issuing Guidelines detailing QoS parameters is, therefore, to contribute to a consistent and harmonized application of the provisions of the EECC, in particular with respect to Article 104 that is broadly a continuation of Article 22 of the Universal Service Directive (2002/22/EC), on the publication of information for end-users on the QoS.

11. More specifically, Article 104(1) of the EECC provides that NRAs in coordination with other competent authorities may require providers of IAS and of publicly available ICS to publish comprehensive, comparable, reliable, user-friendly and upto-date information for end-users on the quality of their services and on measures taken to ensure equivalence in access for end-users with disabilities.

12. Pursuant to information requirements for contracts set out in Article 102 and Annex VIII of the EECC, end-users should be informed, inter alia, of the different levels of the QoS, conditions for promotions and termination of contracts, applicable tariff plans and tariffs for services subject to particular pricing conditions.

13. At the same time, Recital 271 of the EECC provides that NRAs in coordination with other competent authorities should be empowered to monitor the QoS and to systematically collect information about the QoS offered by providers of IAS and of publicly available ICS, to the extent that the latter are able to offer minimum levels of service quality either through control of at least some elements of the network or by virtue of a service level agreement (SLA) to that end, including the quality related to the provision of services to end-users with disabilities. That information should be collected on the basis of criteria which allow comparability between service providers and between Member States. Providers of such electronic communications services, operating in a competitive environment, are likely to make adequate and up-to-date information on their services publicly available for reasons of commercial advantage. NRAs in coordination with other competent authorities should nonetheless be able to require publication of such information where it is demonstrated that such information is not effectively available to the public, including information on equal access and choice for end-users with disabilities. 4

14. Moreover, according to Article 104(1) of the EECC, NRAs in coordination with other competent authorities may require providers of publicly available ICS to inform consumers if the quality of the services they provide depends on any external factors, such as control of signal transmission or network connectivity.

15. It follows from Article 104(1) of the EECC that the information obligations which an NRA may require from a given provider depend on two criteria: • firstly, the relevant service (IAS and/or publicly available ICS); • secondly, whether the provider controls at least some elements of the network either directly or by virtue of an SLA to that effect.

16. In accordance with Article 104(2) of the EECC, NRAs in coordination with other competent authorities shall specify, taking utmost account of the Guidelines, the QoS parameters to be measured, the applicable measurement methods, and the content, form and manner of the information to be published, including possible quality certification mechanisms, using where appropriate, the parameters, definitions and measurement methods set out in Annex X of the EECC.

17. In order to facilitate comparability across the European Union and to reduce compliance cost, according to Recital 272, BEREC should adopt guidelines on relevant QoS parameters which NRAs in coordination with other competent authorities should take utmost account of.

18. Moreover, Article 104(1) provides that “The measures to ensure quality of service shall comply with Regulation (EU) 2015/2120”. Article 4(1) of Regulation (EU) 2015/2120 contains specific transparency obligations for providers of IAS. In the BEREC Guidelines on Net Neutrality adopted in 2016 , BEREC has further explained the transparency obligations contained in Article 4(1), first sentence, (a)- (e).

2.3 Scope of the BEREC Guidelines

19. In light of the above, the Guidelines, in pursuing the goal of providing transparency to consumers on QoS, provide assistance to NRAs on the QoS parameters that NRAs in coordination with other competent authorities could specify for measurement by the providers, together with the applicable methodologies, the information to be published and the possible quality certification mechanisms.

20. Furthermore, QoS can be distinguished from Quality of Experience (QoE) as QoS concerns the network and terminal equipment up to the user interface while QoE focuses on the entire service experience and includes the whole path from user to user including the end-user expectation, perception and context of use. QoS also includes the assistive equipment and the specific services provided to end-users with disabilities where equipment is provided by the ICS Provider (see footnote 36). For more details on QoE see ITU-T Rec P.10/G.100 . Network performance (NP) is more limited in scope because it excludes terminal performance. Figure 1 shows 5 the relationship between these terms. For the purpose of the Guidelines only QoS is taken into consideration. Figure 1: QoS, QoE, NP (source BEREC, 2011) CPE: Customer Premise Equipment, UNI: User-to-Network Interface

21. The Guidelines focus on QoS parameters related to ICS and IAS as well as the corresponding measurement methods and certification mechanisms.

22. ICS can be provided by different technological means. Firstly, in the event where the provider has control over network elements (e.g. because he owns the network) or has an SLA with a network operator - it is possible for the provider of the ICS to give information on QoS parameters. For example, in the case of telephony services which are provided as “traditional” (i.e. non-Over The Top – OTT) telephony services or as specialised services (i.e. managed services), it is possible for the provider to indicate the corresponding QoS parameters in the case where voice connections are originated and terminated within the providers network while endusers use specific terminal equipment. In this scenario, a provider is obliged to fulfil the information requirements set out in Article 104(1) of the EECC, if an NRA in coordination with other competent authorities requires such.

23. Secondly, when the provider has neither control over network elements, nor has an SLA to that effect: this situation may arise if the interpersonal communication services are provided over the internet, i.e. number-independent ICS (NIICS). In this event, the quality of the ICS depends on the quality of the IAS and terminal equipment used. For example, a provider of a messaging service which also has a voice service functionality cannot indicate the QoS of the voice call because the quality of the voice call is influenced by the underlying IAS and terminal equipment used. According to Article 104(1) of the EECC, an NRA in coordination with other competent authorities may require the provider of the NIICS to inform consumers if the quality of the services they provide depends on any external factors, such as control of signal transmission, network connectivity and terminal equipment. If the NRA in coordination with other competent authorities requires so, a NIICS provider is obliged to inform consumers that the voice quality depends e.g. on the quality of the underlying IAS and the terminal equipment used. However, the NIICS provider cannot make a statement or guarantee a QoS as this is outside the area of his control. 6

24. Different standards have been defined to detail methodologies to measure QoS of ICS and IAS. The measurement methods specified by NRAs in coordination with other competent authorities should be based, where appropriate, on standards as set out in Tables 1, 1A, 2 and 3 of the Guidelines.

25. It should be noted that the measures, which an NRA in coordination with other competent authorities may define under Article 104(1) of the EECC and under Article 5 of Regulation (EU) 2015/2120 shall be coherent .

26. IAS and ICS providers should assess all factors that may impact the QoS levels available to end-users, for example, user environment or the bias brought by the location of test servers or interconnection issues, etc. Where possible, providers should take these factors into consideration during the measurement process.

3. QoS Parameters & Measurement Methods for Interpersonal Communication Services and Internet Access Services

27. According to Figure 1 as set out in paragraph 20 of this document and referenced by the EC in its Final Report “BEREC Report on the convergence of Fixed and Mobile Networks” , QoS refers to the effectiveness of performance of a system in support of end-user needs or that contributes positively to another system’s performance.

28. In addition, as set out in Annex 4 to the Guidelines, an extract of the most widely mandated existing QoS indicators across Member States include the following: • Voice - Call set-up time; Unsuccessful call rate; Speech transmission quality; Response time for calls to the operator, customer service and directory assistance; • Mobile - Network availability; Probability of successful connection in an area covered by the network; Dropped call ratio; • Customer service - Time between request for service and start of service; Fault frequency; Time to troubleshoot & eliminate faults; Frequency of complaints about billing; • Internet - Data transfer speed; Web page loading time; Latency; Jitter; Packet loss rate.

29. Depending on the nature of the content to be exchanged (e.g. audio, video, text, data), different ICS quality parameters need to be specified. ETSI EG 202 057 multipart deliverable standards (see Annex X of the EECC) provide guidance on the basic approach to be applied by the providers in order to assess the different aspects of quality.

30. Providers of NIICS and NBICS as well as any other provider of electronic communication services cannot know and influence the technical characteristics of interconnected networks and terminal equipment used at the endpoints of the communication when these are not provided or selected by the providers of NIICS 7 and NBICS. Thus, providers can only specify estimates of the resulting communication quality of actual end-to-end communications. However, such providers are only subject to Article 104 of the EECC in so far as they control parts of the network or have an SLA with a network operator to that effect.

31. Typically, NIICS are designed to compensate for the varying transport quality of packet switched networks and the best effort packet forwarding principle. They do not have stringent requirements for network quality and just require an adequate overall performance level (e.g. a maximum delay value not to be exceeded for realtime communication).

32. Table 1 below lists QoS parameters, definitions and measurement methods from Annex X of the EECC which shall be used, where appropriate, for ICS.

33. For completeness, BEREC has proposed ETSI definitions and measurement methods for two QoS parameters set out in Annex X of the EECC (the call set up failure probability parameter and the call signalling delays parameter), which currently do not have definitions and measurements methods provided, and which shall be used by NRAs in coordination with other competent authorities where appropriate (see Table 1 below).

34. Table 1A below lists QoS parameters from Annex X of the EECC for IAS together 21 7 with definitions and measurement methods from ITU, IETF and BoR (17) 178 .

35. Article 104 contains a specific reference to Regulation (EU) 2015/2120: “the measures to ensure quality of service shall comply with Regulation (EU) . 2015/2120”

36. Annex X of the EECC contains some basic QoS parameters and measurement methods for IAS. Concerning network performance, besides speed, the most important parameters which influence QoS of IAS are delay, delay variation (jitter) and packet loss (see BEREC Guidelines on the Implementation by National Regulators of European Net Neutrality Rules , paragraph 137).

37. In these Guidelines BEREC has considered measurement methods as set out in BEREC Net Neutrality Regulatory Assessment Methodology BoR (17) 178 , both to be consistent with other BEREC reports and to take account of national circumstances and other significant factors (see paragraph 38). A practical implementation of the methodology is currently being developed by BEREC which is based on the Net neutrality measurement tool specification BoR (17) 179 and BoR (18) 32 , Annex 1.

38. It is important to note here that there is a degree of flexibility allowed when deciding which QoS parameters are to be measured by the providers and therefore should be specified by NRAs. To this end NRAs in coordination with other competent authorities are free to choose among the QoS parameters listed in Table 1 and Table 1A, those that are appropriate, taking into account national circumstances and other factors, such as, the meaningfulness and usefulness of the parameter, the underlying costs, time needed to implement the measurement and possible monitoring systems, changes required to adapt and modify current methodologies and allowing for the possibility of comparing new results with previous records. NRAs are therefore not obliged to specify the full list of parameters contained in Table 1 and Table 1A, but can choose the ones that are particularly relevant for the needs of their country. Where NRAs in coordination with other competent authorities 8 choose to impose relevant and appropriate QoS parameters from Table 1 and Table 1A, they shall take utmost account of the Guidelines and of the definitions and the measurement methods listed in Table 1 and Table 1A.

Table 1 QoS Parameters as set out in Annex X of the EECC

QoS Definition Measurement method Parameters Annex X

Supply time ETSI ES 202 057-1 (clause 5.1) ETSI ES 202 057-1 (clause 5.1.3) for initial The duration from the instant of a It is measured by: connection valid service order being received a) the times by which the fastest by a direct service provider to the 50%, 95% and 99% of orders are instant a working service is made completed; available for use. This should b)the percentage of orders exclude cancelled orders. completed by the date agreed with Applicable to both fixed and mobile the customer and, where the services. percentage of orders completed by the date agreed with the customer is below 80%, the average number of days, for the late orders, by which the agreed date is exceeded.

Statistics for both fixed and mobile access networks.

Fault rate per ETSI ES 202 057-1 (clause 5.4) ETSI ES 202 057-1 (clause 5.4.3) access line The number of reported faults per Statistics for all fixed access lines. fixed access line per year.

Fault repair ETSI ES 202 057-1 (clause 5.5) ETSI ES 202 057-1 (clause 5.5.3) time The duration from the instant a fault It is measured by: report has been made to the instant a) the time by which the fastest 80% when the service element or and 95% of valid faults on access service has been restored to lines are repaired (expressed in clock normal working order. hours); b) the percentage of faults cleared Applicable to fixed services only. any time stated as an objective by the service provider;

9

QoS Definition Measurement method Parameters Annex X

c) the provision of information on the hours during which faults may be reported.

Statistics for all access fixed networks.

Call setup ETSI EG 202 057-2 (clause 5.2) ETSI EG 202 057-2 (clause 5.2.3) time The call set up time is the period It is measured by: starting when the address a) the mean value in seconds for information required for setting up a national calls; call is received by the network and b) the time in seconds within which finishing when the called party busy the fastest 95% of national calls are tone or ringing tone or answer set-up; signal is received by the calling c) the mean value in seconds for party. international calls; Where overlap signalling is used d) the time in seconds within which the measurement starts when the fastest 95% of international calls sufficient address information has are set-up; been received to all the network to e)the number of observations begin routeing the call. performed for national and international calls. Applicable to both fixed and mobile calls. Statistics for both fixed and mobile voice services.

3GPP TS 32.454 clause 5.1.2 3GPP TS 32.454 clause 5.1.2 Session setup time It is measured by the mean value Applicable for IMS (VoLTE KPI) The mean setup time of the sessions

Bill ETSI ES 202 057-1 (clause 5.11) ETSI ES 202 057-1 (clause 5.11.3) correctness The proportion of bills resulting in a It is measured by a percentage. complaints customer complaint about the correctness of a given bill per service.

Applicable to both fixed and mobile services.

Voice ETSI EG 202 057-2 (clause 5.3) ETSI EG 202 057-2 (clause 5.3.2)

10

QoS Definition Measurement method Parameters Annex X

connection ETSI TR 102 506 Statistics for: quality Evaluation of speech quality per - Fixed to fixed calls call. - Fixed to mobile calls The end-user perceived voice - Mobile to fixed calls quality. - Mobile to mobile calls

Applicable to fixed and mobile voice ITU-T G.1020: Performance services. parameter definitions for quality of speech and other voice band applications utilizing IP networks.

ITU-T G.1028: End-to-end quality of service for voice over 4G mobile networks.

ITU-T P.863: Perceptual objective listening quality prediction.

Dropped call ETSI EG 202 057-3 (clause 6.4.2) ETSI EG 202 057-3 (clause 6.4.2.2) ratio The proportion of incoming and When using the measurements outgoing calls which, once they based on network element counters, have been correctly established the following statistics should be and therefore have an assigned provided: the percentage of dropped traffic channel, are dropped or calls, calculated from all the calls in interrupted prior to their normal the period. completion by the user, the cause When using test calls, the following of the early termination being within statistics should be provided: the the operator's network. percentage of dropped calls, together with the number of Applicable to mobile networks. observations used and the absolute accuracy limits for 95% confidence calculated from this number.

3GPP TS 32.454 clause 5.2.1 3GPP TS 32.454 clause 5.2.1 Call drop for IMS session It is measured by a percentage. Applicable for IMS (VoLTE KPI) The number of dropped sessions divided by the number of successful session establishments.

Unsuccessful ETSI EG 202 057-2 (clause 5.1) ETSI EG 202 057-2 (clause 5.1.3) call ratio Unsuccessful call ratio is defined as It is measured by: the ratio of unsuccessful calls to the a) the percentage of unsuccessful total number of call attempts in a calls for national calls; specified time period. b) the percentage of unsuccessful calls for international calls; Applicable for both fixed and mobile c)the number of observations used networks. for national and international calls

11

QoS Definition Measurement method Parameters Annex X

together with absolute accuracy.

Call set up ETSI TS 102 024-9 (clause 4.1.1) ETSI TS 102 024-9 (clause 4.1.1) failure The ratio of total call setup attempts probability that result in call setup failure to the total call setup attempts in a population of interest.

Applicable to both fixed and mobile calls.

Call signalling ETSI TS 102 024-9 (clause 4.2) ETSI TS 102 024-9 (clause 4.2) delays It involves three different scenarios: call setup, call answer and call release delays. The Call Setup Delay (CSD) is the time between the calling terminal providing sufficient address information to set up the call, and the calling party receiving a confirmation from the called terminal that the called party is being alerted. The Call Answer Signal Delay (CASD) is the time between the called terminal indicating that it is ready to initiate the call and receipt of that indication by the calling terminal. The Call Release Delay (CRD) is the time between the clearing terminal initiating a call cleardown, and its receipt of clearing confirmation by the called terminal.

Applicable to both fixed and mobile calls.

Table 1A - IAS QoS Parameters as set out in Annex X of the EECC

12

QoS Parameters Definition Measurement method Annex X

Latency Ref. IETF RFC 2681 Ref. BoR (17) 178 Sec 3.2 (delay) The time between the It is recommended that delay is measured first bit of a packet of a using: source entering a • UDP with ICMP or TCP as fall back option, network, being received • at least 10 measurements, and by the destination, which • calculated as an average of recorded roundimmediately sent a bit trip time values (typically expressed in back to the source, and milliseconds). then the last bit of the The measurement server should return any packet arriving at the UDP packet payload immediately, allowing the source across the client to calculate delay. The Unix echo service network (round trip could be used for this function. The delay). measurement setup should be insensitive to (user) clock changes during the measurement.

Delay Ref. IETF RFC 3393 Ref. BoR (17) 178 Sec 3.2 variation The difference between It is recommended that the delay variation (jitter) the delay of the selected (jitter) is calculated as mean deviation based on packets. the samples collected for the delay measurement.

Packet Loss Ref. ITU-T Y.2617 Ref. BoR (17) 178 Sec 3.3 Ratio The total number of If a packet is not received back within a certain packets failing to deliver timeout (e.g. 3 seconds), it is considered as lost through the network for the purpose of packet loss measurements. divided by the total Recommended to send a large number of IP number of transmitted packets (e.g. at least 1000). packets within a specific time window. Delay and packet loss measurements are typically performed over a longer period of time in order to allow for the time varying nature of network performance in packet-switched networks.

13

39. Additionally, NRAs in coordination with other competent authorities who choose to specify other parameters, to be measured by the providers, namely for customer services (ICS and IAS), and for end-users with disabilities that are not included in Annex X of the EECC, shall take utmost account of the QoS parameters listed in Table 2 below and Section 4, “QoS Parameters relevant for end-users with disabilities” (Table 3). To note, NRAs may require providers to publish additional measurements at national level which are not referred to or listed in Tables 1-3 and therefore do not form part of these Guidelines. However, where guidance is provided for specified QoS parameters listed in Tables 1-3 of these Guidelines, the guidance herein must be taken utmost account of.

Table 2 QoS Parameters not set out in Annex X of the EECC

Additional Definition Measurement method QoS Parameters (not in Annex X)

Response time ETSI ES 202 057-1 (clause 5.6.1) ETSI EG 202 057-1 (clause for operator Time elapsed between the end of 5.6.3) 25 services dialling to the instant the human It is measured by: (Customer operator answers the calling user to a) mean time to answers; Care Services provide the service requested. b) percentage of calls answered – Help Desk) Applicable to both fixed and mobile within 20 seconds. services.

Customer ETSI ES 202 057-1 (clause 5.10.1) ETSI ES 202 057-1 (clause complaints The duration from the instant a 5.10.3) resolution time customer complaint is notified to the It is measured by: published point of contact of a service a) the time by which the fastest provider and is not found to be invalid 80% and 95% of complaints to the instant the cause for the have been resolved (expressed complaint has been resolved. in clock hours); b)the percentage of complaints Applicable to both fixed and mobile resolved any time stated as an services. objective by the service provider.

14

4. QoS Parameters relevant for end-users with disabilities 4.1 Legal basis

40. According to Article 3(2)(d) of the EECC, NRAs in coordination with other competent authorities should promote the interests of the citizens of the Union inter alia by ensuring a high and common level of protection for end-users through the necessary sector-specific rules and by addressing the needs, such as affordable prices, of specific social groups, in particular end-users with disabilities, elderly endusers and end-users with special social needs, choice and equivalent access for end-users with disabilities.

41. NRAs should note that accessibility requirements for products and services, including accessibly of electronic communication services, are harmonised in the European Accessibility Act (EAA) ,as stated in Article 85(4) of the EECC: “Member States shall ensure, in light of national conditions, that support is provided, as appropriate, to consumers with disabilities, and that other specific measures are taken, where appropriate, with a view to ensuring that related terminal equipment, and specific equipment and specific services that enhance equivalent access, including where necessary total conversation services and relay services, are available and affordable”.

42. The EAA defines persons with disabilities in line with the United Nations Convention on the Rights of Persons with Disabilities, adopted on 13 December 2006 (UN CRPD). For the purpose of the EAA and the EECC, persons with disabilities shall be understood as persons who have long-term physical, mental, intellectual or sensory impairments, which in interaction with various barriers may hinder their full and effective participation in society on an equal basis with others (Article 3(1) EAA).

4.2. QoS Parameters and Measurement Methods

43. The EECC and the EAA define specific services targeted to address the needs of persons with disabilities that should be of concern to NRAs when considering QoS parameters in coordination with other competent authorities to be measured by the providers. Relay services refer to services which enable two-way communication between remote end-users of different modes of communication (for example text, sign, speech) by providing conversion between those modes of communication, normally by a human operator.

44. Real time text is defined in Article 3(14) of the EAA and refers to a form of text conversation in point to point situations or in multipoint conferencing where the text being entered is sent in such a way that the communication is perceived by the user as being continuous on a character-by-character basis.

45. According to Article 2(35) of the EECC total conversation service means a multimedia real time conversation service that provides bidirectional symmetric real time transfer of motion video, real time text and voice between users in two or more locations. 15

46. NRAs should note that other transparency measures concerning equivalent access for persons with disabilities are set out in Article 102(1) , and Article 103(1) of the EECC, in particular, the competent authority in coordination, where relevant, with the national regulatory authority can oblige service providers to publish details of products and services, including any functions, practices, policies and procedures and alterations in the operation of the service, specifically designed for end-users with disabilities, in accordance with European Union law harmonising accessibility requirements for products and services.

47. NRAs in coordination with other competent authorities could accompany the QoS parameters to be measured by the providers listed in Table 1, Table 1A and Table 2 by appropriate QoS parameters concerning equivalent access for persons with disabilities, choosing among those listed in Table 3, bearing in mind that all QoS indicators set out in the Guidelines, in particular, those related to IAS, should address the specific needs of end-users with disabilities.

48. It should be noted that obligations from the EAA will be in place from 2022 and the measures will be applicable from 2025 .

Table 3 – QoS Parameters relevant for end-users with disabilities

Service QoS Definition Measurement Parameters method

Voice Audio ETSI/CEN/CENELEC EN 301 549 v3.1.1 ETSI/CEN/CENEL communicati bandwidth for (clause 6.1) EC EN 301 549 on speech Where ICT provides two-way voice v3.1.1 (clause communication, in order to provide good audio C.6.1) quality, that ICT shall be able to encode and decode two-way voice communication with a frequency range with an upper limit of at least 7 000 Hz.

16

Service QoS Definition Measurement Parameters method

Real-Time Distinguishable ETSI/CEN/CENELEC EN 301 549 v3.1.1 ETSI/CEN/CENEL Text (RTT) display (clause 6.2.2.1) EC EN 301 549 v3.1.1 (clause Where ICT has RTT send and receive C.6.2.2.1) capabilities, displayed sent text shall be visually differentiated from and separated from received text.

Programmat- ETSI/CEN/CENELEC EN 301 549 v3.1.1 ETSI/CEN/CENEL ically (clause 6.2.2.2) EC EN 301 549 determinable v3.1.1 (clause Where ICT has RTT send and receive send and C.6.2.2.2) capabilities, the send/receive direction of receive transmitted/received text shall be direction programmatically determinable, unless the RTT is implemented as closed functionality .

Interoperability ETS/CEN/CENELEC EN 301 549 v3.1.1 ETSI/CEN/CENEL (clause 6.2.3) EC EN 301 549 v3.1.1 (clause Where ICT with RTT functionality interoperates C.6.2.3) with other ICT with RTT functionality, they shall support the applicable RTT interoperability mechanisms described in clause 6.2.3 .

RTT ETSI/CEN/CENELEC EN 301 549 v3.1.1 ETSI/CEN/CENEL Responsive- (clause 6.2.4) EC EN 301 549 ness v3.1.1 (clause Where ICT utilises RTT input, that RTT input C.6.2.4) shall be transmitted to the ICT network, or platform on which the ICT runs within 500 ms of the time that the smallest reliably composed unit of text entry is available to the ICT for transmission. Delays due to platform or network performance shall not be included in the 500 ms

17

Service QoS Definition Measurement Parameters method

limit.

Video Resolution ETSI/CEN/CENELEC EN 301 549 v3.1.1 ETS/CEN/CENELE communicati (clause 6.5.2) C I EN 301 549 v3.1.1 (clause on Where ICT, that provides two-way voice C.6.5.2) communication, includes real-time video functionality, the ICT:

a) shall support at least QVGA resolution;

b) should preferably support at least VGA resolution.

Frame Rate ETSI/CEN/CENELEC EN 301 549 v3.1.1 ETSI/CEN/CENEL (clause 6.5.3) EC EN 301 549 v3.1.1 (clause Where ICT, that provides two-way voice C.6.5.3) communication, includes real-time video functionality, the ICT:

a) shall support a frame rate of at least 20 frames per second (FPS);

b) should preferably support a frame rate of at least 30 frames per second (FPS) with or without sign language in the video stream.

Synchroniz- ETSI/CEN/CENELEC EN 301 549 v3.1.1 ETSI/CEN/CENEL ation between (clause 6.5.4) EC EN 301 549 audio and v3.1.1 (clause Where ICT that provides two-way voice C.6.5.4) video communication, includes real-time video functionality, the ICT shall ensure a maximum time difference of 100 ms between the speech and video presented to the user.

18

5. Publication of information

49. NRAs should note that the publication requirements that the NRAs may set in accordance with Article 104(1) are in addition to the transparency measures provided for in Articles 102 and 103 of the EECC and the transparency obligations set out in Article 4(1) of Regulation (EU) 2015/2120.

50. According to Recital 271 of the EECC, NRAs should be able to require publication of information described in Article 104(1) of the EECC, where it is demonstrated that such information is not effectively available to the public.

51. Detailed guidelines for the transparency measures for ensuring open internet access can also be found in BEREC Guidelines on the Implementation by National Regulators of European Net Neutrality Rules .

52. According to Article 104(1) of the EECC, the information on QoS required by NRAs should be comprehensive, comparable, reliable, user-friendly and up-to-date.

53. The requirement that information is “comprehensive” and “user-friendly” means that it should be representative as well as understood by members of the intended audience. NRAs in coordination with other competent authorities should look to ensure that service providers adhere to the following practices in order to ensure that information is user friendly: • it should promote the use of relevant standards; • it should be presented, preferably, using clear and plain language, in as simple a manner as possible, avoiding complex sentence and language structures. • the information should be concrete and definitive; • it should not be phrased in abstract or ambivalent terms; • it should avoid unduly technical terminology; and • it should not include excessively detailed information.

54. Information is “comparable” if the same relevant information is presented, by different providers or by the same providers for different offers, for comparison in such a way that it can show differences and similarities. Information should be comparable at least between different offers, and between different service providers.

55. The “reliable” element means that information should be correct and cannot be misleading for end-users. Information shall comply with standards and measurement methodology indicated by NRAs in coordination with other competent authorities, preferably using certified mechanisms if such mechanisms have been introduced in a given Member State.

56. End-users may like to check information on the QoS of the service, and in that regard, the information should be easy to find, easy to understand, up to date and presented in an accessible way.

57. Where relevant, information in respect to the provision of services through special equipment, for example, information in respect to assistive technologies and the provision of augmentative and alternative communication devices, should be available to end-users with disabilities who require it to access the ICS. Information should be available in respect to types of text relay services available. 19

58. All published information shall be up-to-date. NRAs in coordination with other competent authorities shall ensure that service providers are obliged to insert the date of the updated publications and reference the period of update. As well as information concerning QoS parameters, service providers can be obliged to publish information showing the most recent update of data at a minimum frequency on an annual basis.

59. Information should be accessible for the broadest possible group of end-users including in particular end-users with disabilities, elderly end-users and end-users with special social needs. To achieve that aim NRAs in coordination with other competent authorities could oblige service providers to publish information: • in a machine-readable manner and in an accessible format for end-users with disabilities taking into account general accessibility requirements set in Section III of Annex I of the EAA and European standards aiming to address the needs of persons with disabilities and older persons, dealing with accessibility by applying the Design for all approach – ETSI/CEN/CENELEC EN 301 549 version 3.1.1 ; • on the provider websites (no more than one click from the /homepage) and via mobile applications that are viewable, operable, understandable and robust and meets harmonised published standards.

60. NRAs in coordination with other competent authorities could oblige service providers (where warranted) to directly publish information via their own communication channels (direct approach) or oblige service providers to publish information through third parties and provide information to NRAs to publish simultaneously on NRAs websites.

61. According to Recital 271 of the EECC, NRAs in coordination with other competent authorities should be empowered to monitor the QoS and to collect systematically information on the QoS offered by providers on the basis of criteria which allow comparability between service providers and between Member States. To achieve these objectives NRAs in coordination with other competent authorities could require service providers in accordance to Article 104(1) of the EECC to publish information having regard to different levels of aggregation (regional, national) or different groups of end-users (business clients, consumers), depending on the level of availability of information to the public, QoS parameter or service.

62. To that end, and to enhance overall publication, NRAs shall consider the inclusion of QoE (quality of experience) indicators if appropriate.

6. Quality Certification mechanisms

63. Where NRAs require publication as provided for in 104(1), the following applies to quality certification mechanisms (Article 104(2) EECC), “NRAs in coordination with other competent authorities shall specify the quality of service parameters to be measured, the applicable measurement methods, and the content, form and manner of the information to be published, including possible quality certification mechanisms”. Moreover, Article 4(4) of the Regulation (EU) 2015/2120 refers to the quality monitoring mechanism certified by an NRA. 20

64. The EECC does not require Member States or an NRA to establish or certify a monitoring mechanism. Regarding IAS, Article 104(2) of the EECC does not have any impact on monitoring mechanisms in relation to Article 4(4) of Regulation (EU) 2015/2120, which references a monitoring mechanism certified by the NRA. Indeed, the BEREC Guidelines on the Implementation by National Regulators of European Net Neutrality Rules stipulate that if the NRA provides a monitoring mechanism implemented for this purpose, it should be considered as a certified monitoring mechanism in relation to Article 4(4) of Regulation (EU) 2015/2120.

65. Plural “quality certification mechanisms” used in Article 104 of the EECC anticipates the possibility of functioning of more than one certification mechanism, e.g., for internet access services and publicly available interpersonal communications services).

66. EECC provisions do not prescribe who may be a provider of a quality certification mechanism.

67. NRAs in coordination with other competent authorities must take into account the requirement of independence of the provider of the quality certification mechanism from IAS and publicly available ICS providers. In this context, the NRA may take into account not only circumstances pointing to capital or personal links with telecommunications service providers operating in the market, but also the business model of the quality certification mechanism provider.

68. The approach taken by the NRA in coordination with other competent authorities to specify the quality certification mechanism may take many various forms. Provisions of the EECC do not impose requirements on the certification procedure. The level of formalization of the procedure as well as additional requirements, such as the requirement for a specific form of the certification act (e.g. an administrative decision, ordinance) may be determined in national law.

69. The EECC does not set out requirements about the certification period, the conditions for the certification withdrawal, or extending the certification.

70. The NRA in coordination with other competent authorities should determine what factors are to be taken into account when choosing a quality certification mechanism. The certification should ensure that the quality monitoring fulfils requirements, such as: 1. Accuracy - The results of measurements should be accurate as far as it is possible in accordance with the state-of-the-art knowledge and with the reservation that the end-user or consumer should not be loaded with disproportionate obligations associated with performance of measurements, in particular, if these requirements do not have a significant impact on the result. Achieving this objective cannot limit the availability of the mechanism for quality monitoring for all end-users. When assessing the factors that can affect the accuracy and reliability of measuring the quality of the IAS and other publicly available ICS, it may be appropriate for NRAs to specify the requirements that should be met by the end-user environment . 2. Enables comparison of measurements - The quality monitoring mechanism should make it possible to compare the results of the QoS measurements 21 carried out with those QoS parameters which are included in the contract as well as enabling a comparison between different service providers. 3. Openness - The measurement methodology and implementation should be publicly available, and the NRA or other competent entity should consider publishing information on factors which can affect the reliability of results, if such factors have been identified. The publication of its source code contributes to the openness of the quality monitoring mechanism; however, a provider of a quality monitoring mechanism cannot be obliged to publish the source code. 4. Safety – The quality monitoring mechanism should be adequately safeguarded against attacks, and its integrity and the confidentiality of processed personal data against unauthorized access should be guaranteed. 5. Future-proofness - Quality monitoring mechanisms should be based on the current state of technical knowledge, and its design, taking into account the development and evolution of the telecommunications market. 6. Accessibility - The use of the quality monitoring mechanisms should be accessible to people with disabilities.

7. Review of Guidelines

7.1 Review Period

71. The process of undertaking a review of the Guidelines will commence 2 years from the adoption and publication of the Guidelines by BEREC. Subsequent reviews will be determined by BEREC and will be agreed and set out in future BEREC work programmes. 22

Annex 1 EECC Article 104 and Annex X

L 321/178 EN Official Journal of the European Union 17.12.2018

Article 104 - QoS related to IAS and publicly available interpersonal communications services

1. National regulatory authorities in coordination with other competent authorities may require providers of IAS and of publicly available interpersonal communications services to publish comprehensive, comparable, reliable, user-friendly and up-to-date information for end-users on the quality of their services, to the extent that they control at least some elements of the network either directly or by virtue of a service level agreement to that effect, and on measures taken to ensure equivalence in access for end-users with disabilities. National regulatory authorities in coordination with other competent authorities may also require providers of publicly available interpersonal communication services to inform consumers if the quality of the services they provide depends on any external factors, such as control of signal transmission or network connectivity. That information shall, on request, be supplied to the national regulatory and, where relevant, to other competent authorities before its publication. The measures to ensure QoS shall comply with Regulation (EU) 2015/2120.

2. National regulatory authorities in coordination with other competent authorities shall specify, taking utmost account of BEREC guidelines, the QoS parameters to be measured, the applicable measurement methods, and the content, form and manner of the information to be published, including possible quality certification mechanisms. Where appropriate, the parameters, definitions and measurement methods set out in Annex X shall be used. By 21 June 2020, in order to contribute to a consistent application of this paragraph and of Annex X, BEREC shall, after consulting stakeholders and in close cooperation with the Commission, adopt guidelines detailing the relevant QoS parameters, including parameters relevant for end-users with disabilities, the applicable measurement methods, the content and format of publication of the information, and quality certification mechanisms. ANNEX X- QUALITY OF SERVICE PARAMETERS Quality-of-Service Parameters, Definitions and Measurement Methods referred to in Article 104 For providers of access to a public electronic communications network:

PARAMETER DEFINITION MEASUREMENT METHOD (Note 1) Supply time for initial ETSI EG 202 057 ETSI EG 202 057 connection Fault rate per access line ETSI EG 202 057 ETSI EG 202 057 Fault repair time ETSI EG 202 057 ETSI EG 202 057

23

For providers of interpersonal communications services who exert control over at least some elements of the network or have a service level agreement to that effect with undertakings providing access to the network:

PARAMETER DEFINITION MEASUREMENT METHOD (Note 2) Call set up time ETSI EG 202 057 ETSI EG 202 057 Bill correctness complaints ETSI EG 202 057 ETSI EG 202 057 Voice connection quality ETSI EG 202 057 ETSI EG 202 057 Dropped call ratio ETSI EG 202 057 ETSI EG 202 057 Unsuccessful call ratio ETSI EG 202 057 ETSI EG 202 057 (Note 2) Failure probability Call signalling delays PARAMETER DEFINITION MEASUREMENT METHOD Latency (delay) ITU-T Y.2617 ITU-T Y.2617 Jitter ITU-T Y.2617 ITU-T Y.2617 Packet loss ITU-T Y.2617 ITU-T Y.2617

Note 1

Parameters shall allow for performance to be analysed at a regional level (namely, no less than level 2 in the Nomenclature of Territorial Units for Statistics (NUTS) established by Eurostat).

Note 2

Member States may decide not to require up-to-date information concerning the performance for those two parameters to be kept if evidence is available to show that performance in those two areas is satisfactory.

24

Recitals:

(271) National regulatory authorities in coordination with other competent authorities, or where relevant, other competent authorities in co-ordination with national regulatory authorities should be empowered to monitor the quality of services and to collect systematically information on the quality of services offered by providers of internet access services and of publicly available interpersonal communications services, to the extent that the latter are able to offer minimum levels of service quality either through control of at least some elements of the network or by virtue of a service level agreement to that end, including the quality related to the provision of services to end-users with disabilities. That information should be collected on the basis of criteria which allow comparability between service providers and between Member States. Providers of such electronic communications services, operating in a competitive environment, are likely to make adequate and up-to-date information on their services publicly available for reasons of commercial advantage. National regulatory authorities in coordination with other competent authorities, or where relevant, other competent authorities in co-ordination with national regulatory authorities should nonetheless be able to require publication of such information where it is demonstrated that such information is not effectively available to the public. Where the quality of services of publicly available interpersonal communication services depends on any external factors, such as control of signal transmission or network connectivity, national regulatory authorities in coordination with other competent authorities should be able to require providers of such services to inform their consumers accordingly.

(272) National regulatory authorities in coordination with other competent authorities should also set out the measurement methods to be applied by the service providers in order to improve the comparability of the data provided. In order to facilitate comparability across the Union and to reduce compliance cost, BEREC should adopt guidelines on relevant quality of service parameters which national regulatory authorities in coordination with other competent authorities should take into utmost account.

25

Annex 2 Definitions

This Annex contains a selection of terms and definitions used in the Guidelines to support the consistent and harmonised application of the provisions of Article 104 of the EECC.

Internet access services: a publicly available electronic communications service that provides access to the internet, and thereby connectivity to virtually all end points of the internet, irrespective of the network technology and terminal equipment used (Regulation (EU) 2015/2120).

Interpersonal communications services: a service normally provided for remuneration that enables direct interpersonal and interactive exchange of information via electronic communications networks between a finite number of persons, whereby the persons initiating or participating in the communication determine its recipient(s) and does not include services which enable interpersonal and interactive communication merely as a minor ancillary feature that is intrinsically linked to another service (Directive (EU) 2018/1972).

Number-based interpersonal communications service: an interpersonal communications service which connects with publicly assigned numbering resources, namely, a number or numbers in national or international numbering plans, or which enables communication with a number or numbers in national or international numbering plans (Directive (EU) 2018/1972).

Number-independent interpersonal communications service: an interpersonal communications service which does not connect with publicly assigned numbering resources, namely, a number or numbers in national or international numbering plans, or which does not enable communication with a number or numbers in national or international numbering plans (Directive (EU) 2018/1972).

26

Annex 3 Benchmarking

NRAs Questionnaire

NRAs were asked to respond to a questionnaire that contained nine questions. The purpose of the questionnaire was to assess the current level of harmonisation of the QoS standards/guidelines available in Member States for IAS and publicly available ICS under the relevant provisions of Article 22 of the Universal Service Directive . This section contains the aggregated results for each question and analysis of the reported parameters and processes which are currently available in each of the Member States. More specifically the questions covered the following key areas in relation to QoS:

• The QoS parameters in place • Guidelines/measures detailing the relevant QoS parameters • QoS parameters applicable for end-users with disabilities • Applicable measurement methods for these QoS parameters • Content, form and manner of the QoS information to be published • Quality certification mechanisms • Customer satisfaction surveys • Measurement of call waiting times for customer support • Compliance cases with respect to QoS

27 responses were received from the following NRAs. Austria (AT) Germany (DE) Norway (NO) Belgium (BE) Greece (EL) Poland (PL) Bulgaria (BG) Ireland (IE) Portugal (PT) Croatia (HR) Italy (IT) Romania (RO) Cyprus (CY) Latvia (LV Serbia (RS) Czech Republic (CZ) Lithuania (LT) Slovenia (SI) Denmark (DK) Malta (MT) Spain (ES) Finland (FI) The Netherlands (NL) Turkey (TR) France (FR) North Macedonia (MK) United Kingdom (UK)

Quality of Service measures in place

NRAs were asked to indicate what QoS measures are in place in their country. A wide range of QoS measures are listed in the responses for ICS and IAS. However, for fixed ICS, network performance measures stand out as the key measures

in place, i.e., supply time for connection , fault rate per access line and fault repair

times . Indeed some of the responses detail the full range of measures listed in Annex III of Directive 2002/22/EC to include measures relating to directory inquiry services, bill

correctness , working order of public payphones and call set up time . Other

measures include access to Text Relay Services , provision of information to

consumers regarding the indicators measured , publication of performance in relation

to measures , requirement to incorporate QoS parameters in users’ contracts ,

frequency of clients’ complaints , resolution time for complaints , complaints on pre-

paid lines and measures taken by providers to ensure equal access for end-users with

a physical disability .

28

Quality of Service measures in place for Mobile

A number of QoS measures are also in place for mobile networks to include quality of voice and data services such as network coverage, drop call rate, rate of successful SMS/MMS transfer, bit rate error, frame rate error based on ITU technical standards, throughput, packet loss, delay and jitter.

In one specific case , a QoS audit is performed on the mobile services to assess the QoS that mobile operators provide to users for benchmarking purposes so that users experience is reflected in various scenarios (in the city, rural areas, different forms of transport, etc) and to also include the services most used (calling, texting, web browsing, video streaming, file downloads, etc.). The NRA in question also requires the MNO to publish daily a list of antennas that are out of order/malfunctioning. In another case the NRA stated that similar measurements are conducted for the assessment of mobile network quality.

29

Quality of Service measures in place IAS

QoS measures are also in place for IAS to include minimum guaranteed IAS speed values for fixed and mobile networks . Operators are required to measure the achievable speeds on their networks and publish the results periodically/annually on 62 63 their websites . For fixed broadband, measures to include data transmission speeds , availability for internet access, latency, and packet loss are in place. In mobile networks, drive tests/ railway/train journeys are used to measure speeds which are then published on the website and may be used in certain cases to evaluate complaints received in this regard .

Guidelines or measures detailing the relevant quality of service parameters

30

NRAs were asked to indicate if they had guidelines or measures detailing the relevant QoS parameters in their country. In many cases , Regulations/Guidelines are in place specifying the quality levels/targets for ECS, methodologies used to collect and analyse performance data, inspection criteria, the submission of performance reports and the format and publication of same. In other cases , the terms of the USO sets guidance regarding QoS targets and technical requirements. A number of NRAs have issued rules in relation to the publication of internet speeds by IAS providers in line with Article 4 of the NN Regulations. The procedures established in ETSI standards are also relevant.

Parameters applicable for end-users with disabilities

NRAs were asked to indicate what QoS parameters are available for end-users with disabilities. Of the respondents to this question, twelve NRAs stated that there were no specific QoS parameters for end-users with disabilities. However, twelve respondents also provided details of available measures relevant to end-users with disabilities; a number of which, as listed below, identified unique quality of service associated with the services provided. A key theme emerging from the respondents was that service providers are obliged to ensure that end-users with disabilities obtain equivalent levels of access to services and choice of undertakings as those enjoyed by the majority of end-users. Also, in a number of instances , NRAs

31

oblige the USP to provide specific services under the USO for end-users with disabilities to include some of the following: - access to the emergency services via text messages - directory enquiry services and directories - text relay services and sign language services - publish availability of adapted terminal equipment to meet users’ needs - special pricing plans for deaf and blind users - telephone cabins with voice assistance - priority fault repair - accessibility, safety, information, comfort and convenience - accessible bill formats - a visually impaired user shall have unrestricted access to the services required for the customer relationship - providing appropriate devices in entities serving end-users - provide relevant formats of information regarding facilities offered and model contracts - provide appropriate terminal equipment and assistance in configuration of device in premises The following details provided in the responses outline specific examples of the available measures relevant to end-users with disabilities and their associated quality of service metrics to include the following services: a. fault Repair Time of <or equal to 48hours b. providers are obliged to offer ECS for deaf, hearing-impaired, blind-deaf or aphasic end-users with a simultaneous written and visual transcription, free of charge. Conditions of quality associated with the offer include; - QoS target >70% of users accessing an interpreter within 3 minutes - rate of users accessing an interpreter within 30 seconds; - dropout rate; - QoS target >99% for availability; - quality of experience ranging from 1 (poor) to 5 (excellent). c. for hearing or speech impaired users the subscription; - must allow internet connection for the use of video call and remote interpreting services with a speed of at least 512 kilobit/s for incoming and outgoing traffic; - a one-way delay in an IAS provide as a universal service to persons with hearing and speech impairments for the purpose of video conference and remote interpreting services must not exceed 150ms in the access network of a USP. d. accessible obligation including the following: - ensure the website and the information available for end-users with disabilities are compliant with the WCAG, AA Level.

One respondent stated that the General Conditions for service providers specifically require that providers must ensure the following: - access to directory information,

32

- relay services, - mobile SMS access to emergency organisations, - priority fault repair, - third party bill management and - bills and contracts in accessible formats as appropriate.

More generally, one NRA stated that operators are obliged to publish and submit an annual report to the NRA outlining the measures provided for accessibility of end-users with disabilities and state whether they voluntarily include indicators for their actions.

Applicable measurement methods for IAS Quality of Service parameters

NRAs were asked to indicate measurement methods applied. One NRA stated that there were no applicable measurement methods in place and that very few complaints were received. However, they planned to implement the BEREC tool for IAS measurement. Based on the remaining NRA responses , many of the measurement methods for QoS of ICS parameters are based on various iterations of the ETSI standards and where the service providers are required to submit performance data to the NRAs. With regard to IAS measurements, there are a range of methodologies used. These include: - interactive real time performance test tools developed or provided by the NRA , - a testing methodology based on Standard ITU -T Y.1564 , - by means of controlled probes that are installed in each region ,

33

- by users in a crowdsourced approach (based on the BEREC NN Regulatory Assessment Methodology), which provides the user with certified results of the QoS of the connection at the time of the test, - passive data collection method within the IAS network , - drive tests , - surveys based on data collected from internet speed test providers .

Content and format of publication of the information

NRAs were asked to indicate the content and format of the information. It is clear from all the responses received that there are various requirements for content of information with less emphasis on the format of publication of information. Many NRA responses listed the many parameters that are published periodically (some quarterly, every 6 months, or annually) on their websites or the operators websites, with some specifying 82 83 that information must be provided at regional and national level . One NRA stated that content, format, time limits or manner of publishing information regarding QoS is not specified but the information has to be clear, legible and simple. Another NRA listed requirements only for IAS parameters to include upload/download speed, delay, delay variation, limitations regarding terminal equipment usage and response times. Two NRAs indicated that online interactive mapping tools are provided which allow users to view all QoS data collected, with filtering settings for various parameters, and track progress that operators have made on their networks to improve QoS. Some responses provided very detailed requirements about providing terms and conditions of services to be made available in writing with clear and comprehensive information made available at stores for viewing or available online for consultation prior 86 87 to entering into contracts . One NRA publishes a detailed annual report comparing service quality for providers of fixed, mobile and broadband services so that consumers are informed should they wish to choose a new provider, the report is based on provider data, complaints data and market research findings.

34

Quality certification mechanisms

NRAs were asked to indicate if there are quality certification mechanisms in place in their countries. Nine NRAs stated that there are certified quality measurement systems in place for the measurement of IAS. One NRA reported that there is a requirement only for voice call systems (number based communications) to have the metrological certificate. The vast majority of NRAs stated that there are no quality certification mechanisms in place at the time of responding to the questionnaire. One NRA stated that the Regulation in place allows auditing the results of the measurements made, although such auditing is not implemented. Moreover, a number of NRAs have an online tool in place that users can check internet quality parameters.

35

Consumer satisfaction measures/ Indicators for measurement of call waiting times for customer support

NRAs were asked to indicate if consumer satisfaction measures and call waiting times for customer support were measured in their country. Approximately half of the responses received from NRAs indicated that they do not have consumer satisfaction measures in place. However, where measures are in place to ascertain consumer satisfaction, these are conducted in some cases by the NRA , in other cases the operator is obliged to carry out the research and in certain instances market research companies carry out the surveys on behalf of the NRA. Consumer perception/satisfaction surveys are conducted biennially in some instances, annually in other cases and every two years in one case. A number of responses indicated that, apart from questions about general consumer satisfaction, questions are tailored to obtain specific information and concerns that consumers are experiencing in their country at the time of the survey including topics such as: - satisfaction with mobile coverage at home location and travelling locations - the quality of helpdesks - are the correct answers provided and waiting times - impact of fibre optic access on behaviours and activities on the Internet Other responses stated that some measures are in place to gauge consumer satisfaction such as: - indirectly measure consumer satisfaction by assessing the no of users’ complaints - performance against QoS parameters for fixed telephone and internet service access: bill correctness complaints ratio/target value <1%

36

Eight NRAs reported various indicators that are in place for the measurement of call waiting times for customer support to include the following: - duration of the voice response system main menu <or equal to 45sec - time waiting until before option to connect to the person in the call centre <or equal to 20sec - non-binding direction on customer service which requires operators to answer consumer calls within 5 minutes – no measurement in place - comparison of average waiting times reported in the previous calendar year for major landline, mobile and broadband providers - measurements are in place for connections time for USO - average response time by helpdesk (seconds) and the % of calls responded to within 2 minutes - response time for operator service - average response time <20 sec – ETSI EG 201 769-1 - customer service response time – operators customer support service may not be >15 sec annually and % of calls answered by operators customer support service in 20 sec or less may not be <80% annually – ETSI EG 202 057-1

Compliance cases with respect to Quality of Service

NRAs were asked to provide details of compliance cases with respect to QoS. A majority of NRAs responses indicated one of the following: NRA did not respond to the

37

38

Annex 4 Other Benchmarking

2016 Study prepared for the EC – Fixed and Mobile Convergence in Europe – Quality Measurements for 5G and Network Densification Source EC Fixed and Mobile Convergence in Europe Quality Measurements for 5G and Network Densification Table 2.2

39