BEREC Guidelines on Geographical surveys of network deployments. Verification of information

10 June 2021 1 Introduction

1. According to Article 22(7) of the European Electronic Communications Code (‘EECC’), by 21 June 2020, BEREC shall, after consulting stakeholders and in close cooperation with the EC and relevant national authorities, issue guidelines to assist National Regulatory Authorities (NRAs) and/or Other Competent Authorities (OCAs) on the consistent implementation of Article 22 EECC.

2. In March 2020 BEREC issued the BEREC Guidelines on Geographical surveys of network deployments ) which dealt with the consistent application of the obligation for NRAs/OCAs to provide for geographical surveys of the reach of electronic communications networks capable of delivering broadband, and the forecasts of the reach of broadband networks, including very high capacity networks (‘VHCN’), that the authorities may undertake.

3. BoR (20) 42 (hereafter “Core Guidelines”) establishes the definitions for all the indicators that NRA/OCAs must provide for and as well the minimal granularity for the information. They also explain which operators are subject to provide information and deliver important classifications, namely on the kinds of technologies, speed tiers to consider and types of VHCN networks.

4. In the Core Guidelines BEREC concluded that QoS-1 indicators (i.e. theoretical network performance of existing infrastructure/calculated availability of service) would characterise the reach and performance of broadband networks.

5. In the current Guidelines BEREC deals with the verification of QoS-1 indicators as defined in the Core Guidelines. This is the verification of the current reach of broadband networks and the quality of the services that they could offer as declared by the operator, including the verification of the declaration of an area or grid as being covered with a VHCN network. The validation of address databases and the verification of information regarding planned deployments is out of the scope of the current guidelines.

6. It is indispensable that broadband maps of current broadband reach are accurate, since this information is used in the context of many regulatory and policy functions, as detailed in Article 22 (5). However, there are many reasons why the raw data provided by an operator may be partial or incorrect, ranging from unintentional errors made in the data processing, a misunderstanding of the Authority’s requirements or inappropriate assumptions in the modelling of QoS-1 data. Because of all of this, ensuring the quality of the data that nurtures broadband maps is an integral part of the processes leading to the publication and updating of broadband maps.

7. Consequently, BEREC considers that the process of assuring the quality of the data provided by operators is intrinsic to a consistent implementation of the obligations under Article 22 (as required by paragraph 7): collecting and relying on incorrect data would be detrimental to the correct fulfilling of the tasks of the authorities as prescribed by the EECC.

8. The purpose of the data quality assurance process can range from being very general (this is checking overall that an operator’s submission is correct) or quite specific, for example when guided by particular needs (for example, checking the availability of broadband coverage in areas where there are complaints by end users or where some public deployment is being considered).

9. Authorities need to guarantee that the quality assurance process is objective, transparent and non-discriminatory. This is best achieved when the data assurance methodology and a description of the key findings of the verification outcome is published so that operators understand what to expect (see section 5).

10. The process of ensuring the quality of data has five steps: STEP 1) validation of the internal consistency of the database; STEP 2) resorting to external agents to report data inaccuracies; STEP 3) the verification phase, where the data is contrasted against external sources of data; STEP 4) deciding after external validation that the data is not correct and; STEP 5) changing the data if necessary and other consequences. These steps may overlap in time, for example, STEP 2 may happen along the whole process once some information on broadband availability is published, for example when a broadband map is made public.

11. The Guidelines focus on steps 1 to 4. The consequences of data misreporting (STEP 5) depend on national legislation and country specificities , and are therefore not part of these Guidelines.

12. The current guidelines describe different verification methods and provide recommendations to be considered by Authorities when assessing the quality of QoS-1 data provided by operators. Authorities may choose verification methods depending on the information available and their needs. The guidelines stress that Authorities should publish their verification methodologies and outline the verification results in order to establish transparency and help make their proceedings plausible to market participants.

13. Authorities should update their maps and databases at their convenience, when data is corrected by an operator or the Authority. NRAs’ experience is that the process of quality assurance is ongoing and because of this the Geographical Survey information is ”alive”this is, past information may change with new submissions or as a result of verification.

14. It needs to be noted that the Core Guidelines refer to network capability and not to any retail service offered at the address/grid , so this challenges verification. For example: - For fixed broadband, addresses passed sometimes may have no connections or no connections of a certain type, so this makes it difficult to verify with measurements. - For mobile broadband, paragraph 72 in the Core Guidelines specifies that there should be a high likelihood of reception (with a speed of at least 2 Mbps available at least in 95% of the grid area with a successful reception of 95%), so instances of non-reception can happen, it is a matter of their order of magnitude and territorial dispersion. - In general, measurements made by end-users will depend, among other factors, on the characteristics of the broadband service they contract and may not reflect the network capability at a certain point or the best technologies available in a certain area.

15. In 2020, BEREC carried out a survey and found out that most NRAs/OCAs do not carry out verification of the QoS-1 indicators for the fixed broadband indicated in the Core Guidelines. Nine (out of sixteen) NRAs/OCAs carry out a verification of fixed broadband technologies, mainly by using crosschecks with other available databases like available data in public evidences, periodical statistical data collection databases, operators’ homepages or publicly available rollout information. For mobile broadband, 10 NRAs/OCAs (out of sixteen) carry out checks on 3G and 4G availability, (high likelihood of service reception). At the time, no NRA/OCA carried out verification of 5G availability. Measured data (drive tests) is used by four NRAs/OCAs to verify mobile broadband service reception. In the case of fixed broadband the use of measurements is more scarce.

16. While the provisions of the EECC would anticipate and foster also state aid compliance, it is not their main objective to ensure compliance with state aid rules. NRAs/OCAs can use information collected under Article 22 to assist the state aid process but may also need to collect complementary information in line with the State Aid rules.

17. Finally, BEREC has the exclusive EECC mandate to deliver on Article 22 Guidelines and the BEREC Guidelines (BoR (20) 42, BoR (21) 32 and the current BEREC Guidelines on the verification of information) constitute the unique basis for the consistent implementation of broadband mapping activities within the purposes of Article 22 EECC and other related Article 22 activities.

2 What to verify?

18. Paragraph 20 in the Core Guidelines explains that the responsibility of providing reach and performance indicators may rest directly with the operators or with Authorities, when those perform calculations on the basis of their knowledge of infrastructure data or according to assumptions as described in the Core Guidelines. This Guidelines focus on the operatordelivered data, but Authorities may also find useful to carry out some verification of their own calculations to test the assumptions they use in their modelling.

2.1 Fixed broadband

19. According to the Core Guidelines, in case of fixed broadband, the indicators to verify are: a.) Technology availability categorized as per subsection 2.4.1.3 in Core Guidelines; b.) Maximum Download speed class as per Table 7 in Annex 2 of the Core Guidelines; c.) Maximum Upload speed class as per Table 7 in Annex 2 of the Core Guidelines; d.) Expected Peak-Time Download speed class as per Table 7 in Annex 2 of the Core Guidelines; e.) Expected Peak-Time Upload speed class as per Table 7 in Annex 2 of the Core Guidelines); f.) VHCN class declaration as per Table 1 in Core Guidelines (this is, criterion 1, 2, 3 and 4 in the VHCN guidelines, where criterion 2 and 4 are relevant for Fixed Wireless Access). As stated by the Core Guidelines, this information must be produced at address or small grid level (100m x 100m area or smaller). According to the BEREC Guidelines on Very High Capacity Networks (hereafter VHCN Guidelines) , paragraph 19, peak-time is the time of the day with a typical duration of one hour where the network load usually has its maximum. This is relevant to the VHCN class declaration under Criteria 3 and 4 and for the submission of expected peak-time upload and download speeds.

2.2 Mobile broadband

20. According to the Core Guidelines, in case of mobile broadband, the indicators to verify are: a.) Technology availability as per section 2.4.2.3 in the Core Guidelines; b.) VHCN declaration per Table 3 in the Core Guidelines (this is, criteria 2 and 4 in the VHCN Guidelines); c.) Upload Maximum Speed classes as per Table 7 in Annex 2 of the Core Guidelines; d.) Download Maximum Speed classes as per Table 7 in Annex 2 of the Core Guidelines. As stated by the Core Guidelines, this information must be produced at small grid level (100m x 100m area or smaller) and the information on upload and download maximum speed classes (c and d above) are optional to Authorities. According to the VHCN Guidelines, paragraph 19, peak-time is the time of the day with a typical duration of one hour where the network load usually has its maximum. This is relevant for the VHCN declaration under Criteria 4 in the VHCN Guidelines.

3 Internal validation (STEP 1)

21. All authorities should routinely check the internal consistency of QoS-1 data provided by operators.

22. This encompasses activities to ensure the quality of data using only the main data base (this is the data base that results from the collection of QoS-1 information as per the Core Guidelines), for example, checking missing data, looking for possible problems in reporting units (for example, speed or technology codes that have not been required by the Authority) or looking into inconsistencies within the information in the main data base. There could be some algorithms or automatic controls of provided data to assess its consistency to exchange formats / logical tests.

23. A detailed analysis of the main data base can allow identifying possible irregularities or anomalies in the data provided by the operator. For example, declarations as if the whole network of the operator is reported at one (or few) locations and one single broadband connection (network termination point, or a couple of them) which would often indicate that there was an administrative or typing error made by the operator while filling the data in. NRA/OCAs may look for discrepancies and irregularities and then confirm the correctness of such data with the operator. 6

24. Moreover, in so far the main data base allows comparing data declarations for different periods, the Authority can also check that there are no unreasonable discontinuities. In the particular case of georeferenced data, a good practise is to check visually the evolution of the maps to identify potential problems with the data that has been gathered.

25. Checking the integrity of the data base should be done by Authorities before any publication.

4 Use of third parties to find out about inaccuracies in data (STEP 2)

26. When data on the broadband coverage or characteristics on the terrain is made public by the Authority and interaction with the public is most welcome, information from third parties may enable the Authority to suspect that some particular data is inaccurate. For example, after data is publicly available through a Mapping tool, citizens who are not able to get the published speed have the possibility to contact the Authority .

27. These third parties can be end-users, other public administrations and even ECN and ECS providers. The display of public information of good territorial granularity also adds value to all these parties and may also discipline operators in ensuring that they deliver accurate data, as misreported data may upset end-users.

28. Generally, data from third parties is not conclusive on the accuracy of the data reported by an operator, as it may result from measurement or perception problems. Even when a third-party report of inaccurate data is right, in some cases, it can still be compatible with the operator’s declared information, for example when the third-party report is explained by events which are beyond the operator’s control or because the two pieces of information are not exactly the same, and the differences can be explained by the measurement methodology, for example.

29. Yet, third-party information can provide an indication of areas, operators and/or technologies where there may be problems. In those instances, first of all, the Authority 7 should contact the operator to see how he responds to the reported data discrepancies as in many cases this will prove sufficient to solve the problem.

30. When the information provided by third-parties is good and sufficient (for example, there is a number of complaints or those are recurrent) and the operator cannot reasonably explain the differences, this can trigger further investigation by the NRA.

31. For example, once a broadband map is published, direct end-user declarations (which may be sustained by some type of measurement done by the end-user) can help identify problematic data . In order to facilitate the end-users’ submission of geographically referenced data and to ease the internal cross-checking of information, it is good to integrate the third-party reporting of data inaccuracies in the broadband map itself, with a given format and detailed information request such as personal details, operator, mobile/fixed broadband, commercial retail offer name, contractual maximum speeds, measurements if available, technology if known.

32. Some NRAs find that allowing access to the mapping data for reuse by stakeholders enables a better diffusion of the information and more reporting of problems with data. For example, in one MS, an API function has been developed that allows the viewing of the broadband map on other websites. A large website displaying property for sale is using the API and is a source of reports for data inaccuracies.

33. It is advisable to provide this mapping information at the best granularity possible. However, if very detailed or granular maps cannot be published at a given time, either for legal or technical reasons, Authorities can publish information at a more aggregate level, municipal or sub-municipal administrative level, for example. Information on each zone being covered with a certain speed or broadband service of a given characteristic may enable the general public, other companies and other public administrations to respond to this and report discrepancies with their own experience or knowledge.

34. When there are confidentiality concerns and it is necessary and proportionate to do so, it is possible to share some information with other public authorities who have information of their own which may help assess the information of the operator. These other public authorities should safeguard the confidentiality of information and the operator should be informed of the information sharing (which information, with who, and with which purposes).

5 Verification (STEP 3) and decision regarding data accuracy (STEP 4)

35. Verification is the external validation of the main database with other data sources, where data mainly will be placed within the Authority, but that in some cases may be placed with another public body or sourced from operators. Some examples of these data sources are: • georeferenced information on active access connections; • if available, information on the location of access nodes or other relevant infrastructures; • general telecom databases (which may allow checking regional aggregates); • information gathered through state aid proceedings and others.

36. In so far this is possible, BEREC encourages to carry out verification by re-using existing data bases, resident in the Authority or required from other public bodies. Whenever there is a need to require additional data to an operator, the Authority should carefully assess the proportionality of the data request, striking the right balance between the need for information and the burden these requests place upon providers.

37. Additionally, it is advisable that Authorities contact the relevant operators whenever they find discrepancies between their submissions to the main database and the verification information, so that they can provide an explanation for the discrepancies, to assess whether the main database needs to be updated. In any case, the decision to update the main database lies on the judgement of the Authority only.

38. To carry out the verification exercise effectively, it is necessary to integrate the main database and the secondary databases on different layers of the GIS. For new data sources, new databases could be created. 9

5.1 Network infrastructure locations and characteristics

5.1.1 Fixed broadband

39. Where this information is available, knowledge of the geographical coordinates of the active access nodes (such as DPU in case of G.fast, DSLAM for DSL, CMTS in case of DOCSIS or OLT in case of FTTH) and their coverage radii, declared by the providers, makes it possible for the Authority to determine (with GIS tools) the coverage area of an electronic communications network and to carry out a certain quality assurance of the fixed broadband main database (premises/small grids passed with a broadband network and, to a certain extent, the characteristics of the broadband service that could be offered by that network) .

40. However, the granularity of the broadband reach data to be extracted in this way depends, to a great extent, on the availability and accuracy of information on administrative numbers, the outline of built-up areas and the boundaries of administrative-territorial units, information usually beyond the control of the Authority.

41. For this purpose, if the location of the active access nodes is known to the Authority, GIS systems may be used to calculate the distance following roads or pedestrian ways, not just the line of sight, between the nearest active access node and each premise or small grid. Like this, the Authority may verify information on available technologies, the categories of end-users served (business/residential), as well as, in certain circumstances, a number of quality parameters of internet access services that could theoretically be provided by the network (such as bandwidth speeds and VHCN declarations according to criteria 3 of the BEREC VHCN GL ).

42. In any case, if the Authority requires the position of or distances from premises/grids to active access nodes, such information shall be required primarily from the wholesale operators providing access, while the need for additional data requests directed to access 10 takers is subject to the Authority’s judgement on necessity and proportionality. In any event, BEREC advises against the duplication of data requests.

43. If the information on the geographical coordinates of active access nodes is not available, a second-best option is to require the operators to provide, at least, the distance (the farthest being the most relevant) from the nearest active access nodes and antennas to the premises/small grids served. These distances can serve as a means of verification for a declared bandwidth for wireless and DSL networks. Even if there are other factors such as line attenuation or vectoring frequency, which affect the performance of the broadband service offered by the network, a longer distance to the nearest active access nodes is, ceteris paribus, associated with a lower (maximum) data rate, and a sufficiently long distance may imply that the premise/small grid simply cannot be served.

44. Distance is also necessary for characterizing the availability of VHCNs as the endpoints for measuring latency or packet loss (CPE to OLT or metro POP, for example) should be declared in advance. Clearly, the latency depends on the propagation delay, which in turn depends on the distance (amongst other factors), which must be known to derive the latency due to congestion.

45. Furthermore, GIS software can be used to compare the QoS1 data with access node positions and equipment/access node type. Given both the coverage area and the network infrastructure locations, the characteristics on a map, using the spatial analyses tools in the GIS software, it is possible to verify the declared technology (for example, premises/small grids passed). So, for example, if addresses/ small grids have an exclusively copper infrastructure and the provider has declared FTTH in those, there are inaccuracies that may be checked. The first thing to do would be to question the operator about the discrepancy.

46. Also, with the spatial analyses tools in GIS software, it is possible to calculate the distance between the position of the access node and the premise/small grid and to determine the admissible maximum speeds classes (as certain speeds will not be plausible) and compare those with maximum speed class declared by the operator.

47. It is advisable for the Authority to determine the admissible maximum speeds thresholds taking into account the engineering models and in collaboration with operators (what is the admissible maximum speed depending on the technology and the local loop distance, for example: for VDSL2, in case of a 500 m local loop, the speed cannot exceed 100 Mbps). Moreover, such thresholds should be published so that operators are informed on the Authority’s reasonable expectations so that there are less complaints in the verification process.

48. Randomly or in some specific cases, the Authority’s inspections on infrastructure may be needed, which should verify the infrastructure positions on field with or without operators. Authorities inspectors must establish with operators a day or a period of days to verify the data in the field. In this case, it is very useful to have a map with infrastructure (printed or on a device) to verify all the equipment by comparing them on the map with those in the field. The comparison must be done both by visualizing the equipment and by measurements performed with special devices. Even if Authorities have their special devices, inspectors should set a day or more with operators for verification, as most equipment is closed in specially designed spaces, most of the time locked.

5.1.2 Mobile broadband

49. The locations of BTS and its characteristics are important to carry out the theoretical calculation of the network coverage of the mobile networks.

50. The use of specialised tools is recommended, if the Authority decides to do theoretical calculations of the coverage of mobile networks to check the information submitted by operators. These tools could be different from the ones used by the mobile operators. Moreover, some information such as digital maps, are needed to perform these calculations.

51. Some other parameters/information that the Authority would need to calculate mobile coverages (e.g. 4G, 5G) are, amongst others: the location of BTS, the topographical information of the coverage zone and many of the BTS characteristics including their height above ground, and technological parameters such as: their power, the antenna diagrams, frequency, sensitivity per service, use of MIMO, carrier or site aggregation.

52. Additionally, and due to the availability of different propagation models, the mobile operators should provide the Authority with information about the ones they are applying. Other relevant parameters should be provided, for the Authority to be able to simulate the theoretical network coverage, and reflect as much as possible the network coverages calculated by mobile operators.

53. All of the information described in paragraphs 49 to 51, can also be used to carry out the estimation of the network’s capacity (i.e. speeds), that could be calculated either without load in the network, or by simulating different levels of loads.

54. In particular, whenever checking VHCN declarations (a sample of those or specifically in some zone for a particular reason), the Authority should retrieve from the operator the network load that characterises peak-time, so that the speeds at these levels can also be re-calculated to understand if the VHCN thresholds for Criteria 4 are satisfied.

55. The complexity of this kind of analysis (theoretical network coverage and estimation of the network’s capacity) will vary with the Authority’s QoS-1 requirements (for example, whether maximum speed is required or not). BEREC recommends each Authority to use the methodology that better fits their needs. 12

56. When, and where possible, the Authority may carry out inspections on infrastructure in order to check whether the locations of the base stations are correctly reported and also in order to check a declaration of a VHCN network If the BTS is served by fiber (Criteria 2 in the VHCN Guidelines). Other infrastructure’s information verification concerning the equipment placed in the BTS could be deemed appropriate.

5.2 Declared service data by operators

57. For fixed broadband, georeferenced data on active access connections and their characteristics, (sourced from operators), can be used to provide a partial check on the QoS1 information (as per the Core Guidelines). For example, address-based active access connections information (this is connections actually demanded by end-users) inform about the availability of certain broadband technologies and speeds on the terrain. Comparing this information and premises passed information on different GIS layers may allow, at least, the detection of areas where an operator declares the presence of an active service of a certain type (technology or speed), but where there is no declaration of its availability (premise passed).

58. Similarly, declared data regarding services offered to consumers in certain zones (i.e. municipalities or with more granularity) is a useful and cost-efficient means of verification (which is even more important if data regarding e.g. the location of the access nodes, of the FWA antennas, etc. is not available). This is, the operators’ reporting of a specific broadband service (accompanied by a declaration of certain parameters of interest) being available in a particular zone, also provides a means of verification of QoS-1 information for mobile and fixed broadband.

59. According to Net Neutrality Rules (Regulation (EU) 2015/2120 of 25 November 2015) operators are obliged to provide speed information anyway (or at least provide such information to each end user): “a clear and comprehensible explanation of the minimum, normally available, maximum and advertised download and upload speed of the internet access services in the case of fixed networks, or of the estimated maximum and advertised download and upload speed of the internet access services in the case of mobile networks…” (Art. 4/d). So, the provisioning of such data to regulators would not be an excessive burden for operators.

60. It should be noted that the speed definitions used in the Open Internet Regulation differ from the ones provided in BoR (20) 42, as the former are speed definitions reflective of the speeds end-users could expect to receive, while the latter reflect network capabilities.

61. Declared service data can be used as a contrast on the QoS-1 data, just to find out about possible contradictions between the two databases, for which GIS cross checks may be useful. Such verification can help to find some doubtful QoS-1 data in terms of speeds (and other parameters) or network availability, e.g. if an operator is declaring a network 13 (QoS-1 data) in the area without any service provided, or if all services in the given area are provided on significantly lower speed than the speed class reported on QoS-1.

62. The declared service data by operators should correspond to reality and the operator or the Authority should be able to verify it with measurements.

5.3 QoS-2 measurements

63. As reported in BoR (19) 182, BEREC’s response to the Core Guidelines public consultation, BEREC understands that QoS-1 theoretical (speed) data estimates will differ from QoS-2 and QoS-3 measurements, but also considers that measurements, and, in particular, QoS-2 information can be helpful to verify whether the QoS-1 estimates are within a reasonable margin of QoS-2 active network measurements.

64. QoS-2 information has the particularity that they are undertaken under a controlled environment (as opposed to data recovered from crowd-sourcing tools). This is, the tests can be carried out under certain conditions which are known to the tester. For example, for mobile broadband and drive tests, the type of measurement equipment used, the speed of the driving, the specific geo-coordinates and times of the measurements are known. In particular, QoS-2 campaigns carried out by the Authority or by operators under the Authority supervision can be a good means of verification.

65. The VHCN Guidelines specify that in order to determine whether a network has the capabilities specified in Criteria 2 or 4, an Authority may demand that a test service which meets the performance thresholds 1 or the performance thresholds 2 is implemented in the network.

66. However, QoS-2 measurements are costly to implement and because of this, the Authority should clearly define the scope of the verification exercise (area, size of the sample, time of the day, operator, technology) and consider a random sample of measurements, or if necessary, a more complex sample design which could guarantee (at a smaller expense), for example, that all types of areas (rural and urban) are measured or that certain areas of interest are over-represented in the sample.

67. Moreover, since measurements are costly to perform, existing information should be reused as far as possible, for example by considering measurements that have been undertaken by operators or by the authority, if available and adequate for verification purposes.

68. An important point is that the data required in the Core Guidelines is not a point estimation (for example, an average or a median) but rather a declaration that a given parameter falls within a specific range in the area of interest (address or grid). These are, for example, the 6 different speed tiers in Table 7 of the Core Guidelines or for a VHCN declaration (under criteria 2 or 4 of the VHCN Guidelines), that a given variable is above a pre-determined 14 threshold. Therefore, if any QoS2 measurements are carried out, the Authority should expect a sufficiently large proportion of those to abide by the conditions required for the . operator declaration.

69. In general, it would be advisable for the Authority to discuss with operators the conditions that the obtained sample of measurements needs to satisfy in order to conclude that a certain data is inaccurate (for example, a maximum percentage of measurements falling outside the specific class or the formulae leading to such conclusions).

70. Finally, it is also advisable to involve stakeholders in designing QoS-2 campaigns. On one hand, in the case of fixed broadband, the operator collaboration is indispensable to enter the specific locations where the measurements need to take place and on the other, this collaboration strengthens the outcome of the verification process and reduces the possible complaints that may rise. A pilot study is encouraged in the case of new QoS-2 campaigns.

5.3.1 QoS-2 for mobile broadband

71. In accordance with the Core Guidelines, the territory is divided in squares of at most 100 meters on each side, called pixels. Therefore, the QoS-1 information is to be provided for each pixel. All the information provided by the operators at pixel level should reflect the situation at a date specified by the Authority.

72. Section 2.2 describes the information to verify. This includes: - Coverage: A pixel is covered by a mobile broadband technology if a broadband service (at least 2 Mbps) is available in at least 95% of the grid areas with a high likelihood of successful reception, where this means a probability of service reception of 95% - VHCN declaration: For a pixel to be declared as reached by a VHCN, the conditions associated must be satisfied in at least 95% of the pixel, either by base stations connected to fiber lines or, if not, by other technologies, in this case all the performance thresholds specified in Criterion 4 of the VHCN Guidelines must be satisfied in 95% of the grid. - Maximum speed classes, if the Authority has decided to request this information.

73. In particular, within the framework of mapping surveys for mobile coverage, the information to verify provided by operators, as identified in the previous paragraph, would usually be a file per technology, indicating the field strength for each pixel of the territory. Alternatively, operators may provide multiple files (one for each signal level indicated on the map) per technology, indicating if the network covers or not (0 or 1) each pixel.

74. Additionally, the Authority should verify the operator’s information on whether a pixel is covered or not by a VHCN or a certain mobile broadband speed class. 15

75. The Authority can carry out field measures to check the validity of these files, i.e. the accuracy of the information provided by the operators. Yet, it should be noted that for technical and resourcing reasons it may not be possible to make widespread measurements over the country.

76. Measurements should be carried out systematically, with standardised procedures and without human intervention or decision (during the carry out of measurements), and under the same conditions for operators, to provide an objective verification procedure.

77. First, to carry out the verification process, there is a need to define sample areas that include routes that should be chosen randomly, but homogeneously, over the whole territory, in order to ensure a certain level of spatiotemporal statistical significance.

78. Second, to be able to compare the results from one campaign to the other, a number of routes/sample areas, randomly chosen, should remain identical from one campaign to the other.

79. The measurements should be performed by the Authority, or by some entities it recommends and would have to be carried out with a specific measurement tool such as an RF scanner.

80. For the verification of coverage, in order to be able to compare operators’ results, it is important that MNOs are measured at the same time and in a same location (if covered). Moreover, it is necessary that mobile devices/scanners are blocked on the specific network of the operator for which information is verified or enable the identification of this information.

81. For the verification of VHCN it is important that the testing system always chooses the best technology available at the time of each test

82. On the other hand, for the verification of mobile broadband speed classes, it is important that the testing system always chooses the technology that is being checked by the Authority (e.g. 3G, 4G) at the time of each test.

83. All or part of the measurements can be made in a moving vehicle driving at a normal speed based on the type of roads. For each point of measurement, the related GPS coordinates have to be acquired. It is required that the measurements made in a vehicle report an external situation by means of an external antenna.

84. The Authority must define several elements of the process, such as the optimal number of measurements per minute, which must be linked to a GPS position, the vehicle speed, the 16 signal to keep, and the clustering algorithm. The NRA should also calculate a correction factor in order to compare the drive test’s results with the operator’s coverage files.

85. The Authority should inform each of the operators of the validation results for each file that was sent, precisely indicating which correction factor was applied so that operators could check their data and if necessary submit updated data to the Authority.

86. Moreover, it is good practice for the Authority to involve the mobile operators in the definition of some technical parameters (e.g. common signal strength limits) before the measurements are undertaken.

87. The number of measurements to perform by sample area/routes, should be defined by each Authority on a case by case basis, as well as the methodology to verify QoS-1 mapping. Authorities should publish their verification methodologies and the specifications of their drive tests.

5.4 QoS-3 measurements

88. QoS-3 measurements are defined as measured experience of service, including enduser’s environment. For this purpose, end-users are generally required to download crowdsourced measurement tools in order to launch QoS measurements from their customer premise equipment or terminal equipment.

89. By doing so, the measurements will incorporate items from the end-user environment (for example the terminal equipment, cross-traffic, home network limitation in the case of Wi- Fi) and may be influenced by tariff-related speed-cap. Those items are meaningful to understand the real quality of end-users’ access, however they can also lead to misunderstandings by end-users, if those are unable to tell apart the causes of a discrepancy between real speed and theoretical or contractual access speed (as most crowdsourced tools are unable to detect a potential environmental-based source of limitations).

90. QoS-3 measurements can also be associated to Quality of Experience (QoE) measurements, as the crowdsourced measurement tools grow in complexity. The concept is not only to measure the capacity or speed of end-users’ access, but to mimic real-life experience through measurements of diverse online usages (for example by measuring the speed of a link to representative panel of websites, or by measuring not just downloading of files but also content browsing or streaming).

91. Given this, QoS-3 information can indicate that besides the general, theoretical capacity of an access, there can be issues further down or up in the internet chain (for example in the end-user environment, or, at the opposite, at interconnection level) that would go unnoticed under other measurement protocols; and yet those issues are relevant in order to assess to real quality of experience for each and every end-user. 17

92. Especially, as QoS-3 monitoring tools are generally crowdsourced, it is more likely that end-users facing connectivity problems perform those tests, and as a result, connectivity problems might be highlighted quickly to the eye of the Authority.

93. Authorities should be very careful when extrapolating the measurements of Internet users for verification purposes, as it is not possible to guarantee the necessary statistical representativeness of the group, given that the crowdsourcing tests are of a nature voluntary and not random. Moreover, their specific motivations for carrying out the tests are not controllable. In this context, it should be noted that the results may not necessarily reflect the reality of the Internet access service in each region.

94. Moreover, the test results, in addition to the contracted speed, are influenced by other factors, namely 1) the processing and communication capacity of the terminal equipment used which may also be affected by the possible presence of malware, viruses, among others; 2) the type of connection between the equipment and the network connection router (directly via cable or via Wi-Fi); 3) the possible existence of parallel traffic on the same access (existence of other Internet users or other equipment in activity) and 4) the tariff speed cap in the commercial offer that the end-user has contracted.

95. In the case of a mobile access, the speed result is subject to factors related to the quality or strength of the signal, coverage, reception antenna and the number of users who share the same antenna for a given operator at the time of testing.

96. Where Authority sponsored tools have been developed (or the Authority has struck partnerships with tools already on the market), these may enable retrieving useful information at a reasonable cost, acting as a signalling platform for network problems that Authorities can investigate. But for the sake of a more thorough regulatory monitoring of the state of networks, these tools might have downsides in terms of reliability.

97. Various procedures have been put in place by regulators or public bodies to enhance the reliability of these tools: - The practice of setting up representative pools of users is regularly applied. As an example, Ofcom’s Home Broadband Performance study, or the FCC’s Measuring Broadband America, have used sample of users chosen to attain statistic representativity, in terms of geographical coverage, access technologies etc. In these conditions QoS-3 measurement may produce a reliable output on the state of a network. - For the sake of verifying the transparency obligations in article 4 of Regulation 2015/2120, BNetzA has released a QoS-3 measurement tool that is accompanied by a measurement protocol for the end-user. This protocol prescribes the information that the end-user has to disclose (contractual speed, access technology, verification of cross-traffic) and the measurements that the end-user has to perform (a number of measurements during several different times of the day), in order to avoid random results. 18 - It is also possible to specifically diminish interferences stemming from end-users’ environment. As an example, ARCEP has developed an “access ID card” in cooperation with major ISPs and measurement tool providers. This ID card is implemented as an API inside ISP set-top-boxes, and the measurement tools participating in this project can access the information on the ID card (e.g. contractual speed, access technology, cross traffic, set top box model) before each measurement, in order to make it more relevant and understandable by the enduser.

6 Transparency and accountability

98. The Authority should publish the method/s of verification (cf. section 5) that they implement. By publishing the principles of the how the data quality is assured NRAs establish transparency and help make their proceedings plausible to market participants.

99. This publication can include an explanation about which types of data are being verified (cf. section 2.1 and 2.2) or why a specific method has been chosen. The Authority could also indicate if they use data from third parties to verify operator data (cf. section 4).

100. Furthermore, if the Authority re-calculates mobile broadband availability information using its own models, it should publish its methodology, including a reference to the tools and assumptions made in the verification calculation (see section 5.1).

101. To ensure transparent proceedings, the Authority should report publicly and periodically on the outcome of the quality assurance. These reports should focus on the overall results of the verification process and main takeaways for future rounds. To maintain the necessary confidentiality, the Authority should aggregate information at operator level for publications.

102. If possible, the reporting on methods and on verification outcomes should preferably be published online in conjunction with the geographical survey itself (e.g. all on the same websites) so they are easy to find for interested parties. 19

Annex 1 - Glossary of terms

API – Application Programming Interface BTS - Base Transceiver Station CMTS - Cable Modem Termination System CPE – Customer Premises Equipment DPU - Display Processing Unit DSL - Digital Subscriber Line DSLAM - Digital Subscriber Line Access Multiplexer ECN - Electronic Communications Networks ECS - Electronic Communications Services EECC – European Electronic Communications Code FTTH – Fiber to the Home GIS - Geographic Information System IP – Internet Protocol ISP – Internet Service Provider MIMO - Multiple-input and multiple-output MS – Member State NRA - National Regulatory Authorities OCA - Other Competent Authorities OLT - Optical line termination POP – Point of Presence QoE – Quality of Experience QoS1 - Quality of Service 1 - Calculated availability of Service - Theoretical network performance of existing infrastructure QoS2 - Quality of Service 2 - Measured provision of Service - Measurements via panel probes or drive tests, excluding end user’s environment. QoS3 - Quality of Service 3 - Measured experience of Service - Measurements using internet access service including end user’s environment, for example via online speed tests VDSL - Very-high-bit-rate Digital Subscriber Line VHCN - Very High Capacity Networks

20