{'fraction_non_alphanumeric': 0.023741598568734586, 'fraction_numerical': 0.01784246409748078, 'mean_word_length': 4.6446506550218345, 'pattern_counts': {'":': 0, '<': 0, '<?xml version=': 0, '>': 0, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
Supplement of The Tall Tower Dataset: a unique initiative to boost wind energy research

Abstract. A dataset containing quality-controlled wind observations from 222 tall towers has been created. Wind speed and wind direction measurements covering the 1984–2017 period have been collected from existing tall towers around the world in an effort to boost the utilization of these non-standard atmospheric datasets, especially within the wind energy and research fields. Observations taken at several heights greater than 10 m above ground level have been retrieved from various sparse datasets and compiled in a unique collection with a common format, access, documentation and quality control. For the last, a total of 18 quality control checks have been considered to ensure the high quality of the wind records. Non-quality-controlled temperature, relative humidity and barometric pressure data from the towers have also been obtained and included in the dataset. The Tall Tower Dataset (Ramon and Lledó, 2019a) is published in the repository EUDAT and made available at https://doi.org/10.23728/b2share.136ecdeee31a45a7906a773095656ddb.


Wind speed and wind direction records falling outside a physically possible range of values are commonly found within the time series. They are mainly produced by gross errors in the data loggers or storage. This test detects and flags unrealistic values such as negative wind speed values or observations above a maximum allowed threshold. The absolute maximum limit 5 has been set to the maximum wind gust measurement ever recorded on the earth surface, which is 113.3 ms −1 measured in Barrow Island (Australia) under the effects of Olivia cyclone in April 1996 (Courtney et al., 2012). A lower threshold can be selected from which wind speed values can be flagged as suspect. This value is set to 75 ms −1 , which is the one suggested by the WMO (WMO, 2007) and besides, this fixed-value also corresponds to Vaisala's sensors highest measurable value. Wind direction values falling outside the range from 0 to 360 degrees are also flagged as erroneous.

S2.2 Difference between extreme values of the wind distribution
One of the potential uses of the Tall Tower Dataset is the detection of severe weather events by looking at the extreme values of the empirical wind speed distribution. However, some of these extreme measurements might be erroneous and need to be flagged accordingly. This QC check detects and flags unrealistic extreme wind speed values of the time series by checking the difference between the maximum and the second maximum values of the distribution of wind speed values. If the difference 15 between them exceeds the absolute value of the second maximum, the first maximum is flagged as suspect. This test runs iteratively until the previously mentioned condition is not satisfied.

S2.3 Persistence test
Wind time series are usually characterised by strong variability, alternating periods of high and low fluctuations. Nevertheless, the presence of long periods of extremely low variability can be unrealistic since they can be produced by errors in the 20 measuring sensors or instrumental drift. The persistence test detects and flags sequences of wind speed and wind direction observations with abnormally low variability. However, it is important to take into consideration those relatively long periods with very low variability and mean wind speed values close to zero are typical of the observed natural variability (e.g., static high-pressure systems during several days in a row producing weak winds). Hence, these data cannot be considered erroneous. Thus, the persistence test does not introduce any flag to wind speeds weaker than 0.5 ms −1 . These measurements are then 25 flagged as calms.
The WMO proposes that 1-minute data should vary at least 0.5 ms −1 over 60 consecutive wind speed values, and 10 degrees in the case of wind direction records. Otherwise data should be flagged as doubtful. These thresholds have been adapted to the resolutions reported by the towers. Thus, wind speed periods are flagged as suspect if the wind speed does not change more than 0.7 ms −1 in 60 consecutive values. Wind direction values will be considered suspicious when the range between 30 the maximum and the minimum values in a sequence of 60 records is lower than 5 degrees.
The example plotted in Figure S1 shows wind speed observations measured at 18 meters at the top of the Barrow tower (Arctic Circle) during 51 consecutive days. In except of the two spikes on 14 th October and 3 rd November, wind speed values range from 4.8 ms −1 to 5.3 ms −1 . This variability is significantly low when compared with the rest of the wind series (not shown). Although the Persistence test flags the records as a suspect, a visual inspection reveals that they are potentially 35 erroneous and should not be used as reliable data.

S2.4 Flat line
A sequence of numbers with null standard deviation is the extreme case of a period with low variability and indicates that several constant values are observed consecutively. The probability of recording constant values in a row decreases with the number of significant figures that a sensor can record, being almost unlikely to have more than five consecutive exact matches 40 for wind speed (IOSS, 2017) and 40 for wind direction measurements. In this sense, data fail the flat line test when there exist 6 -or more-constant wind speed values in a row. This threshold is increased to 40 for the wind direction variable. Observing Figure S1. Wind speed time series at 18 meters above ground level at Barrow site (71.32ºN 156.61ºW, 11 m), USA 3, 4 or 5 exact consecutive matches is more likely for wind speed values, but still unlikely to happen frequently. Therefore, the tests flags as suspect those flat sequences. Analogously for wind direction data, flat sequences containing 20 to 40 wind direction records are flagged as suspect. It is also frequent to observe an alternation of no data periods with null speed values, which are usually produced by failures in the sensors or data loggers. If the period containing this alternating pattern exceeds 30 days, all the measurements within this period are flagged as erroneous.

5
A detection of a flat line is shown in Figure S2. Various sequences of constant values are encountered at the three different levels between September 14 th and September 20 th . Like that, flat lines are often detected simultaneously at all levels of the tower.

S2.5 Icing
Freezing rain or fog usually frosts the anemometers and vanes placed along the tall tower preventing them from measuring 10 non-zero wind speed values and changes in the wind direction. Hence, these records should be detected by checking wind and temperature observations simultaneously. Based on Jiménez et al. (2010), data are considered wrong when the Icing test detects 4 or more days with 0 ms −1 as the maximum wind speed value and below zero temperatures during all the same period.
Wind speed series at different heights at Hegyhatsal tower are represented in Figure S3. A flat line is observed in the two uppermost levels from December 8 th to December 18 th 2002. However, the air temperature observations ( Figure S4) reveal 15 that negative Celsius temperatures occurred during all the ten days in the two top levels of the tower. Given these conditions, it is very likely that an icing event happened and frosted the two upper anemometers.

S2.6 Abnormal variations
Random and gross errors in the measurements might produce periods of abnormally high or abnormally low variability and usually, appear embedded in the wind speed time series. Various authors have proposed several different thresholds that define 20 a period with extreme variability (see Jiménez et al. (2010)) since the threshold selection should depend on the local wind features. In an attempt of generalisation, in this work it is proposed that these limits are defined by statistical parameters   as the variance) of 30-day periods with the mean variance of all 30-day periods of the time series using moving variances. If the standard deviation of a specific 30-day period departs more than four standard deviations from the mean standard deviation, records within these 30 days are all flagged as suspect.

S2.7 Systematic errors
Another approach to detecting random and systematic errors in the experimental measurements is based on the computation 5 of moving averages. Similar to the abnormal variations check, this QC routine computes the mean wind speeds over a 30-day moving window. Wind speed values within a 30-day period whose average departs more than four standard deviations from the mean value of all 30-day moving means are all considered suspect.
In Figure S5, the Systematic errors check flags as suspect 12 consecutive days of wind speed measurements taken at the top of Hegyhatsal tower. A close inspection reveals that the minimum wind speed record is over 5 ms −1 during all the mentioned period, which is in disagreement with the wind speeds observed at lower levels. Indeed, the three anemometers located at 10, 48, and 82 meters report weaker winds or even calm during these 12 days. It is likely that an offset value could have been inserted in the data logger producing the inconsistency observed in the uppermost wind speed measurements. In this case, these 12 days of winds at 115 meters should not be considered reliable. Figure S6 shows a false detection of a systematic error at WLEF tall tower. Although the test flags as suspect a period of 15 2 months of wind speed data at the 122-meter level, a visual inspection and comparison with winds reported at other tower heights does not reveal any inconsistency in the suspicious observations. Hence, these data should not be discarded unless a sensor failure is reported in the metadata of the site.

S2.8 Quartile occurrences
A third method to detect periods containing gross errors in the measuring process is suggested here by looking at the number of 20 consecutive days where no value is above or below the first, second and third quartiles of the empirical wind speed distribution.   Table S2 summarises the different thresholds (in days) that define the trustworthiness of an observation. As an example, the first row indicates that if all the observations in 30 days fall above the first quartile, data within this period will be flagged as erroneous. Observations are suspicious when the period without any occurrence within the first quartile ranges between 15 and 30 days. Spans shorter than 15 days without any value falling within the first quartile are considered correct by this test.

S2.9 Rate of change
The presence of spikes in the wind series is usually observed during extreme wind phenomena events. However, the magnitude 5 of these peaks is constrained to a specific allowable range of values specially when the very high-frequency wind data are averaged in periods of several minutes (which is the case of the observations within the Tall Tower Dataset). This test compares pairs of adjacent observations. To pass the test successfully, differences between consecutive values must be lower than a specific threshold, that can be either dynamically established or fixed (IOSS, 2017). The Rate of change test uses the interquantile range (IQR) of the considered series, defined as the difference between the third and first quartiles of the empirical distribution.

10
When the difference between two consecutive values exceeds three times the value of the IQR, both values are considered wrong. If the difference is between twofold and threefold the IQR, the pair of observations is considered as suspect.

S2.10
Step test The Step test uses a similar methodology as the Rate of change test to detect spurious peaks of wind speed data. In the Step test, the maximum permissible difference between two consecutive observations is fixed to 20 ms −1 (WMO, 2007), instead of 15 using a statistic derived from the wind series. Although the WMO suggests this limit specifically for 2-minutely averaged wind speed data, their usage has been deemed appropriate for the data within the Tall Tower Dataset since the time stamp samplings observed in this collection are larger. Indeed, by averaging data in longer periods, one can expect a general smoothing of the series, hence reducing the possibility of observing big data spikes.  Figure S7).

25
A careful inspection reveals that the values within the two black rectangles in the top series match perfectly. An analogous situation is noticed for the two lower levels. This is probably due to an standard procedure to fill in no-data periods, which takes previously observed wind speed sequences of data. However, it has been deemed appropriate to the detect and consider erroneous these sequences of data.

30
One of the singularities of the tall tower data is that wind measurements are not taken at the top of a pole where a sensor is placed. Instead, anemometers and wind vanes are distributed along with the vertical structure of the tall tower, which usually consists of a solid vertical cylinder or a lattice structure that inherently produces a wind shadow in the downwind area. If an anemometer is measuring in the shadow area, wind speeds are affected by this shadow and cannot be considered reliable.
To help overcome this handicap, a common practice in the instrumental installation is to place redundant sensors at the same height in booms oriented to different cardinal directions. Shadowed records can then be replaced by those from a sensor not affected. The Tower shadow test identifies first the shadowed directions and anemometers by dividing wind speeds from two 5 sensors at the same level. Ideally, they should measure the same values so that the ratio is expected to be equal to the unit unless the winds from one sensor are shadowed. Then, all wind speed ratios are grouped in wind direction sectors of 1 degree. The 5 th and 95 th percentiles of the distribution generated by all the ratios are calculated next. Those directions showing ratios below the 5 th percentile and above the 95 th are considered to be in the wake of the tower. After identifying the shaded directions for each anemometer, the test marks as suspect those wind speed values affected. Figure S8 exemplifies the previous explanation presenting the ratios between simultaneous wind speeds observations measured by redundant sensors at 60 and 100 meters at the FINO3 met mast in the North Sea. The quotient between wind speeds reported by two different sensors is approximately one for most of the wind directions. However, wind speeds coming from 50±5 and 170±5 degrees of direction are affected by the vertical pole at the two measuring levels. Thus, the anemometer measuring the weakened winds is identified, and those records should not be considered correct.

S2.13 Vertical ratios
QC checks that employ nearby stations are not suitable for meteorological variables with remarkably localised features such as precipitation or wind speed, because the correlation between neighbour series is considerably lower when compared to temperature or pressure time series (Dunn et al., 2012). In addition, those tests require a dense network of stations, which is not the case of the Tall Tower Dataset. However, another particularity of tall tower data is the simultaneous records taken 20 at the same time at different heights along the mast. These series can be compared among them as they are expected to be highly correlated. The Vertical ratios is a particular test which considers pairs of time series measured at different heights and computes the mean ratio (r) of all the pair-wise measurements ratios (r i ). To avoid duplication and save computation time, the test only computes the ratio between one level an all the lower levels. In except local effects such as low-level jets, wind speeds tend to increase in height, so the computed mean ratio is expected to be greater or equal to unity. Taking this assumption into account, the Vertical ratios test will detect and flag as erroneous those pairs which ratio (r i ) satisfies the following condition: Values are considered dubious when the following condition is satisfied: Even though the allowable ranges of ratios was initially chosen somewhat arbitrarily, it has been tested and adjusted using the data within the Tall Tower Dataset to ensure that only gross errors are detected and flagged as erroneous. Wind speeds under 1 ms −1 are not considered in this test.
S2.14 Isolated pass 10 After running some of the QC tests, a certain amount of sequences might be flagged as wrong or dubious. These sequences can be found close in time and encircle values marked as correct by the QC checks. However, it is very likely that those presumably correct values are not be acceptable since a prolonged sensor failure may have occurred, but the previously run QC checks missed it. The Isolated pass check is applied after running at least one QC test and attempts to detect those apparently correct (we note that calms are also identified as good data) sequences of observations surrounded by wrong or suspect values, 15 and change their flag into erroneous or suspect. Besides, we also force to be wrong those scattered individual records appearing randomly within long no-data periods.
A total of 12 predefined sequences (see Table S3) containing data flagged as correct ('Pass' or 'Calm') but surrounded to the left and right by, wrong ('Fail'), dubious ('Suspect') or absent ('Missing') records have been defined. Wherever these series are found, the central 'Pass' or 'Calm' values are changed from 'Pass' to 'Fail'. Table 6 defines similar sequences, but their central records will be changed from 'Pass' to 'Suspect'. Table S3. Explicit definition of the sequences to be searched within the wind time series which central value or values flag will be changed from 'Pass' or 'Calm' to 'Fail'. F ail,F ail,F ail,P ass,F ail,F ail,F ail F ail,...,F ail 5 , P ass, P ass, F ail, ..., F ail 5 F ail, ..., F ail 10 , P ass, P ass, P ass, F ail, ..., F ail 10 F ail, ..., F ail 15 , P ass, P ass, P ass, P ass, F ail, ..., F ail 15 F ail, ..., F ail 25 , P ass, P ass, P ass, P ass, P ass, F ail, ..., F ail  Table S4. Explicit definition of the sequences to be searched within the wind time series which central value or values will be changed from 'Pass' flag to 'Suspect'.

S2.15 Occurrences of 0s and 360s values
The lack of coordination concerning the data storage and formatting conventions in the original data may produce some issues that must be detected. For example, in the wind speed time series, missing records are sometimes set to zero, thus leading 5 to a spurious increase in the occurrence of the zero value. Similarly, some conventions use the value 0 degrees to refer to the northern wind direction while others identify this direction with 360 degrees. Stations with properly detailed metadata information include the convention adopted by the data managers. Regrettably, most of the stations whose data was accessed to be included in the Tall Tower Dataset did not attach such complete information. In those cases, the original basic standards such as assigning the 0 or the 360 value to the north direction need to be inferred.

10
This routine computes the percentage of occurrence of each of these three cases: 1. Occurrences of 0s within the wind speed time series, 2. occurrences of 0s within the wind direction series and 3. occurrences of 360s within the wind direction series.
The Occurrences of 0s and 360s values does not flag individual records, but provides a value for each of the series indicating the percentage of the aforementioned occurrences to the total data. The whole series is considered incorrect if any of these occurrences exceeds 30%, which has been chosen appropriately to take into account that a considerable percentage of calms may exist.
S2.16 Internal consistency 5 Whenever a null wind speed is recorded, the associated wind direction value is meaningless since it is very likely that the wind vane is still pointing to the direction defined by the last non-zero wind speed observation. According to the WMO guidelines, whenever a null wind speed is reported, the simultaneous wind direction measurement must be forced to be null as well. However, in the Tall Tower Dataset the zero wind direction value indicates the true North. Therefore, for null wind speed records, wind direction must be set to NA. We note that the condition must be only applied for wind measurements taken at 10 the same height above ground level.