5. Data Quality
All data and metadata in the TOAR database have been subject to some quality checks. Nevertheless, nobody is perfect and therefore it is not unlikely that you may identify errors, inconsistencies or „weird looking“ data if you only dig deep enough. Most of the data that are kept in the TOAR database originate from quality-controlled repositories, which are maintained by professional data managers. Other data come from resources with fewer resources or potentially less knowledge about the many complex facets of providing FAIR 2 data. Finally, there are data sources, which provide „preliminary“ data in near real-time and such data can obviously not be checked by trained human experts before they are posted.
The TOAR database has been designed with the primary objective to support the Tropospheric Ozone Assessment Report, and therefore our focus lies on providing the data which are most useful for scientific analyses of global air quality and reflect our best knowledge about global air pollutant concentrations. Due to the data curation procedures described below, the data you obtain from the TOAR database may not always be completely identical to data from the same measurements which you might get from the original data providers. Therefore, TOAR data are not suitable for legal purposes, such as the initiation of law suits because of non-attainment of air quality standards.
The TOAR data centre developed a largely automated workflow to process and add new data into the TOAR database (see The TOAR Data Processing Workflow). One step in this workflow is the execution of automated scripts for checking the metadata which describes a measurement site and each individual time series. There is also an automated quality control tool, which performs some basic statistical tests on new data to ensure that at least gross errors are captured and that no „garbage“ enters the database. We are continuously working to improve this quality control tool and plan to add more sophisticated tests in the future. As part of our responsibilities in the TOAR assessment, we will double-check as much data as we can and perform several manual checks through database queries and visualisations at the time when the phase II assessment will be prepared. As TOAR database user you can help us by keeping an eye on the data you download and by informing us about any data or metadata issues you encounter when using the data from the TOAR database. We will try our best to follow your leads and inform the original data providers about any issues that can be confirmed.
During the first phase of TOAR, a semi-quantitative analysis was performed to determine the fraction of erroneous and questionable data among all ground-level ozone time series which are stored in the TOAR database (see 1 ). In general it was found that over 95 % of all data points can be regarded as „trustworthy“ in the sense that they exhibit „typical“ behaviour of ozone time series and show no obvious anomalies. Through the creation of animated maps and trend plots of the TOAR data it could be confirmed that the vast majority of data „fits together“ nicely, which means that errors in the aggregated ozone statistics are likely smaller than 5 parts per billion and trend estimates should be „reasonably accurate“ 3 . As the TOAR database allows downloads of hourly values including the data quality flags, you can always re-assess the quality of the data you obtain from us. You can also re-run our automated quality control tool, which is available from https://gitlab.jsc.fz-juelich.de/esde/toar-public/toarqc.
5.1. Data and Metadata Curation
Data quality is a complex topic and there are many different views about what constitutes „good quality data“. With respect to the metadata describing stations and time series we aim to achieve the best possible consistency through the use of controlled vocabulary (see https://esde.pages.jsc.fz-juelich.de/toar-data/toardb_fastapi/docs/toardb_fastapi.html#controlled-vocabulary) on the one hand, and by performing some algorithmic tests on the other hand. For example, we will compare reported station altitudes with the altitude returned from a fine resolution digital elevation model at the given latitude and longitude coordinates. A warning will be raised if the results differ too much. The development of such algorithmic tests is ongoing and will be documented at a later stage.
The quality of the actual data values can never be assessed with full certainty, but experience and statistical methods can at least provide good clues. In the current version of our automated quality control tests, we check the data ranges and test for outliers as well as unrealistically long periods of constant values and significant step changes. Thresholds for these tests have been developed based on sample data which have been determined to be of high quality due to
trust in the data providers, and
visual inspection of the time series and various descriptive statistics.
The automated quality control tool will not delete any data, but instead change the data quality flag (see Section 5.2). Any such changes applied to the data will be recorded and are made accessible through the time series’ „change log“.
There is some debate in the scientific community of environmental observers and database managers about the roles they have in the data curation procedures and about the respective rights and duties. As a general guiding principle it is often stated that only the first-hand data providers are allowed to make changes to their data and metadata, because they are the only ones who have the full insight into the measurement conditions. On the other hand, many modern data collection efforts place more responsibility on the data curators in the data centres, because only there it is possible to assess different data sets with common standards and to apply additional tests, which involve comparisons with neighbouring sites or with numerical model data. Best practice suggests that the results from such tests are communicated back to the data providers and they are then charged with the task to correct the data and re-send to the data centre. In practice, we have found that it is often more efficient to suggest specific corrections to the data providers and ask for their approval, because this means less work for them. In rare cases, the TOAR data centre may also modify data values without the approval of providers; for example, if the data come from a large monitoring network and there are no direct communication channels with the providers, or if we are convinced that data are erroneous, but the data provider will not react to our inquires. Such changes will only be applied if the correction is obvious. A typical example are unit conversions, which may be necessary if the metadata in the submitted file header is inconsistent with the data values. In any case will we document all of these changes and make this information available to you.
5.2. Data Quality Flags
As described above, the quality of TOAR data is documented via so-called data quality flags. There are numerous flaging schemes in use around the world with varying level of detail. Some of the datasets which we receive for inclusion in the TOAR database provide quality information with their data, others don’t.
We define four possible status code ranges to indicate whether a given data value is appropriate for use or not. In addition, code values greater 100 can be used for aggregated queries (Table 5.1).
Status code range |
Data quality |
|---|---|
0 - 9 |
OK |
11 - 19 |
questionable |
20 - 29 |
erroneous |
90 - 99 |
missing or unknown status |
100 - 140 |
combination of specific data quality flags |
Normally, you will be interested in “OK” data only, which means that you can filter data with quality flag < 10. However, in this case it is easier to request ‘AllOK’ data (flag value 100, see Table 5.2).
As mentioned above, all data are subjected to some automated tests before inclusion in the TOAR database. These tests can only lower the level of confidence in the data, but never change data that were labelled as questionable or erroneous by the data provider into OK values.
The second aspect that might be relevant for assessing the data quality is whether these data have been validated by the provider or not. While in the first phase of TOAR the database only accepted validated data, the expansion to previously uncovered world regions with help of OpenAQ necessitated the inclusion of realtime data, which are never thoroughly validated, although they might have passed some automated quality control checks.
To facilitate the selection of data with a specific quality status, we defined two sets of quality flags. The first set consists of aggregate flags, which allow you to easily select data according to their status as OK, questionable, or erroneous, and to distinguish between validated and preliminary data if you wish to do so (Table 5.1). The second set of flags preserves the information of the original quality assessment by the provider as well as any possible modification introduced through our automated quality control procedures (Table 5.2). These more detailed flag values are the values that are actually stored in the database. You can use both flag sets in the REST interface.
Flag value |
Flag name |
Description |
Combination of original flag values (Table 5.3) |
|---|---|---|---|
100 |
AllOK |
Data values were deemed OK by the provider and the TOAR quality control tool did not find any obvious errors. Note that validated data with no explicit quality information is treated as “provider OK”, whereas preliminary data with no explicit quality information is treated as “not checked by provider”. This status also covers data values which had been erroneous at first but were corrected by the provider or based on feedback by the provider. |
OKValidatedVerified, OKValidatedQCPassed, OKValidatedModified, OKPreliminaryVerified, OKPreliminaryQCPassed, OKPreliminaryModified, OKEstimated |
101 |
ValidatedOK |
Data were sent by provider as validated data, data values were deemed OK by the provider and the TOAR quality control tool did not find any obvious errors. |
OKValidatedVerified, OKValidatedQCPassed, OKValidatedModified |
102 |
PreliminaryOK |
Data were sent by provider as preliminary (or realtime) data, data values were deemed OK by the provider (usually no explicit quality information is given with realtime data) and the TOAR quality control tool did not find any obvious errors. |
OKPreliminaryVerified, OKPreliminaryQCPassed, OKPreliminaryModified |
103 |
NotModifiedOK |
Similar to AllOK, but modified data values are not included. |
OKValidatedVerified, OKValidatedQCPassed, OKPreliminaryVerified, OKPreliminaryQCPassed |
104 |
ModifiedOK |
Data values had been erroneous at first but were corrected by the provider or based on feedback by the provider. |
OKValidatedModified, OKPreliminaryModified, OKEstimated |
110 |
AllQuestionable |
Data were labelled as questionable by provider or marked as questionable by the automated TOAR quality control test. |
QuestionableValidatedConfirmed, QuestionableValidatedUnconfirmed, QuestionableValidatedFlagged, QuestionablePreliminaryConfirmed, QuestionablePreliminaryUnconfirmed, QuestionablePreliminaryFlagged, QuestionablePreliminaryNotChecked |
111 |
ValidatedQuestionable |
Validated data that were labelled as questionable by provider or marked as questionable by the automated TOAR quality control test. |
QuestionableValidatedConfirmed, QuestionableValidatedUnconfirmed, QuestionableValidatedFlagged |
112 |
PreliminaryQuestionable |
Preliminary (realtime) data that were labelled as questionable by provider or marked as questionable by the automated TOAR quality control test. |
QuestionablePreliminaryConfirmed, QuestionablePreliminaryUnconfirmed, QuestionablePreliminaryFlagged, QuestionablePreliminaryNotChecked |
120 |
AllErroneous |
Data were labelled as erroneous by provider or marked as erroneous by the automated TOAR quality control test. |
ErroneousValidatedConfirmed, ErroneousValidatedUnconfirmed, ErroneousValidatedFlagged1, ErroneousValidatedFlagged2, ErroneousPreliminaryConfirmed, ErroneousPreliminaryUnconfirmed, ErroneousPreliminaryFlagged1, ErroneousPreliminaryFlagged2, ErroneousPreliminaryNotChecked |
121 |
ValidatedErroneous |
Validated data that were labelled as erroneous by provider or marked as erroneous by the automated TOAR quality control test. |
ErroneousValidatedConfirmed, ErroneousValidatedUnconfirmed, ErroneousValidatedFlagged1, ErroneousValidatedFlagged2 |
122 |
PreliminaryErroneous |
Preliminary (realtime) data that were labelled as erroneous by provider or marked as erroneous by the automated TOAR quality control test. |
ErroneousPreliminaryConfirmed, ErroneousPreliminaryUnconfirmed, ErroneousPreliminaryFlagged1, ErroneousPreliminaryFlagged2, ErroneousPreliminaryNotChecked |
130 |
AllQuestionableOrErroneous |
Data were labelled as questionable or erroneous by provider or marked as questionable or erroneous by the automated TOAR quality control test. |
QuestionableValidatedConfirmed, QuestionableValidatedUnconfirmed, QuestionableValidatedFlagged, QuestionablePreliminaryConfirmed, QuestionablePreliminaryUnconfirmed, QuestionablePreliminaryFlagged, QuestionablePreliminaryNotChecked, ErroneousValidatedConfirmed, ErroneousValidatedUnconfirmed, ErroneousValidatedFlagged1, ErroneousValidatedFlagged2, ErroneousPreliminaryConfirmed, ErroneousPreliminaryUnconfirmed, ErroneousPreliminaryFlagged1, ErroneousPreliminaryFlagged2, ErroneousPreliminaryNotChecked |
131 |
ValidatedQuestionableOrErroneous |
Validated data that were labelled as questionable or erroneous by provider or marked as questionable or erroneous by the automated TOAR quality control test. |
QuestionableValidatedConfirmed, QuestionableValidatedUnconfirmed, QuestionableValidatedFlagged, ErroneousValidatedConfirmed, ErroneousValidatedUnconfirmed, ErroneousValidatedFlagged1, ErroneousValidatedFlagged2 |
132 |
PreliminaryQuestionableOrErroneous |
Preliminary (realtime) data that were labelled as questionable or erroneous by provider or marked as questionable or erroneous by the automated TOAR quality control test. |
QuestionablePreliminaryConfirmed, QuestionablePreliminaryNotChecked, ErroneousPreliminaryConfirmed, ErroneousPreliminaryUnconfirmed, ErroneousPreliminaryFlagged1, ErroneousPreliminaryFlagged2, ErroneousPreliminaryNotChecked |
140 |
NotChecked |
Preliminary (realtime) data on which no automated quality control procedure has been run due to, for example, an incomplete time series. Note that a simple range check with bounds defined per variable is normally run anyhow, but this simple test cannot lead to the result “QC passed”. |
OKPreliminaryNotChecked, QuestionablePreliminaryNotChecked, ErroneousPreliminaryNotChecked |
Flag value |
Flag name |
Description |
|---|---|---|
0 |
OKValidatedVerified |
Data was received from provider as final validated data and passed the automatic quality control tests of the TOAR data centre. In addition, the data was subjected to manual inspection of the data summary plots. |
1 |
OKValidatedQCPassed |
Data was received from provider as final validated data and passed the automatic quality control tests of the TOAR data centre. |
2 |
OKValidatedModified |
Data was received from provider as final validated data and did not pass the automatic quality control tests of the TOAR data centre in the first pass. The data value was changed according to feedback from the data provider or if an obvious correction was possible. |
3 |
OKPreliminaryVerified |
Data was received from provider as preliminary or near realtime data and passed the automatic quality control tests of the TOAR data centre. In addition, the data was subjected to manual inspection of the data summary plots. |
4 |
OKPreliminaryQCPassed |
Data was received from provider as preliminary or near realtime data and passed the automatic quality control tests of the TOAR data centre. |
5 |
OKPreliminaryModified |
Data was received from provider as preliminary or near realtime data and did not pass the automatic quality control tests of the TOAR data centre in the first pass. The data value was changed according to feedback from the data provider or if an obvious correction was possible. |
6 |
OKEstimated |
Data value derived from an interpolation or modelling tool to fill a data gap. Note: you will never find this flag value in any “original” time series, but the name of the time series will indicate clearly if it contains estimated values. Some statistics may be more reliable if they are based on complete time series and thus avoid sampling biases. |
7 |
OKPreliminaryNotChecked |
Data was received from provider as preliminary or near realtime data and no QC test was run, for example because of an incomplete time series. |
10 |
QuestionableValidatedConfirmed |
Data was received from provider as final validated data with a quality flag indicating potential problems with the data value. The data value was also flagged as suspicious by the automatic quality control tests of the TOAR data centre. |
10 |
QuestionableValidatedConfirmed |
Data was received from provider as final validated data with a quality flag indicating potential problems with the data value. The data value was also flagged as suspicious by the automatic quality control tests of the TOAR data centre. |
12 |
QuestionableValidatedFlagged |
Data was received from provider as final validated data with no indication of potential problems. However, the data value was flagged as suspicious by the automatic quality control tests of the TOAR data centre. |
13 |
QuestionablePreliminaryConfirmed |
Data was received from provider as preliminary or near realtime data with a quality flag indicating potential problems with the data value. The data value was also flagged as suspicious or erroneous by the automatic quality control tests of the TOAR data centre. |
14 |
QuestionablePreliminaryUnconfirmed |
Data was received from provider as preliminary or near realtime data with a quality flag indicating potential problems with the data value. However, the data value was not flagged as suspicious or erroneous by the automatic quality control tests of the TOAR data centre. |
15 |
QuestionablePreliminaryFlagged |
Data was received from provider as preliminary or near realtime data with no indication of potential problems. However, the data value was flagged as suspicious by the automatic quality control tests of the TOAR data centre. |
16 |
QuestionablePreliminaryNotChecked |
Data was received from provider as preliminary or near realtime data with a quality flag indicating potential problems with the data value. No QC test was run, for example because of an incomplete time series. |
20 |
ErroneousValidatedConfirmed |
Data was received from provider as final validated data with a quality flag indicating an erroneous data value. The data value was also flagged as suspicious or erroneous by the automatic quality control tests of the TOAR data centre. |
21 |
ErroneousValidatedUnconfirmed |
Data was received from provider as final validated data with a quality flag indicating an erroneous data value. However, the data value was not flagged as suspicious or erroneous by the automatic quality control tests of the TOAR data centre. |
22 |
ErroneousValidatedFlagged1 |
Data was received from provider as final validated data with no indication of potential problems. However, the data value was flagged as erroneous by the automatic quality control tests of the TOAR data centre. |
23 |
ErroneousValidatedFlagged2 |
Data was received from provider as final validated data flagged as questionable values. However, the data value was flagged as erroneous by the automatic quality control tests of the TOAR data centre. |
24 |
ErroneousPreliminaryConfirmed |
Data was received from provider as preliminary or near realtime data with a quality flag indicating an erroneous data value. The data value was also flagged as suspicious or erroneous by the automatic quality control tests of the TOAR data centre. |
25 |
ErroneousPreliminaryUnconfirmed |
Data was received from provider as preliminary or near realtime data with a quality flag indicating an erroneous data value. However, the data value was not flagged as suspicious or erroneous by the automatic quality control tests of the TOAR data centre. |
26 |
ErroneousPreliminaryFlagged1 |
Preliminary or near realtime data was received from provider with no indication of potential problems. However, the data value was flagged as erroneous by the automatic quality control tests of the TOAR data centre. |
26 |
ErroneousPreliminaryFlagged1 |
Preliminary or near realtime data was received from provider with no indication of potential problems. However, the data value was flagged as erroneous by the automatic quality control tests of the TOAR data centre. |
28 |
ErroneousPreliminaryNotChecked |
Data was received from provider as preliminary or near realtime data with a quality flag indicating an erroneous data value. No QC test was run, for example because of an incomplete time series. |
90 |
MissingValue |
The data provider reported a missing value at this time stamp. Generally, the TOAR database will not explicitly store missing values but instead simply leave out the data value at that timestamp. However, there are situations when missing values are coded in the time series, for example if a new version of a dataset replaces formerly valid values by missing values. |
91 |
UnknownQualityStatus |
Also known as „not checked“. Technical flag to allow setting a quality status to unknown. The data provider did not report the data quality status and no QC test was run, for example because of an incomplete time series. This flag value can only be seen for realtime data, because all validated data are assumed to be OK by default. |
The following two tables summarise how flag values may be modified as a result of the automated quality control tests which are run during data ingestion or as part of a data inspection.
toarqc |
|||
provider |
OK |
Questionable |
Erroneous |
OK |
OKValidatedQCPassed |
QuestionableValidatedFlagged |
ErroneousValidatedFlagged1 |
Questionable |
QuestionableValidatedUnconfirmed |
QuestionableValidatedConfirmed |
ErroneousValidatedFlagged2 |
Erroneous |
ErroneousValidatedUnconfirmed |
ErroneousValidatedConfirmed |
ErroneousValidatedConfirmed |
toarqc |
||||
provider |
OK |
Questionable |
Erroneous |
NotChecked |
OK |
OKPreliminaryQCPassed |
QuestionablePreliminaryFlagged |
ErroneousPreliminaryFlagged1 |
OKPreliminaryNotChecked |
Questionable |
QuestionablePreliminaryUnconfirmed |
QuestionablePreliminaryConfirmed |
ErroneousPreliminaryFlagged2 |
QuestionablePreliminaryNotChecked |
Erroneous |
ErroneousPreliminaryUnconfirmed |
ErroneousPreliminaryConfirmed |
ErroneousPreliminaryConfirmed |
ErroneousPreliminaryNotChecked |
In some situations of realtime data processing the only automated test that can be run is a crude range test (for example if many values from different stations at one specific time step are inserted). This situation does not qualify as full QC test. Therefore, values are only flagged as erroneous (26, 27, or 24 depending on the provider flag) or as not checked (7, 16, 28).
Footnotes
- 1
TOAR V1 is described in Schultz, M. G. et al. (2017) Tropospheric Ozone Assessment Report: Database and Metrics Data of Global Surface Ozone Observations, Elem Sci Anth, 5, p.58. DOI: http://doi.org/10.1525/elementa.244
- 2
Findable, Accessible, Interoperable and Re-usable. For details see https://www.force11.org/group/fairgroup/fairprinciples and the TOAR data FAIRness assessment in Section 6 below.
- 3
In the second phase of TOAR, a dedicated statistics working group will explore more quantitative ways of assessing the accuracy and robustness of ozone trends.
- 4
These flags allow for convenient selection of data with the most relevant quality criteria, i.e. OK, questionable, or erroneous on the one hand and validated or preliminary on the other hand. The flags are composites of more specific flag values which are listed in Table 5.5.