UISH

What this section contains

The Validation section compares the concentrations computed by the model with the station measurements, to quantify how well the model reproduces the observed reality. It gathers the interactive dashboard of comparison charts, the documentation of every chart function and the description of the statistical indices used, with their formulas.

Available tools and how to use them

Note: mutual information (MI) is expressed in nat, the natural unit of information (the natural logarithm is used; 1 nat ≈ 1.44 bit). It measures how much information the model and the observations share: 0 = independent, higher values = stronger (possibly non-linear) link.

How the interface works

The Validation dashboard is organised in three areas.

1. Global controls

• Stratification (Global / Season / Hour): chooses how the metrics are aggregated.
• Metric weights per pollutant: one tab per pollutant with a draggable radar and the list of metrics. For each metric you set the weight (%) and choose the scaling function (drop-down; see Scaling functions) that reshapes the index goodness. The offered functions are filtered by the index domain. The RMSE/MAE reference and the weight rationale are also provided.

2. Score

• Score per station and aggregate: tables combining the indices (weight × reshaped goodness) into a 0–100 score.

3. Charts

• You pick chart type, pollutant and station: the menus filter each other (for a station only the pollutants it measures appear).
• Charts that read raw data from the server are marked 🕒 and triggered with Load data; the heavier ones (3D grid animation, 2D time-lapse) are marked 🕒🕒 (up to ~5 minutes wait).
• Interactions: zoom with the wheel on 3D charts (Three.js/WebGL), 2D/3D views for the wavelet, a Play/Pause/speed player for the animations, legends and tooltips.

Technologies to store, read and process the data

• Storage. The validation indices are pre-computed and stored in PostgreSQL: a table with the metrics per station×pollutant, the metric catalogue, the percentiles, the seasonal decomposition (STL) and the default pollutant weights (JSONB field).
• Reading. Pages read these values through PHP endpoints returning JSON; for the charts that use the raw hourly data (scatter, conditional bias, CDF, wavelet, 3D scatter, animations) the loading is asynchronous (AJAX) and windowed in time.
• Processing. Metrics and decompositions are computed offline in Python (NumPy for the indices, statsmodels for the STL decomposition) and written to the database; client-side rendering uses ECharts, ECharts-GL and Three.js, while formulas are typeset with KaTeX.

Scientific bibliography

• Taylor, K.E. (2001). Summarizing multiple aspects of model performance in a single diagram. Journal of Geophysical Research: Atmospheres, 106(D7), 7183–7192. doi:10.1029/2000JD900719
• Jolliff, J.K. et al. (2009). Summary diagrams for coupled hydrodynamic-ecosystem model skill assessment. Journal of Marine Systems, 76(1–2), 64–82. doi:10.1016/j.jmarsys.2008.05.014
• Willmott, C.J. (1982). Some comments on the evaluation of model performance. Bulletin of the American Meteorological Society, 63(11), 1309–1313. doi:10.1175/1520-0477(1982)063<1309:SCOTEO>2.0.CO;2
• Emery, C. et al. (2017). Recommendations on statistics and benchmarks to assess photochemical model performance. Journal of the Air & Waste Management Association, 67(5), 582–598. doi:10.1080/10962247.2016.1265027
• Harrington, E.C. (1965). The desirability function. Industrial Quality Control, 21(10), 494–498. — foundation of the functions mapping each index to a [0,1] "goodness" (its two-sided form is an exp(−|y|ⁿ), like the scaling curves).
• Derringer, G., & Suich, R. (1980). Simultaneous optimization of several response variables. Journal of Quality Technology, 12(4), 214–219. doi:10.1080/00224065.1980.11980968 — weighted combination of desirability functions.