harp at UWC-West

Overview

• What is UWC-West?
• Point verification using harp:
  • Scripting system and workflow
  • Carlos: harphub/accord-verif-scripts
  • Examples (of non-harp stuff)
  • Operational implementation
• Additional metrics:
  • Corrected upper-air and scatterometer verification
  • “Headline” scores
  • Spatial work: gridded rainfall
• What’s next?

See also the presentation at the 2023 harp working week (https://opensource.umr-cnrm.fr/projects/accord/wiki/MQAWW202302).

What is UWC-West?

• Cooperation between Denmark, Iceland, the Netherlands, and Ireland (DINI).
• Common NWP on a shared HPC:
  • Harmonie-Arome 43h2
    • 2km horizontal grid
    • 90 vertical levels
  • Continuous EPS over western Europe:
    • 1+5 EPS updated every hour
    • 1+30 lagged over 6h
    • IFSENS LBCs, SPP perturbations
  • Deterministic over Greenland
• Common pre-processing, DevOps, dissemination, products, verification etc.

Common operations from late Q1 2024!

Point verification using harp

Motivation:

• “We want all the same scores that are in Monitor.”
• “We want a central tool for viewing the scores.”

Solution:

• A set of verification scripts to cater for all (or most) common point verification within UWC-W:
  • Deterministic and EPS verification
  • Largely based on the verification function Andrew presented yesterday
  • Includes some additions that were/are not in harp

Back in 2022, some features (e.g. vertical profiles) were not available in the harpVis app:

• A decision was made to generate pngs of “standard scores”
• A simply shiny page (similar to Monitor) was developed for visualisation

Point verification using harp

These UWC-W scripts formed the basis for the harphub/accord-verif-scripts repo:

• Created during harp working week in 2022, now quite far behind UWC-W scripts
• These should be harmonised!

Some pros:

• Can add “bespoke” products and react to user requests quickly
• Scripts can be used by non-experts

But, there are many cons:

• Such a black-box/one-size-fits-all approach goes against the overall philosophy of harp
• Generating lots of pngs is not very sensible
• Increasing divergence between harp and “non-harp” products
• Scripts can be used by non-experts

Workflow

graph LR
  config[<b>config.yml</b>]:::def --> vfldscript[<b>vfld2sql.R -OPTIONS</b>]:::def
  config --> vobsscript[<b>vobs2sql.R -OPTIONS</b>]:::def
  config --> verifscript[<b>point_verif.R -OPTIONS</b>]:::def
  classDef def font-size:10pt;

  vfld(<b>vfld</b>):::dataclass --> vfldscript
  vobs(<b>vobs</b>):::dataclass --> vobsscript
  classDef dataclass fill:lightgreen,font-size:10pt;

  vfldscript --> fctable(<b>FCTABLE</b>):::dataclass
  vobsscript --> obstable(<b>OBSTABLE</b>):::dataclass

  fctable --> verifscript
  obstable --> verifscript
  paramfile[<b>set_params.R</b>]:::def --> verifscript
  
  verifscript --> png(<b>.png</b>):::dataclass
  verifscript --> rds(<b>.rds</b>):::dataclass

  png --> visapp{<b>Local app</b>}:::appclass
  rds --> harpVis{<b>harpVis</b>}:::appclass
  classDef appclass fill:orange,font-size:10pt;

-OPTIONS: Command line argments which change from run to run (e.g. start and end date).

config.yml: Specify configuration settings e.g. fcst_model, lead_time, paths, etc.

set_params.R: Specfiy parameters of interest and associated options (e.g. thresholds, scaling).

point_verif.R: Run over all parameters, multiple verification domains (i.e. SID groups), and groupings (e.g. leadtime, validdate, SID). Create scorecards if desired.

Carlos: accord-verif-scripts

Examples

Forecast timeseries and daily variation:

fcst <- join_to_fcst(fcst,obs)
fcst <- expand_date(fcst,valid_dttm)
fcst <- bind(fcst)
fcst <- rename(fcst,"OBS"=all_of(param))

# Daily variation
group_vars <- c("fcst_model","valid_hour")
dv <- fcst %>% group_by_at(group_vars) %>%
summarise(dvar = mean(fcst), 
          mean_obs = mean(OBS),
          num_cases=n())
          

# Forecast timeseries
group_vars <- c("fcst_model","valid_dttm")
ts <- fcst %>% group_by_at(group_vars) %>%
summarise(ts = mean(fcst), 
          mean_obs = mean(OBS),
          num_cases=n())

Examples

Bias/RMSE maps:

fcst <- expand_date(fcst,valid_dttm)
grps_sid <- list(c("SID","valid_hour"),"valid_hour")
verif_sid <- do.call(
  get(verif_fn),
  list(.fcst = fcst, 
       parameter = {{prm_name}},
       thresholds = NULL, 
       groupings = grps_sid)
)

# Add lat/lon to the SIDs in verif object...

# Plotting
score="bias"
p_map <- verif_sid %>% 
ggplot(aes(lon,lat,fill=get(score),
       size=abs(get(score)))) + 
geom_polygon(data = map_data("world"),
             mapping = aes(long, lat, 
             group = group),
             inherit.aes = FALSE) +  
geom_point(colour='grey40',pch=21) + 
coord_map(projection = "lambert")...

Examples: Tile scorecard

“Tile” scorecard:

Operational implementation

Corrected upper-air and scatterometer verif

This work was carried out by Fabíola de Souza Silva (formerly at KNMI), however it is not currently in development and needs to be returned to!

Headline scores

Use harp tools to generate additional “non-harp” metrics. These are generally requested for management/administrative purposes, but are also useful for looking at long-term trends in performance.

fcst <- join_to_fcst(fcst,obs)
fcst <- bind(fcst)
fcst <- rename(fcst,"OBS"=all_of(param))

x <- fcst %>% mutate(hits = 
(abs(fcst-OBS) <= acc_tgt))
  
acc <- x %>% group_by(fcst_model) %>% 
summarise(acc = 100*sum(hits),
          ncases=length(hits)) %>%
mutate(acc = acc/ncases) 

Spatial work: Gridded rainfall

k_up <- array(1,dim=c(upgrid,upgrid))
OBS_up <- focal(OBS_raster,k_up,mean,
                pad=TRUE,padValue=NA,na.rm=TRUE)

Spatial work: FSS heatmaps

What’s next?

• Rewrite/design scripts based on content covered in this course!
• Transition to UWC-W operations.
• Get scatterometer and corrected upper-air verification back on track.
• Spatial verification:
  • DMI/KNMI radar data
  • SEVIRI cloud water path product
• Implement operational verification requested by physics experts:
  • Radiation
  • Comparison at “super-sites” (Cloudnet data)
• Push out point verification scripts to harphub/accord-verif-scripts.
• Revisit the “Screening” approach (see Eoin Whelan’s presentation: https://www.umr-cnrm.fr/accord/?3rd-ACCORD-All-Staff-Workshop-27-31-March-2023-Tallinn-hybrid).