accord verification scripts

C. Peralta, J. Fannon, D. Yazgi, F. Weidle, C. Fortelius, A. Singleton, S. Viana

Introduction

Developed during the 2022 harp working week in Helsinki.
Inspired by the monitor package. Many ideas ported from the uwcw scripts.
Idea was to be able to carry out a deterministic verification using harp.
User configuration in the form of command line options and config files.
Only compatible with harp 0.09 until recently. Ported scripts to harp 0.201 (develop branch)

Scripts structure

git clone git@github.com:harphub/accord-verif-scripts.git

directory tree

Configuration and running the scripts

The scripts need 3 configuration files and one master script.

    main.sh (master script)
    config_atos.sh
    config_atos.R
    config_atos.yml

The data processing is split in three steps:
- pre-processing (generate FCTABLE and OBSTABLE)
- verification (calculate scores)
- visualization (visualize using harp or external visapp)
Recommended: install local R environment using renv.

renv environment installation

module R 
R
renv::init()
exit

Use: renv:clean() if the above does not work. Then attempt again.

renv::install("remotes")
library(remotes)
install_github("harphub/harp")

This should give you a local installation in the renv directory.
Check you see this message like this when doing installation:

Installing package into ‘/etc/ecmwf/nfs/dh1_home_b/nhd/R/dev_accord/accord-verif-scripts/renv/library/R-4.2/x86_64-pc-linux-gnu’

You should see something like this when loading R in the local repository:

- Project '/etc/ecmwf/nfs/dh1_home_b/nhd/R/dev_accord/accord-verif-scripts' loaded. [renv 1.0.3]
- The project is out-of-sync -- use `renv::status()` for details.
[Previously saved workspace restored]

config.sh

Set some environment variables and decide what steps to do

#/bin/bash 
## load modules
module load R

CONFIG_INITIAL=config_atos

#R Env
USING_RENV=${USING_RENV-no}

export VERIF_DIR=/home/nhd/R/dev_accord/accord-verif-scripts #The location of this repo
export CASE_STUDY=testcase <--- the end path where the results will be written

# What to run
export RUN_POINT_VERF=${RUN_VERF-no} <--- SELECT yes TO PRODUCE rds files ONLY
export RUN_POINT_VERF_LOCAL=${RUN_POINT_VERF_LOCAL-no} <---- SELECT yes to produce png files with different scores

export SCORECARDS=${SCORECARDS-no}  <---  SELECT yes to generate scorecards (not tested yet in v021)
export RUN_VOBS2SQL=${RUN_VOBS2SQL-no} <--- SELECT yes to produce OBSTABLE
export RUN_VFLD2SQL=${RUN_VFLD2SQL-no}  <----SELECT yes to produce FCTABLE

export SHOW_WEB_STATIC=no <---- SELECT yes to display static png files
export SHOW_WEB_DYNAMIC=no   <---- SELECT yes to display harp visualization app that reads the rds files 
export SHINY_PORT=3699 # Change this number if port is busy when launching web


## DNOT for USER
MAIN_DIR=$(pwd)
CONFIG_DIR=$MAIN_DIR/config
RS_DIR=$MAIN_DIR/R
CONFIG_YAML=$CONFIG_DIR/$CONFIG_INITIAL.yml
CONFIG_R=$CONFIG_DIR/$CONFIG_INITIAL.R

##
export MAIN_DIR CONFIG_INITIAL  
export USING_RENV
export CONFIG_DIR RS_DIR CONFIG_YAML CONFIG_R

config.yml

Define parameters for verification steps: pre, verification and post

#Section to define paths for the pre-processing
# This section will be used by the scripts under pre_processing
shared:
  start_date: 2021010100
  end_date: 2021010123

pre:
  fclen: 24
  vfld_path: "/ec/res4/scratch/nhd/test_data_harp/vfld"
  vobs_path: "/ec/res4/scratch/nhd/test_data_harp/vobs"
  by_vobs_step: "1h"
  vfld_template:  #include one for each model if they are different. 
    - "vfld"
  do_all: TRUE # if true, it will ignore list of parameters and do all that is available in the vfld files
  params:
    - S10m
    - T2m
    - RH2m
    - Pmsl
    - T

#Section to define paths for the verification
# This section will be used by the scripts under verification
verif:
  by_step: "12h"
  fcst_model:
    - testmodel #enea43h22mbr000
  lead_time: seq(0, 24, 1)
  fcst_type: "det" #det or eps
  grps: "lead_time"
  fcst_path: "/ec/res4/scratch/nhd/test_data_harp/FCTABLE"
  obs_path: "/ec/res4/scratch/nhd/test_data_harp/OBSTABLE"
  verif_path: "/ec/res4/scratch/nhd/test_data_harp/output/verif_results"

#Section to define the paths for the output
# This section will be used by the plotting scripts
post:
  plot_output: "/ec/res4/scratch/nhd/test_data_harp/output"
  rds_path: "/ec/res4/scratch/smcd/verif_rds"
#This section if only for the scorecards
scorecards:
  ref_model: FC1
  fcst_model: FC2
  params:
    - T2m
    - S10m
    - RH2m
    - Pmsl

config.R

Define parameters and unit conversions (hard to include this expressions in the yml file)

# Setting up the parameters and their scaling and thresholds here
# Adding this part not so easy using the yml file
# It is possible to create several nested sections like
# variable:
#   - name
#   - threshold
# etc
# but this make look a bit confusing. Hardcoding this part here


# List of parameters
# Add more parameters below using the same format. For upper air parameters,
# don't forget the vc = "pressure"

library(yaml)

conf_get_config <- function(){
  CONFIG <- yaml.load_file(Sys.getenv('CONFIG_YAML') )
  CONFIG$params_details = conf_get_params_details()
  CONFIG
}


conf_get_params_details <- function(){
    T2m_thr <- c(-20, -10, seq(-5, 25, 5))
    S10m_thr <- seq(0, 25, 5)
    RH2m_thr <- seq(0, 100, 20)

    params <- list(
                T2m = list(
        thresholds = T2m_thr,
        scale_fcst = list(scaling = -273.15, new_units = "degC", mult = FALSE),
        scale_obs  = list(scaling = -273.15, new_units = "degC", mult = FALSE)

                        ),

                S10m = list(
        thresholds = S10m_thr

                        ),
         RH2m  =
         list (
        thresholds = RH2m_thr
         ),

                T = list(
        scale_fcst = list(scaling = -273.15, new_units = "degC", mult = FALSE),
        scale_obs  = list(scaling = -273.15, new_units = "degC", mult = FALSE),
        vc         = "pressure"

                        )


              )

    params

}

Example: convert vfld files

# What to run
export RUN_POINT_VERF=${RUN_VERF-no}   
export RUN_POINT_VERF_LOCAL=${RUN_POINT_VERF_LOCAL-no} 

export SCORECARDS=${SCORECARDS-no}  
export RUN_VOBS2SQL=${RUN_VOBS2SQL-no} 
export RUN_VFLD2SQL=${RUN_VFLD2SQL-yes}

export SHOW_WEB_STATIC=no        
export SHOW_WEB_DYNAMIC=no
export SHINY_PORT=3699 # Change this number if port is busy when launching web

run vfld

Scores generation

# What to run
export RUN_POINT_VERF=${RUN_VERF-yes}   
export RUN_POINT_VERF_LOCAL=${RUN_POINT_VERF_LOCAL-yes} 

export SCORECARDS=${SCORECARDS-no}  
export RUN_VOBS2SQL=${RUN_VOBS2SQL-no} 
export RUN_VFLD2SQL=${RUN_VFLD2SQL-no} 

export SHOW_WEB_STATIC=no        
export SHOW_WEB_DYNAMIC=no
export SHINY_PORT=3699 # Change this number if port is busy when launching web

fig1 fig2

Visualization using harp

# What to run
export RUN_POINT_VERF=${RUN_VERF-no}   
export RUN_POINT_VERF_LOCAL=${RUN_POINT_VERF_LOCAL-no} 

export SCORECARDS=${SCORECARDS-no}  
export RUN_VOBS2SQL=${RUN_VOBS2SQL-no} 
export RUN_VFLD2SQL=${RUN_VFLD2SQL-no} 

export SHOW_WEB_STATIC=no        
export SHOW_WEB_DYNAMIC=yes
export SHINY_PORT=3699 # Change this number if port is busy when launching web

static app

Example 2

static app

Visualization of static png files (S. Viana)

# What to run
export RUN_POINT_VERF=${RUN_VERF-no}   
export RUN_POINT_VERF_LOCAL=${RUN_POINT_VERF_LOCAL-no} 

export SCORECARDS=${SCORECARDS-no}  
export RUN_VOBS2SQL=${RUN_VOBS2SQL-no} 
export RUN_VFLD2SQL=${RUN_VFLD2SQL-no} 

export SHOW_WEB_STATIC=yes        
export SHOW_WEB_DYNAMIC=no
export SHINY_PORT=3699 # Change this number if port is busy when launching web

static app