Chapter 3 Convert observations to SQLite

3.1 Introduction

Point observations also come in many formats. For Hirlam, that format is vobs. Currently this is the only point observations format that harp can deal with. vobs are pretty much the same format as vfld, with files for every observation time. This is considerable file IO if you want a season’s worth of data, for example. Again in harp we use sqlite to store point observations, making it quick and easy to access exactly what you want.

3.2 Converting observations to SQLite

The harp function for converting observations is read_obs_convert. It works similarly to the read_forecast function, but has fewer arguments. In the data directory is a vobs directory containing the vobs files. We have hourly data here from 00 UTC 17 Feb 2019 to 23 UTC 20 Feb. Let’s read them in:

library(tidyverse)
library(here)
library(harpIO)

read_obs_convert(
  start_date  = 2019021700,
  end_date    = 2019022023,
  by          = "1h",
  obs_path    = here("data/vobs"),
  return_data = TRUE
)
## $synop
## # A tibble: 324,138 x 24
##    validdate   SID   lat   lon  elev CCtot  D10m  S10m   T2m  Td2m  RH2m
##        <dbl> <int> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
##  1    1.55e9  1001  70.9 -8.67     9  8      202   8.8  270.  268.  85.3
##  2    1.55e9  1002  80.1 16.2      8 NA      330  13    248.  246   81.0
##  3    1.55e9  1003  77   15.5     10  1.04    30   2    252.  245.  52.1
##  4    1.55e9  1006  78.3 22.8     14 NA       20   3    247.  244.  78.5
##  5    1.55e9  1007  78.9 11.9      8 NA      350   5    250.   NA   NA  
##  6    1.55e9  1008  78.2 15.5     27  7.04   298   7.6  251.  246   61.9
##  7    1.55e9  1009  80.7 25.0      5 NA      330  12    244.  241.  76.5
##  8    1.55e9  1010  69.3 16.1     13 NA      340  11.3  274.  268.  59.9
##  9    1.55e9  1011  80.1 31.5      9 NA      352  11.4  240.  236.  68.9
## 10    1.55e9  1013  78.1 13.6    -99 NA       NA  NA     NA    NA   NA  
## # … with 324,128 more rows, and 13 more variables: Q2m <dbl>, Ps <dbl>,
## #   Pmsl <dbl>, vis <dbl>, AccPcp3h <dbl>, AccPcp6h <dbl>, AccPcp24h <dbl>,
## #   N75 <int>, CClow <int>, Cbase <int>, AccPcp1h <dbl>, Gmax <dbl>,
## #   AccPcp12h <dbl>
## 
## $temp
## # A tibble: 110,568 x 13
##    validdate   SID   lat   lon  elev     p     Z     T    RH     D     S
##        <dbl> <int> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
##  1    1.55e9  1001  70.9 -8.67     9  1000    64  269.  83.3   202     3
##  2    1.55e9  1001  70.9 -8.67     9   925   672  264. 100     205     8
##  3    1.55e9  1001  70.9 -8.67     9   850  1322  262.  86.4   161     3
##  4    1.55e9  1001  70.9 -8.67     9   700  2792  254.  32     251     4
##  5    1.55e9  1001  70.9 -8.67     9   500  5224  240.   8.7   270    18
##  6    1.55e9  1001  70.9 -8.67     9   400  6762  230.  28.1   275    32
##  7    1.55e9  1001  70.9 -8.67     9   300  8656  218   38.6   269    43
##  8    1.55e9  1001  70.9 -8.67     9   250    NA   NA   NA      NA    NA
##  9    1.55e9  1001  70.9 -8.67     9   200 11220  218.   2.9   262    28
## 10    1.55e9  1001  70.9 -8.67     9   150 13048  217.   2.5   257    21
## # … with 110,558 more rows, and 2 more variables: Q <dbl>, Td <dbl>
## 
## $synop_params
##    parameter accum_hours   units
## 1      CCtot           0   oktas
## 2       D10m           0 degrees
## 3       S10m           0     m/s
## 4        T2m           0       K
## 5       Td2m           0       K
## 6       RH2m           0 percent
## 7        Q2m           0   kg/kg
## 8         Ps           0     hPa
## 9       Pmsl           0     hPa
## 10       vis           0       m
## 11  AccPcp3h           3  kg/m^2
## 12  AccPcp6h           6  kg/m^2
## 13 AccPcp24h          24  kg/m^2
## 14       N75           0   oktas
## 15     CClow           0   oktas
## 16     Cbase           0       m
## 17  AccPcp1h           1  kg/m^2
## 18      Gmax           0     m/s
## 19 AccPcp12h          12  kg/m^2
## 
## $temp_params
##   parameter accum_hours   units
## 1         p           0     hPa
## 2         Z           0       m
## 3         T           0       K
## 4        RH           0 percent
## 5         D           0 degrees
## 6         S           0     m/s
## 7         Q           0   kg/kg
## 8        Td           0       K

Your turn:

  • Write the observations to SQLite files in the directory data/OBSTABLE

Solution

read_obs_convert(
  start_date  = 2019021700,
  end_date    = 2019022023,
  by          = "1h",
  obs_path    = here("data/vobs"),
  sqlite_path = here("data/OBSTABLE")
)