Chapter 3 Convert observations to SQLite
3.1 Introduction
Point observations also come in many formats. For Hirlam, that format is vobs. Currently this is the only point observations format that harp can deal with. vobs are pretty much the same format as vfld, with files for every observation time. This is considerable file IO if you want a season’s worth of data, for example. Again in harp we use sqlite to store point observations, making it quick and easy to access exactly what you want.
3.2 Converting observations to SQLite
The harp function for converting observations is read_obs_convert
. It works similarly to the read_forecast
function, but has fewer arguments. In the data directory is a vobs directory containing the vobs files. We have hourly data here from 00 UTC 17 Feb 2019 to 23 UTC 20 Feb. Let’s read them in:
library(tidyverse)
library(here)
library(harpIO)
read_obs_convert(
start_date = 2019021700,
end_date = 2019022023,
by = "1h",
obs_path = here("data/vobs"),
return_data = TRUE
)
## $synop
## # A tibble: 324,138 x 24
## validdate SID lat lon elev CCtot D10m S10m T2m Td2m RH2m
## <dbl> <int> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 1.55e9 1001 70.9 -8.67 9 8 202 8.8 270. 268. 85.3
## 2 1.55e9 1002 80.1 16.2 8 NA 330 13 248. 246 81.0
## 3 1.55e9 1003 77 15.5 10 1.04 30 2 252. 245. 52.1
## 4 1.55e9 1006 78.3 22.8 14 NA 20 3 247. 244. 78.5
## 5 1.55e9 1007 78.9 11.9 8 NA 350 5 250. NA NA
## 6 1.55e9 1008 78.2 15.5 27 7.04 298 7.6 251. 246 61.9
## 7 1.55e9 1009 80.7 25.0 5 NA 330 12 244. 241. 76.5
## 8 1.55e9 1010 69.3 16.1 13 NA 340 11.3 274. 268. 59.9
## 9 1.55e9 1011 80.1 31.5 9 NA 352 11.4 240. 236. 68.9
## 10 1.55e9 1013 78.1 13.6 -99 NA NA NA NA NA NA
## # … with 324,128 more rows, and 13 more variables: Q2m <dbl>, Ps <dbl>,
## # Pmsl <dbl>, vis <dbl>, AccPcp3h <dbl>, AccPcp6h <dbl>, AccPcp24h <dbl>,
## # N75 <int>, CClow <int>, Cbase <int>, AccPcp1h <dbl>, Gmax <dbl>,
## # AccPcp12h <dbl>
##
## $temp
## # A tibble: 110,568 x 13
## validdate SID lat lon elev p Z T RH D S
## <dbl> <int> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 1.55e9 1001 70.9 -8.67 9 1000 64 269. 83.3 202 3
## 2 1.55e9 1001 70.9 -8.67 9 925 672 264. 100 205 8
## 3 1.55e9 1001 70.9 -8.67 9 850 1322 262. 86.4 161 3
## 4 1.55e9 1001 70.9 -8.67 9 700 2792 254. 32 251 4
## 5 1.55e9 1001 70.9 -8.67 9 500 5224 240. 8.7 270 18
## 6 1.55e9 1001 70.9 -8.67 9 400 6762 230. 28.1 275 32
## 7 1.55e9 1001 70.9 -8.67 9 300 8656 218 38.6 269 43
## 8 1.55e9 1001 70.9 -8.67 9 250 NA NA NA NA NA
## 9 1.55e9 1001 70.9 -8.67 9 200 11220 218. 2.9 262 28
## 10 1.55e9 1001 70.9 -8.67 9 150 13048 217. 2.5 257 21
## # … with 110,558 more rows, and 2 more variables: Q <dbl>, Td <dbl>
##
## $synop_params
## parameter accum_hours units
## 1 CCtot 0 oktas
## 2 D10m 0 degrees
## 3 S10m 0 m/s
## 4 T2m 0 K
## 5 Td2m 0 K
## 6 RH2m 0 percent
## 7 Q2m 0 kg/kg
## 8 Ps 0 hPa
## 9 Pmsl 0 hPa
## 10 vis 0 m
## 11 AccPcp3h 3 kg/m^2
## 12 AccPcp6h 6 kg/m^2
## 13 AccPcp24h 24 kg/m^2
## 14 N75 0 oktas
## 15 CClow 0 oktas
## 16 Cbase 0 m
## 17 AccPcp1h 1 kg/m^2
## 18 Gmax 0 m/s
## 19 AccPcp12h 12 kg/m^2
##
## $temp_params
## parameter accum_hours units
## 1 p 0 hPa
## 2 Z 0 m
## 3 T 0 K
## 4 RH 0 percent
## 5 D 0 degrees
## 6 S 0 m/s
## 7 Q 0 kg/kg
## 8 Td 0 K
Your turn:
- Write the observations to SQLite files in the directory data/OBSTABLE
Solution