Skip to contents

Evaluate nowcasts using proper scoring rules

Source: R/model-validation.R
enw_score_nowcast.Rd

Acts as a wrapper to scoringutils::score(). In particular, handling filtering nowcast summary output and linking this output to observed data. See the documentation for the scoringutils package for more on forecast scoring.

Usage

enw_score_nowcast(
  nowcast,
  latest_obs,
  log = FALSE,
  check = FALSE,
  round_to = 3,
  ...
)

Arguments

nowcast

A posterior nowcast or posterior prediction as returned by summary.epinowcast(), when used on the output of epinowcast().

latest_obs

A data.frame of the latest available observations as produced by enw_latest_data() or otherwise.

log

Logical, defaults to FALSE. Should scores be calculated on the log scale (with a 0.01 shift) for both observations and nowcasts. Scoring in this way can be thought of as a relative score vs the more usual absolute measure. It may be useful when targets are on very different scales or when the forecaster is more interested in good all round performance versus good performance for targets with large values.

check

Logical, defaults to FALSE. Should scoringutils::check_forecasts() be used to check input nowcasts.

round_to

Integer defaults to 3. Number of digits to round scoring output to.

...

Arguments passed on to scoringutils::score

data

A data.frame or data.table with the predictions and observations. For scoring using score(), the following columns need to be present:

  • true_value - the true observed values

  • prediction - predictions or predictive samples for one true value. (You only don't need to provide a prediction column if you want to score quantile forecasts in a wide range format.)

For scoring integer and continuous forecasts a sample column is needed:

  • sample - an index to identify the predictive samples in the prediction column generated by one model for one true value. Only necessary for continuous and integer forecasts, not for binary predictions.

For scoring predictions in a quantile-format forecast you should provide a column called quantile:

  • quantile: quantile to which the prediction corresponds

In addition a model column is suggested and if not present this will be flagged and added to the input data with all forecasts assigned as an "unspecified model").

You can check the format of your data using check_forecasts() and there are examples for each format (example_quantile, example_continuous, example_integer, and example_binary).

metrics

the metrics you want to have in the output. If NULL (the default), all available metrics will be computed. For a list of available metrics see available_metrics(), or check the metrics data set.

Value

A data.table as returned by scoringutils::score().

Examples

if (FALSE) { # interactive()
library(data.table)
library(scoringutils)

# Summarise example nowcast
nowcast <- enw_example("nowcast")
summarised_nowcast <- summary(nowcast)

# Load latest available observations
obs <- enw_example("observations")

# Keep the last 7 days of data
obs <- obs[reference_date > (max(reference_date) - 7)]

# score on the absolute scale
scores <- enw_score_nowcast(summarised_nowcast, obs)
summarise_scores(scores, by = "location")

# score overall on a log scale
log_scores <- enw_score_nowcast(summarised_nowcast, obs, log = TRUE)
summarise_scores(log_scores, by = "location")
}