Get and parse multiple VecTraits datasets by ID in chunks — fetch_extract_vt

Retrieve and parse VecTraits datasets specified by their dataset IDs in batches.

This is not usually necessary (generally you just need fetch_vt()) but allows one to release data that is not in use from memory. If you would like more control on extraction or parsing then it is best to wrap fetch_vt() and extract_vt() in your own chunker instead.

Usage

fetch_extract_vt_chunked(
  ids,
  chunksize = 20,
  cols = NA,
  returnunique = FALSE,
  rate = 5,
  connections = 1,
  basereq = NA
)

Arguments

ids: a numeric vector of IDs (preferably in an ohvbd.ids object) indicating the particular datasets to download.
chunksize: an integer defining the size of chunks to retrieve in one iteration.
cols: a character vector of columns to extract from the dataset.
returnunique: whether to return only the unique rows within each dataset according to the filtered columns.
rate: maximum number of calls to the API per second.
connections: number of simultaneous connections to the server at once. Maximum 8. Do not enable unless you really need to as this hits the server significantly harder than usual.
basereq: an httr2 request object, as generated by vb_basereq(). If NA, uses the default request.

Value

An ohvbd.data.frame containing the requested data.

Author

Francis Windram

Examples

if (FALSE) { # \dontrun{
fetch_extract_vt_chunked(c(54,55,56), chunksize = 2, rate=5)
} # }