haver.data {Haver} | R Documentation

## Access Haver Analytics Time Series Data

### Description

`haver.data` queries time series data contained in Haver Analytics databases. The data are returned either as a `HaverData` object, as a `data.frame`, as a `ts` object, or as a `zoo` object.

`haver.hasfullcodes` is a small helper function.

This help entry assumes that you are familiar with Haver_introduction.

### Usage

    haver.data(codes, database, start, end, frequency,
               rtype = "HaverData", aggmode = "strict",
               eop.dates = FALSE, limits = TRUE, cbreak = 1000,
               codeformat = 'auto', caldaily = FALSE)

    haver.hasfullcodes(x)

### Arguments

`codes` | is either a character vector or it is an object that fulfills the conditions of `haver.hasfullcodes`. If a character vector is supplied, the elements can be regular series codes or additionally supply the database where the code resides, in which case the elements must be specified in database:seriescode format. Examples of character vector elements are ‘gdp’ and ‘usecon:gdp’. Haver series codes may also be supplied through an object that passes `haver.hasfullcodes`. The only difference of usage of the `codes` argument in comparison to `haver.metadata` is that it is NOT optional here (see section ‘Details’ below).
`database` | is a character string. This argument is optional and specifies a default Haver database (excluding the path to the database, and excluding any file extension). If this argument is omitted, all elements of ‘codes’ must be in database:seriescode format. Otherwise the function errors out.
`start` | is either a string or an R Date object. It specifies the starting date of the query. If it is supplied as a string, it is interpreted under the date format ‘%Y-%m-%d’. Be careful to adhere to this format. Deviations from it are typically detected by a return value of `NA` of `as.Date` and a subsequent informative Haver package error message, but this is not guaruanteed. Even though `start` is supplied as (or converted to) a Date object, it is interpreted within the context of the frequency of the data set. This is an important point. See section ‘Specifying Query Starting and Ending Periods’ below for details.
`end` | is an R Date object. Specifies the ending date of the query. Analogous comments as for `start` apply.
`frequency` | is a string. Specifies the target frequency of the query. One of ‘caldaily’, ‘daily’, ‘weekly’, ‘monthly’, ‘quarterly’, and ‘annual’, or any abbreviation thereof.
`rtype` | is a string. Specifies the return type of the function. One of ‘HaverData’ (the default), ‘data.frame’, ‘ts’, and ‘zoo’, or any abbreviation thereof.
`aggmode` | is a string. the aggregation mode of the query. One of ‘strict’, ‘relaxed’, and ‘force’, or any abbreviation thereof. See section ‘Aggregation Modes’ below for details.
`eop.dates` | is a logical `TRUE`/`FALSE`. By default (`eop.dates=FALSE`), elements of the time vector are displayed as periods (e.g. ‘1988-Q1’). If `eop.dates=TRUE`, end-of-period dates are used instead (e.g. ‘1988-03-31’).
`limits` | is a logical `TRUE`/`FALSE`. If `limits=FALSE`, `haver.data` ignores settings regarding query limits.
`cbreak` | is an integer. This option is rarely needed and specifies the number of codes after which each C-level function of the Haver package checks for user interrupts. If you want to switch off checking for interrupts, specify a very large value, such as `cbreak=1e8`.
`codeformat` | is a string. The code format used in the column headings of the returned HaverData object. One of ‘auto’ (default), ‘simple’, or ‘full’, or any abbreviation thereof. ‘full’ generates codes in database:series format, whereas ‘simple’ creates column headings using simple Haver codes (i.e. series). ‘auto’ will choose between the two as described in section ‘Duplicate Codes’ below.
`caldaily` | is a logical `TRUE` / `FALSE`. If `TRUE`, treats calendar daily series as such. That is, these series are fully treated as having their own separate (7-day) frequency. The default `caldaily=FALSE` treats them as regular daily (5-day) series. The implications of this are discussed in section ‘Queries Involving Calendar Daily Series’ below. Explicitly requesting a calendar daily or a daily frequency overrides the `caldaily` argument. `frequency='caldaily'` always sets `caldaily=TRUE`. `frequency='daily'` always sets `caldaily=FALSE`.
`x` | a `HaverMetaData` object or a `data.frame` that fulfills certain conditions. See section ‘details’ below.

### Details

Both `haver.data` and `haver.metadata` take arguments `codes` and `database`. They allow you to conveniently specify Haver Analytics series codes. Usage of these arguments in the two functions is very similar, with just one exception: If the argument `codes` is omitted in `haver.metadata`, the function interprets the request as ‘all series codes from database database’. In `haver.data`, such an interpretation is not made, and consequently the argument `codes` is not optional there.

The order of series records within the returned object corresponds to the order of series in the input argument `codes`.

As is generally the case with Haver functions, (character) values of arguments supplied are not case-sensitive.

An object that passes the test of `haver.hasfullcodes` can be used as input argument `codes` instead of a character vector. `haver.hasfullcodes` returns a logical ‘TRUE’ if an object satisfies the following conditions:

- the `class` is `HaverMetaData` or

- the `class` is `data.frame` and in addition the data.frame's first two columns

  - are named ‘database’ and ‘code’

  - have entries of type ‘character’. Note that entries of type ‘factor’ are not allowed.

  - have no empty entries

  - have only alphanumeric entries

  - have at least one entry (the `data.frame` has at least one row).

See the ‘Examples’ section below for more information on this technique.

### Value

The return value of `haver.data`, by default, is a `HaverData` object. You can request different return types using the function argument `rtype`. Besides ‘HaverData’, admitted values are ‘data.frame’, ‘ts’, and ‘zoo’. ‘ts’ will only return a `ts` object if the data set frequency is monthly, quarterly, or annual. Otherwise a warning is issued and a `HaverData` object is returned.

If you specify any other return type than the default, no metadata will be available in the returned objects. Series metadata can only be stored in HaverData objects.

In the case of invalid series code and/or database specifications, a `HaverErrorReport` is returned.

`haver.hasfullcodes` returns a logical `TRUE`/`FALSE`.

### Temporal Aggregation

#### Determination of the Data Set Frequency

If option `frequency` is not specified the data set frequency corresponds to the lowest frequency of the series in the query. For example, querying a daily and a monthly series will result in a monthly data set, with daily values aggregated to monthly ones.

If you do specify option `frequency`, then this setting takes precedence over the rule laid out in the previous paragraph. For example, when querying a daily and a monthly series and specifying `frequency="weekly"`, then the daily series gets aggregated to weekly values, whereas the monthly series gets dropped from the query (Haver supports temporal aggregation, but not temporal disaggregation).

#### Temporal Aggregation Modes

A Haver aggregation mode can either be ‘strict’, ‘relaxed’, or ‘force’. You can invoke a particular aggregation mode by setting the argument `aggmode` to one of these three possible values. The default is ‘strict’.

Note the difference between the argument `aggmode` and the metadata field ‘aggtype’. Examples for ‘aggtype’s are ‘AVG’ and ‘SUM’ (see section ‘Metadata Fields’ in `HaverMetaData`). They define which calculations are carried out when a series is aggregated. `aggmode`s determine how these calculations behave in the presence of NAs. `aggmode` is set for a particular query, whereas ‘aggtype’ is a fixed setting that an individual series has.

For many queries, the default aggregation mode ‘strict’ is the correct one to use. However, temporal aggregation of daily and weekly data to a lower frequency should be done under ‘relaxed’ or ‘force’ in most cases. Most economic daily time series, for example, have about 10 NAs per year because of holidays. Whenever a daily series with ‘aggtype’=AVG or ‘aggtype’=SUM has a missing value, its aggregated monthly value will be NA also, under the default aggregation mode ‘strict’. As a result, aggregating such daily series to monthly series will produce NAs mostly. You can remedy this by specifying `aggmode="relaxed"` or `aggmode="force"`.

The following table describes the behavior that obtains with each combination of `aggmode` and ‘aggtype’. The term ‘aggregated span’ used in the table stands for a time span in the original frequency that aggregates to one observation of the target frequency. For example, 1973-Jan - 1973-Mar is an aggregated span for quarterly aggregation to 1973-Q1.

aggtype | aggmode | rule for returned value
EOP | strict | Returns the value of the last period in the aggregated
|  | span. If this value is missing, it returns missing.
| relaxed | Returns the last nonmissing value of the aggregated
|  | span. Returns missing only if all values in
|  | the aggregated span are missing.
| force | Same as under "relaxed".
AVG | strict | Does not return an aggregated value as soon as
|  | one value in the aggregated span is missing.
| relaxed | Calculates an aggregated value as soon as
|  | one value in the aggregated span is nonmissing.
| force | Same as under "relaxed".
SUM | strict | Does not return an aggregated value as soon as
|  | one value in the aggregated span is missing.
| relaxed | Same as under "strict".
| force | Calculates an aggregated value as soon as one
|  | value in the aggregated span is nonmissing.

### Specifying Query Starting and Ending Periods

Start and end dates have to be supplied as R `Date` object. If the start date is after the last observation of the requested series an error occurs and similarly for the end date. To minimize the possibility of erroneous query specifications start and end dates are restricted to the years 1900-2099.

If argument `enddate` is after the last observation of the implied data set, or if argument `startdate` is before the first observation of the implied data set, observations in the data set returned are padded with `NA`s accordingly. In other words, time periods of the data set will always start with `startdate`, if specified, and will always end with `enddate`, if specified.

Start and ending dates are supplied to `haver.data` as R `Date` objects (i.e. as calendar dates) or equivalent strings, but the interpretation of what these dates mean depends on the frequency of the data set returned. For example, if `start=as.Date("2012-01-01", format="%Y-%m-%d")` and the data set is returned in annual frequency, it is taken to mean ‘2012’. This can be confusing if the frequency of the returned data set is not known in advance. Let's say that you think you have a list of monthly series codes but there is actually one or more annual codes among them, so the data get aggregated to annual frequency. If you specified `start` as above and in addition `end=as.Date("2012-05-31", format="%Y-%m-%d")`, both dates get interpreted as ‘2012’. `haver.data` will return a value for 2012, and it will use all data points from 2012-Jan to 2012-Dec to calculated this value. If there had in addition be daily series in the query, an aggregated value for 2012 would be calculated for them too, based on all data points from 2012-Jan-01 to 2012-Dec-31. To summarize:

- Start and ending periods are supplied as R Date objects (i.e. as calendar dates) or equivalent strings, but are interpreted as periods within the context of the frequency of the data set returned.

- Any aggregation that takes place uses all underlying periods of the ‘aggregated span’, with no exceptions. The rules described in section ‘Aggregation Modes’ apply, with no exceptions.

For daily and weekly data sets it may happen that the dates of the data retrieved are slightly (by a few days) outside of the range specified by arguments `start` and `end`.

### When Data Cannot Be Retrieved

In some instances, `haver.data` cannot retrieve data for a particular series or for a particular query. Depending on the reason why this happens, `haver.data` responds in two different ways: It may either drop some series codes from the query and retrieve data for the other series codes, or it may retrieve no data at all and instead return a `HaverErrorReport` object. The paragraphs below state in which cases each behavior occurs and what you can do to fix up the query.

#### Dropping of Series

There are three instances where a query is successful (i.e. a `HaverData` object is returned) but some of the requested series get dropped from the query:

1. A series does not have any data points. It only occurs if the series has no data point in the database. This is very rarely the case. A series does not get dropped if you specify starting and ending dates in such a way that there are no valid data points in this time span. The series will solely consist of NAs in this case.

2. A series cannot be aggregated to the data set frequency. This occurs, for example, when a series holds some kind of fraction (e.g. the trade balance as a percentage of GDP). Then the aggregated value cannot simply be calculated as a sum or average. Such series have aggtype=NDF or aggtype=NST, see `HaverMetaData`. In these cases, Haver Analytics frequently provides separate series with aggregated values that are based on correct calculations.

3. A series cannot be disaggregated to the data set frequency. Temporal disaggregation is currently not supported by the Haver package.

If at least one series whose data can be retrieved remains, `haver.data` returns a `HaverData` object. If one or more series were dropped, this object contains information on the codes that failed, and on the reasons why this happened. The function `haver.codelists` enables you to get at lists of dropped series codes.

#### Query Errors

As soon as one series code or one database specified in a query cannot be found, `haver.data` returns a `HaverErrorReport`. This object contains details on what went wrong with the query. The `haver.codelists` method of the `HaverErrorReport` object enables you to get at lists of series codes that could not be found.

#### Duplicate Codes

Duplicates of full series codes are removed automatically. A warning is issued. For example, the query
`> hd <- haver.data(codes=c("fxtwb", "fxtwb"), dat="haverd"))`
will return a data set that contains ‘haverd:fxtwb’ just once.

Duplicate regular series codes are allowed. For example,
`> hd <- haver.data(codes=c("daily:ffed", "weekly:ffed")))`
will result in a data set that contains two series. This will, however, also be accompanied by a warning, for the following reasons: First, the `View` command will not work correctly anymore. With regards to the present example, it will display the first occurrence of ‘ffed’ twice. Secondly, if you want to assign aliases, you may not use two different aliases for ‘daily:ffed’ and ‘weekly:ffed’. See section ‘Assigning Aliases’ in `haver.setalias`. Moreover, while coercions to other objects will work correctly you will have multiple time series with identical names. In general, we advice you to avoid duplicate regular series codes within one query.

### Daily and Weekly Queries

When querying a weekly series, the dates assigned to data points correspond to the release day-of-week of the series on which the data source publishes new data. When aggregating a daily series to weekly frequency, this release day is set to Friday. When querying multiple weekly series, the dates shown correspond to the release day of the first series in the data set.

If you supply arguments `start` and `end`, it may happen that the starting and ending dates of the data retrieved are slightly (by a few days) outside of the implied time span.

When aggregating daily or weekly series to a lower frequency, familiarity with Haver aggregation modes is necessary. See section ‘Aggregation Modes’ above.

Any query can span at most 18,900 time periods. This constraint is only binding for daily data sets. It corresponds to about 72 years of data. If a query fails because it implies a data set that spans more than the maximum number of periods, consider explicitly specifying or adjusting the data set span via the parameters `start` and/or `end`. Alternatively, you can split the query up into different parts and then merge the resulting data sets manually.

### Queries Involving Calendar Daily Series

Before reading this section, be sure to be familiar with ‘Calendar Daily Series and Their Weekend Component Series’ in `haver.metadata`.

Calendar daily (7-daily) series are supported in R, but they have to be explicitly switched on. The main relevant option is `caldaily`, which defaults to `FALSE`, but setting option `frequency` to ‘daily’ or ‘caldaily’ also has an impact on the outcome.

If you supply the argument `caldaily=TRUE`, calendar daily series are treated has having their own 7-day frequency (as opposed to the default (5-day) daily frequency). This has a number of implications for queries involving multiple frequencies. All of these implications follow standard package rules for handling temporal aggregation and disaggregation (the latter not being supported and resulting in series being dropped from the query).

Before enumerating the implications of `caldaily=TRUE`, be aware of the default behavior (`caldaily=FALSE`):

1. If you query calendar daily series along with daily series, the resulting frequency will be daily. All series are kept in the query. Only the weekdays of the calendar daily series are used.

2. Since the default is to treat calendar daily series as regular (5-day) series, by default only these values are used in temporal aggregation calculations. If you want to aggregate a calendar daily series to a frequency of weekly or lower, you almost surely want to base calculations on all seven values of each week. You must explicitly supply `caldaily=TRUE` for this to happen.

WARNING: It is worth reiterating the critical importance of the `caldaily` setting for temporal aggregation pointed out in point 2 above. If you aggregate a calendar daily series to weekly or lower, be sure to explicitly use `caldaily=TRUE`.

Because of the importance of the issue, `haver.data` will warn you of potential problems. The messages that are issued differ depending on whether you use local queries or DLX Direct queries. Local queries will warn you whenever a calendar daily (7-day) series is treated as a regular daily (5-day) series; and they will indicate the exact number of series that are concerned. DLX Direct queries will issue a generic warning message whenever daily or calendar daily series are queried under `caldaily=FALSE`. These messages can be switched off using `haver.verbose`.

In contrast to the above two rules, under `caldaily=TRUE` the following rules apply:

1. If your query contains calendar daily series and series with lower frequencies and `frequency='caldaily'` is not specified, temporal aggregation is performed. Calendar daily series that have a valid aggregation type get aggregated using all seven days of the week, unless the target frequency is daily, in which case the calendar daily series get dropped.

2. If your query contains calendar daily series and series with lower frequencies and `frequency='caldaily'` is specified, only the calendar daily series are retrieved and all other series are dropped. This happens also if the query consists only of calendar daily and daily series: The daily series get dropped.

Points 1 and 2 above imply that it is not possible to directly generate a HaverMetaData object that shows all data points of calendar daily series along with daily series that have missings for weekend days. If you want that, you have to issue two separate queries (one for calendar daily series and another one for daily series) using `rtype='data.frame'` and then merge the resulting data.frames manually.

Related to the above, remember that explicitly setting `frequency='caldaily'` will override the `caldaily` setting and set it to `TRUE`. Similarly, explicitly setting `frequency='daily'` will override the `caldaily` setting and set it to `FALSE`.

Lastly, when aggregating calendar daily series to a weekly frequency, the dates assigned to data points correspond to Sundays (the last day of the aggregated span).

Note: The Haver calendar daily frequency in R follows a slightly different set of rules than calendar daily in Excel and DLXVG3. For example, in Excel it is possible to query calendar daily series and regular daily series in a single query. As noted previously, this is not possible in R. Likewise, as pointed out in the above warning, correct aggregation of calendar daily series to lower frequencies require an explicit argument in R (`caldaily=TRUE`). Excel and DLXVG3 always use all seven values of a week for aggregation.

### Query Limits

To provide a safeguard against large data queries that may either take very long or that consume an excessive amount of memory, `haver.data` by default checks settings on limits regarding data queries. There are two limits imposed. The first one specifies the maximum number of series in a query (default value: 5000) and the second one the maximum number of data points (default value: 15 million, which allows e.g. for 1000 daily series that span 50 years). You can query the current state and change these settings using `haver.limits`, whose help entry discusses the issues related to query execution time and memory consumption in more detail.

Note that the above limits never apply to DLX Direct queries. See `haver.direct`.

### Interrupting Execution

If you have specified a query that is taking a long time to conclude, you can cancel it in the usual way (ESC key).

### Time Zone Settings

The datetimemod field of the HaverMetaData object is affected by the time zone settings of your R session. Since each data query internally runs a metadata query, the same remarks on the subject as in `haver.metadata` apply.

### See Also

See Haver_package for all help entries related to the Haver package.

### Examples

    haver.path()
    haver.path("examples")

    ## The following statements illustrate how you can specify series codes and
    ## database names. Argument usage of 'codes' and 'database' is identical to
    ## the statements of the 'Examples' section in 'haver.metadata'.
    ## All queries below are equivalent. To limit the output, we
    ## supply a start and an end date.
    ex_t1 <- "2012-01-10"
    ex_tN <- "2012-01-13"
    haver.data(cod=c(       "fxtwb",        "fxtwm",        "fxtwotp"), dat="haverd"  , sta=ex_t1, end=ex_tN)
    haver.data(cod=c(       "FXTWB",        "FXTWM",        "FXTWOTP"), dat="HAVERD"  , sta=ex_t1, end=ex_tN)
    haver.data(cod=c("haverd:fxtwb",        "fxtwm",        "fxtwotp"), dat="haverd"  , sta=ex_t1, end=ex_tN)
    haver.data(cod=c("haverd:fxtwb", "haverd:fxtwm", "haverd:fxtwotp")                , sta=ex_t1, end=ex_tN)
    haver.data(cod=c("haverd:fxtwb", "haverd:fxtwm", "haverd:fxtwotp"), dat="havermqa", sta=ex_t1, end=ex_tN)

    ## Querying codes from multiple databases is possible:
    hd_1  <- haver.data( cod=c("haverD:fxtwb", "haverW:lic", "fcm1") , dat="haverMQA", start=ex_t1, end="2012-01-31")

    ## Querying all codes of a database is possible with 'haver.metadata'
    ## but not with 'haver.data', so the following produces an error:
    ## Not run:
    haver.data( dat="haverd" )

    ## End(Not run)

    ## You can assign the return value of 'haver.data' to a symbol, of course:
    hd_2  <- haver.data(codes=c("fxtwb", "fxtwm", "fxtwotp") , database="haverd", start=ex_t1, end="2012-01-13")

    ## If you want to look at the data, you can use 'View':
    ## Not run:
    View(hd_2)

    ## End(Not run)

    ## You can also invoke the 'print' method of the object by typing its name:
    hd_2

    ## You can request a lower frequency than daily, e.g. quarterly:
    ## (at this frequency, we do not have to use options 'start' and 'end'
    ##  in order to limit output)
    hd_3 <- haver.data(codes=c("fxtwb", "fxtwm", "fxtwotp") , database="haverd" , freq="q")
    hd_3

    ## The data points are NAs because the default aggregation mode is 'strict'.
    ## When aggregating daily data, it is frequently appropriate to use aggregation
    ## modes that are less restrictive. For details, see section 'Aggregation Modes' above.
    hd_4 <- haver.data(codes=c("fxtwb", "fxtwm", "fxtwotp") ,
                       database="haverd" , freq="q", aggmode="relaxed")
    hd_4

    ## Note that the default aggregation mode remains 'strict'.

    ## Let's see what the database 'HAVERMQA' contains:
    hmd_mqa <- haver.metadata(database="havermqa")
    ## Not run:
    View(hmd_mqa)

    ## End(Not run)

    ## We pick and retrieve a monthly, a quarterly, and an annual series.
    ## The monthly and the quarterly series get aggregated to annual frequency.
    hd_5 <- haver.data(codes=c("c", "fcm1", "ift"), database="havermqa",
                       start=as.Date("2008-01-13", format="%Y-%m-%d"))
    hd_5

    ## When we explicitly request a quarterly frequency, monthly series get aggregated
    ## while annual ones get dropped:
    hd_6 <- haver.data(codes=c("c", "fcm1", "ift"), database="havermqa",
                       start=as.Date("2008-01-13", format="%Y-%m-%d"), freq="q")
    hd_6

    ## We can investigate what series got dropped by using 'haver.codelists':
    haver.codelists(hd_6)

    ## Every time series data retrieval also retrieves the series metadata and
    ## attaches it to the 'HaverData' object.
    ## You can access it using 'haver.datamd':
    haver.datamd(hd_6)

    ## The previous call returned a 'HaverMetaData' object.
    ## When assigning it to a symbol, you can customize its output:
    hmd_6 <- haver.datamd(hd_6)
    print(hmd_6, fields=c("code", "frequency"))

    ## You can retrieve an identical HaverData object that
    ## displays end-of-period dates:
    hd_7 <- haver.data(codes=c("c", "fcm1", "ift"), database="havermqa",
                       start=as.Date("2008-01-13", format="%Y-%m-%d"),
                       freq="q", eop.dates=TRUE)

    ## If you do not need metadata attached to the data object,
    ## you can request other return types, e.g. the R object 'ts':
    ex_ts <- haver.data(codes=c("c", "fcm1", "ift"), database="havermqa",
                        start=as.Date("2008-01-13", format="%Y-%m-%d"),
                        freq="q", rtype="ts")
    ex_ts

    ## 'ts' object data can directly be graphed:
    plot(ex_ts)

    ## We can use a 'HaverMetaData' object to specify codes for 'haver.data' queries:
    hmd_w <- haver.metadata(dat="haverw")
    hd_w  <- haver.data(codes=hmd_w)

    ## We can add codes from other 'HaverMetaData' objects, and manually
    ## add or delete codes once we are working with data.frames
    ## whose columns 1 and 2 contain database names and codes:
    ex_df_d    <- as.data.frame(haver.metadata(dat="haverd"))
    ex_df_dw   <- rbind(ex_df_d[1:2] , as.data.frame(hmd_w)[,1:2])
    ex_df_all  <- rbind(ex_df_dw , data.frame(database="havermqa", code="c"))

    ## If you are not sure whether you can use an object other than a character
    ## vector as input for the 'codes' argument of 'haver.data', you can check with:
    haver.hasfullcodes(ex_df_all)

    ## Since this is 'TRUE', we can pull in the data:
    hd_all <- haver.data(codes=ex_df_all)
    print(haver.datamd(hd_all) , fields.disp=c("database", "code", "descriptor"))

    haver.path("restore")

[Package Haver version 3.2 Index]