dataMeta: Making and appending a data dictionary to an R dataset

Data

The data used for this vignette will be obtained from the Centers for Disease Control and Prevention (CDC) github repository for publicly avaiable Zika data: https://github.com/cdcepi/zika. This repo contains Zika data that has been scraped from the published records of the Health Department of various countries. The dataset used contains information related with Zika infection in the United States Virgin Islands (USVI) as published by their Health Department on their January 03, 2017 report and can be obtained using the code below. Please, note that the data is read with stringsAsFactors = FALSE. Below is a portion of this dataset:

data(my.data)

kable(head(my.data, 10), format = "html", caption = "Portion of dataset")

Another way to load the data is using it’s raw link from the github repo where it is stored:

path = "http://raw.githubusercontent.com/cdcepi/zika/master/"
path2 = "USVI/USVI_Zika/data/USVI_Zika-2017-01-03.csv"
url <- paste0(path, path2, collapse="")

my.data <- read.csv(url, header = TRUE, stringsAsFactors = FALSE)

Linker:

To build a data dictionary for the previous dataset, a linker data frame is constructed. This linker will serve as an intermediary to build the data dictionary. It will contain the names of the variables, a description of each variable provided by the user and a “variable type.” The variable type will give each variable a value of 1 or 0, as follows:

• 0: for those variables that have options that can be portrayed as a range of values. For example, age or dates or any categorical factors that the user does not want to list out (NA values will be maintained).
  
• 1: for those variables that have options that need to be listed and/or descrcibed later on.

The linker is built using one of the following two functions:

• prompt_linker.R will prompt the user to add the description of each variable in the console and the variable type.
  
• build_linker.R will require that the user create two vectors that will fill out the variable descriptions and variable types. This is shown below:

var_desc <- c("Date when report was published", "Regional location", 
             "Description of regional location", "Type of case",
             "A specific code for each data field", "The time period of each week",
             "The type of time period", "The number of cases per data field type",
             "The unit in which cases are reported")

var_type <- c(0, 1, 0, 1, 0, 0, 0, 0, 1)
                          
linker <- build_linker(my.data, variable_description = var_desc, variable_type = var_type)

Dictionary build:

Next, the dicitonary is built using the linker created above and the original dataset using the build_dict. Here, option_description is set to NULL because the options of each variable are self explanatory, thus the dictionary will not contain a description for each variable option. If option_descriptions will be added, then a vector of each option description is constructed. The length of this vector needs to be the same as the number of rows in the dictionary and it will depend on the the variable_type used in the linker. Another way to add option descriptions is to use the prompt_varopts function, which prompt the user for a description of each variable option, without the need to write the descriptions vector beforehand. If using this option, then prompt_varopts must be set to TRUE and option_description must be NULL. The following code builds a data dictionary for the USVI report:

dict <- build_dict(my.data = my.data, linker = linker, option_description = NULL, 
                   prompt_varopts = FALSE)

The dictionary will look as follows:

kable(dict, format = "html", caption = "Data dictionary for original dataset")

Adding the dictionary as a data attribute and incorporating other attributes:

To add the dictionary as an attribute to the original dataset, the function incorporate_attr is used. This function will require that the user add a main description of the dataset as shown below. There is also a prompt_attr function that will prompt the user to add this description through the console. Once the attributes are added to the dataset, the function save_it will save the file as an R data file .rds, which will preserve all attributes to the dataset. Accessing the attributes of the data is done using the following comand: attributes(my.new.data) and single attributes, like the main description of the dataset, can be obtained by typing: attributes(my.new.data)$main, as is shown below:

data_desc = "This data set portrays Zika infection related cases as reported by USVI."

my.new.data <- incorporate_attr(my.data = my.data, data.dictionary = dict, main_string = data_desc)

attributes(my.new.data)

$names
[1] "report_date"      "location"         "location_type"    "data_field"       "data_field_code" 
[6] "time_period"      "time_period_type" "value"            "unit"            

$class
[1] "data.frame"

$row.names
 [1]  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32

$main
[1] "This data set portrays Zika infection related cases as reported by USVI."

$dictionary
      variable_name variable_class                    variable_description
1       report_date      character          Date when report was published
2          location                                      Regional location
3                                                                         
4                                                                         
5                                                                         
6     location_type                       Description of regional location
7        data_field                                           Type of case
8                                                                         
9                                                                         
10                                                                        
11                                                                        
12                                                                        
13                                                                        
14                                                                        
15                                                                        
16                                                                        
17                                                                        
18                                                                        
19                                                                        
20                                                                        
21                                                                        
22                                                                        
23                                                                        
24                                                                        
25                                                                        
26                                                                        
27                                                                        
28                                                                        
29                                                                        
30                                                                        
31                                                                        
32                                                                        
33  data_field_code                    A specific code for each data field
34      time_period        logical            The time period of each week
35 time_period_type                                The type of time period
36            value        integer The number of cases per data field type
37             unit                   The unit in which cases are reported
                            variable_options
1                   2017-01-03 to 2017-01-03
2               United_States_Virgin_Islands
3  United_States_Virgin_Islands-Saint_Thomas
4   United_States_Virgin_Islands-Saint_Croix
5    United_States_Virgin_Islands-Saint_John
6                        county to territory
7                              zika_reported
8                          zika_lab_positive
9                                   zika_not
10                              zika_pending
11                     confirmed_age_under20
12                      confirmed_age_20to39
13                      confirmed_age_40to59
14                      confirmed_age_over59
15                         confirmed_age_unk
16                            confirmed_male
17                          confirmed_female
18                           confirmed_fever
19                     confirmed_acute_fever
20                      confirmed_arthralgia
21                       confirmed_arthritis
22                            confirmed_rash
23                  confirmed_conjunctivitis
24                         confirmed_eyepain
25                        confirmed_headache
26                         confirmed_malaise
27                          zika_no_specimen
28                      zika_tested_pregnant
29                    zika_positive_pregnant
30                    zika_negative_pregnant
31                     zika_pending_pregnant
32                 zika_no_specimen_pregnant
33                          VI0001 to VI0026
34                                  NA to NA
35                                  NA to NA
36                                 0 to 1930
37                                     cases

$last_edit_date
[1] "2025-02-27 03:00:29 UTC"

$author
[1] "root"

The user can also export the data dicitonary using as .csv or .xlsx by itlself and also save the data with all of its attributes, as shown below:

# Exporting dictionary only:
dict_only <- attributes(my.new.data)$dictionary
write.csv(dict_only, "dict_only.csv")

# Saving as .rds (dataset with appended dictionary)
save_it(complete_dict = my.new.data, name_of_file = "My Complete Dataset")