vignettes/articles/when-to-use-rrapply.Rmd
when-to-use-rrapply.RmdThe nested list below outlines the genealogy of several famous mathematicians. Each list element contains an additional "given" attribute with the mathematician’s given name. The numeric values at the leaf elements are the total number of descendants according to the Mathematics Genealogy Project as of June 2020. If no descendants are available there is a missing value present at the leaf node.
students <- list( Bernoulli = structure(list( Bernoulli = structure(list( Bernoulli = structure(1, given = "Daniel"), Euler = structure(list( Euler = structure(NA, given = "Johann"), Lagrange = structure(list( Fourier = structure(68679, given = "Jean-Baptiste"), Plana = structure(NA, given = "Giovanni"), Poisson = structure(113435, given = "Simeon") ), given = "Joseph") ), given = "Leonhard") ), given = "Johann"), Bernoulli = structure(NA, given = "Nikolaus") ), given = "Jacob") ) str(students, give.attr = FALSE) #> List of 1 #> $ Bernoulli:List of 2 #> ..$ Bernoulli:List of 2 #> .. ..$ Bernoulli: num 1 #> .. ..$ Euler :List of 2 #> .. .. ..$ Euler : logi NA #> .. .. ..$ Lagrange:List of 3 #> .. .. .. ..$ Fourier: num 68679 #> .. .. .. ..$ Plana : logi NA #> .. .. .. ..$ Poisson: num 113435 #> ..$ Bernoulli: logi NA
Consider the following exercise in list recursion:
Filter all descendants of ‘Leonhard Euler’ while keeping the original list structure and replace all missing values by zero.
Here is a possible (not so efficient) solution using recursion with the Recall function:
prune_replace_Euler <- function(x) { i <- 1 while(i <= length(x)) { if(identical(names(x)[i], "Euler") & identical(attr(x[[i]], "given"), "Leonhard")) { x[[i]] <- rapply(x[[i]], f = function(x) replace(x, is.na(x), 0), how = "replace") i <- i + 1 } else { if(is.list(x[[i]])) { val <- Recall(x[[i]]) x[[i]] <- val i <- i + !is.null(val) } else { x[[i]] <- NULL } if(all(sapply(x, is.null))) { x <- NULL } } } return(x) } str(prune_replace_Euler(students), give.attr = FALSE) #> List of 1 #> $ Bernoulli:List of 1 #> ..$ Bernoulli:List of 1 #> .. ..$ Euler:List of 2 #> .. .. ..$ Euler : num 0 #> .. .. ..$ Lagrange:List of 3 #> .. .. .. ..$ Fourier: num 68679 #> .. .. .. ..$ Plana : num 0 #> .. .. .. ..$ Poisson: num 113435
This works, but is hardly the kind of code we would like to write for such a seemingly simple data exploration question. Moreover, this code is not very easy to follow or reason about, which makes it time-consuming to update or modify for future tasks.
Another approach would be to unnest the list in a more manageable (e.g. rectangular) format or use specialized packages such as igraph or data.tree to make pruning or modifying node entries more straightforward. Note that attention must be payed to correctly include the node attributes in the transformed object as the node names themselves are not unique in the given example. This is a sensible approach for more complex data analysis or statistical modeling tasks, but ideally we would like to keep the list in its original format for simple data exploration to reduce the number of processing steps and minimize the possibility of introducing mistakes in the code.
The recursive function above makes use of rapply, a member of the base-R apply-family of functions, that allows to apply a function recursively to the elements of a nested list and decide how the returned result is structured. If you are not familiar with the rapply function it might be useful to first read the function documentation help("rapply") or check out the first section of the rrapply-package vignette (browseVignettes(package = "rrapply")).
Although quite useful, the rapply-function is not always sufficiently flexible in practice, e.g. for pruning elements of a nested list (as demonstrated above). The rrapply-function is an attempt to enhance and update base rapply to make it more generally applicable in the context of list recursion. The rrapply-function builds on the native implementation of base rapply in R’s C-interface and for this reason requires no other external dependencies.
rrapply()
For illustration purposes, we will use the dataset renewable_energy_by_country included in the rrapply-package, a nested list containing the shares of renewable energy as a percentage in the total energy consumption per country in 2016. The data is publicly available at the United Nations Open SDG Data Hub UNSD-SDG07. The 249 countries and areas are structured based on their geographical location according to the United Nations M49 standard UNSD-M49 The numeric values listed for each country are percentages, if no data is available the value of the country is NA.
For convenience, we subset only the values for countries and areas in Oceania,
renewable_oceania <- renewable_energy_by_country[["World"]]["Oceania"] str(renewable_oceania, list.len = 3, give.attr = FALSE) #> List of 1 #> $ Oceania:List of 4 #> ..$ Australia and New Zealand:List of 6 #> .. ..$ Australia : num 9.32 #> .. ..$ Christmas Island : logi NA #> .. ..$ Cocos (Keeling) Islands : logi NA #> .. .. [list output truncated] #> ..$ Melanesia :List of 5 #> .. ..$ Fiji : num 24.4 #> .. ..$ New Caledonia : num 4.03 #> .. ..$ Papua New Guinea: num 50.3 #> .. .. [list output truncated] #> ..$ Micronesia :List of 8 #> .. ..$ Guam : num 3.03 #> .. ..$ Kiribati : num 45.4 #> .. ..$ Marshall Islands : num 11.8 #> .. .. [list output truncated] #> .. [list output truncated]
With base rapply, there is no convenient way to prune or filter elements from the input list. The rrapply function adds an option how = "prune" to prune all list elements not subject to application of the function f from a nested list. The original list structure is retained, similar to the non-pruned versions how = "replace" and how = "list". Using how = "prune", we can for instance drop all NA elements from the list while preserving the original list structure:
## Drop all logical NA's while preserving list structure na_drop_oceania <- rrapply( renewable_oceania, f = identity, classes = "numeric", how = "prune" ) str(na_drop_oceania, list.len = 3, give.attr = FALSE) #> List of 1 #> $ Oceania:List of 4 #> ..$ Australia and New Zealand:List of 2 #> .. ..$ Australia : num 9.32 #> .. ..$ New Zealand: num 32.8 #> ..$ Melanesia :List of 5 #> .. ..$ Fiji : num 24.4 #> .. ..$ New Caledonia : num 4.03 #> .. ..$ Papua New Guinea: num 50.3 #> .. .. [list output truncated] #> ..$ Micronesia :List of 7 #> .. ..$ Guam : num 3.03 #> .. ..$ Kiribati : num 45.4 #> .. ..$ Marshall Islands : num 11.8 #> .. .. [list output truncated] #> .. [list output truncated]
Instead, set how = "flatten" to return a flattened unnested version of the pruned list. This is more efficient than first returning the pruned list with how = "prune" and unlisting or flattening the list in a subsequent step.
## Drop all logical NA's and return unnested list na_drop_oceania2 <- rrapply( renewable_oceania, f = identity, classes = "numeric", how = "flatten" ) str(na_drop_oceania2, list.len = 10, give.attr = FALSE) #> List of 22 #> $ Australia : num 9.32 #> $ New Zealand : num 32.8 #> $ Fiji : num 24.4 #> $ New Caledonia : num 4.03 #> $ Papua New Guinea : num 50.3 #> $ Solomon Islands : num 65.7 #> $ Vanuatu : num 33.7 #> $ Guam : num 3.03 #> $ Kiribati : num 45.4 #> $ Marshall Islands : num 11.8 #> [list output truncated]
Or, use how = "melt" to return a melted data.frame of the pruned list similar in format to reshape2::melt applied to a nested list. The rows of the melted data.frame contain the node paths of the elements in the pruned list. The "value" column is a list-column with the values of the terminal nodes analogous to the flattened list returned by how = "flatten".
## Drop all logical NA's and return melted data.frame na_drop_oceania3 <- rrapply( renewable_oceania, f = identity, classes = "numeric", how = "melt" ) head(na_drop_oceania3) #> L1 L2 L3 value #> 1 Oceania Australia and New Zealand Australia 9.32 #> 2 Oceania Australia and New Zealand New Zealand 32.76 #> 3 Oceania Melanesia Fiji 24.36 #> 4 Oceania Melanesia New Caledonia 4.03 #> 5 Oceania Melanesia Papua New Guinea 50.34 #> 6 Oceania Melanesia Solomon Islands 65.73
If no names are present in a sublist of the input list, how = "melt" replaces the names in the melted data.frame by list element indices "..1", "..2", etc.:
## Remove all names at L2 ## (skip this for now, these arguments are explained in the following sections) oceania_unnamed <- rrapply( renewable_oceania, condition = function(x, .xpos) length(.xpos) < 2, f = unname, feverywhere = "break" ) ## Drop all logical NA's and return melted data.frame na_drop_oceania4 <- rrapply( oceania_unnamed, f = identity, classes = "numeric", how = "melt" ) head(na_drop_oceania4) #> L1 L2 L3 value #> 1 Oceania ..1 Australia 9.32 #> 2 Oceania ..1 New Zealand 32.76 #> 3 Oceania ..2 Fiji 24.36 #> 4 Oceania ..2 New Caledonia 4.03 #> 5 Oceania ..2 Papua New Guinea 50.34 #> 6 Oceania ..2 Solomon Islands 65.73
Base rapply allows to apply a function f to list elements of certain types or classes via the classes argument. rrapply generalizes this concept via the condition argument, which accepts any principal argument function to use as a condition or predicate to apply f to a subset of list elements. Conceptually, the f function is applied to all leaf elements for which the condition function exactly evaluates to TRUE simlar to isTRUE. If the condition argument is missing, f is applied to all leaf elements. In combination with how = "prune", the condition function provides additional flexibility in selecting and filtering elements from a nested list. Using the condition argument, we can update the above function call to better reflect our purpose:
## drop all NA elements using condition function na_drop_oceania3 <- rrapply( renewable_oceania, condition = Negate(is.na), f = identity, how = "prune" ) str(na_drop_oceania3, list.len = 3, give.attr = FALSE) #> List of 1 #> $ Oceania:List of 4 #> ..$ Australia and New Zealand:List of 2 #> .. ..$ Australia : num 9.32 #> .. ..$ New Zealand: num 32.8 #> ..$ Melanesia :List of 5 #> .. ..$ Fiji : num 24.4 #> .. ..$ New Caledonia : num 4.03 #> .. ..$ Papua New Guinea: num 50.3 #> .. .. [list output truncated] #> ..$ Micronesia :List of 7 #> .. ..$ Guam : num 3.03 #> .. ..$ Kiribati : num 45.4 #> .. ..$ Marshall Islands : num 11.8 #> .. .. [list output truncated] #> .. [list output truncated]
More interesting is to consider a condition that cannot also be defined using the classes argument. For instance, we can filter all countries with values that satisfy a certain numeric condition:
## filter all countries with values above 85% renewable_energy_above_85 <- rrapply( renewable_energy_by_country, condition = function(x) x > 85, how = "prune" ) str(renewable_energy_above_85, give.attr = FALSE) #> List of 1 #> $ World:List of 1 #> ..$ Africa:List of 1 #> .. ..$ Sub-Saharan Africa:List of 3 #> .. .. ..$ Eastern Africa:List of 7 #> .. .. .. ..$ Burundi : num 89.2 #> .. .. .. ..$ Ethiopia : num 91.9 #> .. .. .. ..$ Rwanda : num 86 #> .. .. .. ..$ Somalia : num 94.7 #> .. .. .. ..$ Uganda : num 88.6 #> .. .. .. ..$ United Republic of Tanzania: num 86.1 #> .. .. .. ..$ Zambia : num 88.5 #> .. .. ..$ Middle Africa :List of 2 #> .. .. .. ..$ Chad : num 85.3 #> .. .. .. ..$ Democratic Republic of the Congo: num 97 #> .. .. ..$ Western Africa:List of 1 #> .. .. .. ..$ Guinea-Bissau: num 86.5 ## or by passing arguments to condition via ... renewable_energy_equal_0 <- rrapply( renewable_energy_by_country, condition = "==", e2 = 0, how = "prune" ) str(renewable_energy_equal_0, give.attr = FALSE) #> List of 1 #> $ World:List of 4 #> ..$ Americas:List of 1 #> .. ..$ Latin America and the Caribbean:List of 1 #> .. .. ..$ Caribbean:List of 1 #> .. .. .. ..$ Antigua and Barbuda: num 0 #> ..$ Asia :List of 1 #> .. ..$ Western Asia:List of 4 #> .. .. ..$ Bahrain: num 0 #> .. .. ..$ Kuwait : num 0 #> .. .. ..$ Oman : num 0 #> .. .. ..$ Qatar : num 0 #> ..$ Europe :List of 2 #> .. ..$ Northern Europe:List of 1 #> .. .. ..$ Channel Islands:List of 1 #> .. .. .. ..$ Guernsey: num 0 #> .. ..$ Southern Europe:List of 1 #> .. .. ..$ Gibraltar: num 0 #> ..$ Oceania :List of 2 #> .. ..$ Micronesia:List of 1 #> .. .. ..$ Northern Mariana Islands: num 0 #> .. ..$ Polynesia :List of 1 #> .. .. ..$ Wallis and Futuna Islands: num 0
Remark: Note that the NA elements are not returned, as the condition function does not evaluate to TRUE for NA values. Also, the f argument is allowed to be missing, in which case no function is applied to the leaf elements.
As the condition function is a generalization of the classes argument to have more flexible control of the predicate, it is also possible to use the deflt argument together with how = "list" or how = "unlist" to set a default value to all leaf elements for which the condition does not evaluate to TRUE:
## replace all NA elements by zero na_zero_oceania_list <- rrapply( renewable_oceania, condition = Negate(is.na), deflt = 0, how = "list" ) str(na_zero_oceania_list, list.len = 3, give.attr = FALSE) #> List of 1 #> $ Oceania:List of 4 #> ..$ Australia and New Zealand:List of 6 #> .. ..$ Australia : num 9.32 #> .. ..$ Christmas Island : num 0 #> .. ..$ Cocos (Keeling) Islands : num 0 #> .. .. [list output truncated] #> ..$ Melanesia :List of 5 #> .. ..$ Fiji : num 24.4 #> .. ..$ New Caledonia : num 4.03 #> .. ..$ Papua New Guinea: num 50.3 #> .. .. [list output truncated] #> ..$ Micronesia :List of 8 #> .. ..$ Guam : num 3.03 #> .. ..$ Kiribati : num 45.4 #> .. ..$ Marshall Islands : num 11.8 #> .. .. [list output truncated] #> .. [list output truncated]
To be consistent with base rapply, the deflt argument can still only be used together with how = "list" or how = "unlist".
... argumentThe first argument to f always evaluates to the content of the leaf element to which f is applied. Any further arguments (besides the special arguments .xname and .xpos discussed below) that are independent of the node content can be supplied via the ... argument. Since rrapply accepts a function in two of its arguments f and condition, any further arguments defined via the ... also need to be defined as function arguments in both the f and condition function (if existing), even if they are not used in the function itself.
To clarify, consider the following example where we replace all NA elements by a value defined in a separate argument newvalue:
## this is not ok! tryCatch({ rrapply( renewable_oceania, condition = is.na, f = function(x, newvalue) newvalue, newvalue = 0, how = "replace" ) }, error = function(error) error$message) #> [1] "2 arguments passed to 'is.na' which requires 1" ## this is ok na_zero_oceania_replace3 <- rrapply( renewable_oceania, condition = function(x, newvalue) is.na(x), f = function(x, newvalue) newvalue, newvalue = 0, how = "replace" ) str(na_zero_oceania_replace3, list.len = 3, give.attr = FALSE) #> List of 1 #> $ Oceania:List of 4 #> ..$ Australia and New Zealand:List of 6 #> .. ..$ Australia : num 9.32 #> .. ..$ Christmas Island : num 0 #> .. ..$ Cocos (Keeling) Islands : num 0 #> .. .. [list output truncated] #> ..$ Melanesia :List of 5 #> .. ..$ Fiji : num 24.4 #> .. ..$ New Caledonia : num 4.03 #> .. ..$ Papua New Guinea: num 50.3 #> .. .. [list output truncated] #> ..$ Micronesia :List of 8 #> .. ..$ Guam : num 3.03 #> .. ..$ Kiribati : num 45.4 #> .. ..$ Marshall Islands : num 11.8 #> .. .. [list output truncated] #> .. [list output truncated]
.xname, .xpos and .xparents
The .xparents argument is currently only available in the development version of rrapply installed with devtools::install_github("JorisChau/rrapply").
In base rapply, the f function only has access to the content of the list element under evaluation through its principal argument, and there is no convenient way to access its name or location in the nested list from inside the f function. This makes rapply impractical if we want to apply a function f that relies on e.g. the name or position of a node as in the students example above. To overcome this limitation, rrapply allows the use of the special arguments .xname, .xpos and .xparents inside the f and condition functions (in addition to the principal function argument). .xname evaluates to the name of the list element, .xpos evaluates to the position of the element in the nested list structured as an integer vector, and .xparents evaluates to a vector of all parent node names in the path to the list element.
Using the .xname and .xpos arguments, we can transform or filter list elements based on their names or positions in the nested list:
## apply a function using the name of the node renewable_oceania_text <- rrapply( renewable_oceania, f = function(x, .xname) sprintf("Renewable energy in %s: %.2f%%", .xname, x), condition = Negate(is.na), how = "flatten" ) str(renewable_oceania_text, list.len = 10) #> List of 22 #> $ Australia : chr "Renewable energy in Australia: 9.32%" #> $ New Zealand : chr "Renewable energy in New Zealand: 32.76%" #> $ Fiji : chr "Renewable energy in Fiji: 24.36%" #> $ New Caledonia : chr "Renewable energy in New Caledonia: 4.03%" #> $ Papua New Guinea : chr "Renewable energy in Papua New Guinea: 50.34%" #> $ Solomon Islands : chr "Renewable energy in Solomon Islands: 65.73%" #> $ Vanuatu : chr "Renewable energy in Vanuatu: 33.67%" #> $ Guam : chr "Renewable energy in Guam: 3.03%" #> $ Kiribati : chr "Renewable energy in Kiribati: 45.43%" #> $ Marshall Islands : chr "Renewable energy in Marshall Islands: 11.75%" #> [list output truncated] ## extract values based on country names renewable_benelux <- rrapply( renewable_energy_by_country, condition = function(x, .xname) .xname %in% c("Belgium", "Netherlands", "Luxembourg"), how = "prune" ) str(renewable_benelux, give.attr = FALSE) #> List of 1 #> $ World:List of 1 #> ..$ Europe:List of 1 #> .. ..$ Western Europe:List of 3 #> .. .. ..$ Belgium : num 9.14 #> .. .. ..$ Luxembourg : num 13.5 #> .. .. ..$ Netherlands: num 5.78
Knowing that Europe is located under the node renewable_energy_by_country[[c(1, 5)]], we can filter all European countries with a renewable energy share above 50 percent by using the .xpos argument,
## filter European countries with value above 50% renewable_europe_above_50 <- rrapply( renewable_energy_by_country, condition = function(x, .xpos) identical(head(.xpos, 2), c(1L, 5L)) & x > 50, how = "prune" ) str(renewable_europe_above_50, give.attr = FALSE) #> List of 1 #> $ World:List of 1 #> ..$ Europe:List of 2 #> .. ..$ Northern Europe:List of 3 #> .. .. ..$ Iceland: num 78.1 #> .. .. ..$ Norway : num 59.5 #> .. .. ..$ Sweden : num 51.4 #> .. ..$ Western Europe :List of 1 #> .. .. ..$ Liechtenstein: num 62.9
This can be done more conveniently using the .xparents argument, as this does not require us to look up the location of Europe in the list beforehand,
## filter European countries with value above 50% renewable_europe_above_50 <- rrapply( renewable_energy_by_country, condition = function(x, .xparents) "Europe" %in% .xparents & x > 50, how = "prune" ) str(renewable_europe_above_50, give.attr = FALSE) #> List of 1 #> $ World:List of 1 #> ..$ Europe:List of 2 #> .. ..$ Northern Europe:List of 3 #> .. .. ..$ Iceland: num 78.1 #> .. .. ..$ Norway : num 59.5 #> .. .. ..$ Sweden : num 51.4 #> .. ..$ Western Europe :List of 1 #> .. .. ..$ Liechtenstein: num 62.9
Using the .xpos argument, we could look up the location of a particular country in the nested list,
## look up position of Sweden in list (xpos_sweden <- rrapply( renewable_energy_by_country, condition = function(x, .xname) identical(.xname, "Sweden"), f = function(x, .xpos) .xpos, how = "flatten" )) #> $Sweden #> [1] 1 5 2 14 renewable_energy_by_country[[xpos_sweden$Sweden]] #> [1] 51.35 #> attr(,"M49-code") #> [1] "752"
We could even use the .xpos argument to determine the maximum depth of the list or the length of the longest sublist,
## maximum list depth depth_all <- rrapply( renewable_energy_by_country, f = function(x, .xpos) length(.xpos), how = "unlist" ) max(depth_all) #> [1] 5 ## longest sublist length sublist_count <- rrapply( renewable_energy_by_country, f = function(x, .xpos) max(.xpos), how = "unlist" ) max(sublist_count) #> [1] 28
By default, both base rapply and rrapply recurse into any list-like element. To override this behavior, we can set feverywhere = "break" to apply f to any element of the nested list (e.g. a sublist) that satisfies the condition and classes arguments. If the condition of classes arguments are not satisfied for a list-like element, rrapply will recurse deeper into the sublist, apply the f function to the nodes that satisfy condition and classes, and so on.
This option is useful to calculate summary statistics across nodes or to look up the position of intermediate nodes in the nested list. To illustrate, we can return the mean and standard deviation of the renewable energy share in Europe as follows:
## compute mean value of Europe rrapply( renewable_energy_by_country, condition = function(x, .xname) .xname == "Europe", f = function(x) list( mean = mean(unlist(x), na.rm = TRUE), sd = sd(unlist(x), na.rm = TRUE) ), how = "flatten", feverywhere = "break" ) #> $Europe #> $Europe$mean #> [1] 22.36565 #> #> $Europe$sd #> [1] 17.12639
Remark: Note that the principal x argument in the f function can evaluate to an entire sublist for the node satisfying the condition. For this reason, we first unlist the sublist before passing it to mean and sd.
We can use the .xpos argument to apply the f function only at specific locations or depths in the nested list. For instance, we could return the mean renewable energy shares for each continent by making use of the fact that the .xpos vector of each continent has length (i.e. depth) 2:
## compute mean value of each continent renewable_continent_summary <- rrapply( renewable_energy_by_country, condition = function(x, .xpos) length(.xpos) == 2, f = function(x) mean(unlist(x), na.rm = TRUE), feverywhere = "break" ) ## Antarctica has a missing value str(renewable_continent_summary, give.attr = FALSE) #> List of 1 #> $ World:List of 6 #> ..$ Africa : num 54.3 #> ..$ Americas : num 18.2 #> ..$ Antarctica: num NaN #> ..$ Asia : num 17.9 #> ..$ Europe : num 22.4 #> ..$ Oceania : num 17.8
If feverywhere = "break", rrapply will not recurse further into list-like elements after application of the f function. This makes it for instance impossible to recursively update the name of each element in the nested list, as rrapply stops recursing after updating the first list layer. For this purpose, we can set feverywhere = "recurse" in which case rrapply recurses further into any updated list-like element after application of the f function. In this context, the condition function should be interpreted as a passing criterion (i.e. the opposite of a stopping criterion). As long as the condition and classes arguments are satisfied, rrapply will try to recurse further into any list-like elements.
Using feverywhere = "recurse", we can recursively replace all names in the nested list by their M49-codes:
## replace country names by M-49 codes renewable_M49 <- rrapply( list(renewable_energy_by_country), condition = is.list, f = function(x) { names(x) <- vapply(x, attr, character(1L), which = "M49-code") return(x) }, feverywhere = "recurse" ) str(renewable_M49[[1]], max.level = 3, list.len = 3, give.attr = FALSE) #> List of 1 #> $ 001:List of 6 #> ..$ 002:List of 2 #> .. ..$ 015:List of 7 #> .. ..$ 202:List of 4 #> ..$ 019:List of 2 #> .. ..$ 419:List of 3 #> .. ..$ 021:List of 5 #> ..$ 010: logi NA #> .. [list output truncated]
Remark: Here we passed list(renewable_energy_by_country) to the call of rrapply in order to start application of the f function at the level of the list renewable_energy_by_country itself, instead of starting at its list elements.
By default, rrapply and rapply recurse into all list-like objects. Since data.frames are list-like objects, the f function is applied to the individual columns instead of the data.frame object as a whole. To change this behavior, rrapply includes a convenience argument dfaslist. If dfaslist = TRUE, rrapply behaves the same as rapply by recursing into the individual columns of a data.frame. If dfaslist = FALSE, the f and condition functions are applied directly to the data.frame object itself and not its columns.
## create a list of data.frames oceania_df <- list( Oceania = lapply( renewable_oceania[["Oceania"]], FUN = function(x) data.frame( Name = names(x), value = unlist(x), stringsAsFactors = FALSE ) ) ) ## this does not work! tryCatch({ rrapply( oceania_df, f = function(x) subset(x, !is.na(value)), ## filter NA-rows of data.frame how = "replace", dfaslist = TRUE ) }, error = function(error) error$message) #> [1] "object 'value' not found" ## this does work rrapply( oceania_df, f = function(x) subset(x, !is.na(value)), how = "replace", dfaslist = FALSE ) #> $Oceania #> $Oceania$`Australia and New Zealand` #> Name value #> Australia Australia 9.32 #> New Zealand New Zealand 32.76 #> #> $Oceania$Melanesia #> Name value #> Fiji Fiji 24.36 #> New Caledonia New Caledonia 4.03 #> Papua New Guinea Papua New Guinea 50.34 #> Solomon Islands Solomon Islands 65.73 #> Vanuatu Vanuatu 33.67 #> #> $Oceania$Micronesia #> Name value #> Guam Guam 3.03 #> Kiribati Kiribati 45.43 #> Marshall Islands Marshall Islands 11.75 #> Micronesia (Federated States of) Micronesia (Federated States of) 1.64 #> Nauru Nauru 31.44 #> Northern Mariana Islands Northern Mariana Islands 0.00 #> Palau Palau 0.02 #> #> $Oceania$Polynesia #> Name value #> American Samoa American Samoa 1.00 #> Cook Islands Cook Islands 1.90 #> French Polynesia French Polynesia 11.06 #> Niue Niue 22.07 #> Samoa Samoa 27.30 #> Tonga Tonga 1.98 #> Tuvalu Tuvalu 11.76 #> Wallis and Futuna Islands Wallis and Futuna Islands 0.00
Remark: Note that the same result can also be obtained using feverywhere = "break" and checking that the list element under evaluation is a data.frame:
rrapply( oceania_df, condition = function(x) class(x) == "data.frame", f = function(x) subset(x, !is.na(value)), how = "replace", feverywhere = "break" )
Using the dfaslist argument better reflects our purpose and is also slightly more efficient as the class of the list element is checked directly analogous to the classes argument.
Base rapply may produce different results when using how = "replace" or how = "list" when working with list attributes. The former preserves intermediate list attributes whereas the latter does not. To avoid unexpected behavior, rrapply always preserves intermediate list attributes when using how = "replace", how = "list" or how = "prune". If how = "flatten", how = "melt" or how = "unlist" intermediate list attributes cannot be preserved as the result is no longer a nested list.
## how = "list" now preserves all list attributes na_drop_oceania_list_attr2 <- rrapply( renewable_oceania, f = function(x) replace(x, is.na(x), 0), how = "list" ) str(na_drop_oceania_list_attr2, max.level = 2) #> List of 1 #> $ Oceania:List of 4 #> ..$ Australia and New Zealand:List of 6 #> .. ..- attr(*, "M49-code")= chr "053" #> ..$ Melanesia :List of 5 #> .. ..- attr(*, "M49-code")= chr "054" #> ..$ Micronesia :List of 8 #> .. ..- attr(*, "M49-code")= chr "057" #> ..$ Polynesia :List of 10 #> .. ..- attr(*, "M49-code")= chr "061" #> ..- attr(*, "M49-code")= chr "009" ## how = "prune" also preserves list attributes na_drop_oceania_attr <- rrapply( renewable_oceania, condition = Negate(is.na), how = "prune" ) str(na_drop_oceania_attr, max.level = 2) #> List of 1 #> $ Oceania:List of 4 #> ..$ Australia and New Zealand:List of 2 #> .. ..- attr(*, "M49-code")= chr "053" #> ..$ Melanesia :List of 5 #> .. ..- attr(*, "M49-code")= chr "054" #> ..$ Micronesia :List of 7 #> .. ..- attr(*, "M49-code")= chr "057" #> ..$ Polynesia :List of 8 #> .. ..- attr(*, "M49-code")= chr "061" #> ..- attr(*, "M49-code")= chr "009"
rrapply on data.framesThe dfaslist argument can be used to avoid recursing into the individual columns of a data.frame object. However, it can also be useful to exploit exactly this property of rapply. A convenient way to apply a function to columns of a data.frame of a certain class is through the use of the classes argument in base rapply.
For instance, suppose we wish to standardize all numeric columns in the iris dataset by their sample mean and standard deviation:
iris_standard <- rapply(iris, f = scale, classes = "numeric", how = "replace") head(iris_standard) #> Sepal.Length Sepal.Width Petal.Length Petal.Width Species #> 1 -0.8976739 1.01560199 -1.335752 -1.311052 setosa #> 2 -1.1392005 -0.13153881 -1.335752 -1.311052 setosa #> 3 -1.3807271 0.32731751 -1.392399 -1.311052 setosa #> 4 -1.5014904 0.09788935 -1.279104 -1.311052 setosa #> 5 -1.0184372 1.24503015 -1.335752 -1.311052 setosa #> 6 -0.5353840 1.93331463 -1.165809 -1.048667 setosa
Using the condition argument in rrapply, we gain additional flexibility in selecting the columns to which f is applied. For instance, we can now easily apply the f function only to the Sepal columns using the .xname argument:
iris_standard_sepal <- rrapply( iris, condition = function(x, .xname) grepl("Sepal", .xname), f = scale ) head(iris_standard_sepal) #> Sepal.Length Sepal.Width Petal.Length Petal.Width Species #> 1 -0.8976739 1.01560199 1.4 0.2 setosa #> 2 -1.1392005 -0.13153881 1.4 0.2 setosa #> 3 -1.3807271 0.32731751 1.3 0.2 setosa #> 4 -1.5014904 0.09788935 1.5 0.2 setosa #> 5 -1.0184372 1.24503015 1.4 0.2 setosa #> 6 -0.5353840 1.93331463 1.7 0.4 setosa
Instead of mutating columns, we can also transmute columns (i.e. keeping only the columns to which f is applied) by setting how = "prune":
iris_standard_transmute <- rrapply( iris, f = scale, classes = "numeric", how = "prune" ) head(iris_standard_transmute) #> Sepal.Length Sepal.Width Petal.Length Petal.Width #> 1 -0.8976739 1.01560199 -1.335752 -1.311052 #> 2 -1.1392005 -0.13153881 -1.335752 -1.311052 #> 3 -1.3807271 0.32731751 -1.392399 -1.311052 #> 4 -1.5014904 0.09788935 -1.279104 -1.311052 #> 5 -1.0184372 1.24503015 -1.335752 -1.311052 #> 6 -0.5353840 1.93331463 -1.165809 -1.048667
In order to summarize a set of selected columns, use how = "flatten" instead of how = "prune", as the latter preserves list attributes –including data.frame dimensions– which should not be kept.
## summarize columns with how = "flatten" iris_standard_summarize <- rrapply( iris, f = summary, classes = "numeric", how = "flatten" ) iris_standard_summarize #> $Sepal.Length #> Min. 1st Qu. Median Mean 3rd Qu. Max. #> 4.300 5.100 5.800 5.843 6.400 7.900 #> #> $Sepal.Width #> Min. 1st Qu. Median Mean 3rd Qu. Max. #> 2.000 2.800 3.000 3.057 3.300 4.400 #> #> $Petal.Length #> Min. 1st Qu. Median Mean 3rd Qu. Max. #> 1.000 1.600 4.350 3.758 5.100 6.900 #> #> $Petal.Width #> Min. 1st Qu. Median Mean 3rd Qu. Max. #> 0.100 0.300 1.300 1.199 1.800 2.500
To conclude, let us return to the list recursion exercise in the first section. Using rrapply, we can solve the task in a more readable and less error-prone fashion. A possible approach is to split up the question in two steps as follows:
## look up the position of Euler (Leonhard) xpos_Euler <- rrapply( students, condition = function(x, .xname) .xname == "Euler" && attr(x, "given") == "Leonhard", f = function(x, .xpos) .xpos, feverywhere = "break", how = "flatten" )[[1]] xpos_Euler #> [1] 1 1 2 ## filter descendants of Euler (Leonhard) and replace missing values by zero students_Euler <- rrapply( students, condition = function(x, .xpos) identical(.xpos[seq_along(xpos_Euler)], xpos_Euler), f = function(x) replace(x, is.na(x), 0), how = "prune" ) str(students_Euler, give.attr = FALSE) #> List of 1 #> $ Bernoulli:List of 1 #> ..$ Bernoulli:List of 1 #> .. ..$ Euler:List of 2 #> .. .. ..$ Euler : num 0 #> .. .. ..$ Lagrange:List of 3 #> .. .. .. ..$ Fourier: num 68679 #> .. .. .. ..$ Plana : num 0 #> .. .. .. ..$ Poisson: num 113435
Knowing that Johann Euler is a descendant of Leonhard Euler, we can further simplify this into a single function call using the .xparents argument:
## filter descendants of Euler (Leonhard) and replace missing values by zero students_Euler <- rrapply( students, condition = function(x, .xparents) "Euler" %in% .xparents, f = function(x) replace(x, is.na(x), 0), how = "prune" ) str(students_Euler, give.attr = FALSE) #> List of 1 #> $ Bernoulli:List of 1 #> ..$ Bernoulli:List of 1 #> .. ..$ Euler:List of 2 #> .. .. ..$ Euler : num 0 #> .. .. ..$ Lagrange:List of 3 #> .. .. .. ..$ Fourier: num 68679 #> .. .. .. ..$ Plana : num 0 #> .. .. .. ..$ Poisson: num 113435
For additional information and details check out the package vignette (browseVignettes(package = "rapply")). The package vignette also includes additional examples that might serve as useful inspiration for list recursion tasks in practice.