which list element is being processed when using snowfall::sfLapply? - list

Question

Assume we have a list (mylist) that is use as input object for a lapply function. Is there a way to know which element in mylist is being evaluated? The method should work on lapply and snowfall::sfApply (and possible others apply family members) as well.
On chat, Gavin Simpson suggested the following method. This works great for lapply but not so much for sfApply. I would like to avoid extra packages or fiddling with the list. Any suggestions?
mylist <- list(a = 1:10, b = 1:10)
foo <- function(x) {
deparse(substitute(x))
}
bar <- lapply(mylist, FUN = foo)
> bar
$a
[1] "X[[1L]]"
$b
[1] "X[[2L]]"
This is the parallel version that isn't cutting it.
library(snowfall)
sfInit(parallel = TRUE, cpus = 2, type = "SOCK") # I use 2 cores
sfExport("foo", "mylist")
bar.para <- sfLapply(x = mylist, fun = foo)
> bar.para
$a
[1] "X[[1L]]"
$b
[1] "X[[1L]]"
sfStop()

Answer 1

I think you are going to have to use Shane's solution/suggestion in that chat session. Store your objects in a list such that each component of the top list contains a component with the name or ID or experiment contained in that list component, plus a component containing the object you want to process:
obj <- list(list(ID = 1, obj = 1:10), list(ID = 2, obj = 1:10),
list(ID = 3, obj = 1:10), list(ID = 4, obj = 1:10),
list(ID = 5, obj = 1:10))
So we have the following structure:
> str(obj)
List of 5
$ :List of 2
..$ ID : num 1
..$ obj: int [1:10] 1 2 3 4 5 6 7 8 9 10
$ :List of 2
..$ ID : num 2
..$ obj: int [1:10] 1 2 3 4 5 6 7 8 9 10
$ :List of 2
..$ ID : num 3
..$ obj: int [1:10] 1 2 3 4 5 6 7 8 9 10
$ :List of 2
..$ ID : num 4
..$ obj: int [1:10] 1 2 3 4 5 6 7 8 9 10
$ :List of 2
..$ ID : num 5
..$ obj: int [1:10] 1 2 3 4 5 6 7 8 9 10
The have something like the first line in the following function, followed by your
foo <- function(x) {
writeLines(paste("Processing Component:", x$ID))
sum(x$obj)
}
Which will do this:
> res <- lapply(obj, foo)
Processing Component: 1
Processing Component: 2
Processing Component: 3
Processing Component: 4
Processing Component: 5
Which might work on snowfall.

Answer 2

I could also alter the attributes like so.
mylist <- list(a = 1:10, b = 1:10)
attr(mylist[[1]], "seq") <- 1
attr(mylist[[2]], "seq") <- 2
foo <- function(x) {
writeLines(paste("Processing Component:", attributes(x)))
}
bar <- lapply(mylist, FUN = foo)
(and the parallel version)
mylist <- list(a = 1:10, b = 1:10)
attr(mylist[[1]], "seq") <- 1
attr(mylist[[2]], "seq") <- 2
foo <- function(x) {
x <- paste("Processing Component:", attributes(x))
}
sfExport("mylist", "foo")
bar <- sfLapply(mylist, fun = foo)