Appending data frames that satisfy regular expressions in loop in R? - regex

Question

I asked a similar question earlier but asked it confusingly. So now I'm trying to do it in a more orderly fashion.
I'm running a loop that imports up to 6 dataframes based on 650 ID variables. I want to append these 6 dataframes for every one of the 650 cases. I import the data like this:
for(i in 1:650){
try(part1 <- read.csv(file = paste0("Twitter Scrapes/searchTwitter/09July/",MP.ID[i],".csv")))
try(part2 <- read.csv(file = paste0("Twitter Scrapes/userTimeline/08July/",MP.ID[i],".csv")))
try(part3 <- read.csv(file = paste0("Twitter Scrapes/userTimeline/16July/",MP.ID[i],".csv")))
try(part4 <- read.csv(file = paste0("Twitter Scrapes/searchTwitter/17July/",MP.ID[i],".csv")))
try(part5 <- read.csv(file = paste0("Twitter Scrapes/userTimeline/24July/",MP.ID[i],".csv")))
try(part6 <- read.csv(file = paste0("Twitter Scrapes/searchTwitter/24July/",MP.ID[i],".csv")))
This all works fine. If any part doesn't exist, the try-arguments makes sure that the loop continues to execute.
So, for some cases, not all 6 datasets exist. This means I can't simply have the next line read
combinedData <- rbind(part1, part2, part3, part4, part5, part6)
as one of these elements may not exist and therefore mean that the appended dataset can't be produced. This is why I thought it would be good to have the rbind command run for any dataframe that satisfied a regular expression requirement, i.e. partX. In this case, even if, say, part5 doesn't exist, it can simply append the existing other dataframes and then move on to the next ID in the loop.
However, I have no idea how to do this. It would be amazing if you could help me with this, and I'm really sorry for posting the confusing question earlier.

Answer 1

I might use the recursive argument in list.files instead and use lists:
(lf <- list.files('~/desktop/test', recursive = TRUE, full.names = TRUE))
# [1] "/Users/rawr/desktop/test/feb/three.csv"
# [2] "/Users/rawr/desktop/test/jan/one.csv"
# [3] "/Users/rawr/desktop/test/jan/three.csv"
# [4] "/Users/rawr/desktop/test/jan/two.csv"
# [5] "/Users/rawr/desktop/test/jul/one.csv"
# [6] "/Users/rawr/desktop/test/jul/two.csv"
you can group by ids by grep'ing
id <- c('one','two','three')
for (ii in id) {
print(lf[grepl(ii, lf)])
cat('\n')
}
# [1] "/Users/rawr/desktop/test/jan/one.csv" "/Users/rawr/desktop/test/jul/one.csv"
#
# [1] "/Users/rawr/desktop/test/jan/two.csv" "/Users/rawr/desktop/test/jul/two.csv"
#
# [1] "/Users/rawr/desktop/test/feb/three.csv" "/Users/rawr/desktop/test/jan/three.csv"
so using this idea, you can use lapply to read them in resulting in one object with all the data frames
ll <- lapply(id, function(ii) {
files <- lf[grepl(ii, lf)]
setNames(lapply(files, function(x)
read.csv(x, header = FALSE)), files)
})
setNames(ll, id)
# $one
# $one$`/Users/rawr/desktop/test/jan/one.csv`
# V1
# 1 one
#
# $one$`/Users/rawr/desktop/test/jul/one.csv`
# V1
# 1 one
# 2 one
# 3 one
#
#
# $two
# $two$`/Users/rawr/desktop/test/jan/two.csv`
# V1
# 1 two
#
# $two$`/Users/rawr/desktop/test/jul/two.csv`
# V1
# 1 two
#
#
# $three
# $three$`/Users/rawr/desktop/test/feb/three.csv`
# V1
# 1 three
#
# $three$`/Users/rawr/desktop/test/jan/three.csv`
# V1
# 1 three
And then rbind them:
lapply(ll, function(x) `rownames<-`(do.call('rbind', x), NULL))
# [[1]]
# V1
# 1 one
# 2 one
# 3 one
# 4 one
#
# [[2]]
# V1
# 1 two
# 2 two
#
# [[3]]
# V1
# 1 three
# 2 three
or you can rbind them in the step before