split strings on first and last commas - regex

I would like to split strings on the first and last comma. Each string has at least two
commas. Below is an example data set and the desired result.
A similar question here asked how to split on the first comma: Split on first comma in string
Here I asked how to split strings on the first two colons: Split string on first two colons
Thank you for any suggestions. I prefer a solution in base R. Sorry if this is a duplicate.
my.data <- read.table(text='
my.string some.data
123,34,56,78,90 10
87,65,43,21 20
a4,b6,c8888 30
11,bbbb,ccccc 40
uu,vv,ww,xx 50
j,k,l,m,n,o,p 60', header = TRUE, stringsAsFactors=FALSE)
desired.result <- read.table(text='
my.string1 my.string2 my.string3 some.data
123 34,56,78 90 10
87 65,43 21 20
a4 b6 c8888 30
11 bbbb ccccc 40
uu vv,ww xx 50
j k,l,m,n,o p 60', header = TRUE, stringsAsFactors=FALSE)

You can use the \K operator which keeps text already matched out of the result and a negative look ahead assertion to do this (well almost, there is an annoying comma at the start of the middle portion which I am yet to get rid of in the strsplit). But I enjoyed this as an exercise in constructing a regex...
x <- '123,34,56,78,90'
strsplit( x , "^[^,]+\\K|,(?=[^,]+$)" , perl = TRUE )
#[[1]]
#[1] "123" ",34,56,78" "90"
Explantion:
^[^,]+ : from the start of the string match one or more characters that are not a ,
\\K : but don't include those matched characters in the match
So the first match is the first comma...
| : or you can match...
,(?=[^,]+$) : a , so long as it is followed by [(?=...)] one or more characters that are not a , until the end of the string ($)...

Here is a relatively simple approach. In the first line we use sub to replace the first and last commas with semicolons producing s. Then we read s using sep=";" and finally cbind the rest of my.data to it:
s <- sub(",(.*),", ";\\1;", my.data[[1]])
DF <- read.table(text=s, sep =";", col.names=paste0("mystring",1:3), as.is=TRUE)
cbind(DF, my.data[-1])
giving:
mystring1 mystring2 mystring3 some.data
1 123 34,56,78 90 10
2 87 65,43 21 20
3 a4 b6 c8888 30
4 11 bbbb ccccc 40
5 uu vv,ww xx 50
6 j k,l,m,n,o p 60

Here is code to split on the first and last comma. This code draws heavily from an answer by #bdemarest here: Split string on first two colons The gsub pattern below, which is the meat of the answer, contains important differences. The code for creating the new data frame after strings are split is the same as that of #bdemarest
# Replace first and last commas with colons.
new.string <- gsub(pattern="(^[^,]+),(.+),([^,]+$)",
replacement="\\1:\\2:\\3", x=my.data$my.string)
new.string
# Split on colons
split.data <- strsplit(new.string, ":")
# Create data frame
new.data <- data.frame(do.call(rbind, split.data))
names(new.data) <- paste("my.string", seq(ncol(new.data)), sep="")
my.data$my.string <- NULL
my.data <- cbind(new.data, my.data)
my.data
# my.string1 my.string2 my.string3 some.data
# 1 123 34,56,78 90 10
# 2 87 65,43 21 20
# 3 a4 b6 c8888 30
# 4 11 bbbb ccccc 40
# 5 uu vv,ww xx 50
# 6 j k,l,m,n,o p 60
# Here is code for splitting strings on the first comma
my.data <- read.table(text='
my.string some.data
123,34,56,78,90 10
87,65,43,21 20
a4,b6,c8888 30
11,bbbb,ccccc 40
uu,vv,ww,xx 50
j,k,l,m,n,o,p 60', header = TRUE, stringsAsFactors=FALSE)
# Replace first comma with colon
new.string <- gsub(pattern="(^[^,]+),(.+$)",
replacement="\\1:\\2", x=my.data$my.string)
new.string
# Split on colon
split.data <- strsplit(new.string, ":")
# Create data frame
new.data <- data.frame(do.call(rbind, split.data))
names(new.data) <- paste("my.string", seq(ncol(new.data)), sep="")
my.data$my.string <- NULL
my.data <- cbind(new.data, my.data)
my.data
# my.string1 my.string2 some.data
# 1 123 34,56,78,90 10
# 2 87 65,43,21 20
# 3 a4 b6,c8888 30
# 4 11 bbbb,ccccc 40
# 5 uu vv,ww,xx 50
# 6 j k,l,m,n,o,p 60
# Here is code for splitting strings on the last comma
my.data <- read.table(text='
my.string some.data
123,34,56,78,90 10
87,65,43,21 20
a4,b6,c8888 30
11,bbbb,ccccc 40
uu,vv,ww,xx 50
j,k,l,m,n,o,p 60', header = TRUE, stringsAsFactors=FALSE)
# Replace last comma with colon
new.string <- gsub(pattern="^(.+),([^,]+$)",
replacement="\\1:\\2", x=my.data$my.string)
new.string
# Split on colon
split.data <- strsplit(new.string, ":")
# Create new data frame
new.data <- data.frame(do.call(rbind, split.data))
names(new.data) <- paste("my.string", seq(ncol(new.data)), sep="")
my.data$my.string <- NULL
my.data <- cbind(new.data, my.data)
my.data
# my.string1 my.string2 some.data
# 1 123,34,56,78 90 10
# 2 87,65,43 21 20
# 3 a4,b6 c8888 30
# 4 11,bbbb ccccc 40
# 5 uu,vv,ww xx 50
# 6 j,k,l,m,n,o p 60

You can do a simple strsplit here on that column
popshift<-sapply(strsplit(my.data$my.string,","), function(x)
c(x[1], paste(x[2:(length(x)-1)],collapse=","), x[length(x)]))
desired.result <- cbind(data.frame(my.string=t(popshift)), my.data[-1])
I just split up all the values and make a new vector with the first, last and middle strings. Then i cbind that with the rest of the data. The result is
my.string.1 my.string.2 my.string.3 some.data
1 123 34,56,78 90 10
2 87 65,43 21 20
3 a4 b6 c8888 30
4 11 bbbb ccccc 40
5 uu vv,ww xx 50
6 j k,l,m,n,o p 60

Using str_match() from package stringr, and a little help from one of your links,
> library(stringr)
> data.frame(str_match(my.data$my.string, "(.+?),(.*),(.+?)$")[,-1],
some.data = my.data$some.data)
# X1 X2 X3 some.data
# 1 123 34,56,78 90 10
# 2 87 65,43 21 20
# 3 a4 b6 c8888 30
# 4 11 bbbb ccccc 40
# 5 uu vv,ww xx 50
# 6 j k,l,m,n,o p 60

Related

regex to split on anything not a digit

I would like to split strings on anything not a digit. In this particular case the strings were dates and times read in from an external .csv file and are not currently in as.POSIXct format.
Ideally I would like to split the strings using regex, but if there is a simpler way to convert them to six columns of numbers using a date / time function that would be of interest as well.
I have already succeeded in creating a regex that splits the strings into six columns, but this regex is not general.
Here are the data:
my.data <- read.csv(text = '
Date_Time
18/05/2011 07:32:40
19/05/2011 13:26:02
19/05/2011 13:32:47
19/05/2011 13:45:24
19/05/2011 14:57:27
19/05/2011 15:03:18
', header=TRUE, stringsAsFactors = FALSE, na.strings = 'NA', strip.white = TRUE)
Here is a regex statement that splits the strings into six columns:
my.date.time <- data.frame(do.call(rbind, strsplit(my.data$Date_Time,"[/|:|[:space:]]+") ))
The above statement is not general. Here is an unsuccessful attempt at making the regex general by specifying a split on anything that is not a digit:
data.frame(do.call(rbind, strsplit(my.data$Date_Time,"[^\\d]+") ))
After I split the strings into six columns I still need what seems like an excessive number of statements to convert the columns into numeric format:
colnames(my.date.time) <- c('my.day', 'my.month', 'my.year', 'my.hour', 'my.minute', 'my.second')
revised.data <- data.frame(my.data, my.date.time, stringsAsFactors = FALSE)
revised.data$my.day <- as.numeric(as.character(revised.data$my.day))
revised.data$my.month <- as.numeric(as.character(revised.data$my.month))
revised.data$my.year <- as.numeric(as.character(revised.data$my.year))
revised.data$my.hour <- as.numeric(as.character(revised.data$my.hour))
revised.data$my.minute <- as.numeric(as.character(revised.data$my.minute))
revised.data$my.second <- as.numeric(as.character(revised.data$my.second))
revised.data
str(revised.data)
Thank you for any assistance in generalizing the above regex (or streamlining the procedure using date / time functions). The apply function probably can eliminate most of the as.numeric(as.character) statements, although that is a relatively minor issue.
Give a try to \\D+
> x <- "18/05/2011 07:32:40"
> strsplit(x, "\\D+")
[[1]]
[1] "18" "05" "2011" "07" "32" "40"
or
> strsplit(x, "[^0-9]+")
[[1]]
[1] "18" "05" "2011" "07" "32" "40"
Maybe I missed something but here is my solution:
lisda <- apply(my.data, 1, strsplit, "[^[:digit:]]")
my.data2 <- t(data.frame(lisda))
my.data2
[,1] [,2] [,3] [,4] [,5] [,6]
Date_Time "18" "05" "2011" "07" "32" "40"
Date_Time.1 "19" "05" "2011" "13" "26" "02"
Date_Time.2 "19" "05" "2011" "13" "32" "47"
Date_Time.3 "19" "05" "2011" "13" "45" "24"
Date_Time.4 "19" "05" "2011" "14" "57" "27"
Date_Time.5 "19" "05" "2011" "15" "03" "18"
Just in case you want to convert them all to numeric.
apply(my.data2, 2, function(x) as.numeric(as.character(x)))
Using cSplit
library(splitstackshape)
tmp = cSplit(my.data, "Date_Time", "/")
out = cSplit(tmp, "Date_Time_3", ":")
if you read your data like this
my.data <- read.csv(text = 'Date Time
18/05/2011 07:32:40
19/05/2011 13:26:02
19/05/2011 13:32:47
19/05/2011 13:45:24
19/05/2011 14:57:27
19/05/2011 15:03:18', header=TRUE, sep =' ' ,stringsAsFactors = FALSE, na.strings = 'NA', strip.white = TRUE)
you could do
library(splitstackshape)
out = cSplit(my.data, splitCols = c("Date", "Time"), sep = c("/", ":"))
#> out
# Date_1 Date_2 Date_3 Time_1 Time_2 Time_3
#1: 18 5 2011 7 32 40
#2: 19 5 2011 13 26 2
#3: 19 5 2011 13 32 47
#4: 19 5 2011 13 45 24
#5: 19 5 2011 14 57 27
#6: 19 5 2011 15 3 18
You might consider using read.pattern from the gsubfn package for this:
library(gsubfn)
read.pattern(text = my.data$Date_Time, pattern = "\\d+")
# V1 V2 V3 V4 V5 V6
# 1 18 5 2011 7 32 40
# 2 19 5 2011 13 26 2
# 3 19 5 2011 13 32 47
# 4 19 5 2011 13 45 24
# 5 19 5 2011 14 57 27
# 6 19 5 2011 15 3 18
Then you can simply assign the column names as you desire.

How to remove non-alphabetic characters and convert all letter to lowercase in R?

In the following string:
"I may opt for a yam for Amy, May, and Tommy."
How to remove non-alphabetic characters and convert all letter to lowercase and sort the letters within each word in R?
Meanwhile, I try to sort words in sentence and removes the duplicates.
You could use stringi
library(stringi)
unique(stri_sort(stri_trans_tolower(stri_extract_all_words(txt, simplify = TRUE))))
Which gives:
## [1] "a" "amy" "and" "for" "i" "may" "opt" "tommy" "yam"
Update
As per mentionned by #DavidArenburg, I overlooked the "sort the letters within words" part of your question. You didn't provide a desired output and no immediate application comes to mind but, assuming you want to identify which words have a matching counterpart (string distance of 0):
unique(stri_sort(stri_trans_tolower(stri_extract_all_words(txt, simplify = TRUE)))) %>%
stringdistmatrix(., ., useNames = "strings", method = "qgram") %>%
# a amy and for i may opt tommy yam
# a 0 2 2 4 2 2 4 6 2
# amy 2 0 4 6 4 0 6 4 0
# and 2 4 0 6 4 4 6 8 4
# for 4 6 6 0 4 6 4 6 6
# i 2 4 4 4 0 4 4 6 4
# may 2 0 4 6 4 0 6 4 0
# opt 4 6 6 4 4 6 0 4 6
# tommy 6 4 8 6 6 4 4 0 4
# yam 2 0 4 6 4 0 6 4 0
apply(., 1, function(x) sum(x == 0, na.rm=TRUE))
# a amy and for i may opt tommy yam
# 1 3 1 1 1 3 1 1 3
Words with more than one 0 per row ("amy", "may", "yam") have a scrambled counterpart.
str <- "I may opt for a yam for Amy, May, and Tommy."
## Clean the words (just keep letters and convert to lowercase)
words <- strsplit(tolower(gsub("[^A-Za-z ]", "", str)), " ")[[1]]
## split the words into characters and sort them
sortedWords <- sapply(words, function(word) sort(unlist(strsplit(word, ""))))
## Join the sorted letters back together
sapply(sortedWords, paste, collapse="")
# i may opt for a yam for amy may and
# "i" "amy" "opt" "for" "a" "amy" "for" "amy" "amy" "adn"
# tommy
# "mmoty"
## If you want to convert result back to string
do.call(paste, lapply(sortedWords, paste, collapse=""))
# [1] "i amy opt for a amy for amy amy adn mmoty"
stringr will let you work on all character sets in R and at C-speed, and magrittr will let you use a piping idiom that works well for your needs:
library(stringr)
library(magrittr)
txt <- "I may opt for a yam for Amy, May, and Tommy."
txt %>%
str_to_lower %>% # lowercase
str_replace_all("[[:punct:][:digit:][:cntrl:]]", "") %>% # only alpha
str_replace_all("[[:space:]]+", " ") %>% # single spaces
str_split(" ") %>% # tokenize
extract2(1) %>% # str_split returns a list
sort %>% # sort
unique # unique words
## [1] "a" "amy" "and" "for" "i" "may" "opt" "tommy" "yam"
The qdap package that I maintain has the bag_o_words function that works well for this:
txt <- "I may opt for a yam for Amy, May, and Tommy."
library(qdap)
unique(sort(bag_o_words(txt)))
## [1] "a" "amy" "and" "for" "i" "may" "opt" "tommy" "yam"

match a word an not the entire line in R

I have a text file formatted as such:
# title "Secondary Structure"
# xaxis label "Time (ns)"
# yaxis label "Number of Residues"
#TYPE xy
# subtitle "Structure = A-Helix + B-Sheet + B-Bridge + Turn"
# view 0.15, 0.15, 0.75, 0.85
# legend on
# legend box on
# legend loctype view
# legend 0.78, 0.8
# legend length 2
# s0 legend "Structure"
# s1 legend "Coil"
# s2 legend "B-Sheet"
# s3 legend "B-Bridge"
# s4 legend "Bend"
# s5 legend "Turn"
# s6 legend "A-Helix"
# s7 legend "5-Helix"
# s8 legend "3-Helix"
# s9 legend "Chain_Separator"
0 637 180 201 7 94 129 300 0 47 1
1 617 189 191 11 99 121 294 5 48 1
2 625 183 198 7 97 130 290 0 53 1
3 625 180 195 5 102 125 300 0 51 1
4 622 185 196 5 99 117 304 0 52 1
5 615 192 190 5 106 121 299 0 45 1
6 629 187 196 7 102 122 304 0 40 1
I'm trying to to match the lines starting with "s+number" (s0,s1,s2,...s9) and save the values between "" in a list so I can then use this list for naming the columns.
list <- c("Structure", "Coil","B-Sheet", ..., "Chain_Separato")
names(data) <- list
The problem is that I can't match the single words but only the entire lines.
grep('s\\d\\s[a-z]{6}\\s\"([A-z-9]+)\"',readLines("file.xvg"),perl=T,value=T)
[1] "# s0 legend \"Structure\"" "# s1 legend \"Coil\""
[3] "# s2 legend \"B-Sheet\"" "# s3 legend \"B-Bridge\""
[5] "# s4 legend \"Bend\"" "# s5 legend \"Turn\""
[7] "# s6 legend \"A-Helix\"" "# s9 legend \"Chain_Separator\""
I tried several regex, like '# s[0-9] [a-z]+ "([A-z-9]+)"', all working in perl but in R I'm always matching the entire line and not the word.
Isn't the () used to capture the value? What am I doing wrong?
You can do this:
conn = file(fileName,open="r")
lines=readLines(conn)
lst = Filter(function(u) grepl('^# s[0-9]+', u), lines)
result = gsub('.*\"(.*)\".*','\\1',lst)
close(conn)
#> result
#[1] "Structure" "Coil" "B-Sheet" "B-Bridge" "Bend" "Turn" "A-Helix" "5-Helix"
#[9] "3-Helix" "Chain_Separator"
You can use a system command in fread(). For example, on a file named "file.txt" you can do
library(data.table)
fread("grep '^# s[0-9]\\+' file.txt", header = FALSE, select = 4)[[1]]
# [1] "Structure" "Coil" "B-Sheet"
# [4] "B-Bridge" "Bend" "Turn"
# [7] "A-Helix" "5-Helix" "3-Helix"
# [10] "Chain_Separator"
Note: This uses data.table dev version 1.9.5
Basically the area you're looking for in the text has four columns. ^# s[0-9]\\+ looks for lines that begin with # and then a space, then s, then any number of digits. select = 4 takes the last column, and [[1]] drops it down from a single column data table into a character vector.
Thanks to #BrodieG for help with the regex.
If you use linux, awk commands can be combined with read.table using pipe
read.table(pipe("awk 'BEGIN {FS=\" \"}/# s[0-9]/ { print$4 }' fra.txt"),
stringsAsFactors=FALSE)$V1
# [1] "Structure" "Coil" "B-Sheet" "B-Bridge"
# [5] "Bend" "Turn" "A-Helix" "5-Helix"
# [9] "3-Helix" "Chain_Separator"
The above command also works with fread
fread("awk 'BEGIN {FS=\" \"}/# s[0-9]/ { print$4 }' fra.txt",
header=FALSE)$V1

Creating A Dataframe From A Text Dataset

I have a dataset that has hundreds of thousands of fields. The following is a simplified dataset
dataSet <- c("Plnt SLoc Material Description L.T MRP Stat Auto MatSG PC PN Freq Qty CFreq CQty Cur.RPt New.RPt CurRepl NewRepl Updt Cost ServStock Unit OpenMatResb DFStorLocLevel",
"0231 0002 GB.C152260-00001 ASSY PISTON & SEAL/O-RING 44 PD X A A A 18 136 30 29 50 43 24.88 51.000 EA",
"0231 0002 WH.112734 MOTOR REDUCER, THREE-PHAS 41 PD X B B A 16 17 3 3 5 4 483.87 1.000 EA X",
"0231 0002 WH.920569 SPINDLE MOTOR MINI O 22 PD X A A A 69 85 15 9 25 13 680.91 21.000 EA",
"0231 0002 GB.C150583-00001 VALVE-AIR MDI 64 PD X A A A 16 113 50 35 80 52 19.96 116.000 EA",
"0231 0002 FG.124-0140 BEARING 32 PD X A A A 36 205 35 32 50 48 21.16 55.000 EA",
"0231 0002 WP.254997 BEARING,BALL .9843 X 2.04 52 PD X A A A 18 155 50 39 100 58 2.69 181.000 EA"
)
I would like to create a dataframe out of this dataSet for further calculation. The approach I am following is as follows:
I split the dataSet by space and then recombine it.
dataSetSplit <- strsplit(dataSet, "\\s+")
The header (which is the first line) splits correctly and produces 25 characters. This can be seen by the str() function.
str(dataSetSplit)
I will then intend to combine all the rows together using the folloing script
combinedData <- data.frame(do.call(rbind, dataSetSplit))
Please note that the above script "combinedData " errors because the split did not produce equal number of fields.
For this approach to work all the fields must split correctly into 25 fields.
If you think this is a sound approach please let me know how to split the fileds into 25 fields.
It is worth mentioning that I do not like the approach of splitting the data set with the function strsplit(). It is an extremely time consuming step if used with a large data set. Can you please recommend an alternate approach to create a data frame out of the supplied data?
By the looks of it, you have a header row that is actually helpful. You can easily use gregexpr to calculate your "widths" to use with read.fwf.
Here's how:
## Use gregexpr to find the position of consecutive runs of spaces
## This will tell you the starting position of each column
Widths <- gregexpr("\\s+", dataSet[1])[[1]]
## `read.fwf` doesn't need the starting position, but the width of
## each column. We can use `diff` to calculate this.
Widths <- c(Widths[1], diff(Widths))
## Since there are no spaces after the last column, we need to calculate
## a reasonable width for that column too. We can do this with `nchar`
## to find the widest row in the data. From this, subtract the `sum`
## of all the previous values.
Widths <- c(Widths, max(nchar(dataSet)) - sum(Widths))
Let's also extract the column names. We could do this in read.fwf, but it would require us to substitute the spaces in the first line with a "sep" character.
Names <- scan(what = "", text = dataSet[1])
Now, read in everything except the first line. You would use the actual file instead of textConnection, I would suppose.
read.fwf(textConnection(dataSet), widths=Widths, strip.white = TRUE,
skip = 1, col.names = Names)
# Plnt SLoc Material Description L.T MRP Stat Auto MatSG PC PN Freq Qty
# 1 231 2 GB.C152260-00001 ASSY PISTON & SEAL/O-RING 44 PD NA X A A A 18 136
# 2 231 2 WH.112734 MOTOR REDUCER, THREE-PHAS 41 PD NA X B B A 16 17
# 3 231 2 WH.920569 SPINDLE MOTOR MINI O 22 PD NA X A A A 69 85
# 4 231 2 GB.C150583-00001 VALVE-AIR MDI 64 PD NA X A A A 16 113
# 5 231 2 FG.124-0140 BEARING 32 PD NA X A A A 36 205
# 6 231 2 WP.254997 BEARING,BALL .9843 X 2.04 52 PD NA X A A A 18 155
# CFreq CQty Cur.RPt New.RPt CurRepl NewRepl Updt Cost ServStock Unit OpenMatResb
# 1 NA NA 30 29 50 43 NA 24.88 51 EA <NA>
# 2 NA NA 3 3 5 4 NA 483.87 1 EA X
# 3 NA NA 15 9 25 13 NA 680.91 21 EA <NA>
# 4 NA NA 50 35 80 52 NA 19.96 116 EA <NA>
# 5 NA NA 35 32 50 48 NA 21.16 55 EA <NA>
# 6 NA NA 50 39 100 58 NA 2.69 181 EA <NA>
# DFStorLocLevel
# 1 NA
# 2 NA
# 3 NA
# 4 NA
# 5 NA
# 6 NA
Many thanks to Ananda Mahto, he provided many pieces to this answer.
widthMinusFirst <- diff(gregexpr('(\\s[A-Z])+', dataSet[1])[[1]])
widthFirst <- gregexpr('\\s+', dataSet[1])[[1]][1]
Width <- c(widthFirst, widthMinusFirst)
Widths <- c(Width, max(nchar(dataSet)) - sum(Width))
columnNames <- scan(what = "", text = dataSet[1])
read.fwf(textConnection(dataSet[-1]), widths = Widths, strip.white = FALSE,
skip = 0, col.names = columnNames)

Working with dataframes in a list: Rename variables

Define:
dats <- list( df1 = data.frame(A=sample(1:3), B = sample(11:13)),
df2 = data.frame(AA=sample(1:3), BB = sample(11:13)))
s.t.
> dats
$df1
A B
1 2 12
2 3 11
3 1 13
$df2
AA BB
1 1 13
2 2 12
3 3 11
I would like to change all variable names from all caps to lower. I can do this with a loop but somehow cannot get this lapply call to work:
dats <- lapply(dats, function(x)
names(x)<-tolower(names(x)))
which results in:
> dats
$df1
[1] "a" "b"
$df2
[1] "aa" "bb"
while the desired result is:
> dats
$df1
a b
1 2 12
2 3 11
3 1 13
$df2
aa bb
1 1 13
2 2 12
3 3 11
If you don't use return at the end of a function, the last evaluated expression returned. So you need to return x.
dats <- lapply(dats, function(x) {
names(x)<-tolower(names(x))
x})