I'm trying to use the stringi package to split on a delimiter (potentially the delimiter is repeated) yet keep the delimiter. This is similar to this question I asked moons ago: R split on delimiter (split) keep the delimiter (split) but the delimiter can be repeated. I don't think base strsplit can handle this type of regex. The stringi package can but I can't figure out how to format the regex to it splits on the delimiter if there are repeats and also not to leave an empty string at the end of the string.
Base R solutions, stringr, stringi etc. solutions all welcomed.
The later problem occurs because I use greedy * on the \\s but the space isn't garunteed so I could only think to leave it in:
MWE
text.var <- c("I want to split here.But also||Why?",
"See! Split at end but no empty.",
"a third string. It has two sentences"
)
library(stringi)
stri_split_regex(text.var, "(?<=([?.!|]{1,10}))\\s*")
# Outcome
## [[1]]
## [1] "I want to split here." "But also|" "|" "Why?"
## [5] ""
##
## [[2]]
## [1] "See!" "Split at end but no empty." ""
##
## [[3]]
## [1] "a third string." "It has two sentences"
# Desired Outcome
## [[1]]
## [1] "I want to split here." "But also||" "Why?"
##
## [[2]]
## [1] "See!" "Split at end but no empty."
##
## [[3]]
## [1] "a third string." "It has two sentences"
Using strsplit
strsplit(text.var, "(?<=[.!|])( +|\\b)", perl=TRUE)
#[[1]]
#[1] "I want to split here." "But also||" "Why?"
#[[2]]
#[1] "See!" "Split at end but no empty."
#[[3]]
#[1] "a third string." "It has two sentences"
Or
library(stringi)
stri_split_regex(text.var, "(?<=[.!|])( +|\\b)")
#[[1]]
#[1] "I want to split here." "But also||" "Why?"
#[[2]]
#[1] "See!" "Split at end but no empty."
#[[3]]
#[1] "a third string." "It has two sentences"
Just use a pattern that finds inter-character locations that: (1) are preceded by one of ?.!|; and (2) are not followed by one of ?.!|. Tack on \\s* to match and eat up any number of consecutive space characters, and you're good to go.
## (look-behind)(look-ahead)(spaces)
strsplit(text.var, "(?<=([?.!|]))(?!([?.!|]))\\s*", perl=TRUE)
# [[1]]
# [1] "I want to split here." "But also||" "Why?"
#
# [[2]]
# [1] "See!" "Split at end but no empty."
#
# [[3]]
# [1] "a third string." "It has two sentences"
Related
I am trying something I thought would be easy. I'm looking for a single regex solution (though others are welcomed for completeness). I want to split on n occurrences of a delimiter.
Here is some data:
x <- "I like_to see_how_too"
pat <- "_"
Desired outcome
Say I want to split on first occurrence of _:
[1] "I like" "to see_how_too"
Say I want to split on second occurrence of _:
[1] "I like_to see" "how_too"
Ideally, if the solution is a regex one liner generalizable to nth occurrence; the solution will use strsplit with a single regex.
Here's a solution that doesn't fit my parameters of single regex that works with strsplit
x <- "I like_to see_how_too"
y <- "_"
n <- 1
loc <- gregexpr("_", x)[[1]][n]
c(substr(x, 1, loc-1), substr(x, loc + 1, nchar(x)))
Here is another solution using the gsubfn package and some regex-fu. To change the nth occurrence of the delimiter, you can simply swap the number that is placed inside of the range quantifier — {n}.
library(gsubfn)
x <- 'I like_to see_how_too'
strapply(x, '((?:[^_]*_){1})(.*)', c, simplify =~ sub('_$', '', x))
# [1] "I like" "to see_how_too"
If you would like the nth occurrence to be user defined, you could use the following:
n <- 2
re <- paste0('((?:[^_]*_){',n,'})(.*)')
strapply(x, re, c, simplify =~ sub('_$', '', x))
# [1] "I like_to see" "how_too"
Non-Solution
Since R is using PCRE, you can use \K to remove everything that matches the pattern before \K from the main match result.
Below is the regex to split the string at the 3rd _
^[^_]*(?:_[^_]*){2}\K_
If you want to split at the nth occurrence of _, just change 2 to (n - 1).
Demo on regex101
That was the plan. However, strsplit seems to think differently.
Actual execution
Demo on ideone.com
x <- "I like_to see_how_too but_it_seems to_be_impossible"
strsplit(x, "^[^_]*(?:_[^_]*)\\K_", perl=TRUE)
strsplit(x, "^[^_]*(?:_[^_]*){1}\\K_", perl=TRUE)
strsplit(x, "^[^_]*(?:_[^_]*){0}\\K_", perl=TRUE)
# strsplit(x, "^[^_]*(?:_[^_]*)\\K_", perl=TRUE)
# [[1]]
# [1] "I like_to see" "how_too but" "it_seems to" "be_impossible"
# strsplit(x, "^[^_]*(?:_[^_]*){1}\\K_", perl=TRUE)
# [[1]]
# [1] "I like_to see" "how_too but" "it_seems to" "be_impossible"
# strsplit(x, "^[^_]*(?:_[^_]*){0}\\K_", perl=TRUE)
# [[1]]
# [1] "I like" "to see" "how" "too but" "it"
# [6] "seems to" "be" "impossible"
It still fails to work on a stronger assertion \A
strsplit(x, "\\A[^_]*(?:_[^_]*){0}\\K_", perl=TRUE)
# [[1]]
# [1] "I like" "to see" "how" "too but" "it"
# [6] "seems to" "be" "impossible"
Explanation?
This behavior hints at the fact that strsplit find the first match, do a substring to extract the first token and the remainder part, and find the next match in the remainder part.
This removes all the states from the previous matches, and leaves us with a clean state when it tries to match the regex on the remainder. This makes the task of stopping the strsplit function at first match and achieving the task at the same time impossible. There is not even a parameter in strsplit to limit the number of splits.
Rather than split you do match to get your split strings.
Try this regex:
^((?:[^_]*_){1}[^_]*)_(.*)$
Replace 1 by n-1 where you're trying to get split on nth occurrence of underscore.
RegEx Demo
Update: It seems R also supports PCRE and in that case you can do split as well using this PCRE regex:
^((?:[^_]*_){1}[^_]*)(*SKIP)(*F)|_
Replace 1 by n-1 where you're trying to get split on nth occurrence of underscore.
(*FAIL) behaves like a failing negative assertion and is a synonym for (?!)
(*SKIP) defines a point beyond which the regex engine is not allowed to backtrack when the subpattern fails later
(*SKIP)(*FAIL) together provide a nice alternative of restriction that you cannot have a variable length lookbehind in above regex.
RegEx Demo2
x <- "I like_to see_how_too"
strsplit(x, "^((?:[^_]*_){0}[^_]*)(*SKIP)(*F)|_", perl=TRUE)
strsplit(x, "^((?:[^_]*_){1}[^_]*)(*SKIP)(*F)|_", perl=TRUE)
## > strsplit(x, "^((?:[^_]*_){0}[^_]*)(*SKIP)(*F)|_", perl=TRUE)
## [[1]]
## [1] "I like" "to see" "how" "too"
## > strsplit(x, "^((?:[^_]*_){1}[^_]*)(*SKIP)(*F)|_", perl=TRUE)
## [[1]]
## [1] "I like_to see" "how_too"
This uses gsubfn to to preprocess the input string so that strsplit can handle it. The main advantage is that one can specify a vector of numbers, k, indicating which underscores to split on.
It replaces the occurrences of underscore defined by k by a double underscore and then splits on double underscore. In this example we split at the 2nd and 4th underscore:
library(gsubfn)
k <- c(2, 4) # split at 2nd and 4th _
p <- proto(fun = function(., x) if (count %in% k) "__" else "_")
strsplit(gsubfn("_", p, "aa_bb_cc_dd_ee_ff"), "__")
giving:
[[1]]
[1] "aa_bb" "cc_dd" "ee_ff"
If empty fields are allowed then use any other character sequence not in the string, e.g. "\01" in place of the double underscore.
See section 4 of the gusbfn vignette for more info on using gusbfn with proto objects to retain state between matches.
Could anyone maybe know how to extract x and y from this character: "x and y" using grep function (not using stringi package) if x and y are random characters?
I am so not skilled in regular expressions.
Thanks for any response.
The regex here matches any chars "and" chars and then extracts them with regmatches:
txt <- c("x and y", "a and b", " C and d", "qq and rr")
matches <- regexec("([[:alpha:]]+)[[:blank:]]+and[[:blank:]]+([[:alpha:]]+)", txt)
regmatches(txt, matches)[[1]][2:3]
## [1] "x" "y"
regmatches(txt, matches)[[2]][2:3]
## [1] "a" "b"
regmatches(txt, matches)[[3]][2:3]
## [1] "C" "d"
regmatches(txt, matches)[[4]][2:3]
## [1] "qq" "rr"
([[:alpha:]]+) matches one or more alpha characters and places it in a match group. [[:blank:]]+ matches one or more "whitespace" characters. There are less verbose ways to write these regexes but the expanded ones (to me) help make it easier to grok if there will be folks reading the code that aren't familiar with regexes.
I also didn't need to call regmatches 4x, but it was faster to cut/paste for a toy example.
As #MrFlick commented, grep is not the right function to extract these substrings.
You can use regmatches and do something like this:
> x <- c('x and y', 'abc and def', 'foo and bar')
> regmatches(x, gregexpr('and(*SKIP)(*F)|\\w+', x, perl=T))
# [[1]]
# [1] "x" "y"
# [[2]]
# [1] "abc" "def"
# [[3]]
# [1] "foo" "bar"
Or if " and " is always constant, then use strsplit as suggested in the comments.
> x <- c('x and y', 'abc and def', 'foo and bar')
> strsplit(x, ' and ', fixed=T)
# [[1]]
# [1] "x" "y"
# [[2]]
# [1] "abc" "def"
# [[3]]
# [1] "foo" "bar"
I have a list of city, state data in a data frame. I need to extract only the state abbreviation and store into a new variable column called state. From visual inspection it looks like the state is always the last 2 characters in the string and they are both capitalized. The city, state data looks like the following:
test <- c("Anchorage, AK", "New York City, NY", "Some Place, Another Place, LA")
I tried the following
pattern <- "[, (A-Z){2}]"
strsplit(test, pattern)
The output was:
[[1]]
[1] "Anchorage, "
[[2]]
[1] "New York City, "
[[3]]
[1] "Some Place, Another Place, "
EDI:
I used another regular expresson:
pattern2 <- "([a-z, ])"
sp <- strsplit(test, pattern2)
I get these results:
[[1]]
[1] "A" "" "" "" "" "" "" "" "" "" "AK"
[[2]]
[1] "N" "" "" "Y" "" "" "" "C" "" "" "" "" "NY"
[[3]]
[1] "S" "" "" "" "P" "" "" "" "" "" "A" "" "" "" "" "" ""
[18] "P" "" "" "" "" "" "LA"
So, the abbreviation is there, but when I try to extract using sapply(), I am not sure how to get the last element of a list. I know how to get the first:
sapply(sp, "[[", 1)
I'm not sure you really need a regular expression here. If you always just want the last two characters of the string, just use
substring(test, nchar(test)-1, nchar(test))
[1] "AK" "NY" "LA"
If you really insist on a regular expression, at least consider using regexec rather than strsplit since you're not really interested in splitting, you only want to extract the state.
m <- regexec("[A-Z]+$", test)
unlist(regmatches(test,m))
# [1] "AK" "NY" "LA"
Try:
tt = strsplit(test, ', ')
tt
[[1]]
[1] "Anchorage" "AK"
[[2]]
[1] "New York City" "NY"
[[3]]
[1] "Some Place" "Another Place" "LA"
z = list()
for(i in tt) z[length(z)+1] = i[length(i)]
z
[[1]]
[1] "AK"
[[2]]
[1] "NY"
[[3]]
[1] "LA"
This can work:
regmatches(test, gregexpr("(?<=[,][\\s+])([A-Z]{2})", test, perl = TRUE))
## [[1]]
## [1] "AK"
##
## [[2]]
## [1] "NY"
##
## [[3]]
## [1] "LA"
Explanation compliments of: http://liveforfaith.com/re/explain.pl
(?<= look behind to see if there is:
[,] any character of: ','
[\\s+] any character of: whitespace (\n, \r,
\t, \f, and " "), '+'
) end of look-behind
( group and capture to \1:
[A-Z]{2} any character of: 'A' to 'Z' (2 times)
) end of \1
I think you understood reversely the meaning of '[]' and '()'. '()' means to match a group of characters; '[]' means to match any one character from a class. What you need is
"(, [A-Z]{2})".
library(stringr)
str_extract(test, perl('[A-Z]+(?=\\b$)'))
#[1] "AK" "NY" "LA"
here is a regex for the same
Regex
(?'state'\w{2})(?=")
Test String
"Anchorage, AK", "New York City, NY", "Some Place, Another Place, LA"
Result
MATCH 1
state [12-14] AK
MATCH 2
state [33-35] NY
MATCH 3
state [66-68] LA
live demo here
you may remove the named capture to make it smaller if required
eg
(\w{2})(?=")
Data <- c("My name is Ernst.","I love chicken","Hello, my name is Stan!","Who?","I Love you!","Winner")
The Function should add a "." if at the end of the Sentence is none of those signs [.?!] to end the sentence.
I was trying do build a function in R with help of Regex but i had some issues to only look at the End of the String.
The below gsub function would add a dot at the end of the sentence only if the sentence is not ended with a . or ? or ! symbols.
> Data <- c("My name is Ernst.","I love chicken","Hello, my name is Stan!","Who?","I Love you!","Winner")
> gsub("^(?!.*[.?!]$)(.*)$", "\\1.", Data, perl=TRUE)
[1] "My name is Ernst." "I love chicken."
[3] "Hello, my name is Stan!" "Who?"
[5] "I Love you!" "Winner."
In regex, lookaheads are used for condition checking purposes. The negative lookahead (?!.*[.?!]$) would checks for the presence of . or ? or ! at the line end. If it's present at the last, then it skips the sentence and the replacement would never happen on that corresponding line. The replacement would occur only if there is no . or ? or ! symbols at the last.
OR
Through negative lookbehind and positive lookahead,
> Data <- c("My name is Ernst.","I love chicken","Hello, my name is Stan!","Who?","I Love you!","Winner")
> sub("(?<![!?.])(?=$)", ".", Data, perl=TRUE)
[1] "My name is Ernst." "I love chicken."
[3] "Hello, my name is Stan!" "Who?"
[5] "I Love you!" "Winner."
using stringi
library(stringi)
stri_replace_all_regex(Data, "(?<![^!?.])\\b$", ".")
#[1] "My name is Ernst." "I love chicken."
#[3] "Hello, my name is Stan!" "Who?"
#[5] "I Love you!" "Winner."
Here is another solution.
x <- c('My name is Ernst.', 'I love chicken',
'Hello, my name is Stan!', 'Who?', 'I Love you!', 'Winner')
r <- sub('[^?!.]\\K$', '.', x, perl=T)
## [1] "My name is Ernst." "I love chicken."
## [3] "Hello, my name is Stan!" "Who?"
## [5] "I Love you!" "Winner."
Here are some possible approaches:
1) If the last character is not dot, ? or ! then replace it with that character followed by dot:
sub("([^.!?])$", "\\1.", Data)
For the data in the question we get:
[1] "My name is Ernst." "I love chicken."
[3] "Hello, my name is Stan!" "Who?"
[5] "I Love you!" "Winner."
2) A gsubfn solution is even simpler. It replaces the empty () with a dot if the last character is not a dot, ! or ? .
library(gsubfn)
gsubfn("[^.!?]()$", ".", Data)
3) This one uses grepl. If dot, ! or ? is the last character then append the empty string and otherwise append dot.
paste0(Data, ifelse(grepl("[.!?]$", Data), "", "."))
4) This one does not use regular expressions at all. It picks off the last character and if its one of dot, ! or ? it appends the empty string and otherwise appends dot:
paste0(Data, ifelse(substring(Data, nchar(Data)) %in% c(".", "!", "?"), "", "."))
Common sense and a sanity-check using gregexpr() indicate that the look-behind and look-ahead assertions below should each match at exactly one location in testString:
testString <- "text XX text"
BB <- "(?<= XX )"
FF <- "(?= XX )"
as.vector(gregexpr(BB, testString, perl=TRUE)[[1]])
# [1] 9
as.vector(gregexpr(FF, testString, perl=TRUE)[[1]][1])
# [1] 5
strsplit(), however, uses those match locations differently, splitting testString at one location when using the lookbehind assertion, but at two locations -- the second of which seems incorrect -- when using the lookahead assertion.
strsplit(testString, BB, perl=TRUE)
# [[1]]
# [1] "text XX " "text"
strsplit(testString, FF, perl=TRUE)
# [[1]]
# [1] "text" " " "XX text"
I have two questions: (Q1) What's going on here? And (Q2) how can one get strsplit() to be better behaved?
Update: Theodore Lytras' excellent answer explains what's going on, and so addresses (Q1). My answer builds on his to identify a remedy, addressing (Q2).
I am not sure whether this qualifies as a bug, because I believe this is expected behaviour based on the R documentation. From ?strsplit:
The algorithm applied to each input string is
repeat {
if the string is empty
break.
if there is a match
add the string to the left of the match to the output.
remove the match and all to the left of it.
else
add the string to the output.
break.
}
Note that this means that if there is a match at the beginning of
a (non-empty) string, the first element of the output is ‘""’, but
if there is a match at the end of the string, the output is the
same as with the match removed.
The problem is that lookahead (and lookbehind) assertions are zero-length. So for example in this case:
FF <- "(?=funky)"
testString <- "take me to funky town"
gregexpr(FF,testString,perl=TRUE)
# [[1]]
# [1] 12
# attr(,"match.length")
# [1] 0
# attr(,"useBytes")
# [1] TRUE
strsplit(testString,FF,perl=TRUE)
# [[1]]
# [1] "take me to " "f" "unky town"
What happens is that the lonely lookahead (?=funky) matches at position 12. So the first split includes the string up to position 11 (left of the match), and it is removed from the string, together with the match, which -however- has zero length.
Now the remaining string is funky town, and the lookahead matches at position 1. However there's nothing to remove, because there's nothing at the left of the match, and the match itself has zero length. So the algorithm is stuck in an infinite loop. Apparently R resolves this by splitting a single character, which incidentally is the documented behaviour when strspliting with an empty regex (when argument split=""). After this the remaining string is unky town, which is returned as the last split since there's no match.
Lookbehinds are no problem, because each match is split and removed from the remaining string, so the algorithm is never stuck.
Admittedly this behaviour looks weird at first glance. Behaving otherwise however would violate the assumption of zero length for lookaheads. Given that the strsplit algorithm is documented, I belive this does not meet the definition of a bug.
Based on Theodore Lytras' careful explication of substr()'s behavior, a reasonably clean workaround is to prefix the to-be-matched lookahead assertion with a positive lookbehind assertion that matches any single character:
testString <- "take me to funky town"
FF2 <- "(?<=.)(?=funky)"
strsplit(testString, FF2, perl=TRUE)
# [[1]]
# [1] "take me to " "funky town"
Looks like a bug to me. This doesn't appear to just be related to spaces, specifically, but rather any lonely lookahead (positive or negative):
FF <- "(?=funky)"
testString <- "take me to funky town"
strsplit(testString,FF,perl=TRUE)
# [[1]]
# [1] "take me to " "f" "unky town"
FF <- "(?=funky)"
testString <- "funky take me to funky funky town"
strsplit(testString,FF,perl=TRUE)
# [[1]]
# [1] "f" "unky take me to " "f" "unky "
# [5] "f" "unky town"
FF <- "(?!y)"
testString <- "xxxyxxxxxxx"
strsplit(testString,FF,perl=TRUE)
# [[1]]
# [1] "xxx" "y" "xxxxxxx"
Seems to work fine if given something to capture along with the zero-width assertion, such as:
FF <- " (?=XX )"
testString <- "text XX text"
strsplit(testString,FF,perl=TRUE)
# [[1]]
# [1] "text" "XX text"
FF <- "(?= XX ) "
testString <- "text XX text"
strsplit(testString,FF,perl=TRUE)
# [[1]]
# [1] "text" "XX text"
Perhaps something like that might function as a workaround.