Parsing case statements in scala - regex

Parsing case statements in scala
CASE WHEN col1 <> 0 AND col2 <> 0 THEN 'COL1 & COL2 IS NOT ZERO' ELSE 'COL1 & COL2 IS ZERO'
challenge here is to give all the scenarios where case statement can come for e.g. it can come inside a function. Also case statements/functions etc. can come inside another case statements which has to be handled.

This problem can be solved with scala parser combinator
first define the classes needed to map experssions
sealed trait Exp {
def asStr: String
override def toString: String = asStr
}
case class OperationExp(a: Exp, op: String, b: Exp, c: Option[String]) extends Exp { override def asStr = s"$a $op $b ${c.getOrElse("")}" }
case class CaseConditions(conditionValue: List[(String, String)] , elseValue: String, asAlias: Option[Exp]) extends Exp {
override def asStr = "CASE " + conditionValue.map(c => s"WHEN ${c._1} THEN ${c._2}").mkString(" ") + s" ELSE ${elseValue} END ${asAlias.getOrElse("")}"
}
now the solution
case class OperationExp(a: Exp, op: String, b: Exp, c: Option[String]) extends Exp { override def asStr = s"$a $op $b ${c.getOrElse("")}" }
case class CaseConditions(conditionValue: List[(String, String)] , elseValue: String, asAlias: Option[Exp]) extends Exp {
override def asStr = "CASE " + conditionValue.map(c => s"WHEN ${c._1} THEN ${c._2}").mkString(" ") + s" ELSE ${elseValue} END ${asAlias.getOrElse("")}"
}
val identifiers: Parser[String] = "[a-zA-Z0-9_~\\|,'\\-\\+:.()]+".r
val operatorTokens: Parser[String] = "[<>=!]+".r | ("IS NOT" | "IN" | "IS")
val conditionJoiner: Parser[String] = ( "AND" | "OR" )
val excludeKeywords = List("CASE","WHEN", "THEN", "ELSE", "END")
val identifierWithoutCaseKw: Parser[Exp] = Parser(input =>
identifiers(input).filterWithError(
!excludeKeywords.contains(_),
reservedWord => s"$reservedWord encountered",
input
)
) ^^ StrExp
val anyStrExp: Parser[Exp] = "[^()]*".r ^^ StrExp
val funcIdentifier: Parser[Exp] = name ~ ("(" ~> (caseConditionExpresionParser | funcIdentifier | anyStrExp) <~ ")") ^^ {case func ~ param => FunCallExp(func, Seq(param))}
val identifierOrFunctions = funcIdentifier | identifierWithoutCaseKw
val conditionParser: Parser[String] =
identifierOrFunctions ~ operatorTokens ~ identifierOrFunctions ~ opt(conditionJoiner) ^^ {
case a ~ op ~ b ~ c => s"$a $op $b ${c.getOrElse("")}"
}
def caseConditionExpresionParser: Parser[CaseConditions] = "CASE" ~ rep1("WHEN" ~ rep(conditionParser) ~ "THEN" ~ rep(identifierWithoutCaseKw)) ~ "ELSE" ~ rep(identifierWithoutCaseKw) ~ "END" ~ opt("AS" ~> identifierWithoutCaseKw)^^ {
case "CASE" ~ conditionValuePair ~ "ELSE" ~ falseValue ~ "END" ~ asName =>
CaseConditions(
conditionValuePair.map(cv => (
cv._1._1._2.mkString(" "),
parsePipes(cv._2.mkString(" ")).isRight match {
case true => parsePipes(cv._2.mkString(" ")).right.get
case _ => cv._2.mkString(" ")
}
)),
parsePipes(falseValue.mkString("")).isRight match {
case true => parsePipes(falseValue.mkString(" ")).right.get
case _ => falseValue.mkString("")
}, asName)
}
//this parser can be used to get the results
val caseExpression = caseConditionExpresionParser | funcIdentifier
def parsePipes(input: String): Either[Seq[ParsingError], String] = {
parse(caseExpression, input) match {
case Success(parsed, _) => Right(parsed.asStr)
case Failure(msg, next) => Left(Seq(ParsingError(s"Failed to parse $pipedStr: $msg, next: ${next.source}.")))
case Error(msg, next) => Left(Seq(ParsingError(s"Error in $pipedStr parse: $msg, next: ${next.source}.")))
}
}

Related

Regex RDD using Apache Spark Scala

I have the following RDD:
x: Array[String] =
Array("Et: NT=grouptoClassify,hadoop-exec,sparkConnection,Ready
group: NT=app_1,hadoop-exec,sparkConnection,Ready
group: NT=app_exmpl_2,DB-exec,MDBConnection,NR
group: NT=apprexec,hadoop-exec,sparkConnection,Ready
group: NT=nt_prblm_app,hadoop-exec,sparkConnection,NR
I just want to get the first part of every part of this RDD as you can see in the next example:
Et: NT=grouptoClassify
group: NT=app_1
group: NT=app_exmpl_2
group: NT=apprexec
group: NT=nt_prblm_app
To do it I am trying it in this way.
//Here I get the RDD:
val x = spark.sparkContext.parallelize(List(value)).collect()
//Try to use regex on it, this regex is to get until the first comma
val regex1 = """(^(.+?),)"""
val rdd_1 = x.map(g => g.matches(regex1))
This is what I am trying but is not working for me because I just get an Array of Boolean. What am I doing wrong?
I am new with Apache Spark Scala. If you need something more just tell me it. Thanks in advance!
try this.
val x: Array[String] =
Array(
"Et: NT=grouptoClassify,hadoop-exec,sparkConnection,Ready",
"group: NT=app_1,hadoop-exec,sparkConnection,Ready",
"group: NT=app_exmpl_2,DB-exec,MDBConnection,NR",
"group: NT=apprexec,hadoop-exec,sparkConnection,Ready",
"group: NT=nt_prblm_app,hadoop-exec,sparkConnection,NR")
val rdd = sc.parallelize(x)
val result = rdd.map(lines => {
lines.split(",")(0)
})
result.collect().foreach(println)
output:
Et: NT=grouptoClassify
group: NT=app_1
group: NT=app_exmpl_2
group: NT=apprexec
group: NT=nt_prblm_app
Try with this regex :
^\s*([^,]+)(_\w+)?
Demo
To implement this regex in your example, you can try :
val arr = Seq("Et: NT=grouptoClassify,hadoop-exec,sparkConnection,Ready",
"group: NT=app_1,hadoop-exec,sparkConnection,Ready",
"group: NT=app_exmpl_2,DB-exec,MDBConnection,NR",
"group: NT=apprexec,hadoop-exec,sparkConnection,Ready",
"group: NT=nt_prblm_app,hadoop-exec,sparkConnection,NR")
val rd_var = spark.sparkContext.parallelize((arr).map((Row(_))))
val pattern = "^\s*([^,]+)(_\w+)?".r
rd_var.map {
case Row(str) => str match {
case pattern(gr1, _) => gr1
}
}.foreach(println(_))
With RDD:
val spark = SparkSession.builder().master("local[1]").getOrCreate()
val pattern = "([a-zA-Z0-9=:_ ]+),(.*)".r
val el = Seq("Et: NT=grouptoClassify,hadoop-exec,sparkConnection,Ready",
"group: NT=app_1,hadoop-exec,sparkConnection,Ready",
"group: NT=app_exmpl_2,DB-exec,MDBConnection,NR",
"group: NT=apprexec,hadoop-exec,sparkConnection,Ready",
"group: NT=nt_prblm_app,hadoop-exec,sparkConnection,NR")
def main(args: Array[String]): Unit = {
val rdd = spark.sparkContext.parallelize((el).map((Row(_))))
rdd.map {
case Row(str) => str match {
case pattern(gr1, _) => gr1
}
}.foreach(println(_))
}
It gives:
Et: NT=grouptoClassify
group: NT=app_1
group: NT=app_exmpl_2
group: NT=apprexec
group: NT=nt_prblm_app

Parsing error when trying to parse comma-delimited key-value pairs with values containing list separated by commas

I have this class:
import scala.util.parsing.combinator.JavaTokenParsers
class RequestMappingParser extends JavaTokenParsers {
def key: Parser[String] = "value" | "method" | "consumes" | "params"
def singleValue: Parser[String] = """[^),]*""".r
def multipleValues: Parser[String] = "{" ~ repsep(ident, ",") ~ "}" ^^ {
case "{" ~ lst ~ "}" => lst.mkString(", ")
}
def value: Parser[String] = multipleValues | singleValue
def keyValue: Parser[(String, String)] = (key ~ "=" ~ value).map {
case k ~ _ ~ v => k -> v
}
def commaDelimitedSeq: Parser[Map[String, String]] = repsep(keyValue, ",").map(_.toMap)
def requestMapping: Parser[MethodRequestMapping] = ("#RequestMapping(" ~ commaDelimitedSeq ~ ")").map {
case _ ~ map ~ _ =>
val consumes = if (map.contains("consumes")) Some(map("consumes")) else None
val value = if (map.contains("value")) Some(map("value")) else None
val method = if (map.contains("method")) Some(map("method")) else None
val params = if (map.contains("params")) Some(map("params")) else None
new MethodRequestMapping(value = value, method = method, consumes = consumes, params = params)
}
}
I have this test:
test("#RequestMapping – params") {
val parser = new RequestMappingParser()
val requestMappingStr = """#RequestMapping(
| value = "/ex/bars",
| params = { "id", "second" },
| method = GET)""".stripMargin
val parseResult = parser.parse(parser.requestMapping, requestMappingStr)
val result = parseResult.get
assert(result.value.get.equals("\"/ex/bars\""))
assert(result.method.get.equals("GET"))
assert(result.params.get.equals("{ \"id\", \"second\" }"))
}
But the test is failing with this parsing error:
[3.18] failure: ')' expected but ',' found
params = { "id", "second" },
^
I am using Scala 2.13. What do I have wrong here?
import scala.util.parsing.combinator.JavaTokenParsers
class RequestMappingParser extends JavaTokenParsers {
def key: Parser[String] = "value" | "method" | "consumes" | "params"
def singleValue: Parser[String] = """[^),]*""".r
def multipleValues: Parser[String] = "{" ~ repsep(stringLiteral, ",") ~ "}" ^^ {
case "{" ~ lst ~ "}" => "{ " + lst.mkString(", ") + " }"
}
def value: Parser[String] = multipleValues | singleValue
def keyValue: Parser[(String, String)] = (key ~ "=" ~ value).map {
case k ~ _ ~ v => k -> v
}
def commaDelimitedSeq: Parser[Map[String, String]] = repsep(keyValue, ",").map(_.toMap)
def requestMapping: Parser[MethodRequestMapping] = ("#RequestMapping(" ~ commaDelimitedSeq ~ ")").map {
case _ ~ map ~ _ =>
val consumes = if (map.contains("consumes")) Some(map("consumes")) else None
val value = if (map.contains("value")) Some(map("value")) else None
val method = if (map.contains("method")) Some(map("method")) else None
val params = if (map.contains("params")) Some(map("params")) else None
new MethodRequestMapping(value = value, method = method, consumes = consumes, params = params)
}
}

How to do pattern matching on regex in a foreach function in scala?

I don't understand why this doesn't work (i have two "no match" here) :
val a = "aaa".r
val b = "bbb".r
List("aaa", "bbb").foreach {
case a(t) => println(t)
case b(t) => println(t)
case _ => println("no match")
}
Variable in parentheses is supposed to be capturing group.
Change your regexes to val a = "(aaa)".r; val b = "(bbb)".r, that'll make it do what you want.
Alternatively, change the match patterns:
List("aaa", "bbb").foreach {
case a() => println("aaa")
case b() => println("bbb")
case _ => println("no match")
}
Your pattern contains no capture group, you need to put parenthesis around the pattern you want to capture in order for the the pattern matching to work:
val a = "(aaa)".r
// a: scala.util.matching.Regex = (aaa)
val b = "(bbb)".r
// b: scala.util.matching.Regex = (bbb)
List("aaa", "bbb").foreach {
case b(t) => println(t)
case a(t) => println(t)
case _ => println("no match")
}
//aaa
//bbb

Regex not matching

I expect regex "[a-zA-Z]\\d{6}" to match "z999999" but it is not matching as an empty List is mapped :
val lines = List("z999999"); //> lines : List[String] = List(z999999)
val regex = """[a-zA-Z]\d{6}""".r //> regex : scala.util.matching.Regex = [a-zA-Z]\d{6}
val fi = lines.map(line => line match { case regex(group) => group case _ => "" })
//> fi : List[String] = List("")
Is there a problem with my regex or how I'm using it with Scala ?
val l="z999999"
val regex = """[a-zA-Z]\d{6}""".r
regex.findAllIn(l).toList
res1: List[String] = List(z999999)
The regex seems valid.
lines.map( _ match { case regex(group) => group; case _ => "" })
res2: List[String] = List("")
How odd. Let's see what happens with a capturing group around the whole expression we defined in regex.
val regex2= """([a-zA-Z]\d{6})""".r
regex2: scala.util.matching.Regex = ([a-zA-Z]\d{6})
lines.map( _ match { case regex2(group) => group; case _ => "" })
res3: List[String] = List(z999999)
Huzzah.
The unapply method on a regex is for getting the results of capturing groups.
There are other methods on a regex object that just get matches (e.g. findAllIn, findFirstIn, etc)

Scala : how to expose regex variables bound within a case statement

Scala regex works great under either of two conditions:
unconditionally executed code:
e.g.
val lineregx = """([\w]+)\t([/\w.+-]+)\t([/\w+=\-!%# ]+)""".r
val anotherregx = """([\w]+)\t([/\w+=\-!%# ]+)""".r
val lineregx(category, aaUrl, title)
or if inside a case statement we consume the expressions (and don't need them again..)
val lineregx = """([\w]+)\t([/\w.+-]+)\t([/\w+=\-!%# ]+)""".r
line match {
case lineregx(category, aaUrl, title) => // do something with category, aaUrl and title in here!
case anotherRegx(category, aaUrl) => // do something different with category, aaUrl and title in here!
case _ => { println("did not match line %s".format(line)); return 0 }
}
But what about if i need to 'surface' the matches to variables outside of the case statement? Specifically the var's shown below,
val lineregx = """([\w]+)\t([/\w.+-]+)\t([/\w+=\-!%# ]+)""".r
var category = "" ; var aaUrl = "";var title = ""
line match {
case lineregx(category, aaUrl, title) => val lineregx(category, aaUrl, title) = line
case anotherRegx(category, aaUrl) => val lineregx(category, aaUrl) = line
case _ => { println("did not match line %s".format(line)); return 0 }
}
// Do something with category, aaUrl, title HERE after the case statement.
Problem is , the syntax for applying the lineregx/anotherregex makes those variables local to the case statement only.
Roughly,
val lineregx = """([\w]+)\t([/\w.+-]+)\t([/\w+=\-!%# ]+)""".r
val (category, aaUrl, title) = line match {
case lineregx(category, aaUrl, title) => (category, aaUrl, title)
case anotherRegx(category, aaUrl) => (category, aaUrl, ???)
case _ => { println("did not match line %s".format(line)); return 0 }
}
// Do something with category, aaUrl, title HERE after the case statement.
But that code is quite disorganized. For one thing, there's the question of the value of title for the second case. For another, there's the early return. Instead, the code would probably be best organized like this:
// method declaration
// ...
val lineregx = """([\w]+)\t([/\w.+-]+)\t([/\w+=\-!%# ]+)""".r
line match {
case lineregx(category, aaUrl, title) => f(category, aaUrl, title)
case anotherRegx(category, aaUrl) => f(category, aaUrl, ???)
case _ =>
println("did not match line %s".format(line))
0
}
} // end of method
def f(category: String, aaUrl: String, title: String): Int = {
// Do something with category, aaUrl, title HERE
}
You could use Option:
val lineregx = """([\w]+)\t([/\w.+-]+)\t([/\w+=\-!%# ]+)""".r
val (maybeCat, maybeUrl, maybeTitle) =
line match {
case lineregx(category, aaUrl, title) => (Some(category), Some(aaUrl), Some(title))
case anotherRegx(category, aaUrl) => (Some(category), Some(aaUrl), None)
case _ =>
println("did not match line %s".format(line))
(None, None, None)
}
var category = maybeCat getOrElse ""
var aaUrl = maybeURL getOrElse ""
var title = maybeTitle getOrElse ""
Slightly more verbose, but this way you can get the variables in the same scope.