Replace a number combination with a colon in pyspark dataframe - regex

Question

I have a column in pyspark as:
column_a
force is 12 N and weight is 5N 4455 6700 and second force is 12N 6700 3460
weight is 14N and force is 5N 7000 10000
acceleration due to gravity is 10 and force is 6N 15000 4500
force is 12 4 N and weight is 7N 9000 17000 and second force is 12N
I want to replace the numbers which are in the range of (1000, 20000) and which occur one after another by a colon (;). For example in 4th row 12 and 4 are one after another but, they do not fall into the range so we will not replace them with a colon (;).
So my final output will be
column_a
force is 12 N and weight is 5N ; and second force is 12N ;
weight is 14N and force is 5N ;
acceleration due to gravity is 10 and force is 6N ;
force is 12 4 N and weight is 7N ; and second force is 12N
How do I achieve this in pyspark?

Answer 1

You can use regexp_replace to replace the specificed format with ;.
The hardest part is coming up with the regex, we can use Numeric Range Regex Generator to find the regex pattern to match the condition.
from pyspark.sql import functions as F
data = [("force is 12 N and weight is 5N 4455 6700 and second force is 12N 6700.010 3460",),
("weight is 14N and force is 5N 7000 10000",),
("acceleration due to gravity is 10 and force is 6N 15000 4500.1999999901",),
("force is 12 4 N and weight is 7N 9000 17000 and second force is 12N",),
("handle zero padded decimals 20000.000000 20000.00",),
("Wont be replaced as outside range 20001 17000 even for decimal 20000.01 2000",),]
df = spark.createDataFrame(data, ("column_a", ))
df = spark.createDataFrame(data, ("column_a", ))
# This pattern matches whole and decimal numbers between 1000 and 20000 inclusive
numeric_pattern ="(((100[0-9]|10[1-9][0-9]|1[1-9][0-9]{2}|[2-9][0-9]{3}|1[0-9]{4})(\\.\\d+)?)|(20000)(\\.0*)?)"
# This pattern matches 2 numeric patterns separated by a space
pattern = f".({numeric_pattern}\\s{numeric_pattern})\\b"
df.withColumn("column_a", F.regexp_replace(F.col("column_a"), pattern, " ;")).show(truncate=False)
"""
+----------------------------------------------------------------------------+
|column_a |
+----------------------------------------------------------------------------+
|force is 12 N and weight is 5N ; and second force is 12N ; |
|weight is 14N and force is 5N ; |
|acceleration due to gravity is 10 and force is 6N ; |
|force is 12 4 N and weight is 7N ; and second force is 12N |
|handle zero padded decimals ; |
|Wont be replaced as outside range 20001 17000 even for decimal 20000.01 2000|
+----------------------------------------------------------------------------+
"""