I have a long list of objects around 300, with each object in the list having this data structure:
case class MyObject(id: String,
name: String,
colour: String,
price: Int
height: Int
width: Int,
desc: String)
I can’t work out what is the best way to go through the list and for each object remove any other object that has the same name, colour, price, height and width. Note that this isn’t a simple dedupe as the ids and desc can be different. The input and output need to remain List[MyObject] and I do not know beforehand which objects are the duplicated ones.
This is my initial solution which works, but not sure its the most efficient way of doing it when it comes to dealing with large list.
def removeDuplicates(originalList: List[MyObject]): List[MyObject] = {
def loop(remaining: List[MyObject], acc: List[MyObject]): List[MyObject] = {
remaining match {
case head :: tail =>
val listOfDuplicates = tail.filter{ x =>
x.name == head.name &&
x.colour == head.colour &&
x.price == head.price &&
x.height == head.height &&
x.width == head.width
}
val deDupedTail = tail.filter(!listOfDuplicates.contains(_))
loop(deDupedTail, acc ::: listOfDuplicates)
case Nil => acc
}
}
val listOfDuplicateObjects = loop(originalList, List())
originalList.filter(!listOfDuplicateObjects.contains(_))
}
Not sure if it's most efficent, but IMHO it's elegant:
originalList.distinctBy(o => (o.name, o.colour, o.price, o.height, o.width))
I need to modify the data to give input to CEP system, my current data looks like below
val rdd = {"var":"system-ready","value":0.0,"objectID":"2018","partnumber":2,"t":"2017-08-25 11:27:39.000"}
I need output like
t = "2017-08-25 11:27:39.000
Check = { var = "system-ready",value = 0.0, objectID = "2018", partnumber = 2 }
I have to write RDD map operations to achieve this if anybody suggests better option welcome. colcount is the number of columns.
rdd.map(x => x.split("\":").mkString("\" ="))
.map((f => (f.dropRight(1).split(",").last.toString, f.drop(1).split(",").toSeq.take(colCount-1).toString)))
.map(f => (f._1, f._2.replace("WrappedArray(", "Check = {")))
.map(f => (f._1.drop(0).replace("\"t\"", "t"), f._2.dropRight(1).replace("(", "{"))) /
.map(f => f.toString().split(",C").mkString("\nC").replace(")", "}").drop(0).replace("(", "")) // replacing , with \n, droping (
.map(f => f.replace("\" =\"", "=\"").replace("\", \"", "\",").replace("\" =", "=").replace(", \"", ",").replace("{\"", "{"))
Scala's JSON parser seems to be a good choice for this problem:
import scala.util.parsing.json
rdd.map( x => {
JSON.parseFull(x).get.asInstanceOf[Map[String,String]]
})
This will result in an RDD[Map[String, String]]. You can then access the t field from the JSON, for example, using:
.map(dict => "t = "+dict("t"))
Current question is relative with next one, but now I need to read the data from database instead of insert.
I have next three case classes:
case class A (id: Long, bList: List[B])
case class B (id: Long, aId: cList: List[C])
case class C (id: Long, bId: Long)
And query with two leftJoin functions and incomingAId for filtering aTable results:
val query = (for {
((aResult,bResult),cResult) <- aTable.filter(_.id === incomigAId)
.joinLeft(bTable).on(_.id === _.aId)
.joinLeft(cTable).on(_._2.map(_.id) === _.bId)
} yield ((aResult,bResult),cResult)).result.transactionally
Next query works and the result looks valid, but isn't easy to handle it to the case classes. Also, executionResult has Seq[Nothing] type and process of mapping requires something like that:
database.run(query).map{ executionResult =>
executionResult.map { vectorElement: [Tuple2[Tuple2[A, Option[B]], Option[C]]]
...
}
}
Is there any proper way to prevent Seq[Nothing] (changes in query)?
Or if the query result type is fine, could you please share solution how to map it to the case classes above?
Right now I'm using next solution, but I suppose that some part of code can be optimized (e.g. groupBy replacing with something else).
execute(query).mapTo[Vector[((A, Option[B]), Option[C])]]
.flatMap { insideFuture =>
insideFuture.groupBy(_._1._1).mapValues { values =>
//scala groupBy losts ordering
val orderedB = values.groupBy(_._1._2).toSeq.sortBy(_._1.map(_.id)).toMap
orderedB.mapValues(_.map(_._2))
}.headOption match {
case Some(data) => Future.successful {
data._1.copy(bList = data._2.map {
case (Some(bElement), optionCElements) =>
bElement.copy(cList = optionCElements.toList.flatten)
case _ =>
throw new Exception("Invalid query result. Unable to find B elements")
}.toList)
}
case None => Future.failed(new Exception("Unable to find A with next id " + incomigAId))
}
}
}
I've got a query result of List[(Int,String,Double)] that I need to convert to a Map[String,String] (for display in an html select list)
My hacked solution is:
val prices = (dao.getPricing flatMap {
case(id, label, fee) =>
Map(id.toString -> (label+" $"+fee))
}).toMap
there must be a better way to achieve the same...
How about this?
val prices: Map[String, String] =
dao.getPricing.map {
case (id, label, fee) => (id.toString -> (label + " $" + fee))
}(collection.breakOut)
The method collection.breakOut provides a CanBuildFrom instance that ensures that even if you're mapping from a List, a Map is reconstructed, thanks to the type annotation, and avoids the creation of an intermediary collection.
A little more concise:
val prices =
dao.getPricing.map { case (id, label, fee) => ( id.toString, label+" $"+fee)} toMap
shorter alternative:
val prices =
dao.getPricing.map { p => ( p._1.toString, p._2+" $"+p._3)} toMap
If I have a list of items like this:
local items = { "apple", "orange", "pear", "banana" }
how do I check if "orange" is in this list?
In Python I could do:
if "orange" in items:
# do something
Is there an equivalent in Lua?
You could use something like a set from Programming in Lua:
function Set (list)
local set = {}
for _, l in ipairs(list) do set[l] = true end
return set
end
Then you could put your list in the Set and test for membership:
local items = Set { "apple", "orange", "pear", "banana" }
if items["orange"] then
-- do something
end
Or you could iterate over the list directly:
local items = { "apple", "orange", "pear", "banana" }
for _,v in pairs(items) do
if v == "orange" then
-- do something
break
end
end
Use the following representation instead:
local items = { apple=true, orange=true, pear=true, banana=true }
if items.apple then
...
end
You're seeing firsthand one of the cons of Lua having only one data structure---you have to roll your own. If you stick with Lua you will gradually accumulate a library of functions that manipulate tables in the way you like to do things. My library includes a list-to-set conversion and a higher-order list-searching function:
function table.set(t) -- set of list
local u = { }
for _, v in ipairs(t) do u[v] = true end
return u
end
function table.find(f, l) -- find element v of l satisfying f(v)
for _, v in ipairs(l) do
if f(v) then
return v
end
end
return nil
end
Write it however you want, but it's faster to iterate directly over the list, than to generate pairs() or ipairs()
#! /usr/bin/env lua
local items = { 'apple', 'orange', 'pear', 'banana' }
local function locate( table, value )
for i = 1, #table do
if table[i] == value then print( value ..' found' ) return true end
end
print( value ..' not found' ) return false
end
locate( items, 'orange' )
locate( items, 'car' )
orange found
car not found
Lua tables are more closely analogs of Python dictionaries rather than lists. The table you have create is essentially a 1-based indexed array of strings. Use any standard search algorithm to find out if a value is in the array. Another approach would be to store the values as table keys instead as shown in the set implementation of Jon Ericson's post.
This is a swiss-armyknife function you can use:
function table.find(t, val, recursive, metatables, keys, returnBool)
if (type(t) ~= "table") then
return nil
end
local checked = {}
local _findInTable
local _checkValue
_checkValue = function(v)
if (not checked[v]) then
if (v == val) then
return v
end
if (recursive and type(v) == "table") then
local r = _findInTable(v)
if (r ~= nil) then
return r
end
end
if (metatables) then
local r = _checkValue(getmetatable(v))
if (r ~= nil) then
return r
end
end
checked[v] = true
end
return nil
end
_findInTable = function(t)
for k,v in pairs(t) do
local r = _checkValue(t, v)
if (r ~= nil) then
return r
end
if (keys) then
r = _checkValue(t, k)
if (r ~= nil) then
return r
end
end
end
return nil
end
local r = _findInTable(t)
if (returnBool) then
return r ~= nil
end
return r
end
You can use it to check if a value exists:
local myFruit = "apple"
if (table.find({"apple", "pear", "berry"}, myFruit)) then
print(table.find({"apple", "pear", "berry"}, myFruit)) -- 1
You can use it to find the key:
local fruits = {
apple = {color="red"},
pear = {color="green"},
}
local myFruit = fruits.apple
local fruitName = table.find(fruits, myFruit)
print(fruitName) -- "apple"
I hope the recursive parameter speaks for itself.
The metatables parameter allows you to search metatables as well.
The keys parameter makes the function look for keys in the list. Of course that would be useless in Lua (you can just do fruits[key]) but together with recursive and metatables, it becomes handy.
The returnBool parameter is a safe-guard for when you have tables that have false as a key in a table (Yes that's possible: fruits = {false="apple"})
function valid(data, array)
local valid = {}
for i = 1, #array do
valid[array[i]] = true
end
if valid[data] then
return false
else
return true
end
end
Here's the function I use for checking if data is in an array.
Sort of solution using metatable...
local function preparetable(t)
setmetatable(t,{__newindex=function(self,k,v) rawset(self,v,true) end})
end
local workingtable={}
preparetable(workingtable)
table.insert(workingtable,123)
table.insert(workingtable,456)
if workingtable[456] then
...
end
The following representation can be used:
local items = {
["apple"]=true, ["orange"]=true, ["pear"]=true, ["banana"]=true
}
if items["apple"] then print("apple is a true value.") end
if not items["red"] then print("red is a false value.") end
Related output:
apple is a true value.
red is a false value.
You can also use the following code to check boolean validity:
local items = {
["apple"]=true, ["orange"]=true, ["pear"]=true, ["banana"]=true,
["red"]=false, ["blue"]=false, ["green"]=false
}
if items["yellow"] == nil then print("yellow is an inappropriate value.") end
if items["apple"] then print("apple is a true value.") end
if not items["red"] then print("red is a false value.") end
The output is:
yellow is an inappropriate value.
apple is a true value.
red is a false value.
Check Tables Tutorial for additional information.
function table.find(t,value)
if t and type(t)=="table" and value then
for _, v in ipairs (t) do
if v == value then
return true;
end
end
return false;
end
return false;
end
you can use this solution:
items = { 'a', 'b' }
for k,v in pairs(items) do
if v == 'a' then
--do something
else
--do something
end
end
or
items = {'a', 'b'}
for k,v in pairs(items) do
while v do
if v == 'a' then
return found
else
break
end
end
end
return nothing
A simple function can be used that :
returns nil, if the item is not found in table
returns index of item, if item is found in table
local items = { "apple", "orange", "pear", "banana" }
local function search_value (tbl, val)
for i = 1, #tbl do
if tbl[i] == val then
return i
end
end
return nil
end
print(search_value(items, "pear"))
print(search_value(items, "cherry"))
output of above code would be
3
nil