"Deinterlacing" a list in Scala - list

I have a list of bytes that represent raw samples read in from an audio interface. Depending on the use case and H/W, each sample can be anywhere from 1 to 4 bytes long, and the total number of channels in the "stream" can be more or less arbitrary. The amount of channels and bits per sample are both known at runtime.
I'll give an example of what I mean. There are four channels in the stream and each sample is two bytes.
List(A1, A2, B1, B2, C1, C2, D1, D2, A3, A4, B3, B4, C3, C4, D3, D4)
so A1 is the first byte of channel A's first sample, A2 is the second byte of the same sample and so on.
What I need to do is extract each channel's samples into their own lists, like this:
List(List(A1, A2, A3, A4), List(B1, B2, B3, B4), List(C1, C2, C3, C4), List(D1, D2, D3, D4))
How would I go about doing this in idiomatic Scala? I just started learning Scala a few hours ago, and the only non-imperative solution I've come up with is clearly nonoptimal:
def uninterleave(samples: Array[Byte], numChannels: Int, bytesPerSample: Int) = {
val dropAmount = numChannels * bytesPerSample
def extractChannel(n: Int) = {
def extrInner(in: Seq[Byte], acc: Seq[Byte]): Seq[Byte] = {
if(in == List()) acc
else extrInner(in.drop(dropAmount), in.take(bytesPerSample) ++ acc)
}
extrInner(samples.drop(n * bytesPerSample), Nil)
}
for(i <- 0 until numChannels) yield extractChannel(i)
}

I would do
samples.grouped(bytesPerSample).grouped(numChannels).toList
.transpose.map(_.flatten)
I would not vouch for its performance though. I would rather avoid lists, unfortunately grouped produces them.
Maybe
samples.grouped(bytesPerSample).map(_.toArray)
.grouped(numChannels).map(_.toArray)
.toArray
.transpose
.map(flatten)
Still, lots of lists.

didierd's answer is just about perfect, but, alas, I think one can improve it a bit. He is concerned with all the list creation, and transpose is a rather heavy operation as well. If you can process all the data at the same time, it might well be good enough.
However, I'm going with Stream, and use a little trick to avoid transposing.
First of all, the grouping is the same, only I'm turning stuff into streams:
def getChannels[T](input: Iterator[T], elementsPerSample: Int, numOfChannels: Int) =
input.toStream.grouped(elementsPerSample).toStream.grouped(numOfChannels).toStream
Next, I'll give you a function to extract one channel from that:
def streamN[T](s: Stream[Stream[Stream[T]]])(channel: Int) = s flatMap (_(channel))
With those, we can decode the streams like this:
// Sample input
val input = List('A1, 'A2, 'B1, 'B2, 'C1, 'C2, 'D1, 'D2, 'A3, 'A4, 'B3, 'B4, 'C3, 'C4, 'D3, 'D4)
// Save streams to val, to avoid recomputing the groups
val streams = getChannels(input.iterator, elementsPerSample = 2, numOfChannels = 4)
// Decode each one
def demuxer = streamN(streams) _
val aa = demuxer(0)
val bb = demuxer(1)
val cc = demuxer(2)
val dd = demuxer(3)
This will return separate streams for each channel without having the whole stream at hand. This might be useful if you need to process the input in real time. Here's some input source to test how far into the input it reads to get at a particular element:
def source(elementsPerSample: Int, numOfChannels: Int) = Iterator.from(0).map { x =>
"" + ('A' + x / elementsPerSample % numOfChannels).toChar +
(x % elementsPerSample
+ (x / (numOfChannels * elementsPerSample)) * elementsPerSample
+ 1)
}.map { x => println("Saw "+x); x }
You can then try stuff like this:
val streams = getChannels(source(2, 4), elementsPerSample = 2, numOfChannels = 4)
def demuxer = streamN(streams) _
val cc = demuxer(2)
println(cc take 20 toList)
val bb = demuxer(1)
println(bb take 30 toList)

Related

How to randomly generate an Oct-Tuple with SML

Edit: Here is the code I have so far for generating the Patient Oct-Tuples.
(thanks Anon for giving me the bost on how to calculate weighted probability/setting the seed)
fun genPatients(x:int) =
let
val seed=let
val m=Date.minute(Date.fromTimeLocal(Time.now()))
val s=Date.second(Date.fromTimeLocal(Time.now()))
in Random.rand(m,s)
end;
val survivalrate = ref(1)
val numl = ref(1)
val td = ref(1)
val xray = ref(false)
val count= ref(0)
val emnum= ref(1000)
val ageList = [1, 2, 3, 3];
val xrayList=[false,true];
val age = Random.randRange (0, 3) seed;(* random age*)
val nextInt1 = Random.randRange(0, 1)(* random xray*)
val r1 = Random.rand(1,1)
val nextInt2 = Random.randRange(1, 10000000)(* random td*)
val r2 = Random.rand(1,1)
val r1hold= ref(1);
in
while !count < x do
(
count:= !count + 1;
List.nth(ageList, age);
r1hold:= nextInt1 r1;
td:= nextInt2 r2;
(!emnum,age,survivalrate,numl,[],[],xray,td);
emnum:= !emnum + 1
)
end;
My question now is now how to go about indexing a boolean list?
So I was looking for some help defining my Oct-tuple to finish up my project and lo and behold I find someone posting the entirety of my project hoping for a handout answer. Not only that, but I'm almost certain we're in the same class, and you think posting this the night before the morning the project is due is what a responsible student does? Pretty sure nobody on SO is gonna do your homework for you anyway, in fact I'm not even sure it's allowed.
Maybe do some work and then ask for help when you've actually done anything. Or maybe in the next phase try a little harder.
EDIT: I'll give you something to get you started.
To calculate weighted probability you need a seed.
val seed=let
val m=Date.minute(Date.fromTimeLocal(Time.now()))
val s=Date.second(Date.fromTimeLocal(Time.now()))
in Random.rand(m,s)
end;
Here's one. Then you can calculate probability, at least for the age, like this:
val ageList = [1, 2, 3, 3];
val ageInt = Random.randRange (0, 3) seed;
List.nth(ageList, ageInt)
This was how I decided to do the weighted probability portion, you can equate this to the other weighted sections if you're creative. Good luck.

Make a new list from two other lists of different types by comparing values of each type

I have two Lists of objects that both implement an interface, but are otherwise unrelated. How can I create a new collection of objects containing only the objects of one of the lists that match a value in the other list?
Obviously I could use a for loop & do this manually, but I'd like to know how I can do this using Kotlin's standard library collection filtering functions.
So here's an example:
interface Ids
{
val id: Int
}
data class A(override val id: Int, val name: String) : Ids
data class B(override val id: Int, val timestamp: Long) : Ids
fun main(args: Array<String>) {
val a1 = A(1, "Steve")
val a2 = A(2, "Ed")
val aCol = listOf(a1, a2)
val b2 = B(2, 12345)
val b3 = B(3, 67890)
val bCol = listOf(b2, b3)
val matches = mutableListOf<B>()
// This is where I'm stuck.
// I want to filter bCol using objects from aCol as a filter.
// The result should be that matches contains only a single object: b2
// because bCol[0].id == aCol[1].id
// I'm guessing I need to start with something like this:
bCol.filterTo(matches) { ??? }
}
A straightforward approach would be to search aCol for an object with the same id for each b in bCol:
bCol.filter { b -> aCol.any { a -> a.id == b.id } }
However that may become too slow if your lists are big enough.
To make it more scalable you can first build a set of all ids in aCol:
val aColIds = aCol.map { it.id }.toSet()
And then use Set.contains method to determine whether b.id is in aColIds:
bCol.filter { it.id in aColIds }
// or equivalent
bCol.filter { aColIds.contains(it.id) }

Merge Two Pandas Dataframes when two columns are list

I have two Pandas data frames and they need to be merged. Example data frames are:
c1 c2
pd1 = [[1, [1,2]]
c3 c4
pd2 = [[1, [1,3]],
[2,[2,3]]
result = [[1,1], [1,2]]
The join condition is that lists in c2 and c4 have at lease one common element.
I've tried:
result = pd.merge(pd1, pd2, left_on=list('c2'),right_on=list('c4'), how='inner')
However, this seems to only join them when the rows in each column are single values like a float, int or string.
I've attacked this problem using nested loops. This runs like a dog when the sets get large. Is there a faster way to perform this merge exploiting data frames or is there another way that's better?
pd1 = pd.DataFrame([[1, [1,2]]], columns=['c1', 'c2'])
pd1
pd2 = pd.DataFrame([[1, [1, 2]], [2, [2, 3]]], columns=['c3', 'c4'])
pd2
Setup for a merge
s2 = pd2.c4.apply(pd.Series).stack() \
.rename_axis(['idx2', 'lst2']).reset_index(name='val')
s2
s1 = pd1.c2.apply(pd.Series).stack() \
.rename_axis(['idx1', 'lst1']).reset_index(name='val')
s1
mrg = s1.merge(s2)[['idx1', 'idx2']].drop_duplicates()
mrg
a1 = pd1.c1.loc[mrg.idx1].values
a2 = pd2.c3.loc[mrg.idx2]
pd.DataFrame(dict(c1=a1, c3=a2))

How can I get upper and lower limits in 1 line?

How can I get the min and max in one line?
d = {'k1': ['2000-01-01', '2003-01-01'],
'k2': ['2001-01-01', '2003-01-21'],
'k3': ['2001-11-01', '2002-01-01'],
}
d0 = min((a[0] for a in d.itervalues())) # '2000-01-01'
d1 = max((a[1] for a in d.itervalues())) # '2003-01-21'
Of course, the real dict is much longer than in this example. And of course too d0, d1 = min(...), max(...) is not the tricky I'm asking for :)
reduce(lambda lst, val: [min(lst[0],val[0]), max(lst[1],val[1])],iter(d.itervalues()))

How to maintain an immutable list when you impact object linked to each other into this list

I'm trying to code the fast Non Dominated Sorting algorithm (NDS) of Deb used in NSGA2 in immutable way using Scala.
But the problem seems more difficult than i think, so i simplify here the problem to make a MWE.
Imagine a population of Seq[A], and each A element is decoratedA with a list which contains pointers to other elements of the population Seq[A].
A function evalA(a:decoratedA) take the list of linkedA it contains, and decrement value of each.
Next i take a subset list decoratedAPopulation of population A, and call evalA on each. I have a problem, because between each iteration on element on this subset list decoratedAPopulation, i need to update my population of A with the new decoratedA and the new updated linkedA it contain ...
More problematic, each element of population need an update of 'linkedA' to replace the linked element if it change ...
Hum as you can see, it seem complicated to maintain all linked list synchronized in this way. I propose another solution bottom, which probably need recursion to return after each EvalA a new Population with element replaced.
How can i do that correctly in an immutable way ?
It's easy to code in a mutable way, but i don't find a good way to do this in an immutable way, do you have a path or an idea to do that ?
object test extends App{
case class A(value:Int) {def decrement()= new A(value - 1)}
case class decoratedA(oneAdecorated:A, listOfLinkedA:Seq[A])
// We start algorithm loop with A element with value = 0
val population = Seq(new A(0), new A(0), new A(8), new A(1))
val decoratedApopulation = Seq(new decoratedA(population(1),Seq(population(2),population(3))),
new decoratedA(population(2),Seq(population(1),population(3))))
def evalA(a:decoratedA) = {
val newListOfLinked = a.listOfLinkedA.map{e => e.decrement()
new decoratedA(a.oneAdecorated,newListOfLinked)}
}
def run()= {
//decoratedApopulation.map{
// ?
//}
}
}
Update 1:
About the input / output of the initial algorithm.
The first part of Deb algorithm (Step 1 to Step 3) analyse a list of Individual, and compute for each A : (a) domination count, the number of A which dominate me (the value attribute of A) (b) a list of A i dominate (listOfLinkedA).
So it return a Population of decoratedA totally initialized, and for the entry of Step 4 (my problem) i take the first non dominated front, cf. the subset of elements of decoratedA with A value = 0.
My problem start here, with a list of decoratedA with A value = 0; and i search the next front into this list by computing each listOfLinkedA of each of this A
At each iteration between step 4 to step 6, i need to compute a new B subset list of decoratedA with A value = 0. For each , i decrementing first the domination count attribute of each element into listOfLinkedA, then i filter to get the element equal to 0. A the end of step 6, B is saved to a list List[Seq[DecoratedA]], then i restart to step 4 with B, and compute a new C, etc.
Something like that in my code, i call explore() for each element of B, with Q equal at the end to new subset of decoratedA with value (fitness here) = 0 :
case class PopulationElement(popElement:Seq[Double]){
implicit def poptodouble():Seq[Double] = {
popElement
}
}
class SolutionElement(values: PopulationElement, fitness:Double, dominates: Seq[SolutionElement]) {
def decrement()= if (fitness == 0) this else new SolutionElement(values,fitness - 1, dominates)
def explore(Q:Seq[SolutionElement]):(SolutionElement, Seq[SolutionElement])={
// return all dominates elements with fitness - 1
val newSolutionSet = dominates.map{_.decrement()}
val filteredSolution:Seq[SolutionElement] = newSolutionSet.filter{s => s.fitness == 0.0}.diff{Q}
filteredSolution
}
}
A the end of algorithm, i have a final list of seq of decoratedA List[Seq[DecoratedA]] which contain all my fronts computed.
Update 2
A sample of value extracted from this example.
I take only the pareto front (red) and the {f,h,l} next front with dominated count = 1.
case class p(x: Int, y: Int)
val a = A(p(3.5, 1.0),0)
val b = A(p(3.0, 1.5),0)
val c = A(p(2.0, 2.0),0)
val d = A(p(1.0, 3.0),0)
val e = A(p(0.5, 4.0),0)
val f = A(p(0.5, 4.5),1)
val h = A(p(1.5, 3.5),1)
val l = A(p(4.5, 1.0),1)
case class A(XY:p, value:Int) {def decrement()= new A(XY, value - 1)}
case class ARoot(node:A, children:Seq[A])
val population = Seq(
ARoot(a,Seq(f,h,l),
ARoot(b,Seq(f,h,l)),
ARoot(c,Seq(f,h,l)),
ARoot(d,Seq(f,h,l)),
ARoot(e,Seq(f,h,l)),
ARoot(f,Nil),
ARoot(h,Nil),
ARoot(l,Nil))
Algorithm return List(List(a,b,c,d,e), List(f,h,l))
Update 3
After 2 hour, and some pattern matching problems (Ahum...) i'm comming back with complete example which compute automaticaly the dominated counter, and the children of each ARoot.
But i have the same problem, my children list computation is not totally correct, because each element A is possibly a shared member of another ARoot children list, so i need to think about your answer to modify it :/ At this time i only compute children list of Seq[p], and i need list of seq[A]
case class p(x: Double, y: Double){
def toSeq():Seq[Double] = Seq(x,y)
}
case class A(XY:p, dominatedCounter:Int) {def decrement()= new A(XY, dominatedCounter - 1)}
case class ARoot(node:A, children:Seq[A])
case class ARootRaw(node:A, children:Seq[p])
object test_stackoverflow extends App {
val a = new p(3.5, 1.0)
val b = new p(3.0, 1.5)
val c = new p(2.0, 2.0)
val d = new p(1.0, 3.0)
val e = new p(0.5, 4.0)
val f = new p(0.5, 4.5)
val g = new p(1.5, 4.5)
val h = new p(1.5, 3.5)
val i = new p(2.0, 3.5)
val j = new p(2.5, 3.0)
val k = new p(3.5, 2.0)
val l = new p(4.5, 1.0)
val m = new p(4.5, 2.5)
val n = new p(4.0, 4.0)
val o = new p(3.0, 4.0)
val p = new p(5.0, 4.5)
def isStriclyDominated(p1: p, p2: p): Boolean = {
(p1.toSeq zip p2.toSeq).exists { case (g1, g2) => g1 < g2 }
}
def sortedByRank(population: Seq[p]) = {
def paretoRanking(values: Set[p]) = {
//comment from #dk14: I suppose order of values isn't matter here, otherwise use SortedSet
values.map { v1 =>
val t = (values - v1).filter(isStriclyDominated(v1, _)).toSeq
val a = new A(v1, values.size - t.size - 1)
val root = new ARootRaw(a, t)
println("Root value ", root)
root
}
}
val listOfARootRaw = paretoRanking(population.toSet)
//From #dk14: Here is convertion from Seq[p] to Seq[A]
val dominations: Map[p, Int] = listOfARootRaw.map(a => a.node.XY -> a.node.dominatedCounter) //From #dk14: It's a map with dominatedCounter for each point
val listOfARoot = listOfARootRaw.map(raw => ARoot(raw.node, raw.children.map(p => A(p, dominations.getOrElse(p, 0)))))
listOfARoot.groupBy(_.node.dominatedCounter)
}
//Get the first front, a subset of ARoot, and start the step 4
println(sortedByRank(Seq(a, b, c, d, e, f, g, h, i, j, k, l, m, n, o, p)).head)
}
Talking about your problem with distinguishing fronts (after update 2):
val (left,right) = population.partition(_.node.value == 0)
List(left, right.map(_.copy(node = node.copy(value = node.value - 1))))
No need for mutating anything here. copy will copy everything but fields you specified with new values. Talking about the code, the new copy will be linked to the same list of children, but new value = value - 1.
P.S. I have a feeling you may actually want to do something like this:
case class A(id: String, level: Int)
val a = A("a", 1)
val b = A("b", 2)
val c = A("c", 2)
val d = A("d", 3)
clusterize(List(a,b,c,d)) === List(List(a), List(b,c), List(d))
It's simple to implement:
def clusterize(list: List[A]) =
list.groupBy(_.level).toList.sortBy(_._1).map(_._2)
Test:
scala> clusterize(List(A("a", 1), A("b", 2), A("c", 2), A("d", 3)))
res2: List[List[A]] = List(List(A(a,1)), List(A(b,2), A(c,2)), List(A(d,3)))
P.S.2. Please consider better naming conventions, like here.
Talking about "mutating" elements in some complex structure:
The idea of "immutable mutating" some shared (between parts of a structure) value is to separate your "mutation" from the structure. Or simply saying, divide and conquerror:
calculate changes in advance
apply them
The code:
case class A(v: Int)
case class AA(a: A, seq: Seq[A]) //decoratedA
def update(input: Seq[AA]) = {
//shows how to decrement each value wherever it is:
val stats = input.map(_.a).groupBy(identity).mapValues(_.size) //domination count for each A
def upd(a: A) = A(a.v - stats.getOrElse(a, 0)) //apply decrement
input.map(aa => aa.copy(aa = aa.seq.map(upd))) //traverse and "update" original structure
}
So, I've introduced new Map[A, Int] structure, that shows how to modify the original one. This approach is based on highly simplified version of Applicative Functor concept. In general case, it should be Map[A, A => A] or even Map[K, A => B] or even Map[K, Zipper[A] => B] as applicative functor (input <*> map). *Zipper (see 1, 2) actually could give you information about current element's context.
Notes:
I assumed that As with same value are same; that's default behaviour for case classess, otherwise you need to provide some additional id's (or redefine hashCode/equals).
If you need more levels - like AA(AA(AA(...)))) - just make stats and upd recursive, if dеcrement's weight depends on nesting level - just add nesting level as parameter to your recursive function.
If decrement depends on parent node (like decrement only A(3)'s, which belongs to A(3)) - add parent node(s) as part of stats's key and analise it during upd.
If there is some dependency between stats calculation (how much to decrement) of let's say input(1) from input(0) - you should use foldLeft with partial stats as accumulator: val stats = input.foldLeft(Map[A, Int]())((partialStats, elem) => partialStats ++ analize(partialStats, elem))
Btw, it takes O(N) here (linear memory and cpu usage)
Example:
scala> val population = Seq(A(3), A(6), A(8), A(3))
population: Seq[A] = List(A(3), A(6), A(8), A(3))
scala> val input = Seq(AA(population(1),Seq(population(2),population(3))), AA(population(2),Seq(population(1),population(3))))
input: Seq[AA] = List(AA(A(6),List(A(8), A(3))), AA(A(8),List(A(6), A(3))))
scala> update(input)
res34: Seq[AA] = List(AA(A(5),List(A(7), A(3))), AA(A(7),List(A(5), A(3))))