Question
Would like to understand the cause of the error and meaning of the error messages. Initially thought it was because of mixing String and Character, but mixing String and Integer was possible.
Problem
Expected below would produce List(tako, saba, ika, h, a, m, a, c, h, i) like List(tako, saba, ika, 1, 2, 3) mixing string and integer. However it caused errors.
println(
List(
List("tako", "saba", "ika"),
"hamachi"
).flatten)
^^^^^^^^
No implicit view available from java.io.Serializable => scala.collection.GenTraversableOnce[B].
not enough arguments for method flatten: (implicit asTraversable: java.io.Serializable => scala.collection.GenTraversableOnce[B])List[B]. Unspecified value parameter asTraversable.
Below worked.
println(
List(
"hamachi"
).flatten)
----
List(h, a, m, a, c, h, i)
println(
List(
List("tako", "saba", "ika")
).flatten)
----
List(tako, saba, ika)
println(
List(
List("tako", "saba", "ika"),
List(1,2,3)
).flatten)
----
List(tako, saba, ika, 1, 2, 3)
The signature of flatten is
def flatten[B](implicit asTraversable: A => GenTraversableOnce[B]): CC[B]
You can check out more about this in the source. So, about the error you get: when you make List(List("tako", "saba", "ika"), "hamachi"), scala has to infer the generic type of the List you are making. The way it does this is that it takes the least common ancestor of all the types in the list. In this case, it so happens that the closest class/trait that List("tako", "saba", "aka"): List[String] and "hamachi: String" share is java.io.Serializable.
Now, scala has to look for an implicit parameter asTraversable: A => GenTraversableOnce[B], but it doesn't find one. Since it already figured out that A = java.io.Serializable, it gives you as helpful an error message as it can muster: it didn't find any implicit of type java.io.Serializable => scala.collection.GenTraversableOnce[B] to give to asTraversable.
Now, what about the other calls? Where are they getting their implicits? Mostly from the Predef I think. This IMO is one of Scala's weaknesses - noticing, tracing, and debugging implicits. They are absolutely lovely until you try something that requires some complex implicit no one thought of defining. Oh well, no such thing as a free lunch.
Related
I have created MyList abstract class to implement the list, the reason for not using already present list implementation is I am learning Scala and this was exercise for the same course. I am writing a zipWith function to create a new list with concatenation of individual items for example:
list 1: list = [1,2,3]
list 2: listOfStrings = ["Hello", "This is", "Scala"]
and I am expecting output like: [1-Hello, 2-This is, 3-Scala]
I wrote zipWith function as mentioned below:
override def zipWith[B, C](list: MyList[B], zip: (A, B) => C): MyList[C] = {
if(list.isEmpty) throw new RuntimeException("Lists do not have the same length")
else new Cons(zip(h, list.head), t.zipWith(list.tail, zip))
}
And I am trying to call this function using this statement:
println(list.zipWith[String, String](listOfStrings, (Int,String)=>_+"-"+_))
But I am getting an error:
I could not infer the type of the parameter $3 of expanded function:
($3, _$4) => _$3 + "-" + _$4.
Type for this variable is clearly mentioned as Int still I am getting this error. This could be solved using:
println(list.zipWith[String, String](listOfStrings, _+"-"+_))
I am not able to understand why earlier statement fails, even after giving the type for the required variable
The syntax (Int,String)=>_+"-"+_ doesn't mean what you think.
It represents a function taking two parameters with some name but unknown type: (Int: ???, String: ???) => _+"-"+_.
Thus the compiler is raising an error because it indeed have no clue about the types.
You should either:
write it with explicit variable names: (i: Int, s: String) => s"$i-$s". (Notice the usage of interpolation which is recommended over adding int and string),
or declare the function separately like this: val f: (Int, String) => String = _+"-"+_.
I think the compiler is confused on which variable to match each underscore. This explicit expression works for me:
println(list.zipWith[String, String](listOfStrings, (a:Int, b:String) => a+"-"+b))
I have a function unlist taking as first argument a list of any type:
def unlist(xs: List[Any], ys: List[Any]): List[Any] = ...
and I call this function on the first element of an external list, which may or may not be a list of lists. Thus, I first need to check if this head element is itself a list, in which case I can call my unlist function.
I do it with InsintanceOf method, like so:
...
if (xs.head.isInstanceOf[List[Any]]) unlist(xs.head, ys)
However, this does not compile because of a type mismatch on xs.head:
Error: type mismatch;
found : Any
required: List[Any]
What am I doing wrong?
P.S.: since many of you have suggested to avoid type Any , I have to specify that this is part of a coding exercise aiming at having a function as general as possible
To illustrate Mateusz's comment
isInstanceOf is not being remembered
it is meant we would have to follow up with asInstanceOf like so
if (list.head.isInstanceOf[List[_]]) unlist(list.head.asInstanceOf[List[_]])
Pattern matching, as demonstrated by Tim, implicitly performs isInstanceOf/asInstanceOf combination. Also consider related answer.
If you are unable to refactor out Any, which is the weakest of types, maybe try to recover as much typing information as early as possible, perhaps like so
val lists: List[List[_]] = list.collect { case xs: List[_] => xs }
val ints: List[Int] = list.collect { case i: Int => i }
val strings: List[String] = list.collect { case s: String => s }
Note due to type erasure we cannot easily do much better than List[List[_]], for example, the following assertion passes
val list: List[Any] = List(List("woo", "hoo"), 42, "kerfuffle")
assert(list.head.isInstanceOf[List[Double]])
In this last case, a compiler warning will tell you that type argument Double in type List[Double] (the underlying of List[Double]) is unchecked since it is eliminated by erasure. It is usually not a good idea to ignore such warnings.
As mentioned in the comments, using Any like this is the sign of some bad design choices. But for the specific problem you have, you can change your if to a match like this:
def unlist(xs: List[Any], ys: List[Any]): List[Any] = ???
val xs: List[Any] = ???
xs.head match {
case l: List[Any] => unlist(l, ???)
case _ =>
}
The match checks that the head value is List[Any] then assigns the value to a variable l which has type List[Any] and can therefore be used in the unlist call.
Note: This only works because you are testing for List[Any]. You cannot test for a list of a specific type (e.g. List[Int]) because a process called type erasure removes the runtime information that would be required for this to work. The compiler will warn you when it can't implement a match for this reason.
The real solution to this question is to fix the design to use specific types rather than Any.
in a Spark process I have an RDD[Try[(A, B)]]. I have to transform this RDD using a function f: B => List[C]. What I want to obtain is an RDD[Try[(A, B, C)], in which I have to flatMap the list obtained from the application of function f.
I tryed this:
val tryRdd = // Obtain the RDD[Try[(A, B)]]
val transformedRdd =
tryRdd.map {
pair =>
for {
(a, b) <- pair
c <- f(b)
} yield {
(a, b, c)
}
}
Unfortunately what I am obtaining is an RDD[Try[Nothing]]. Why? Can anyone help me to understand where I am wrong?
I suppose that problem is not really related to RDD. Probabily RDD with List will end with the same result.
The for-comprehension is translated to
pair.flatMap { case (a, b) => f(b).map { case c => (a, b, c) } }
But f(b).map(...) will give you a List[(A, B, C)], not a Try[(A, B, C)] which you want for the argument of pair.flatMap. So the code shouldn't compile at all (unless you have a strange implicit conversion in scope).
But if you are using, say, IntelliJ, it can fail to show the error and show an incorrect type (or the other way around, it can show errors in working code): you need to actually build the project to see the real errors.
Have you tried to formally type your RDD ?
val transformedRdd : RDD[Try[Tuple3]] = ...
Edit :
If this does raise you an error, then the output of the map is wrong.
The pair variable's type is Try.
Since scala doesn't do it for you, you must add some instruction to interact with its content (the tuple (A,B)).
Plus, you don't have to keep the Try type.
I would use a flatMap to keep successes and clean my RDD.
Something like
val transformedRdd = tryRdd.flatMap {value =>
value match {
case Success((a,b)) => ...
}
}
Watch http://www.scala-lang.org/api/2.9.3/scala/util/Try.html for more informations about the Try class.
I would have thought that a list of tuples could easily be flattened:
scala> val p = "abcde".toList
p: List[Char] = List(a, b, c, d, e)
scala> val q = "pqrst".toList
q: List[Char] = List(p, q, r, s, t)
scala> val pq = p zip q
pq: List[(Char, Char)] = List((a,p), (b,q), (c,r), (d,s), (e,t))
scala> pq.flatten
But instead, this happens:
<console>:15: error: No implicit view available from (Char, Char) => scala.collection.GenTraversableOnce[B].
pq.flatten
^
I can get the job done with:
scala> (for (x <- pq) yield List(x._1, x._2)).flatten
res1: List[Char] = List(a, p, b, q, c, r, d, s, e, t)
But I'm not understanding the error message. And my alternative solution seems convoluted and inefficient.
What does that error message mean and why can't I simply flatten a List of tuples?
If the implicit conversion can't be found you can supply it explicitly.
pq.flatten {case (a,b) => List(a,b)}
If this is done multiple times throughout the code then you can save some boilerplate by making it implicit.
scala> import scala.language.implicitConversions
import scala.language.implicitConversions
scala> implicit def flatTup[T](t:(T,T)): List[T]= t match {case (a,b)=>List(a,b)}
flatTup: [T](t: (T, T))List[T]
scala> pq.flatten
res179: List[Char] = List(a, p, b, q, c, r, d, s, e, t)
jwvh's answer covers the "coding" solution to your problem perfectly well, so I am not going to go into any more detail about that. The only thing I wanted to add was clarifying why the solution that both you and jwvh found is needed.
As stated in the Scala library, Tuple2 (which (,) translates to) is:
A tuple of 2 elements; the canonical representation of a Product2.
And following up on that:
Product2 is a cartesian product of 2 components.
...which means that Tuple2[T1,T2] represents:
The set of all possible pairs of elements whose components are members of two sets (all elements in T1 and T2 respectively).
A List[T], on the other hand, represents an ordered collections of T elements.
What all this means practically is that there is no absolute way to translate any possible Tuple2[T1,T2] to a List[T], simply because T1 and T2 could be different. For example, take the following tuple:
val tuple = ("hi", 5)
How could such tuple be flattened? Should the 5 be made a String? Or maybe just flatten to a List[Any]? While both of these solutions could be used, they are working around the type system, so they are not encoded in the Tuple API by design.
All this comes down to the fact that there is no default implicit view for this case and you have to supply one yourself, as both jwvh and you already figured out.
We needed to do this recently. Allow me to explain the use case briefly before noting our solution.
Use case
Given a pool of items (which I'll call type T), we want to do an evaluation of each one against all others in the pool. The result of these comparisons is a Set of failed evaluations, which we represent as a tuple of the left item and the right item in said evaluation: (T, T).
Once these evaluations are complete, it becomes useful for us to flatten the Set[(T, T)] into another Set[T] that highlights all the items that have failed any comparisons.
Solution
Our solution for this was a fold:
val flattenedSet =
set.foldLeft(Set[T]())
{ case (acc, (x, y)) => acc + x + y }
This starts with an empty set (the initial parameter to foldLeft) as the accumulator.
Then, for each element in the consumed Set[(T, T)] (named set) here, the fold function is passed:
the last value of the accumulator (acc), and
the (T, T) tuple for that element, which the case deconstructs into x and y.
Our fold function then returns acc + x + y, which returns a set containing all the elements in the accumulator in addition to x and y. That result is passed to the next iteration as the accumulator—thus, it accumulates all the values inside each of the tuples.
Why not Lists?
I appreciated this solution in particular since it avoided creating intermediate Lists while doing the flattening—instead, it directly deconstructs each tuple while building the new Set[T].
We could also have changed our evaluation code to return List[T]s containing the left and right items in each failed evaluation—then flatten would Just Work™. But we thought the tuple more accurately represented what we were going for with the evaluation—specifically one item against another, rather than an open-ended type which could conceivably represent any number of items.
(Using Julia 0.3.11)
I'm having trouble type-annotating correctly some of my code, in an initial version we've been using ASCIIString - to annotate any String, to "avoid" abstract types, but let's start with the example, this might be related to what I've seen refereed to as "triangular dispatch" in some discussions here:
# How to type annotate this (sortof "Dictionary with default)
function pushval!(dict, key, val)
key in keys(dict) ? push!(dict[key], val) : dict[key] = [val]
return dict
end
d1 = Dict{ASCIIString, Vector{Int}}()
d2 = Dict{String, Vector{Int}}()
pushval!(d1, "a", 1)
pushval!(d2, "a", 1)
Ok (firstly - if there's a more idiomatic way to construct a dictionary with defaults, in this case an empty array, I'd love to hear about it)
So now, I've tried to type annotated it:
function pushval!{K, V} (dict::Dict{K, Vector{V}} , key, val)
Much more documenting, and works.
But now comes the trickier part - I want 'key' to be any subtype of K, and val - any subtypes of V (right?) eg - I would like to make a dictionary of String - which is an abstract type, but use concrete keys - which are ASCIIString/ByteString/UTF8String,
I thought I should write one of the followings:
function pushval!{K, V} (dict::Dict{K, Vector{V}} , key::KK <: K, val:: VV <: V)
function pushval!{K, V, KK <: K, VV <: V} (dict::Dict{K, Vector{V}} , key::KK, val::VV)
One solution would be as suggest in ( Can I use a subtype of a function parameter in the function definition? ) something with 'convert'.
But this whole thing made me wonder about the Julia code I'm writing, I've started a write a system - using String, FloatingPoint, Number and such abstract types, when I actually tried running it, I've reverted to convert everything to concrete types just to get thing running for now...
Is there a recommended codebase to read as a reference to idiomatic Julia code?
Like the very implementation of Julia's dictionary-assign operator even. Is there a part of the standard library considered good to start with as a reference? thanks
Although I don't like this this solution very much (low performance), it can be helpful:
function pushval!{K, V} (dict::Dict{K, Vector{V}} , key , val)
fun = function inner{tt1<:K,tt2<:V}(key::tt1,val::tt2)
key in keys(dict) ? push!(dict[key], val) : dict[key] = [val]
return dict
end
return fun(key,val)
end
# => pushval! (generic function with 1 method)
d1 = Dict{ASCIIString, Vector{Int}}()
# => Dict{ASCIIString,Array{Int32,1}} with 0 entries
d2 = Dict{String, Vector{Int}}()
# => Dict{String,Array{Int32,1}} with 0 entries
pushval!(d1, "a", 1)
# => Dict{ASCIIString,Array{Int32,1}} with 1 entry:
# "a" => [1]
pushval!(d2, "a", 1)
# => Dict{String,Array{Int32,1}} with 1 entry:
# "a" => [1]
I know this is only partly what you asked for, but maybe you find it sufficient.
What you call pushval! can be achieved using push!(get!(d1, "a", []), 1) (although it will return the dictionary value that was appended to instead of the dictionary itself). If you need to constrain the type of the inner collection's values, you can, for example, use:
push!(get!(d1, "a", Number[]), 1)
If you really need to define this as a function, I am afraid that, at the moment, you cannot define the types in the way you describe. As the accepted answer to the question you referenced notes, Julia does not implement triangular dispatch yet, although it is targeted for 0.5.
I could recommend looking at Julia Style Guide. There are some advices about using type annotations.
For your case you don't need type annotations for pushval! function at all. Julia will get enough info from Dict creation and deduce appropriate types for pushval! arguments.
function pushval!(dict, key, val)
key in keys(dict) ? push!(dict[key], val) : dict[key] = [val]
return dict
end
d = Dict{String, Vector{Int} # Here is all annotations you need.
pushval!(d, "a", 1) # OK, "a" is ASCIIString which is subtype of AbstractString
pushval!(d, 1, 1) # ERROR, 1 is Int64 which is not subtype of AbstractString