I come from SML background and feel quite comfortable with high-order functions. But I don't really get the idea of list comprehension. Is there any situation where list comprehension is more suitable than high-order functions on List and vice versa?
I heard somewhere that list comprehension is slower than high-order functions, should I avoid to use it when writing performance-critical functions?
For the example' sake, take a look at Projecting a list of lists efficiently in F# where #cfern's answer contains two versions using list comprehension and high-order functions respectively:
let rec cartesian = function
| [] -> [[]]
| L::Ls -> [for C in cartesian Ls do yield! [for x in L do yield x::C]]
and:
let rec cartesian2 = function
| [] -> [[]]
| L::Ls -> cartesian2 Ls |> List.collect (fun C -> L |> List.map (fun x->x::C))
Choosing between comprehensions and higher-order functions is mostly a matter of style. I think that comprehensions are sometimes more readable, but that's just a personal preference. Note that the cartesian function could be written more elegantly like this:
let rec cartesian = function
| [] -> [[]]
| L::Ls ->
[ for C in cartesian Ls do for x in L do yield x::C ]
The interesting case is when writing recursive functions. If you use sequences (and sequence comprehensions), they remove some unnecessary allocation of temporary lists and if you use yield! in a tail-call position, you can also avoid stack overflow exceptions:
let rec nums n =
if n = 100000 then []
else n::(nums (n+1))
// throws StackOverflowException
nums 0
let rec nums n = seq {
if n < 100000 then
yield n
yield! nums (n+1) }
// works just fine
nums 0 |> List.ofSeq
This is quite an interesting pattern, because it cannot be written in the same way using lists. When using lists, you cannot return some element and then make a recursive call, because it corresponds to n::(nums ...), which is not tail-recursive.
Looking at the generated code in ILSpy, you can see that list comprehensions are compiled to state machines (like methods using yield return in C#), then passed to something like List.ofSeq. Higher-order functions, on the other hand, are hand-coded, and frequently use mutable state or other imperative constructs to be as efficient as possible. As is often the case, the general-purpose mechanism is more expensive.
So, to answer your question, if performance is critical there is usually a higher-order function specific to your problem that should be preferred.
Adding to Tomas Petricek's answer. You can make the list version tail recursive.
let nums3 n =
let rec nums3internal acc n =
if n = 100000 then
acc
else
nums3internal (n::acc) (n+1) //Tail Call Optimization possible
nums3internal [] n |> List.rev
nums3 0
With the added benefit of a considerable speedup. At least when I measured with the stopwatch tool I get. (nums2 being the algorithm using Seq).
Nums2 takes 81.225500ms
Nums3 takes 4.948700ms
For higher numbers this advantage shrinks, because List.rev is inefficient. E.g. for 10000000 I get:
Nums2 takes 11054.023900ms
Nums3 takes 8256.693100ms
Related
I'm working in OCaml and need to write a function
let rev lst = ...
which reverses the list lst without using recursion. I don't think I can use iterative methods either, like a for-loop. And I can't use List library functions. And I can't define some kind of data structure that allows me to interface with the elements in reverse order. It has to be a very from-bare-OCaml implementation.
Given these constraints I really can't think of any way to do this. I really don't even know where to start. The only two things in my bag of tricks, when dealing with arbitrary lists, are recursion and iteration.
The only loophole I can see here is to define another function that uses recursion, and then have rev use it such that rev itself is not recursive. List.fold_left is easy enough to reimplement such that your rev function also doesn't use any functions from the List module. This should satisfy the requirements.
let rec foldl f i =
function
| [] -> i
| x::xs -> foldl f (f i x) xs
And then rev:
let rev lst = foldl (fun i x -> x::i) [] lst
If you feel like being clever, you could reimplement Fun.flip as well, and create a cons function. Both simple enough.
let flip f a b = f b a
let cons a b = a :: b
let rev lst = foldl (flip cons) [] lst
I am to use combinators and no for/while loops, recursion or defined library functions from F#'s List module, except constructors :: and []
Ideally I want to implement map
I am trying to write a function called llength that returns the list of the lengths of the sublists. For example llength [[1;2;3];[1;2];[1;2;3]] should return [3;2,3]. I also have function length that returns the length of a list.
let Tuple f = fun a b -> f (a, b)
let length l : int =
List.fold (Tuple (fst >> (+) 1)) 0 l
currently have
let llength l : int list =
List.map (length inner list) list
Not sure how I should try accessing my sublists with my restraints and should I use my other method on each sublist? any help is greatly appreciated, thanks!
Since this is homework, I don't want to just give you a fully coded solution, but here are some hints:
First, since fold is allowed you could implement map via fold. The folding function would take the list accumulated "so far" and prepend the next element transformed with mapping function. The result will come out reversed though (fold traverses forward, but you prepend at every step), so perhaps that wouldn't work for you if you're not allowed List.rev.
Second - the most obvious, fundamental way: naked recursion. Here's the way to think about it: (1) when the argument is an empty list, result should be an empty list; (2) when the argument is a non-empty list, the result should be length of the argument's head prepended to the list of lengths of the argument's tail, which can be calculated recursively. Try to write that down in F#, and there will be your solution.
Since you can use some functions that basically have a loop (fold, filter ...), there might be some "cheated & dirty" ways to implement map. For example, via filter:
let mymap f xs =
let mutable result = []
xs
|> List.filter (fun x ->
result <- f x :: result
true)
|> ignore
result |> List.rev
Note that List.rev is required as explained in the other answer.
EDIT: see this followup question that simplifies the problem I am trying to identify here, and asks for input on a GHC modification proposal.
So I was trying to write a generic breadth-first search function and came up with the following:
bfs :: (a -> Bool) -> (a -> [a]) -> [a] -> Maybe a
bfs predf expandf xs = find predf bfsList
where bfsList = xs ++ concatMap expandf bfsList
which I thought was pretty elegant, however in the does-not-exist case it blocks forever.
After all the terms have been expanded to [], concatMap will never return another item, so concatMap is blocking waiting for another item from itself? Could Haskell be made smart enough to realize the list generation is blocked reading the self-reference and terminate the list?
The best replacement I've been able to come up with isn't quite as elegant, since I have to handle the termination case myself:
where bfsList = concat.takeWhile (not.null) $ iterate (concatMap expandf) xs
For concrete examples, the first search terminates with success, and the second one blocks:
bfs (==3) (\x -> if x<1 then [] else [x/2, x/5]) [5, 3*2**8]
bfs (==3) (\x -> if x<1 then [] else [x/2, x/5]) [5, 2**8]
Edited to add a note to explain my bfs' solution below.
The way your question is phrased ("could Haskell be made smart enough"), it sounds like you think the correct value for a computation like:
bfs (\x -> False) (\x -> []) []
given your original definition of bfs should be Nothing, and Haskell is just failing to find the correct answer.
However, the correct value for the above computation is bottom. Substituting the definition of bfs (and simplifying the [] ++ expression), the above computation is equal to:
find (\x -> False) bfsList
where bfsList = concatMap (\x -> []) bfsList
Evaluating find requires determining if bfsList is empty or not, so it must be forced to weak head normal form. This forcing requires evaluating the concatMap expression, which also must determine if bfsList is empty or not, forcing it to WHNF. This forcing loop implies bfsList is bottom, and therefore so is find.
Haskell could be smarter in detecting the loop and giving an error, but it would be incorrect to return [].
Ultimately, this is the same thing that happens with:
foo = case foo of [] -> []
which also loops infinitely. Haskell's semantics imply that this case construct must force foo, and forcing foo requires forcing foo, so the result is bottom. It's true that if we considered this definition an equation, then substituting foo = [] would "satisfy" it, but that's not how Haskell semantics work, for the same reason that:
bar = bar
does not have value 1 or "awesome", even though these values satisfy it as an "equation".
So, the answer to your question is, no, this behavior couldn't be changed so as to return an empty list without fundamentally changing Haskell semantics.
Also, as an alternative that looks pretty slick -- even with its explicit termination condition -- maybe consider:
bfs' :: (a -> Bool) -> (a -> [a]) -> [a] -> Maybe a
bfs' predf expandf = look
where look [] = Nothing
look xs = find predf xs <|> look (concatMap expandf xs)
This uses the Alternative instance for Maybe, which is really very straightforward:
Just x <|> ... -- yields `Just x`
Nothing <|> Just y -- yields `Just y`
Nothing <|> Nothing -- yields `Nothing` (doesn't happen above)
so look checks the current set of values xs with find, and if it fails and returns Nothing, it recursively looks in their expansions.
As a silly example that makes the termination condition look less explicit, here's its double-monad (Maybe in implicit Reader) version using listToMaybe as the terminator! (Not recommended in real code.)
bfs'' :: (a -> Bool) -> (a -> [a]) -> [a] -> Maybe a
bfs'' predf expandf = look
where look = listToMaybe *>* find predf *|* (look . concatMap expandf)
(*>*) = liftM2 (>>)
(*|*) = liftM2 (<|>)
infixl 1 *>*
infixl 3 *|*
How does this work? Well, it's a joke. As a hint, the definition of look is the same as:
where look xs = listToMaybe xs >>
(find predf xs <|> look (concatMap expandf xs))
We produce the results list (queue) in steps. On each step we consume what we have produced on the previous step. When the last expansion step added nothing, we stop:
bfs :: (a -> Bool) -> (a -> [a]) -> [a] -> Maybe a
bfs predf expandf xs = find predf queue
where
queue = xs ++ gen (length xs) queue -- start the queue with `xs`, and
gen 0 _ = [] -- when nothing in queue, stop;
gen n q = let next = concatMap expandf (take n q) -- take n elemts from queue,
in next ++ -- process, enqueue the results,
gen (length next) (drop n q) -- advance by `n` and continue
Thus we get
~> bfs (==3) (\x -> if x<1 then [] else [x/2, x/5]) [5, 3*2**8]
Just 3.0
~> bfs (==3) (\x -> if x<1 then [] else [x/2, x/5]) [5, 2**8]
Nothing
One potentially serious flow in this solution is that if any expandf step produces an infinite list of results, it will get stuck calculating its length, totally needlessly so.
In general, just introduce a counter and increment it by the length of solutions produced at each expansion step (length . concatMap expandf or something), decrementing by the amount that was consumed. When it reaches 0, do not attempt to consume anything anymore because there's nothing to consume at that point, and you should instead terminate.
This counter serves in effect as a pointer back into the queue being constructed. A value of n indicates that the place where the next result will be placed is n notches ahead of the place in the list from which the input is taken. 1 thus means that the next result is placed directly after the input value.
The following code can be found in Wikipedia's article about corecursion (search for "corecursive queue"):
data Tree a b = Leaf a | Branch b (Tree a b) (Tree a b)
bftrav :: Tree a b -> [Tree a b]
bftrav tree = queue
where
queue = tree : gen 1 queue -- have one value in queue from the start
gen 0 _ = []
gen len (Leaf _ : s) = gen (len-1) s -- consumed one, produced none
gen len (Branch _ l r : s) = l : r : gen (len+1) s -- consumed one, produced two
This technique is natural in Prolog with top-down list instantiation and logical variables which can be explicitly in a not-yet-set state. See also tailrecursion-modulo-cons.
gen in bfs can be re-written to be more incremental, which is usually a good thing to have:
gen 0 _ = []
gen n (y:ys) = let next = expandf y
in next ++ gen (n - 1 + length next) ys
bfsList is defined recursively, which is not in itself a problem in Haskell. It does, however, produce an infinite list, which, again, isn't in itself a problem, because Haskell is lazily evaluated.
As long as find eventually finds what it's looking for, it's not an issue that there's still an infinity of elements, because at that point evaluation stops (or, rather, moves on to do other things).
AFAICT, the problem in the second case is that the predicate is never matched, so bfsList just keeps producing new elements, and find keeps on looking.
After all the terms have been expanded to [] concatMap will never return another item
Are you sure that's the correct diagnosis? As far as I can tell, with the lambda expressions supplied above, each input element always expand to two new elements - never to []. The list is, however, infinite, so if the predicate goes unmatched, the function will evaluate forever.
Could Haskell be made smart enough to realize the list generation is blocked reading the self-reference and terminate the list?
It'd be nice if there was a general-purpose algorithm to determine whether or not a computation would eventually complete. Alas, as both Turing and Church (independently of each other) proved in 1936, such an algorithm can't exist. This is also known as the Halting problem. I'm not a mathematician, though, so I may be wrong, but I think it applies here as well...
The best replacement I've been able to come up with isn't quite as elegant
Not sure about that one... If I try to use it instead of the other definition of bfsList, it doesn't compile... Still, I don't think the problem is the empty list.
Its possible to create infinite, circular lists using let rec, without needing to resort to mutable references:
let rec xs = 1 :: 0 :: xs ;;
But can I use this same technique to write a function that receives a finite list and returns an infinite, circular version of it? I tried writing
let rec cycle xs =
let rec result = go xs and
go = function
| [] -> result
| (y::ys) -> y :: go ys in
result
;;
But got the following error
Error: This kind of expression is not allowed as right-hand side of `let rec'
Your code has two problems:
result = go xs is in illegal form for let rec
The function tries to create a loop by some computation, which falls into an infinite loop causing stack overflow.
The above code is rejected by the compiler because you cannot write an expression which may cause recursive computation in the right-hand side of let rec (see Limitations of let rec in OCaml).
Even if you fix the issue you still have a problem: cycle does not finish the job:
let rec cycle xs =
let rec go = function
| [] -> go xs
| y::ys -> y :: g ys
in
go xs;;
cycle [1;2];;
cycle [1;2] fails due to stack overflow.
In OCaml, let rec can define a looped structure only when its definition is "static" and does not perform any computation. let rec xs = 1 :: 0 :: xs is such an example: (::) is not a function but a constructor, which purely constructs the data structure. On the other hand, cycle performs some code execution to dynamically create a structure and it is infinite. I am afraid that you cannot write a function like cycle in OCaml.
If you want to introduce some loops in data like cycle in OCaml, what you can do is using lazy structure to prevent immediate infinite loops like Haskell's lazy list, or use mutation to make a loop by a substitution. OCaml's list is not lazy nor mutable, therefore you cannot write a function dynamically constructs looped lists.
If you do not mind using black magic, you could try this code:
let cycle l =
if l = [] then invalid_arg "cycle" else
let l' = List.map (fun x -> x) l in (* copy the list *)
let rec aux = function
| [] -> assert false
| [_] as lst -> (* find the last cons cell *)
(* and set the last pointer to the beginning of the list *)
Obj.set_field (Obj.repr lst) 1 (Obj.repr l')
| _::t -> aux t
in aux l'; l'
Please be aware that using the Obj module is highly discouraged. On the other hand, there are industrial-strength programs and libraries (Coq, Jane Street's Core, Batteries included) that are known to use this sort of forbidden art.
camlspotter's answer is good enough already. I just want to add several more points here.
First of all, for the problem of write a function that receives a finite list and returns an infinite, circular version of it, it can be done in code / implementation level, just if you really use the function, it will have stackoverflow problem and will never return.
A simple version of what you were trying to do is like this:
let rec circle1 xs = List.rev_append (List.rev xs) (circle1 xs)
val circle: 'a list -> 'a list = <fun>
It can be compiled and theoretically it is correct. On [1;2;3], it is supposed to generate [1;2;3;1;2;3;1;2;3;1;2;3;...].
However, of course, it will fail because its run will be endless and eventually stackoverflow.
So why let rec circle2 = 1::2::3::circle2 will work?
Let's see what will happen if you do it.
First, circle2 is a value and it is a list. After OCaml get this info, it can create a static address for circle2 with memory representation of list.
The memory's real value is 1::2::3::circle2, which actually is Node (1, Node (2, Node (3, circle2))), i.e., A Node with int 1 and address of a Node with int 2 and address of a Node with int 3 and address of circle2. But we already know circle2's address, right? So OCaml just put circle2's address there.
Everything will work.
Also, through this example, we can also know a fact that for a infinite circled list defined like this actually doesn't cost limited memory. It is not generating a real infinite list to consume all memory, instead, when a circle finishes, it just jumps "back" to the head of the list.
Let's then go back to example of circle1. Circle1 is a function, yes, it has an address, but we do not need or want it. What we want is the address of the function application circle1 xs. It is not like circle2, it is a function application which means we need to compute something to get the address. So,
OCaml will do List.rev xs, then try to get address circle1 xs, then repeat, repeat.
Ok, then why we sometimes get Error: This kind of expression is not allowed as right-hand side of 'let rec'?
From http://caml.inria.fr/pub/docs/manual-ocaml/extn.html#s%3aletrecvalues
the let rec binding construct, in addition to the definition of
recursive functions, also supports a certain class of recursive
definitions of non-functional values, such as
let rec name1 = 1 :: name2 and name2 = 2 :: name1 in expr which
binds name1 to the cyclic list 1::2::1::2::…, and name2 to the cyclic
list 2::1::2::1::…Informally, the class of accepted definitions
consists of those definitions where the defined names occur only
inside function bodies or as argument to a data constructor.
If you use let rec to define a binding, say let rec name. This name can be only in either a function body or a data constructor.
In previous two examples, circle1 is in a function body (let rec circle1 = fun xs -> ...) and circle2 is in a data constructor.
If you do let rec circle = circle, it will give error as circle is not in the two allowed cases. let rec x = let y = x in y won't do either, because again, x not in constructor or function.
Here is also a clear explanation:
https://realworldocaml.org/v1/en/html/imperative-programming-1.html
Section Limitations of let rec
Several times I've wanted to traverse a list and pick out elements that have some property which also relies on, say, the next element in the list. For a simple example I have some code which counts how many times a function f changes sign over a specified interval [a,b]. This is fairly obvious in an imperative language like C:
for(double x=a; x<=b; x+=(b-a)/n){
s*f(x)>0 ? : printf("%e %e\n",x, f(x)), s=sgn(f(x));
}
In Haskell my first instinct was to zip the list with its tail and then apply the filter and extract the elements with fst or whatever. But that seems clumsy and inefficient, so I shoehorned it into being a fold:
signChanges f a b n = tail $
foldl (\(x:xs) y -> if (f x*f y)<0 then y:x:xs else x:xs) [a] [a,a+(b-a)/n..b]
Either way I feel there is a "right" way to do this (as there so often is in Haskell) and that I don't know (or just haven't realised) what it is. Any help with how to express this in a more idiomatic or elegant way would be greatly appreciated, as would advice on how, in general, to find the "right" way to do things.
Zipping is efficient if you run with -O2 as list fusion engages. No need to resort to folds in this case is one of essential advantages of Haskell as it improves modularity.
So zipping is the right way to do it.
Here is a "version" using a paramorphism (not quite the same as the question - but it should illustrate a paramorphism usefully enough), first we need para as it is not in the standard libraries:
-- paramorphism (generalizes fold)
para :: (a -> ([a], b) -> b) -> b -> [a] -> b
para phi b = step
where step [] = b
step (x:xs) = phi x (xs, step xs)
Using a paramorphism is much like using a fold but as well as seeing the accumulator we can see the rest of input:
countSignChanges :: [Int] -> Int
countSignChanges = para phi 0
where
phi x ((y:_),st) = if signum x /= signum y then st+1 else st
phi x ([], st) = st
demo = countSignChanges [1,2,-3,4,-5,-6]
The nice thing about para compared to zipping against the tail is that we can peek as far as we want into the rest of input.
if you need to calculate value for i-th element, but depending on j-th element of the list, it's better to convert list to Array, either mutable or immutable.
So you will be able to do arbitrary computation based on index of current element either in fold, or in recursive calls.