Found some odd behavior in midje, not sure if it's midje related, or due to my misunderstanding of some clojure constructs, but it's puzzling:
Inside a facts statement, a for loop is not getting called:
(ns t1
(:require [midje.sweet :refer :all ] )
)
(facts
(println "ok") ; -- this prints fine
(for [val '(1 2 3)] (println val)) ; this does not
(fact "junk"
(> (.length "aaaaha") 3) => true ))
Thought maybe it had something to do with the for being overwritten in the ns but calling clojure.core/for behaves similarly.
clojure.core/for "...yields a lazy sequence..."
You need to realize the sequence to see its side effects.
(doall (for [val '(1 2 3)] (println val)))
I'd suggest using something more appropriate like clojure.core/doseq:
(doseq [val '(1 2 3)] (println val))
Related
I'm learning core.async and have written a simple producer consumer code:
(ns webcrawler.parallel
(:require [clojure.core.async :as async
:refer [>! <! >!! <!! go chan buffer close! thread alts! alts!! timeout]]))
(defn consumer
[in out f]
(go (loop [request (<! in)]
(if (nil? request)
(close! out)
(do (print f)
(let [result (f request)]
(>! out result))
(recur (<! in)))))))
(defn make-consumer [in f]
(let [out (chan)]
(consumer in out f)
out))
(defn process
[f s no-of-consumers]
(let [in (chan (count s))
consumers (repeatedly no-of-consumers #(make-consumer in f))
out (async/merge consumers)]
(map #(>!! in %1) s)
(close! in)
(loop [result (<!! out)
results '()]
(if (nil? result)
results
(recur (<!! out)
(conj results result))))))
This code works fine when I step in through the process function in debugger supplied with Emacs' cider.
(process (partial + 1) '(1 2 3 4) 1)
(5 4 3 2)
However, if I run it by itself (or hit continue in the debugger) I get an empty result.
(process (partial + 1) '(1 2 3 4) 1)
()
My guess is that in the second case for some reason producer doesn't wait for consumers before exiting, but I'm not sure why. Thanks for help!
The problem is that your call to map is lazy, and will not run until something asks for the results. Nothing does this in your code.
There are 2 solutions:
(1) Use the eager function mapv:
(mapv #(>!! in %1) items)
(2) Use the doseq, which is intended for side-effecting operations (like putting values on a channel):
(doseq [item items]
(>!! in item))
Both will work and produce output:
(process (partial + 1) [1 2 3 4] 1) => (5 4 3 2)
P.S. You have a debug statement in (defn consumer ...)
(print f)
that produces a lot of noise in the output:
<#clojure.core$partial$fn__5561 #object[clojure.core$partial$fn__5561 0x31cced7
"clojure.core$partial$fn__5561#31cced7"]>
That is repeated 5 times back to back. You probably want to avoid that, as printing function "refs" is pretty useless to a human reader.
Also, debug printouts in general should normally use println so you can see where each one begins and ends.
I'm going to take a safe stab that this is being caused by the lazy behavior of map, and this line that's carrying out side effects:
(map #(>!! in %1) s)
Because you never explicitly use the results, it never runs. Change it to use mapv, which is strict, or more correctly, use doseq. Never use map to run side effects. It's meant to lazily transform a list, and abuse of it leads to behaviour like this.
So why is it working while debugging? I'm going to guess because the debugger forces evaluation as part of its operation, which is masking the problem.
As you can read from docstring map returns a lazy sequence. And I think the best way is to use dorun. Here is an example from clojuredocs:
;;map a function which makes database calls over a vector of values
user=> (map #(db/insert :person {:name %}) ["Fred" "Ethel" "Lucy" "Ricardo"])
JdbcSQLException The object is already closed [90007-170] org.h2.message.DbE
xception.getJdbcSQLException (DbException.java:329)
;;database connection was closed before we got a chance to do our transactions
;;lets wrap it in dorun
user=> (dorun (map #(db/insert :person {:name %}) ["Fred" "Ethel" "Lucy" "Ricardo"]))
DEBUG :db insert into person values name = 'Fred'
DEBUG :db insert into person values name = 'Ethel'
DEBUG :db insert into person values name = 'Lucy'
DEBUG :db insert into person values name = 'Ricardo'
nil
I want to check if the right instance is used.
But while the repl gives me true, the actual test returns nil.
Idk why
(ns my-app.queue)
(def queue (atom clojure.lang.PersistentQueue/EMPTY))
(ns my-app.queue-test
(:require [my-app.queue :as sut]
[clojure.test :refer :all]))
(deftest queue-type-test
(is (instance? clojure.lang.PersistentQueue #sut/queue)))
;repl output
;(instance? clojure.lang.PersistentQueue #sut/queue) ;=> true
The command lein test gives me:
FAIL in (queue-type-test) (queue_test.clj:6)
expected: (instance? clojure.lang.PersistentQueue (clojure.core/deref sut/queue))
actual: nil
I know the test itself is not really useful, but I can't figure out why this behaves like this.
tl;dr: both is and # are macros. Most obvious fix would be something like:
(deftest queue-type-test
(let [queue #sut/queue]
(is (instance? clojure.lang.PersistentQueue queue)))
clojure.test docs don't make it immediately obvious, but is is a pretty tricky macro that looks inside its body. Here's a snippet from the docstring:
user> (is (= 5 (+ 2 2)))
FAIL in (:1)
expected: (= 5 (+ 2 2))
actual: (not (= 5 4))
false
Notice actual: (not (= 5 4)). It's clear that it did not evaluate = so that it can show us the result of (+ 2 2). Here are all the special cases for assert-expr multimethod.
If any step in my test setup fails, I want to report this as a failure, and cease any subsequent tests in the current deftest block (or current namespace). One way to do this now:
(if some-condition-is-ok
(do
... do tests)
(is (= 1 0) "Failure, condition not met")
The above:
Reports failure if some-condition-is-ok is not met
Does not run any tests, since the setup condition was not met
Except that it does not flow well, and does not work well for multiple conditions. I'd like something like:
(let [;; setup here...]
(assert-or-stop-tests some-condition-is-ok)
... continue with tests here
Any ideas on a clean way to do this?
You could use Mark Engelberg's better-cond for this:
(require '[better-cond.core :as b]
'[clojure.test :refer [is]])
(def some-condition-is-ok true)
(def some-other-condition-is-ok false)
(deftest a-test
(b/cond
:let [#_"setup here..."]
:when (is some-condition-is-ok)
:let [_ (is (= 0 1))]
:when (is some-other-condition-is-ok)
:let [_ (is (= 1 2))]))
Or if you want to avoid the :let [_ ,,,], you could define your own macro:
(defmacro ceasing [& exprs]
(when-let [[left & [right & less :as more]] (seq exprs)]
(if (= :assert left)
`(when (is ~right)
(ceasing ~#less))
`(do
~left
(ceasing ~#more)))))
(deftest b-test
(let [#_"setup here..."]
(ceasing
:assert some-condition-is-ok
(is (= 0 1))
:assert some-other-condition-is-ok
(is (= 1 2)))))
For the most part I understand what Clojure is telling me with it's error messages. But I am still clueless as to find out where the error happened.
Here is an example of what I mean
(defn extract [m]
(keys m))
(defn multiple [xs]
(map #(* 2 %) xs))
(defn process [xs]
(-> xs
(multiple) ; seq -> seq
(extract))) ; map -> seq ... fails
(process [1 2 3])
Statically typed languages would now tell me that I tried to pass a sequence to a function that expects a map on line X. And Clojure does this in a way:
ClassCastException java.lang.Long cannot be cast to java.util.Map$Entry
But I still have no idea where the error happened. Obviously for this instance it's easy because there are just 3 functions involved, you can easily just read through all of them but as programs grow bigger this gets old very quickly.
Is there a way find out where the errors happened other than just proof reading the code from top to bottom? (which is my current approach)
You can use clojure.spec. It is still in alpha, and there's still a bunch of tooling support coming (hopefully), but instrumenting functions works well.
(ns foo.core
(:require
;; For clojure 1.9.0-alpha16 and higher, it is called spec.alpha
[clojure.spec.alpha :as s]
[clojure.spec.test.alpha :as stest]))
;; Extract takes a map and returns a seq
(s/fdef extract
:args (s/cat :m map?)
:ret seq?)
(defn extract [m]
(keys m))
;; multiple takes a coll of numbers and returns a coll of numbers
(s/fdef multiple
:args (s/cat :xs (s/coll-of number?))
:ret (s/coll-of number?))
(defn multiple [xs]
(map #(* 2 %) xs))
(defn process [xs]
(-> xs
(multiple) ; seq -> seq
(extract))) ; map -> seq ... fails
;; This needs to come after the definition of the specs,
;; but before the call to process.
;; This is something I imagine can be handled automatically
;; by tooling at some point.
(stest/instrument)
;; The println is to force evaluation.
;; If not it wouldn't run because it's lazy and
;; not used for anything.
(println (process [1 2 3]))
Running this file prints (among other info):
Call to #'foo.core/extract did not conform to spec: In: [0] val: (2
4 6) fails at: [:args :m] predicate: map? :clojure.spec.alpha/spec
#object[clojure.spec.alpha$regex_spec_impl$reify__1200 0x2b935f0d
"clojure.spec.alpha$regex_spec_impl$reify__1200#2b935f0d"]
:clojure.spec.alpha/value ((2 4 6)) :clojure.spec.alpha/args ((2 4
6)) :clojure.spec.alpha/failure :instrument
:clojure.spec.test.alpha/caller {:file "core.clj", :line 29,
:var-scope foo.core/process}
Which can be read as: A call to exctract failed because the value passed in (2 4 6) failed the predicate map?. That call happened in the file "core.clj" at line 29.
A caveat that trips people up is that instrument only checks function arguments and not return values. This is a (strange if you ask me) design decision from Rich Hickey. There's a library for that, though.
If you have a REPL session you can print a stack trace:
(clojure.stacktrace/print-stack-trace *e 30)
See http://puredanger.github.io/tech.puredanger.com/2010/02/17/clojure-stack-trace-repl/ for various different ways of printing the stack trace. You will need to have a dependency such as this in your project.clj:
[org.clojure/tools.namespace "0.2.11"]
I didn't get a stack trace using the above method, however just typing *e at the REPL will give you all the available information about the error, which to be honest didn't seem very helpful.
For the rare cases where the stack trace is not helpful I usually debug using a call to a function that returns the single argument it is given, yet has the side effect of printing that argument. I happen to call this function probe. In your case it can be put at multiple places in the threading macro.
Re-typing your example I have:
(defn extract [m]
(keys m))
(defn multiply [xs]
(mapv #(* 2 %) xs))
(defn process [xs]
(-> xs
(multiply) ; seq -> seq
(extract))) ; map -> seq ... fails ***line 21***
(println (process [1 2 3]))
;=> java.lang.ClassCastException: java.lang.Long cannot be cast
to java.util.Map$Entry, compiling:(tst/clj/core.clj:21:21)
So we get a good clue in the exception where is says the file and line/col number tst.clj.core.clj:21:21 that the extract method is the problem.
Another indispensible tool I use is Plumatic Schema to inject "gradual" type checking into clojure. The code becomes:
(ns tst.clj.core
(:use clj.core tupelo.test)
(:require
[tupelo.core :as t]
[tupelo.schema :as tsk]
[schema.core :as s]))
(t/refer-tupelo)
(t/print-versions)
(s/defn extract :- [s/Any]
[m :- tsk/Map]
(keys m))
(s/defn multiply :- [s/Num]
[xs :- [s/Num]]
(mapv #(* 2 %) xs))
(s/defn process :- s/Any
[xs :- [s/Num]]
(-> xs
(multiply) ; seq -> seq
(extract))) ; map -> seq ... fails
(println (process [1 2 3]))
clojure.lang.ExceptionInfo: Input to extract does not match schema:
[(named (not (map? [2 4 6])) m)] {:type :schema.core/error, :schema [#schema.core.One{:schema {Any Any},
:optional? false, :name m}],
:value [[2 4 6]], :error [(named (not (map? [2 4 6])) m)]},
compiling:(tst/clj/core.clj:23:17)
So, while the format of the error message is a bit lengthy, it tells right away that we passed a parameter of the wrong type and/or shape into the method extract.
Note that you need a line like this:
(s/set-fn-validation! true) ; enforce fn schemas
I create a special file test/tst/clj/_bootstrap.clj so it is always in the same place.
For more information on Plumatic Schema please see:
https://github.com/plumatic/schema
https://youtu.be/o_jtwIs2Ot8
https://github.com/plumatic/schema/wiki/Basics-Examples
https://github.com/plumatic/schema/wiki/Defining-New-Schema-Types-1.0
I am really not sure what is the problem here. I started to experience this "issue" with this kind of code:
First I did define that string with some metadata:
(def ^{:meta-attr ["foo" "bar"]
:meta-attr2 "some value"} foo "some value")
Then I did create the following two functions:
(defn second-fn [values]
(for [x values] (println x)))
(defn first-fn [value]
(doseq [[meta-key meta-val] (seq (meta value))]
(if (= meta-key :meta-attr)
(second-fn meta-val))))
Now when I run this command in the REPL:
(first-fn #'foo)
I am getting nil.
However, if I change second-fn for:
(defn second-fn [values]
(println values))
And if I run that command again, I am getting this in the REPL:
user> (first-fn #'foo)
[foo bar]
nil
What I was expecting to get in the REPL with the first version of my function is the following:
user> (first-fn #'foo)
foo
bar
nil
But somehow, I think there is something I don't get that is related to the bound variable by doseq.
Here is another set of functions that has exactly the same behavior:
(defn test-2 [values]
; (println values))
(for [x values] (println x)))
(defn test-1 [values]
(doseq [x values]
(test-2 x)))
(test-1 [["1.1" "1.2"] ["2"] ["3"]])
I think I am missing some Clojure knowledge to understand what is going on here. Why it looks like good when I println or pprint the value in the second function, but the for is not working...
Update and Final Thoughts
As answered for this question, the problem has to do with lazyness of the for function. Let's take the simplest example to illustrate what is going on.
(defn test-2 [values]
(for [x values] (println x)))
(defn test-1 [values]
(doseq [x values]
(test-2 x)))
What happens there is that in test-1, every time that doseq "iterate", then a new non-lazy sequence is being created. That means that they are accessible like any other collection during the "looping".
doseq should generally be used when you work with non-pure functions that may have side effects, or I think when you are playing with relatively small collections.
Then when test-2 is called, the for will create a lazy-seq. That means that the sequence exists, but that it never did get realized (so, each step hasn't been computed yet). As is, nothing will happen with these two functions, since none of the values returned by the for have been realized.
If we want to keep this doseq and this for loops, then we have to make sure that for get realized in test-2. We can do this that way:
(defn test-2 [values]
(doall (for [x values] (println x))))
(defn test-1 [values]
(doseq [x values]
(test-2 x)))
That doall does here, is to force the full realization of the sequence returned by the for loop. That way, we will end with the expected result.
Additionally, we could realize the lazy-seq returned by for using other functions like:
(defn test-2 [values]
(first (for [x values] (println x))))
(defn test-2 [values]
(count (for [x values] (println x))))
None of this make sense, but all of these examples for the realization of the lazy-seq returned by the for.
Additionally, we could have simply used two doseq like this:
(defn test-2 [values]
(doseq [x values] (println x)))
(defn test-1 [values]
(doseq [x values]
(test-2 x)))
That way, we don't use any lazy-seq and so we don't have to realize anything since nothing is evaluated lazilly.
for is lazy, while doseq is eager.
for is "functional" (values) and doseq is "imperative" (side-effects).
In other words, you should not be using for in second-fn, since you seem to be worried only with side-effects. What you are actually doing there is building a lazy sequence (which, it seems, is never executed).
See Difference between doseq and for in Clojure for further info.