I am learning clojure, and implementing my standard test project, Tic Tac Toe 10. I have written the same AI in many languages before and have had problems scaling it beyond 6 moves ahead in the other languages also.
I got the AI algorithm basically working, but I'm trying to improve the speed with pmap. Since everything is immutable, I should be able to just drop in pmap in place of map and get the same results, but I'm not seeing that.
(defn get-spot
[board player win-cond levels]
(def avail (get-available board))
(def final
(apply merge
(map #(array-map % (calc-score board player win-cond levels %)) avail)))
final)
But pmap in that spot returns inconsistent results. Not sure where to start looking. I can post more of the code if it's needed.
Replacing def with let everywhere solved the problem. My functions had lots of inconsistent side effects if I didn't use let.
(defn get-spot
[board player win-cond levels]
(let [avail (get-available board)
final (apply merge
(pmap #(array-map % (calc-score board player win-cond levels %)) avail))]
final))
I've written a bunch of clojure code, and read through many tutorials. I don't know how I missed that really important detail.
Please read a book or two about clojure before you even consider to do performance tuning. Having immutability built into a language does not mean you can just replace map by pmap.
Two more things to consider are the absence of side effects for your code and the fact that operations will have to be commutativ.
If they are not, you will have to consider this in your merge algorithm.
As others already said, don't create def bindings insider functions, use let instead. Also, the last expression will be returned from a function anyway, so you don't need to bind final before returning it.
I am sorry for not giving a direct solution to your problem, I just think you need to understand that replacement of map is not as easy as it may seem.
If you still want your problem solved I guess we need to see more code here.
Related
I have recently watched Rich Hickeys talk at Cojure Conj 2016 and although it was very interesting, I didn't really understand the point in clojure.spec or when you'd use it. It seemed like most of the ideas, such as conform, valid etc, had similar functions in Clojure already.
I have only been learning clojure for around 3 months now so maybe this is due to lack of programming/Clojure experience.
Do clojure.spec and cljs.spec work in similar ways to Clojure and Cljs in that, although they are not 100% the same, they are based on the same underlying principles.
Are you tired of documenting your programs?
Does the prospect of making up yet more tests cause procrastination?
When the boss says "test coverage", do you cower with fear?
Do you forget what your data names mean?
For smooth expression of hard specifications, you need Clojure.Spec!
Clojure.spec gives you a uniform method of documenting, specifying, and automatically testing your programs, and of validating your live data.
It steals virtually every one of its ideas. And it does nothing you can't do for yourself.
But in my - barely informed - opinion, it changes the economy of specification, making it worth while doing properly. A game-changer? - quite possibly.
At the clojure/conj conference last week, probably half of the presentations featured spec in some way, and it's not even out of alpha yet. spec is a major feature of clojure; it is here to stay, and it is powerful.
As an example of its power, take static type checking, hailed as a kind of safety net by so many, and a defining characteristic of so many programming languages. It is incredibly limited in that it's only good at compile time, and it only checks types. spec, on the other hand, validates and conforms any predicate (not just type) for the args, the return, and can also validate relationships between the two. All of this is external to the function's code, separating the logic of the function from being commingled with validation and documentation about the code.
Regarding WORKFLOW:
One archetypal example of the benefits of relationship-checking, versus only type-checking, is a function which computes the substring of a string. Type checking ensures that in (subs s start end) the s is a string and start and end are integers. However, additional checking must be done within the function to ensure that start and end are positive integers, that end is greater than start, and that the resulting substring is no larger than the original string. All of these things can be spec'd out, for example (forgive me if some of this is a bit redundant or maybe even inaccurate):
(s/fdef clojure.core/subs
:args (s/and (s/cat :s string? :start nat-int? :end (s/? nat-int?))
(fn [{:keys [s start end]}]
(if end
(<= 0 start end (count s))
(<= 0 start (count s)))))
:ret string?
:fn (fn [{{:keys [s start end]} :args, substring :ret}]
(and (if end
(= (- end start) (count substring))
(= (- (count s) start) (count substring)))
(<= (count substring) (count s)))))
Call the function with sample data meeting the above args spec:
(s/exercise-fn `subs)
Or run 1000 tests (this may fail a few times, but keep running and it will work--this is due to the built-in generator not being able to satisfy the second part of the :args predicate; a custom generator can be written if needed):
(stest/check `subs)
Or, want to see if your app makes calls to subs that are invalid while it's running in real time? Just run this, and you'll get a spec exception if the function is called and the specs are not met:
(stest/instrument `subs)
We have not integrated this into our work flow yet, and can't in production since it's still alpha, but the first goal is to write specs. I'm putting them in the same namespace but in separate files currently.
I foresee our work flow being to run the tests for spec'd functions using this (found in the clojure spec guide):
(-> (stest/enumerate-namespace 'user) stest/check)
Then, it would be advantageous to turn on instrumenting for all functions, and run the app under load as we normally would test it, and ensure that "real world" data works.
You can also use s/conform to destructure complex data in functions themselves, or use s/valid as pre- and post- conditions for running functions. I'm not too keen on this, as it's overhead in a production system, but it is a possibility.
The sky's the limit, and we've just scratched the surface! Cool things coming in the next months and years with spec!
Clojure has a number of libraries for generative testing such as test.check, test.generative or data.generators.
It is possible to use higher order functions to create random data generators that are composable such as:
(defn gen [create-fn content-fn lazy]
(fn [] (reduce #(create-fn %1 %2) (for [a lazy] (content-fn)))))
(def a (gen str #(rand-nth [\a \b \c]) (range 10)))
(a)
(def b (gen vector #(rand-int 10) (range 2)))
(b)
(def c (gen hash-set b (range (rand-int 10))))
(c)
This is just an example and could be modified with different parameters, filters, partials, etc to create data generating functions which are quite flexible.
Is there something that any of the generative libraries can do that isn't also just as (or more) succinctly achievable by composing some higher order functions?
As a side note to the stackoverflow gods: I don't believe this question is subjective. I'm not asking for an opinion on which library is better. I want to know what specific feature(s) or technique(s) of any/all data generative libraries differentiate them from composing vanilla higher order functions. An example answer should illustrate generating random data using any of the libraries with an explanation as to why this would be more complex to do by composing HOFs in the way I have illustrated above.
test.check does this way better. Most notably, suppose you generate a random list of 100 elements, and your test fails: something about the way you handled that list is wrong. What now? How do you find the basic bug? It surely doesn't depend on exactly those 100 inputs; you could probably reproduce it with a list of just a few elements, or even an empty list if something is wrong with your base case.
The feature that makes all this actually useful isn't the random generators, it is the "shrinking" of those generators. Once test.check finds an input that breaks your tests, it tries to simplify the input as much as possible while still making your tests break. For a list of integers, the shrinks are simple enough you could maybe do them yourself: remove any element, or decrease any element. Even that may not be true: choosing the order to do shrinks in is probably a harder problem than I realize. And for larger inputs, like a list of maps from vectors to a 3-tuple of [string, int, keyword], you'll find it totally unmanageable, whereas test.check has done all the hard work already.
I've been developing in Java and Perl for a long time but wanted to learn something new so I've begun looking into clojure. One of the first things I've tried was a solution for the Towers of Hanoi puzzle but I've been getting strange behavior on my pretty-print function. Basically, my for loop is never entered when I run it with 'lein run' but it seems to work fine when I run it from the repl. Here's a stripped down example:
(ns test-app.core
(:gen-class))
(defn for-print
"Print the same thing 3 times"
[ p-string ]
(println (str "Checkpoint: " p-string))
(for
[x [1 2 3]]
(printf "FOR: %s\n" p-string)
))
(defn -main
"I don't do a whole lot ... yet."
[& args]
(for-print "Haldo wurld!"))
When I run this with 'lein run' I only see the output from the "Checkpoint" println. If I remove that line I get no output at all. But, if I run 'lein repl' and then type (-main) it prints the string 3 times, as expected:
test-app.core=> (-main)
Checkpoint: Haldo wurld!
(FOR: Haldo wurld!
FOR: Haldo wurld!
FOR: Haldo wurld!
nil nil nil)
test-app.core=>
What's going on here? I have a feeling that I'm approaching this the wrong way, trying to use my past Perl/Java mentality to write clojure. What would be an idiomatic way to run the same task a set number of times?
The for loop is returning a lazy sequence that is evaluated only as needed.
When you run the program inside the repl the for result is realized in order to display the result on screen.
But when you run using lein run result is never used so the collection is not realized.
You have a couple alternatives:
1) use doall outside the for loop for force lazy sequence realization
Ex:
(defn for-print
"Print the same thing 3 times"
[ p-string ]
(println (str "Checkpoint: " p-string))
(doall (for
[x [1 2 3]]
(printf "FOR: %s\n" p-string))))
2) since you're only printing which is a side effect and not really creating a collection you can use doseq.
Ex:
(defn for-print
"Print the same thing 3 times"
[ p-string ]
(println (str "Checkpoint: " p-string))
(doseq [x [1 2 3]]
(printf "FOR: %s\n" p-string)))
Clojure for is not an imperative loop (you should avoid thinking about loops in Clojure at all), it's a list comprehension, which returns lazy sequence. It's made for creating a sequence, not for printing anything. You can realize it, and make it work, but it is bad way to go.
As Guillermo said, you're looking for doseq macro, which is designed for side effects. It's probably the most idiomatic Clojure in your particular case.
From my point of view the most similar to imperative loops construction in Clojure is tail recursion, made with loop/recur. Still it's rather low level construct in Clojure and certainly should not be used in a imperative loop manner. Have a better look into functional programming principles, as well as Clojure core functions. Trying to transfer Java/Perl thinking to Clojure may harm you.
The other answers are correct, and provide details. I want to add some higher-level clarification that might be helpful. In most languages, "for" means "for such and conditions, perform such and such actions (which could be of an arbitrary type, including side-effects)." This kind of thing can be done in Clojure, and I have seen experienced Clojure programmers do it when it's useful and convenient. However, using loops with side-effects works usually against the strengths of the language. So in Clojure the word "for" has a different meaning in most languages: Generating (lazy) sequences. It means something like "For these inputs, when/while/etc. they meet such and such conditions, bind then temporarily to these variables, generate a (lazy) sequence by processing each set of values in such and such a way."
This is a follow up to my question "Recursively reverse a sequence in Clojure".
Is it possible to reverse a sequence using the Clojure "for" macro? I'm trying to better understand the limitations and use-cases of this macro.
Here is the code I'm starting from:
((defn reverse-with-for [s]
(for [c s] c))
Possible?
If so, I assume the solution may require wrapping the for macro in some expression that defines a mutable var, or that the body-expr of the for macro will somehow pass a sequence to the next iteration (similar to map).
Clojure for macro is being used with arbitrary Clojure sequences.
These sequences may or may not expose random access like vectors do. So, in general case, you do not have access to the last element of a Clojure sequence without traversing all the way to it, which would make making a pass through it in reverse order not possible.
I'm assumming you had something like this in mind (Java-like pseudocode):
for(int i = n-1; i--; i<=0){
doSomething(array[i]);
}
In this example we know array size n in advance and we can access elements by its index. With Clojure sequences we don't know that. In Java it makes sense to do that with arrays and ArrayLists. Clojure sequences are however much more like linked lists - you have an element, and a reference to next one.
Btw, even if there were a (probably non-idiomatic)* way to do that, its time complexity would be something like O(n^2) which is just not worth the effort compared to much easier solution in the linked post which is O(n^2) for lists and a much better O(n) for vectors (and it is quite elegant and idiomatic. In fact, the official reverse has that implementation).
EDIT:
A general advice: Don't try to do imperative programming in Clojure, it wasn't designed for it. Although many things may seem strange or counter-intuitive (as opposed to well known idioms from imperative programming) once you get used to the functional way of doing things it is a lot, and I mean a lot easier.
Specifically for this question, despite the same name Java (and other C-like) for and Clojure for are not the same thing! First is an actual loop - it defines a flow control. The second one is a comprehension - look at it conceptually as a higher function of a sequence and a function f to be done for each of its element, which returns another sequence of f(element) s. Java for is a statement, it doesn't evaluate to anything, Clojure for (as well as anything else in Clojure) is an expression - it evaluates to the sequence of f(element) s.
Probably the easiest way to get the idea is to play with sequence functions library: http://clojure.org/sequences. Also, you can solve some problems on http://www.4clojure.com/. The first problems are very easy but they gradually get harder as you progress through them.
*As shown in Alexandre's answer the solution to the problem in fact is idiomatic and quite clever. Kudos for that! :)
Here's how you could reverse a string with for:
(defn reverse-with-for [s]
(apply str
(for [i (range (dec (count s)) -1 -1)]
(get s i))))
Note that this code is mutation free. It's the same as:
(defn reverse-with-map [s]
(apply str
(map (partial get s) (range (dec (count s)) -1 -1))))
A simpler solution would be:
(apply str (reverse s))
First of all, as Goran said, for is not a statement - it is an expression, namely sequence comprehension. It construct sequences by iteration through other sequences. So in the form it is meant to be used it is pure function (without side-effects). for can be seen as enhanced map infused with filter. Because of this it cannot be used to hold iteration state as e.g. reduce do.
Secondly, you can express sequence reversal using for and mutable state, e.g. using an atom, which is rough equivalent (not taking into account its concurrency properties) of java variable. But doing so you are facing several problems:
You are breaking main language paradigm so you will definitely get worse looking and behaving code.
Since all clojure mutable state cells are designed to be thread-safe, they all use some kind of illegal concurrent modification protection, and there is no ability to remove it. Consequently, you will get poorer performance characteristics.
In this particular case, like Goran said, sequences are one of the wide-used Clojure abstractions. For example, there are lazy sequences, which could be potentially infinite, so you just cannot walk them to the end. You certainly will have difficulties trying to work with such sequences with imperative techniques.
So don't do it, at least in Clojure :)
EDIT: I forgot to mention it. for returns lazy sequence, so you have to evaluate it in some way in order to apply all state mutations you do in it. Another reason not to do so :)
I have a defrecord called a bag. It behaves like a list of item to count. This is sometimes called a frequency or a census. I want to be able to do the following
(def b (bag/create [:k 1 :k2 3])
(keys bag)
=> (:k :k1)
I tried the following:
(defrecord MapBag [state]
Bag
(put-n [self item n]
(let [new-n (+ n (count self item))]
(MapBag. (assoc state item new-n))))
;... some stuff
java.util.Map
(getKeys [self] (keys state)) ;TODO TEST
Object
(toString [self]
(str ("Bag: " (:state self)))))
When I try to require it in a repl I get:
java.lang.ClassFormatError: Duplicate interface name in class file compile__stub/techne/bag/MapBag (bag.clj:12)
What is going on? How do I get a keys function on my bag? Also am I going about this the correct way by assuming clojure's keys function eventually calls getKeys on the map that is its argument?
Defrecord automatically makes sure that any record it defines participates in the ipersistentmap interface. So you can call keys on it without doing anything.
So you can define a record, and instantiate and call keys like this:
user> (defrecord rec [k1 k2])
user.rec
user> (def a-rec (rec. 1 2))
#'user/a-rec
user> (keys a-rec)
(:k1 :k2)
Your error message indicates that one of your declarations is duplicating an interface that defrecord gives you for free. I think it might actually be both.
Is there some reason why you cant just use a plain vanilla map for your purposes? With clojure, you often want to use plain vanilla data structures when you can.
Edit: if for whatever reason you don't want the ipersistentmap included, look into deftype.
Rob's answer is of course correct; I'm posting this one in response to the OP's comment on it -- perhaps it might be helpful in implementing the required functionality with deftype.
I have once written an implementation of a "default map" for Clojure, which acts just like a regular map except it returns a fixed default value when asked about a key not present inside it. The code is in this Gist.
I'm not sure if it will suit your use case directly, although you can use it to do things like
user> (:earth (assoc (DefaultMap. 0 {}) :earth 8000000000))
8000000000
user> (:mars (assoc (DefaultMap. 0 {}) :earth 8000000000))
0
More importantly, it should give you an idea of what's involved in writing this sort of thing with deftype.
Then again, it's based on clojure.core/emit-defrecord, so you might look at that part of Clojure's sources instead... It's doing a lot of things which you won't have to (because it's a function for preparing macro expansions -- there's lots of syntax-quoting and the like inside it which you have to strip away from it to use the code directly), but it is certainly the highest quality source of information possible. Here's a direct link to that point in the source for the 1.2.0 release of Clojure.
Update:
One more thing I realised might be important. If you rely on a special map-like type for implementing this sort of thing, the client might merge it into a regular map and lose the "defaulting" functionality (and indeed any other special functionality) in the process. As long as the "map-likeness" illusion maintained by your type is complete enough for it to be used as a regular map, passed to Clojure's standard function etc., I think there might not be a way around that.
So, at some level the client will probably have to know that there's some "magic" involved; if they get correct answers to queries like (:mars {...}) (with no :mars in the {...}), they'll have to remember not to merge this into a regular map (merge-ing the other way around would work fine).