I'm writing a Genetic Algorithm. To help play with the settings during runtime, I'm trying to save the settings (mutation rate, cross chance, fitness function...) to disk. That would allow me to modify the file, then read the file occasionally to update the settings at runtime. Getting input from the console would be easier, but Intellij's ability to take input is atrocious and proved too difficult to use, so this is the route I'm trying.
My problem is this: The settings are represented by a record. Many of the individual settings are numbers, so they're easy to save to disk. Some settings however are functions, which don't save nicely:
(defrecord Settings [
gene-set
sequence-length
fitness-f])
(with-out-str (pr
(->Settings #{1 2 3}
10
#(count %)))))
Yields:
"#ai.genetic_algorithm.genetic_algorithm2_2.Settings{:gene-set #{1 3 2}, :sequence-length 10, :fitness-f #object[ai.genetic_algorithm.genetic_algorithm2_2$eval776$fn__777$fn__778 0x1b30c92 \"ai.genetic_algorithm.genetic_algorithm2_2$eval776$fn__777$fn__778#1b30c92\"]}"
For the most part this is fine, except for the way the fitness function is represented. If I try to read the file back in:
(read-string
(with-out-str (pr
(->Settings #{1 2 3}
10
#(count %)))))
I get:
ai.genetic-algorithm.genetic-algorithm2-2=> RuntimeException No reader function for tag object clojure.lang.LispReader$CtorReader.readTagged (LispReader.java:1245)
And I only get this error when trying to save/read a function. I'm guessing it has no way of knowing what I'm asking it to read the function as, so it barfs.
Is there any way of reliably saving anonymous functions to file?
And yes, I know read-string is potentially dangerous. I'm using it solely for a personal project though, so I'm not worried about the vulnerability.
If you change your code to instead of generating a function and then later extracting that function from the namespace to save it to a file, you rather generate the expression for the settings and save the expression to a file right then. Then use the result of loading that file as the new state.
user> (let [new-config '{:gene-set #{1 2 3}
:sequence-length 10
:fitness-f (fn [x] (count x))}]
(with-open [w (clojure.java.io/writer "/tmp/config.edn")]
(binding [*out* w]
(pr new-config)))
(load-file "/tmp/config.edn"))
{:gene-set #{1 3 2},
:sequence-length 10,
:fitness-f #function[user/eval111126/fn--111127]}
when you are defining the new fitness function to save, generate it as a sequence of symbols and save that sequence before it is compiled into a function class. Let the only time it is compiled be when it is loaded so you know it will always be something loadable. This will catch bugs.
If you find yourself wanting to mechanically generate compositions of these you may want to consider giving them names an only saving the composition order in the config.
Related
I'm learning Clojure by following the Hackerrank 30 days of code, and lost some hours due to a behavior I neither understand nor found any documentation or explanation about:
(read) returns a symbol:
user=> (def key-read (read))
sam
#'user/key-read
user=> (type key-read)
clojure.lang.Symbol
(read-line) returns a string
user=> (def key-line (read-line))
sam
#'user/key-line
user=> (type key-line)
java.lang.String
As a result, parsing a line with (read) (read) to get map keys and values results in the keys to be symbols, than will never be matched by a further (read-line).
Why is this so? And also, where can I find the return value? (this is not documented in (doc read)).
TL;DR
clojure.core/read is used to read "code" by Clojure itself
clojure.edn/read is used to read "data" (EDN)
read-line is used to read text lines as string; it's your problem to
decipher them
What can read do for you
read does not only read symbols, but anything, that Clojure uses to
represent code. If you give it a symbol to parse, it will give you
symbol back:
(type (read))
test
clojure.lang.Symbol
But also other things
(type (read))
5
java.lang.Long
(type (read))
{:a 42}
clojure.lang.PersistentArrayMap
(type (read))
"hello"
java.lang.String
So you can get back a string with read too, if you feed it a string.
real-world use of read
Usually read is used by Clojure itself and that's it. Reading
EDN is usually done using clojure.edn/read, which does not allow code
execution and therefor is no security risk if handling EDN from
untrusted sources.
docs
For good measure, here are the docs:
(doc read)
-------------------------
clojure.core/read
([] [stream] [stream eof-error? eof-value] [stream eof-error? eof-value recursive?] [opts stream])
Reads the next object from stream, which must be an instance of
java.io.PushbackReader or some derivee. stream defaults to the
current value of *in*.
Opts is a persistent map with valid keys:
:read-cond - :allow to process reader conditionals, or
:preserve to keep all branches
:features - persistent set of feature keywords for reader conditionals
:eof - on eof, return value unless :eofthrow, then throw.
if not specified, will throw
Note that read can execute code (controlled by *read-eval*),
and as such should be used only with trusted sources.
For data structure interop use clojure.edn/read
(doc read-line)
-------------------------
clojure.core/read-line
([])
Reads the next line from stream that is the current value of *in* .
You can find the Clojure API documentation at https://clojure.github.io/clojure/clojure.core-api.html. Both read and read-line are there.
Your specific goal isn't quite clear, but in general, application software prefers read-line and parses the results in whatever way makes sense... perhaps with re-matches for regular expressions. Clojure itself reads program code with read.
The other 2 answers are good. I didn't even know that clojure.core/read existed!
I only wanted to add in a list of my favorite documentation sources.
Please review & study the Clojure CheatSheet, which links to examples on clojuredocs.org.
Unfortunately, the API docs at clojure.org are not as descriptive and it is harder to find things unless you already know the name and location.
for an assignment I need to create a map from a text file in clojure, which I am new to. I'm specifically using a hash-map...but it's possible I should be using another type of map. I'm hoping someone here can answer that for me. I did try changing my hash-map to sorted-map but it gave me the same problem.
The first character in every line in the file is the key and the whole line is the value. The key is a number from 0-9999. There are 10,000 lines and each number after the first number in a line is a random number between 0 and 9999.
I've created the hashmap successfully I think. At least, its not giving me an error when I just run that code. However when I try to iterate through it, printing every value for keys 0-9999 it gives me a stack overflow error right at the middle of line 2764(in the text file). I'm hoping someone can tell me why it's doing this and a better way to do it?
Here's my code:
(ns clojure-project-441.core
(:gen-class))
(defn -main
[& args]
(def pages(def hash-map (file)))
(iter 0)
)
(-main)
(defn file []
(with-open [rdr (clojure.java.io/reader "pages.txt")]
(reduce conj [] (line-seq rdr))))
(defn iter [n]
(doseq [keyval (pages n)] (print keyval))
(if (< n 10000)
(iter (inc n))
)
)
here's a screenshot of my output
If it's relevant at all I'm using repl.it as my IDE.
Here are some screenshots of the text file, for clarity.
beginning of text file
where the error is being thrown
Thanks.
I think the specific problem that causes the exception to be thrown is caused because iter calls itself recursively too many times before hitting the 10,000 line limit.
There some issues in your code that are very common to all people learning Clojure; I'll try to explain:
def is used to define top-level names. They correspond with the concept of constants in the global scope on other programming languages. Think of using def in the same way you would use defn to define functions. In your code, you probably want to use let to give names to intermediate results, like:
(let [uno 1
dos 2]
(+ uno dos)) ;; returns 3
You are using the name hash-map to bind it to some result, but that will get in the way if you want to use the function hash-map that is used to create maps. Try renaming it to my-map or similar.
To call a function recursively without blowing the stack you'll need to use recur for reasons that are a bit long to explain. See the factorial example here: https://clojuredocs.org/clojure.core/recur
My advice would be to think of this assignment as a pipeline composed of the following small functions:
A function that reads the lines from the file (you already have this)
A function that, given a line, returns a pair: the first element of the pair is the first number of the line, the second element is the whole line (the input parameter) OR
A function that reads the first number of the line
To build the map, you have a few options; two off the top of my mind:
Use a loop construct and, for each line, "update" the hash-map to include a new key-value pair (the key is the first number, the value is the whole line), then return the whole hash-map you've built
Use a reduce operation: you create a collection of key-value pairs, then tell reduce to merge, one step at a time, into the original hash-map. The result is the hash-map you want
I think the key is to get familiar with the functions that you can use and build small functions that you can test in isolation and try to group them conveniently to solve your problem. Try to get familiar with functions like hash-map, assoc, let, loop and recur. There's a great documentation site at https://clojuredocs.org/ that also includes examples that will help you understand each function.
I'm building a ClojureScript app and I'm having trouble with using reagent to fill a table with data. The two issues I'm having are TONS of warnings of the form
Every element in a seq should have a unique :key
And also, as soon as I call the function that does the rendering, it renders correctly, then my entire page freezes and reloading the page is the only way to fix it. These are my two functions:
(defn foo
[]
[:table
(for [i (range 10)]
[:tr (for [j (range 3)]
[:td (str "Row " i ", Col " j)])])])
And when I call the following, I get the warnings and the page freezes, though it does render correctly:
(reagent/render [foo] (dom/getElement "results"))
Am I approaching the process of filling in data the wrong way? Is there an easier way?
The warning you are getting is due to reagent requiring a unique key value for dynamic elements created along these lines. There are a couple of ways you can fix this. The other thing you need to watch out for is possible issues with using for because it generates lazy sequences. While this is working for you in this context, it can create subtle issues with re-rendering.
My advice would be to create a function to render the td element and prefix the rendering with
^{:key (str i j)} [:td (str "Row " i ", Col " j)]
The (str i j) will create a unique key for each td element. The other thing I find useful is to use into i.e.
(into [:tr]
(for [j (range 3)]
^{:key (str i j)} [:td ....])))
I've been developing my own app using reagent. It isn't great code and it still needs a lot of re-factoring, but I have done tables like this as well as paginated tables and a few other reagent components, such as tabs, sidebar menus etc. It can be found at my github arcis project It should give you some ideas if nothing else
There is also some good documentation regarding reagent and how it does rendering and some of the subtle 'gotchas' at re-frame wiki
I am trying to 'purify' some of my Clojure functions. I would like to get to a point where all my side-effecting code is explicitly declared in one function. It's easy to get some data at the start and to write it to a db at the end and just have a pure function transforming in between. However, the usual situation is that the transformation function requires another DB read somewhere in the middle of the logic:
(defn transform-users
[users]
(let [ids (map :id users)
profiles (db/read :profiles ids)]
(profiles->something profiles)))
(->> (db/read :users)
(transform-users)
(db/write :something)
Obviously this is a very simple example but the point is, how do I get the side-effecting db/read function out of there, how can I make transform-users pure (and as a benefit, easily testable)?
One thing you could do here would be a dependency-injection-like approach of supplying the (potentially) side-effectful function as an optional parameter, e.g.:
(defn transform-users
[users & {:keys [ids->profiles]
:or {ids->profiles #(db/read :profiles %)}]
(let [ids (map :id users)
profiles (ids->profiles ids)]
(profiles->something profiles)))
This should be easily testable since you can mock the injected functions without a lot of effort. And as a bonus, by supplying the default value, you're documenting what you're expecting and making the function convenient to call.
Why couple the reading of the profiles with transforming profiles?
(->> (db/read :users)
(map :id)
(db/read :profiles)
(profile->something)
(db/write :something)
(This also exposes the fact that you are doing two round trips to the db. Where is db/read :user-profiles ?)
(->> (db/read :user-profiles)
(profile->something)
(db/write :something)
or perhaps:
(->> (read-profiles-from-users)
(profile->something)
(db/write :something)
I am working through the first edition of this book and while I enjoy it, some of the examples given seem out-dated. I would give up and find another book to learn from, but I am really interested in what the author is talking about and want to make the examples work for myself, so I am trying to update them as I go along.
The following code is a map/reduce approach to analyzing text that depends on clojure.contrib. I have tried changing the .split function to re-seq with #"\w+", used line-seq instead of read-lines, and changed the .toLowerCase to string/lower-case. I tried to follow my problems to the source code and read the docs thoroughly to learn that the read-lines function closes after you consume the entire sequence and that line-seq returns a lazy sequence of strings, implementing java.io.BufferedReader. The most helpful thing for my problem was post about how to read files after clojure 1.3. Even still, I can't get it to work.
So here's my question: What dependencies and/or functions do I need to change in the following code to make it contemporary, reliable, idiomatic Clojure?
First namespace:
(ns chapter-data.word-count-1
(:use clojure.contrib.io
clojure.contrib.seq-utils))
(defn parse-line [line]
(let [tokens (.split (.toLowerCase line) " ")]
(map #(vector % 1) tokens)))
(defn combine [mapped]
(->> (apply concat mapped)
(group-by first)
(map (fn [[k v]]
{k (map second v)}))
(apply merge-with conj)))
(defn map-reduce [mapper reducer args-seq]
(->> (map mapper args-seq)
(combine)
(reducer)))
(defn sum [[k v]]
{k (apply + v)})
(defn reduce-parsed-lines [collected-values]
(apply merge (map sum collected-values)))
(defn word-frequency [filename]
(map-reduce parse-line reduce-parsed-lines (read-lines filename)))
Second namespace:
(ns chapter-data.average-line-length
(:use rabbit-x.data-anal
clojure.contrib.io))
(def IGNORE "_")
(defn parse-line [line]
(let [tokens (.split (.toLowerCase line) " ")]
[[IGNORE (count tokens)]]))
(defn average [numbers]
(/ (apply + numbers)
(count numbers)))
(defn reducer [combined]
(average (val (first combined))))
(defn average-line-length [filename]
(map-reduce parse-line reducer (read-lines filename)))
But when I compile and run it in light table I get a bevy of errors:
1) In the word-count-1 namespace I get this when I try to reload the ns function after editing:
java.lang.IllegalStateException: spit already refers to: #'clojure.contrib.io/spit in namespace: chapter-data.word-count-1
2) In the average-line-length namespace I get similar name collision errors under the same circumstances:
clojure.lang.Compiler$CompilerException: java.lang.IllegalStateException: parse-line already refers to: #'chapter-data.word-count-1/parse-line in namespace: chapter-data.average-line-length, compiling:(/Users/.../average-line-length.clj:7:1)
3) Oddly, when I quit and restart light table, copy and paste the code directly into the files (replacing what's there) and call instances of their top level functions the word-count-1 namespace runs fine, giving me the number of occurrences of certain words in the test.txt file but the average-line-length name-space gives me this:
"Warning: *default-encoding* not declared dynamic and thus is not dynamically rebindable, but its name suggests otherwise. Please either indicate ^:dynamic *default-encoding* or change the name. (clojure/contrib/io.clj:73)...
4) At this point when I call the word-frequency functions of the first namespace it returns nil instead of the number of word occurrences and when I call the average-line-length function of the second namespace it returns
java.lang.NullPointerException: null
core.clj:1502 clojure.core/val
As far as I can tell, clojure.contrib.io and clojure.contrib.seq-utils are no longer updated, and in fact they may be conflicting with clojure.core functions like spit. I would recommend taking out those dependencies and seeing if you can do this using only core functions. spit should just work -- the error that you're getting is caused by useing clojure.contrib.io, which contains its own spit function, which looks to be roughly equivalent; perhaps the current version in clojure.core is a "new and improved" version of clojure.contrib.io/spit.
Your problem with the parse-line function looks to be caused by the fact that you've defined two functions with the same name, in two different namespaces. The namespaces don't depend on one another, but you can still run into a conflict if you load both namespaces in a REPL. If you only need to use one at a time, try using one of them, and then when you want to use the other one, make sure you do a (remove-ns name-of-first-ns) first to free up the vars so there is no conflict. Alternatively, you could make parse-line a private function in each namespace, by changing (defn parse-line ... to (defn- parse-line ....
EDIT: If you still need any functions that were in clojure.contrib.io or clojure.contrib.seq-utils that aren't available in core or elsewhere, you can always copy the source over into your namespace. See clojure.contrib.io and clojure.contrib.seq-utils on github.