clojure: implementing jmp in a simple stack based vm - clojure

New to clojure, I am currently implementing a simple stack based vm in clojure just for practice and fun.
I tried to be as functional as possible. And I constructed the following code:
(defmulti execute-byte (fn [stack b] b))
(defmethod execute-byte 0x00 [stack _] ;; nop
stack)
(defmethod execute-byte 0x01 [stack _] ;; unimplemented
())
(defmethod execute-byte 0x02 [stack _] ;; del
(rest stack))
(defmethod execute-byte 0x03 [stack _] ;; jmp, honestly I don't know how to implement this
())
(defmethod execute-byte 0x10 [stack _] ;; add
(let [f (first stack)
s (second stack)]
(cons (+ f s) (nthrest stack 2))))
(defmethod execute-byte 0x11 [stack _] ;; sub
(let [f (first stack)
s (second stack)]
(cons (- f s) (nthrest stack 2))))
(defmethod execute-byte 0x12 [stack _] ;; multi
(let [f (first stack)
s (second stack)]
(cons (* f s) (nthrest stack 2))))
(defmethod execute-byte 0x13 [stack _] ;; div
(let [f (first stack)
s (second stack)]
(cons (/ f s) (nthrest stack 2))))
(defmethod execute-byte :default [stack bc] ;; just a testing workaround
(cons bc stack))
(defn execute-single-stack-step
[[inst stack]]
[(rest inst) (execute-byte stack (first inst))])
(defn execute-bytes
[inst stack step]
(last (take step (iterate execute-single-stack-step [inst stack]))))
and it kind of works. "Kind of" because it currently functions only as a reversed polish notation calculator
(execute-bytes [0x50 0x50 0x10] [] 4) ;; equivalent to [0x50 0x50 +] and gets [() (160)] as a result
I want the vm at least to be able to perform "jmp", reading a value popped from the stack and move the "cursor" to where the value refers, but it seems impossible with my current design. (the current design functions only like "reduce", and there is no such "cursor" at all).
For a better representation of the method I used
(execute-bytes [0x50 0x50 0x10] [] 1)
;;[[80 80 16] []]
(execute-bytes [0x50 0x50 0x10] [] 2)
;;[(80 16) (80)]
(execute-bytes [0x50 0x50 0x10] [] 3)
;;[(16) (80 80)]
(execute-bytes [0x50 0x50 0x10] [] 4)
;;[() (160)]
so is there any way I can implement "jmp" by this method?

I managed to make it work.
For a demonstration of the result:
lunadial.logic> (execute-bytes [0x01 0x10 0x01 0x10 0x10 0x03 0x00] [] 1) ;; push 0x10 push 0x10 add jump 0x00
[[] [1 16 1 16 16 3 0] 0]
lunadial.logic> (execute-bytes [0x01 0x10 0x01 0x10 0x10 0x03 0x00] [] 2)
[(16) [1 16 1 16 16 3 0] 2]
lunadial.logic> (execute-bytes [0x01 0x10 0x01 0x10 0x10 0x03 0x00] [] 3)
[(16 16) [1 16 1 16 16 3 0] 4]
lunadial.logic> (execute-bytes [0x01 0x10 0x01 0x10 0x10 0x03 0x00] [] 4)
[(32) [1 16 1 16 16 3 0] 5]
lunadial.logic> (execute-bytes [0x01 0x10 0x01 0x10 0x10 0x03 0x00] [] 5)
[(32) [1 16 1 16 16 3 0] 0]
lunadial.logic> (execute-bytes [0x01 0x10 0x01 0x10 0x10 0x03 0x00] [] 6)
[(16 32) [1 16 1 16 16 3 0] 2]
lunadial.logic> (execute-bytes [0x01 0x10 0x01 0x10 0x10 0x03 0x00] [] 7)
[(16 16 32) [1 16 1 16 16 3 0] 4]
lunadial.logic> (execute-bytes [0x01 0x10 0x01 0x10 0x10 0x03 0x00] [] 8)
[(32 32) [1 16 1 16 16 3 0] 5]
lunadial.logic> (execute-bytes [0x01 0x10 0x01 0x10 0x10 0x03 0x00] [] 9)
[(32 32) [1 16 1 16 16 3 0] 0]
lunadial.logic> (execute-bytes [0x01 0x10 0x01 0x10 0x10 0x03 0x00] [] 10)
[(16 32 32) [1 16 1 16 16 3 0] 2]
and the code:
(defmulti execute-byte (fn [stack inst point] (nth inst point)))
(defmethod execute-byte 0x00 [stack inst point] ;; nop
[stack inst (inc point)])
(defmethod execute-byte 0x01 [stack inst point] ;; push
(let [the-bc-after (nth inst (inc point))]
[(cons the-bc-after stack) inst (inc (inc point))]))
(defmethod execute-byte 0x02 [stack inst point] ;; del
[(rest stack) inst (inc point)])
(defmethod execute-byte 0x03 [stack inst point] ;; jmp
(let [the-bc-after (nth inst (inc point))]
[stack inst the-bc-after]))
(defmethod execute-byte 0x10 [stack inst point] ;; add
(let [f (first stack)
s (second stack)]
[(cons (+ f s) (nthrest stack 2)) inst (inc point)]))
(defmethod execute-byte 0x11 [stack inst point] ;; sub
(let [f (first stack)
s (second stack)]
[(cons (- f s) (nthrest stack 2)) inst (inc point)]))
(defmethod execute-byte 0x12 [stack inst point] ;; multi
(let [f (first stack)
s (second stack)]
[(cons (* f s) (nthrest stack 2)) inst (inc point)]))
(defmethod execute-byte 0x13 [stack inst point] ;; div
(let [f (first stack)
s (second stack)]
[(cons (/ f s) (nthrest stack 2)) inst (inc point)]))
(defmethod execute-byte :default [stack inst point] ;; just a testing workaround
[(cons (nth inst point) stack) inst (inc point)])
(defn execute-single-stack-step ;; the iterator wrapper for the multimethod
[[stack inst point]]
(execute-byte stack inst point))
(defn execute-bytes
[inst stack step]
(last (take step (iterate execute-single-stack-step [stack inst 0]))))
I just need to pass all the instructions and pointer.. and everything into the function, and modify the multimethods. It does not look elegant, but it works.

Related

Macro expansion fails to compile even though it expands correctly?

As practice I wanted to implement some of the macros described in Doug Hoytes "Let over Lambda" which I've read a while ago.
When I started to play around with anaphoric macros I ran into a weird problem. I've implemented the alet macro described in the book as follows:
(defmacro a-let
"Anaphoric let, `this` refers to the last form in body, which should be a
function"
[bindings & body]
`(let [~'this (atom nil) ~#bindings]
(reset! ~'this ~(last body))
~#(butlast body)
(fn [& params]
(apply ~'#this params))))
This compiles fine. However, if I try to use it in code, for instance in this simple example
(a-let [a 1, b 2]
(fn [] (+ a b)))
Cider protests and throws an error with the stack trace below:
2. Unhandled clojure.lang.Compiler$CompilerException
Error compiling /home/dasbente/Dokumente/Informatik/Clojure/let-over-lambda.clj at (39:1)
Compiler.java: 6891 clojure.lang.Compiler/checkSpecs
Compiler.java: 6907 clojure.lang.Compiler/macroexpand1
Compiler.java: 6989 clojure.lang.Compiler/analyzeSeq
Compiler.java: 6773 clojure.lang.Compiler/analyze
Compiler.java: 6729 clojure.lang.Compiler/analyze
Compiler.java: 6100 clojure.lang.Compiler$BodyExpr$Parser/parse
Compiler.java: 6420 clojure.lang.Compiler$LetExpr$Parser/parse
Compiler.java: 7003 clojure.lang.Compiler/analyzeSeq
Compiler.java: 6773 clojure.lang.Compiler/analyze
Compiler.java: 6729 clojure.lang.Compiler/analyze
Compiler.java: 6100 clojure.lang.Compiler$BodyExpr$Parser/parse
Compiler.java: 5460 clojure.lang.Compiler$FnMethod/parse
Compiler.java: 4022 clojure.lang.Compiler$FnExpr/parse
Compiler.java: 7001 clojure.lang.Compiler/analyzeSeq
Compiler.java: 6773 clojure.lang.Compiler/analyze
Compiler.java: 7059 clojure.lang.Compiler/eval
Compiler.java: 7025 clojure.lang.Compiler/eval
core.clj: 3206 clojure.core/eval
core.clj: 3202 clojure.core/eval
main.clj: 243 clojure.main/repl/read-eval-print/fn
main.clj: 243 clojure.main/repl/read-eval-print
main.clj: 261 clojure.main/repl/fn
main.clj: 261 clojure.main/repl
main.clj: 177 clojure.main/repl
RestFn.java: 1523 clojure.lang.RestFn/invoke
interruptible_eval.clj: 87 clojure.tools.nrepl.middleware.interruptible-eval/evaluate/fn
AFn.java: 152 clojure.lang.AFn/applyToHelper
AFn.java: 144 clojure.lang.AFn/applyTo
core.clj: 657 clojure.core/apply
core.clj: 1965 clojure.core/with-bindings*
core.clj: 1965 clojure.core/with-bindings*
RestFn.java: 425 clojure.lang.RestFn/invoke
interruptible_eval.clj: 85 clojure.tools.nrepl.middleware.interruptible-eval/evaluate
interruptible_eval.clj: 55 clojure.tools.nrepl.middleware.interruptible-eval/evaluate
interruptible_eval.clj: 222 clojure.tools.nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
interruptible_eval.clj: 190 clojure.tools.nrepl.middleware.interruptible-eval/run-next/fn
AFn.java: 22 clojure.lang.AFn/run
ThreadPoolExecutor.java: 1149 java.util.concurrent.ThreadPoolExecutor/runWorker
ThreadPoolExecutor.java: 624 java.util.concurrent.ThreadPoolExecutor$Worker/run
Thread.java: 748 java.lang.Thread/run
1. Caused by clojure.lang.ExceptionInfo
Call to clojure.core/fn did not conform to spec: In: [0 1] val:
user/params fails spec: :clojure.core.specs.alpha/local-name at:
[:args :bs :arity-1 :args :varargs :form :sym] predicate:
simple-symbol? In: [0 1] val: user/params fails spec:
:clojure.core.specs.alpha/seq-binding-form at: [:args :bs :arity-1
:args :varargs :form :seq] predicate: vector? In: [0 1] val:
user/params fails spec: :clojure.core.specs.alpha/map-bindings at:
[:args :bs :arity-1 :args :varargs :form :map] predicate: coll?
In: [0 1] val: user/params fails spec:
:clojure.core.specs.alpha/map-special-binding at: [:args :bs
:arity-1 :args :varargs :form :map] predicate: map? In: [0 0] val:
& fails spec: :clojure.core.specs.alpha/arg-list at: [:args :bs
:arity-n :args] predicate: vector?
#:clojure.spec.alpha{:problems
({:path
[:args :bs :arity-1 :args :varargs :form :sym],
:pred clojure.core/simple-symbol?,
:val user/params,
:via
[:clojure.core.specs.alpha/args+body
:clojure.core.specs.alpha/arg-list
:clojure.core.specs.alpha/arg-list
:clojure.core.specs.alpha/binding-form
:clojure.core.specs.alpha/binding-form
:clojure.core.specs.alpha/local-name],
:in [0 1]}
{:path
[:args :bs :arity-1 :args :varargs :form :seq],
:pred clojure.core/vector?,
:val user/params,
:via
[:clojure.core.specs.alpha/args+body
:clojure.core.specs.alpha/arg-list
:clojure.core.specs.alpha/arg-list
:clojure.core.specs.alpha/binding-form
:clojure.core.specs.alpha/binding-form
:clojure.core.specs.alpha/seq-binding-form],
:in [0 1]}
{:path
[:args :bs :arity-1 :args :varargs :form :map],
:pred clojure.core/coll?,
:val user/params,
:via
[:clojure.core.specs.alpha/args+body
:clojure.core.specs.alpha/arg-list
:clojure.core.specs.alpha/arg-list
:clojure.core.specs.alpha/binding-form
:clojure.core.specs.alpha/binding-form
:clojure.core.specs.alpha/map-binding-form
:clojure.core.specs.alpha/map-bindings],
:in [0 1]}
{:path
[:args :bs :arity-1 :args :varargs :form :map],
:pred map?,
:val user/params,
:via
[:clojure.core.specs.alpha/args+body
:clojure.core.specs.alpha/arg-list
:clojure.core.specs.alpha/arg-list
:clojure.core.specs.alpha/binding-form
:clojure.core.specs.alpha/binding-form
:clojure.core.specs.alpha/map-binding-form
:clojure.core.specs.alpha/map-special-binding],
:in [0 1]}
{:path [:args :bs :arity-n :args],
:pred clojure.core/vector?,
:val &,
:via
[:clojure.core.specs.alpha/args+body
:clojure.core.specs.alpha/args+body
:clojure.core.specs.alpha/args+body
:clojure.core.specs.alpha/arg-list
:clojure.core.specs.alpha/arg-list],
:in [0 0]}),
:spec
#object[clojure.spec.alpha$regex_spec_impl$reify__2436 0x3517c752 "clojure.spec.alpha$regex_spec_impl$reify__2436#3517c752"],
:value
([& user/params]
(clojure.core/apply #this user/params)),
:args
([& user/params]
(clojure.core/apply #this user/params))}
core.clj: 4739 clojure.core/ex-info
core.clj: 4739 clojure.core/ex-info
alpha.clj: 689 clojure.spec.alpha/macroexpand-check
alpha.clj: 681 clojure.spec.alpha/macroexpand-check
AFn.java: 156 clojure.lang.AFn/applyToHelper
AFn.java: 144 clojure.lang.AFn/applyTo
Var.java: 702 clojure.lang.Var/applyTo
Compiler.java: 6889 clojure.lang.Compiler/checkSpecs
Compiler.java: 6907 clojure.lang.Compiler/macroexpand1
Compiler.java: 6989 clojure.lang.Compiler/analyzeSeq
Compiler.java: 6773 clojure.lang.Compiler/analyze
Compiler.java: 6729 clojure.lang.Compiler/analyze
Compiler.java: 6100 clojure.lang.Compiler$BodyExpr$Parser/parse
Compiler.java: 6420 clojure.lang.Compiler$LetExpr$Parser/parse
Compiler.java: 7003 clojure.lang.Compiler/analyzeSeq
Compiler.java: 6773 clojure.lang.Compiler/analyze
Compiler.java: 6729 clojure.lang.Compiler/analyze
Compiler.java: 6100 clojure.lang.Compiler$BodyExpr$Parser/parse
Compiler.java: 5460 clojure.lang.Compiler$FnMethod/parse
Compiler.java: 4022 clojure.lang.Compiler$FnExpr/parse
Compiler.java: 7001 clojure.lang.Compiler/analyzeSeq
Compiler.java: 6773 clojure.lang.Compiler/analyze
Compiler.java: 7059 clojure.lang.Compiler/eval
Compiler.java: 7025 clojure.lang.Compiler/eval
core.clj: 3206 clojure.core/eval
core.clj: 3202 clojure.core/eval
main.clj: 243 clojure.main/repl/read-eval-print/fn
main.clj: 243 clojure.main/repl/read-eval-print
main.clj: 261 clojure.main/repl/fn
main.clj: 261 clojure.main/repl
main.clj: 177 clojure.main/repl
RestFn.java: 1523 clojure.lang.RestFn/invoke
interruptible_eval.clj: 87 clojure.tools.nrepl.middleware.interruptible-eval/evaluate/fn
AFn.java: 152 clojure.lang.AFn/applyToHelper
AFn.java: 144 clojure.lang.AFn/applyTo
core.clj: 657 clojure.core/apply
core.clj: 1965 clojure.core/with-bindings*
core.clj: 1965 clojure.core/with-bindings*
RestFn.java: 425 clojure.lang.RestFn/invoke
interruptible_eval.clj: 85 clojure.tools.nrepl.middleware.interruptible-eval/evaluate
interruptible_eval.clj: 55 clojure.tools.nrepl.middleware.interruptible-eval/evaluate
interruptible_eval.clj: 222 clojure.tools.nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
interruptible_eval.clj: 190 clojure.tools.nrepl.middleware.interruptible-eval/run-next/fn
AFn.java: 22 clojure.lang.AFn/run
ThreadPoolExecutor.java: 1149 java.util.concurrent.ThreadPoolExecutor/runWorker
ThreadPoolExecutor.java: 624 java.util.concurrent.ThreadPoolExecutor$Worker/run
Thread.java: 748 java.lang.Thread/run
This is all still not really weird, mistakes happen when writing macros or something like that. However, when I expanded the macro using macroexpand-1, the resulting code made no problems whatsoever:
(macroexpand-1 '(a-let [a 1, b 2]
(fn [] (+ a b))))
;; => (clojure.core/let [this (clojure.core/atom nil) a 1 b 2]
;; (clojure.core/reset! this (fn [] (+ a b)))
;; (clojure.core/fn [& user/params] (clojure.core/apply (clojure.core/deref this) user/params)))
;; Without namespaces for readability
;; => (let [this (atom nil) a 1 b 2]
;; (reset! this (fn [] (+ a b)))
;; (fn [& params] (apply #this params)))
Which also works perfectly fine outside of macroexpand-1
(def f *) ;; => #'user/f
(f) ;; => 1
I'm not that familiar with the details of Clojures macro system so I'd be happy to be enlightened as to why this weird behaviour occurs because I am pretty lost on this one.
Thanks in advance!
I'm not sure where that mess of an error message came from. I can't say I've ever seen an error like that before.
When I run it, I get:
CompilerException java.lang.RuntimeException: Can't use qualified name as parameter: mandelbrot-redo.seesaw-main.first-main/params, compiling:(C:\Users\slomi\AppData\Local\Temp\form-init395175488607706237.clj:1:1)
Then the error is obvious. When creating bindings inside of a ` quoted form, they're automatically namespaced to the current namespace. Function parameters can't be namespaced however, as the error says.
Change the last bit to:
(fn [& params#]
(apply ~'#this params#))))
Note the #. Those turn params into a unique, non-namespaced symbol.
Now, it appears to work fine:
(a-let [a 1, b 2]
(fn [] (+ a b)))
=>
#object[mandelbrot_redo.seesaw_main.first_main$eval8162$fn__8165
0x63cf5b7e
"mandelbrot_redo.seesaw_main.first_main$eval8162$fn__8165#63cf5b7e"]
You can also use a promise instead of an atom. It's a little neater, and more correct, since you only want to set it once:
(defmacro my-a-let
"Anaphoric let, `this` refers to the last form in body, which should be a function"
[bindings & body]
`(let [~'this (promise)
~#bindings]
(deliver ~'this ~(last body))
~#(butlast body)
(fn [& params#]
(apply ~'#this params#))))

trouble displaying type that implements IPersistentMap interface

I am implementing a type that is really just a wrapper for hash-maps
(defn as-pairs [m]
(when-not (empty? m)
(seq (persistent! (reduce (fn [s [k vs]]
(reduce (fn [s v] (conj! s [k v])) s vs)) (transient []) m)))))
(deftype Rel [m]
clojure.lang.Seqable
(seq [this] (as-pairs m))
clojure.lang.ILookup
(valAt [this k] (get m k))
(valAt [this k default] (get m k default))
clojure.lang.IPersistentMap
(assoc [this k v] (Rel. (update m k #(conj (or % #{}) v))))
(assocEx [this k v] (throw (Exception.)))
(without [this k] (Rel. (dissoc m k))))
(defn relation [] (Rel. (hash-map)))
It seems to be working as expected
state-machines.maps> (def r (relation))
#'state-machines.maps/r
state-machines.maps> (type r)
state_machines.maps.Rel
state-machines.maps> r
{} ; interesting that it actually displays a map! -- source of problem?
state-machines.maps> (type (assoc r :foo 1 :foo 2 :bar 1))
state_machines.maps.Rel
state-machines.maps> (get (assoc r :foo 1 :foo 2 :bar 1) :foo)
#{1 2}
state-machines.maps> (seq (assoc r :foo 1 :foo 2 :bar 1))
([:foo 1] [:foo 2] [:bar 1])
state-machines.maps> (assoc r :foo 1 :foo 2 :bar 1)
ClassCastException clojure.lang.PersistentVector cannot be cast to java.util.Map$Entry clojure.core/key (core.clj:1518)
state-machines.maps>
looking at the stack trace
1. Unhandled java.lang.ClassCastException
clojure.lang.PersistentVector cannot be cast to java.util.Map$Entry
core.clj: 1518 clojure.core/key
core_print.clj: 212 clojure.core/print-map/fn
core_print.clj: 59 clojure.core/print-sequential
core_print.clj: 208 clojure.core/print-map
core_print.clj: 217 clojure.core/fn
core_print.clj: 217 clojure.core/fn
MultiFn.java: 233 clojure.lang.MultiFn/invoke
pr_values.clj: 35 clojure.tools.nrepl.middleware.pr-values/pr-values/fn/reify
interruptible_eval.clj: 113 clojure.tools.nrepl.middleware.interruptible-eval/evaluate/fn/fn
main.clj: 241 clojure.main/repl/read-eval-print
main.clj: 258 clojure.main/repl/fn
main.clj: 258 clojure.main/repl
main.clj: 174 clojure.main/repl
RestFn.java: 137 clojure.lang.RestFn/applyTo
core.clj: 646 clojure.core/apply
core.clj: 641 clojure.core/apply
regrow.clj: 18 refactor-nrepl.ns.slam.hound.regrow/wrap-clojure-repl/fn
RestFn.java: 1523 clojure.lang.RestFn/invoke
interruptible_eval.clj: 87 clojure.tools.nrepl.middleware.interruptible-eval/evaluate/fn
AFn.java: 152 clojure.lang.AFn/applyToHelper
AFn.java: 144 clojure.lang.AFn/applyTo
core.clj: 646 clojure.core/apply
core.clj: 1881 clojure.core/with-bindings*
core.clj: 1881 clojure.core/with-bindings*
RestFn.java: 425 clojure.lang.RestFn/invoke
interruptible_eval.clj: 85 clojure.tools.nrepl.middleware.interruptible-eval/evaluate
interruptible_eval.clj: 55 clojure.tools.nrepl.middleware.interruptible-eval/evaluate
interruptible_eval.clj: 222 clojure.tools.nrepl.middleware.interruptible-eval/interruptible-eval/fn/fn
interruptible_eval.clj: 190 clojure.tools.nrepl.middleware.interruptible-eval/run-next/fn
AFn.java: 22 clojure.lang.AFn/run
ThreadPoolExecutor.java: 1142 java.util.concurrent.ThreadPoolExecutor/runWorker
ThreadPoolExecutor.java: 617 java.util.concurrent.ThreadPoolExecutor$Worker/run
Thread.java: 745 java.lang.Thread/run
I am assuming that something is happening in one of
core_print.clj: 212 clojure.core/print-map/fn
core_print.clj: 59 clojure.core/print-sequential
core_print.clj: 208 clojure.core/print-map
Is this an interface that I need to implement?
After updating as-pairs to
(defn as-pairs [m]
(when-not (empty? m)
(seq (persistent! (reduce (fn [s [k vs]]
(reduce (fn [s v] (conj! s (clojure.lang.MapEntry/create k v))) s vs)) (transient []) m)))))
the result is:
state-machines.maps> (assoc (relation) :foo 1 :foo 2 :bar 1)
{:foo 1, :foo 2, :bar 1}
As you suspect, the error is arising in print-map, which calls seq on its argument and then calls key and val on each element of that sequence. As the error message states, these two functions expect their argument to conform to the java.util.Map$Entry interface, and Clojure vectors do not conform to that interface.
The solution is to take the [k v] in your as-pairs function and replace it with an expression that creates a map entry; see this question for details.

In Clojure core.logic, why can't I use threading macros?

When I run the following code, I get the ensuing error:
(run 3 [q]
(fresh [a0 a1 a2
b0 b1 b2
c0 c1 c2]
(== q [[a0 a1 a2] [b0 b1 b2] [c0 c1 c2]])
(fd/in a0 a1 a2 b0 b1 b2 c0 c1 c2 (fd/interval 1 9))
(fd/distinct [a0 a1 a2 b0 b1 b2 c0 c1 c2])
(fd/eq
(= a0 4)
(= 22 (- (* a0 a1) a2))
(= -1 (-> b0 (* b1) (- b2)))
)))
error:
2. Unhandled clojure.lang.Compiler$CompilerException
Error compiling src/euler/core.clj at (1392:5)
1. Caused by java.lang.IllegalArgumentException
Can't call nil
core.clj: 3081 clojure.core/eval
main.clj: 240 clojure.main/repl/read-eval-print/fn
main.clj: 240 clojure.main/repl/read-eval-print
main.clj: 258 clojure.main/repl/fn
main.clj: 258 clojure.main/repl
RestFn.java: 1523 clojure.lang.RestFn/invoke
interruptible_eval.clj: 87 clojure.tools.nrepl.middleware.interruptible-eval/evaluate/fn
AFn.java: 152 clojure.lang.AFn/applyToHelper
AFn.java: 144 clojure.lang.AFn/applyTo
core.clj: 630 clojure.core/apply
core.clj: 1868 clojure.core/with-bindings*
RestFn.java: 425 clojure.lang.RestFn/invoke
...
Notice the line with the threading macro ->, in CIDER I'll macro expand it and everything looks fine, but in the end the code crashes. I'm assuming this is the fault of macros, but I'm not sure why. Any ideas?
You should look at the source of clojure.core.logic.fd. The error occurs in the macro eq, which process all of its forms before macroexpansion can occur.
For a quick solution, I've created a version of eq which calls macroexpand-all on all of its forms before doing anything else. It seems to work for your code, although I have not tested it in other contexts:
(defmacro mac-eq [& forms]
(let [exp-forms (map clojure.walk/macroexpand-all forms)]
`(fd/eq ~#exp-forms)))
Let's try it!
stack-prj.logicThreadMacro> (run 3 [q]
(fresh [a0 a1 a2
b0 b1 b2
c0 c1 c2]
(== q [[a0 a1 a2] [b0 b1 b2] [c0 c1 c2]])
(fd/in a0 a1 a2 b0 b1 b2 c0 c1 c2 (fd/interval 1 9))
(fd/distinct [a0 a1 a2 b0 b1 b2 c0 c1 c2])
(mac-eq
(= a0 4)
(= 22 (- (* a0 a1) a2))
(= -1 (-> b0 (* b1) (- b2))))))
()

Clojure Lazy Seq Results in Stack Overflow

I want to compute a lazy sequence of primes.
Here is the interface:
user=> (take 10 primes)
(2 3 5 7 11 13 17 19 23 29)
So far, so good.
However, when I take 500 primes, this results in a stack overflow.
core.clj: 133 clojure.core/seq
core.clj: 2595 clojure.core/filter/fn
LazySeq.java: 40 clojure.lang.LazySeq/sval
LazySeq.java: 49 clojure.lang.LazySeq/seq
RT.java: 484 clojure.lang.RT/seq
core.clj: 133 clojure.core/seq
core.clj: 2626 clojure.core/take/fn
LazySeq.java: 40 clojure.lang.LazySeq/sval
LazySeq.java: 49 clojure.lang.LazySeq/seq
Cons.java: 39 clojure.lang.Cons/next
LazySeq.java: 81 clojure.lang.LazySeq/next
RT.java: 598 clojure.lang.RT/next
core.clj: 64 clojure.core/next
core.clj: 2856 clojure.core/dorun
core.clj: 2871 clojure.core/doall
core.clj: 2910 clojure.core/partition/fn
LazySeq.java: 40 clojure.lang.LazySeq/sval
LazySeq.java: 49 clojure.lang.LazySeq/seq
RT.java: 484 clojure.lang.RT/seq
core.clj: 133 clojure.core/seq
core.clj: 2551 clojure.core/map/fn
LazySeq.java: 40 clojure.lang.LazySeq/sval
LazySeq.java: 49 clojure.lang.LazySeq/seq
RT.java: 484 clojure.lang.RT/seq
core.clj: 133 clojure.core/seq
core.clj: 3973 clojure.core/interleave/fn
LazySeq.java: 40 clojure.lang.LazySeq/sval
I'm wondering what is the problem here and, more generally, when working with lazy sequences, how should I approach this class of error?
Here is the code.
(defn assoc-nth
"Returns a lazy seq of coll, replacing every nth element by val
Ex:
user=> (assoc-nth [3 4 5 6 7 8 9 10] 2 nil)
(3 nil 5 nil 7 nil 9 nil)
"
[coll n val]
(apply concat
(interleave
(map #(take (dec n) %) (partition n coll)) (repeat [val]))))
(defn sieve
"Returns a lazy seq of primes by Eratosthenes' method
Ex:
user=> (take 4 (sieve (iterate inc 2)))
(2 3 5 7)
user=> (take 10 (sieve (iterate inc 2)))
(2 3 5 7 11 13 17 19 23 29)
"
[s]
(lazy-seq
(if (seq s)
(cons (first s) (sieve
(drop-while nil? (assoc-nth (rest s) (first s) nil))))
[])))
(def primes
"Returns a lazy seq of primes
Ex:
user=> (take 10 primes)
(2 3 5 7 11 13 17 19 23 29)
"
(concat [2] (sieve (filter odd? (iterate inc 3)))))
You are generating a lot of lazy sequences that all take up stack space.
Breaking it down:
user=> (iterate inc 3)
;; produces first lazy seq (don't run in repl!)
(3 4 5 6 7 8 9 10 11 12 13 ...)
user=> (filter odd? (iterate inc 3))
;; again, don't run in repl, but lazy seq #2
(3 5 7 9 11 13 ...)
I personally would have done (iterate #(+ 2 %) 3) to cut a sequence out, but that's a drop in the ocean compared to the overall problem.
Now we start into the sieve which starts creating a new lazy seq for our output
(lazy-seq
(cons 3 (sieve (drop-while nil? (assoc-nth '(5 7 9 ...) 3 nil)))))
Jumping into assoc-nth, we start creating more lazy sequences
user=> (partition 3 [5 7 9 11 13 15 17 19 21 23 25 ...])
;; lazy seq #4
((5 7 9) (11 13 15) (17 19 21) (23 25 27) ...)
user=> (map #(take 2 %) '((5 7 9) (11 13 15) (17 19 21) (23 25 27) ...))
;; lazy seq #5
((5 7) (11 13) (17 19) (23 25) ...)
user=> (interleave '((3 5) (9 11) (15 17) (21 23) ...) (repeat [nil]))
;; lazy seq #6 + #7 (repeat [nil]) is a lazy seq too
((5 7) [nil] (11 13) [nil] (17 19) [nil] (23 25) [nil] ...)
user=> (apply concat ...
;; lazy seq #8
(5 7 nil 11 13 nil 17 19 nil 23 25 nil ...)
So already you have 8 lazy sequences, and we've only applied first sieve.
Back to sieve, and it does a drop-while which produces lazy sequence number 9
user=> (drop-while nil? '(5 7 nil 11 13 nil 17 19 nil 23 25 nil ...))
;; lazy seq #9
(5 7 11 13 17 19 23 25 ...)
So for one iteration of sieve, we generated 9 sequences.
In the next iteration of sieve, you generate a new lazy-seq (not the original!), and the process begins again, generating another 7 sequences per loop.
The next problem is the efficiency of your assoc-nth function. A sequence of numbers and nils isn't an efficient way to mark multiples of a particular factor. You end up very quickly with a lot of very long sequences that can't be released, mostly filled with nils, and you need to read longer and longer sequences to decide if a candidate is a factor or not - this isn't efficient for lazy sequences which read chunks of 32 entries to be efficient, so when your factoring out 33 and above, you're pulling in multiple chunks of the sequence to work on.
Also, each time you're doing this in a completely new sequence.
Adding some debug to your assoc-nth, and running it on a small sample will illustrate this quickly.
user=> (sieve (take 50 (iterate #(+ 2 %) 3)))
assoc-nth n: 3 , coll: (5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59 61 63 65 67 69 71 73 75 77 79 81 83 85 87 89 91 93 95 97 99 101)
returning r: (5 7 nil 11 13 nil 17 19 nil 23 25 nil 29 31 nil 35 37 nil 41 43 nil 47 49 nil 53 55 nil 59 61 nil 65 67 nil 71 73 nil 77 79 nil 83 85 nil 89 91 nil 95 97 nil)
assoc-nth n: 5 , coll: (7 nil 11 13 nil 17 19 nil 23 25 nil 29 31 nil 35 37 nil 41 43 nil 47 49 nil 53 55 nil 59 61 nil 65 67 nil 71 73 nil 77 79 nil 83 85 nil 89 91 nil 95 97 nil)
returning r: (7 nil 11 13 nil 17 19 nil 23 nil nil 29 31 nil nil 37 nil 41 43 nil 47 49 nil 53 nil nil 59 61 nil nil 67 nil 71 73 nil 77 79 nil 83 nil nil 89 91 nil nil)
assoc-nth n: 7 , coll: (nil 11 13 nil 17 19 nil 23 nil nil 29 31 nil nil 37 nil 41 43 nil 47 49 nil 53 nil nil 59 61 nil nil 67 nil 71 73 nil 77 79 nil 83 nil nil 89 91 nil nil)
returning r: (nil 11 13 nil 17 19 nil 23 nil nil 29 31 nil nil 37 nil 41 43 nil 47 nil nil 53 nil nil 59 61 nil nil 67 nil 71 73 nil nil 79 nil 83 nil nil 89 nil)
;; ...
assoc-nth n: 17 , coll: (19 nil 23 nil nil 29 31 nil nil 37 nil 41 43 nil 47 nil nil 53 nil nil 59 61 nil nil)
returning r: (19 nil 23 nil nil 29 31 nil nil 37 nil 41 43 nil 47 nil nil)
assoc-nth n: 19 , coll: (nil 23 nil nil 29 31 nil nil 37 nil 41 43 nil 47 nil nil)
returning r: ()
(3 5 7 11 13 17 19)
This illustrates how you need more and more elements of the sequence to produce a list of primes, as I started with odd numbers 3 to 99, but only ended up with primes 3 to 19, but on the last iteration n=19, there weren't enough elements in my finite sequence to nil out further multiples.
Is there a solution?
I think you're looking for answers to "how do I make lazy sequences better at doing what I want to do?". Lazy sequences are going to be a trade off in this case. Your algorithm works, but generates too much stack. You're not using any recur, so you generate sequences on sequences. The first thing to look at would be how to make your methods more tail recursive, and lose some of the sequences. I can't offer solutions to that here, but I can link other solutions to the same problem and see if there are areas they do better than yours.
There are 2 decent (bound numbers) solutions at this implementation of Eratosthenes primes. One is using a range, and sieving it, the other using arrays of booleans (which is 40x quicker). The associated article with it (in Japenese but google translates well in Chrome) is a good read, and shows the times for naive approaches vs a very focused version using arrays directly and type hints to avoid further unboxing issues in the jvm.
There's another SO question that has a good implementation using transient values to improve efficiency. It has similar filtering methods to yours.
In each case, different sieving methods are used that avoid lazy sequences, however as there's an upper bound on the primes they generate, they can swap out generality for efficiency.
For unbound case, look at this gist for an infinite sequence of primes. In my tests it was 6x slower than the previous SO question, but still pretty good. The implementation however is very different.

Group-by and data get

I think I am misunderstanding GET in clojure - I am trying to mould 1 data set from another, from the below A to B.
A
ID REGION MIN
10346 GLBL 106
10346 ASPAC 106
10346 NA 106
10346 LATAM 106
10346 EMEA 106
10347 GLBL 32
10347 ASPAC 32
10347 NA 32
10347 LATAM 32
10347 EMEA 32
10349 NA 10
10327 NA
10344 EMEA 8
10342 ASPAC 292
10342 EMEA 292
10348 ASPAC 15
10422 EMEA 37
10438 NA 0
B
ID EMEA NA ASPAC GLBL LATAM
10346 106 106 106 106 106
10347 32 32 32 32 32
10349 0 10 0 0 0
10327 0 0 0 0 0
10344 8 0 0 0 0
10342 292 0 292 0 0
10348 0 0 15 0 0
10422 37 0 0 0 0
10438 0 0 0 0 0
The group by is working but I am getting null values for all the regions, I though filtering on the region I could use get to obtain that value for MIN in that record and map it to the new region field - any advise on what I am doing wrong here? Or what I should be using instead of GET?
(defn- create-summary [data]
(->> data
(group-by :ID
vals
(map
(fn [recs]
(let [a (fn [priority](get :MIN (filter #(= priority (:REGION %)) recs)))]
{:ID (:ID (first recs))
:EMEA (a "EMEA")
:NA (a "NA")
:GLBL (a "GLBL")
:LATAM (a "LATAM")
:ASPAC (a "ASPAC")
})))
))
This:
(let [a (fn [priority](get :MIN (filter #(= priority (:REGION %)) recs)))]
Should be
(let [a (fn [priority](get (first (filter #(= priority (:REGION %)) recs)) :MIN))]