I am currently going through the book Clojure for the Brave and True, in an attempt to learn the language, but I'm a bit hung up on lazy seqs, and I'm afraid the book does a poor job explaining them. But, according to the book, something like this:
(defn wait-for-a-bit [arg]
(Thread/sleep 1000))
(defn get-map [seq]
(map wait-for-a-bit seq))
(time (first (get-map [1 2 3 4 5 6 7 8 9 0])))
Should only take about one second to process, because a value for a lazy seq isn't calculated (realized?) until it's accessed. However, when I run the above code, it takes about ten seconds, so clearly, deferred computation isn't happening. I've looked at the docs on clojuredocs.org, and I think I understand lazy-seq's, but I guess just not in the context of map, reduce, etc.
Lazy sequences are chunked by default. So the actual values are computed in chunks of 30 or so. This greatly reduces the context switching overhead in processing them.
here I'll define a sequence 100 items long and look at the first two items:
hello.core> (def foo (map #(do (println "calculating " %)
%)
(range 100)))
#'hello.core/foo
hello.core> (first foo)
calculating 0
calculating 1
calculating 2
calculating 3
calculating 4
calculating 5
calculating 6
calculating 7
...
calculating 26
calculating 27
calculating 28
calculating 29
calculating 30
calculating 31
0
hello.core> (second foo)
1
This shows that it calculates the first chunk the first time any of the items are realised.
Some sequences are chunked while others are not. It's up to the function that initially creates the seq to decide if it can be chunked. range creates chunked sequences while iterate does not. If we look at the same example again, this time generating the seq using iterate rather than map, we get a non-chunked sequence:
hello.core> (def foo (map #(do (println "calculating " %)
%)
(take 100 (iterate inc 0))))
#'hello.core/foo
hello.core> (first foo)
calculating 0
0
hello.core> (second foo)
calculating 1
1
and each item is calculated on when it is read. In theory this has efficiency impacts though I have been writing Clojure full time about as long as anyone and have never seen a case where this made a difference in something that was not otherwise poorly designed.
Related
Let's say I have a huge lazy seq and I want to iterate it so I can process on the data that I get during the iteration.
The thing is I want to lose head(GC'd) of lazy seq(that processed) so I can work on seqs that have millions of data without having OutofMemoryException.
I have 3 examples that I'm not sure.
Could you provide best practices(examples) for that purpose?
Do these functions lose head?
Example 1
(defn lose-head-fn
[lazy-seq-coll]
(when (seq (take 1 lazy-seq-coll))
(do
;;do some processing...(take 10000 lazy-seq-coll)
(recur (drop 10000 lazy-seq-coll)))))
Example 2
(defn lose-head-fn
[lazy-seq-coll]
(loop [i lazy-seq-coll]
(when (seq (take 1 i))
(do
;;do some processing...(take 10000 i)
(recur (drop 10000 i))))))
Example 3
(doseq [i lazy-seq-coll]
;;do some processing...
)
Update: Also there is an explanation in this answer here
copy of my above comments
As far as I know, all of the above would lose head (first two are obvious, since you manually drop the head, while doseq's doc claims that it doesn't retain head).
That means that if the lazy-seq-coll you pass to the function isn't bound somewhere else with def or let and used later, there should be nothing to worry about. So (lose-head-fn (range)) won't eat all your memory, while
(def r (range))
(lose-head-fn r)
probably would.
And the only best practice I could think of is not to def possibly infinite (or just huge) sequences, because all of their realized items would live forever in the var.
In general, you must be careful not to retain a reference either locally or globally for a part of a lazy seq that precedes another which involves excessive computation.
For example:
(let [nums (range)
first-ten (take 10 nums)]
(+ (last first-ten) (nth nums 100000000)))
=> 100000009
This takes about 2 seconds on a modern machine. How about this though? The difference is the last line, where the order of arguments to + is swapped:
;; Don't do this!
(let [nums (range)
first-ten (take 10 nums)]
(+ (nth nums 100000000) (last first-ten)))
You'll hear your chassis/cpu fans come to life, and if you're running htop or similar, you'll see memory usage grow rather quickly (about 1G in the first several seconds for me).
What's going on?
Much like a linked list, elements in a lazy seq in clojure reference the portion of the seq that comes next. In the second example above, first-ten is needed for the second argument to +. Thus, even though nth is happy to hold no references to anything (after all, it's just finding an index in a long list), first-ten refers to a portion of the sequence that, as stated above, must hold onto references to the rest of the sequence.
The first example, by contrast, computes (last first-ten), and after this, first-ten is no longer used. Now the only reference to any portion of the lazy sequence is nums. As nth does its work, each portion of the list that it's finished with is no longer needed, and since nothing else refers to the list in this block, as nth walks the list, the memory taken by the sequence that has been examined can be garbage collected.
Consider this:
;; Don't do this!
(let [nums (range)]
(time (nth nums 1e8))
(time (nth nums 1e8)))
Why does this have a similar result as the second example above? Because the sequence will be cached (held in memory) on the first realization of it (the first (time (nth nums 1e8))), because nums is being used on the next line. If, instead, we use a different sequence for the second nth, then there is no need to cache the first one, so it can be discarded as it's processed:
(let [nums (range)]
(time (nth nums 1e8))
(time (nth (range) 1e8)))
"Elapsed time: 2127.814253 msecs"
"Elapsed time: 2042.608043 msecs"
So as you work with large lazy seqs, consider whether anything is still pointing to the list, and if anything is (global vars being a common one), then it will be held in memory.
I am trying to print the value of the fibonacci number at position 400, using memoization. At first, I was only able to print the number until the 94th position, because I got an integer overflow error. After reading online, I am now able to print the value up to position 247 by converting into BigInt. But for some reason, when I try 248 or above, I get a huge error. Here is my code:
(def m-fib
(memoize (fn [n]
(condp = n
0N 0N
1N 1N
(+ (m-fib (dec n)) (m-fib (- n 2)))))))
(println(m-fib 248N))
And here is a sample of the error I get (It's too long to put all of it into the question, and it won't specifically tell me what the problem is.
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:646)
at clojure.core$memoize$fn__5708.doInvoke(core.clj:6107)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at fibonacci.core$_main$fn__28.invoke(core.clj:11)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:646)
at clojure.core$memoize$fn__5708.doInvoke(core.clj:6107)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at fibonacci.core$_main$fn__28.invoke(core.clj:11)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:646)
at clojure.core$memoize$fn__5708.doInvoke(core.clj:6107)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at fibonacci.core$_main$fn__28.invoke(core.clj:11)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:646)
at clojure.core$memoize$fn__5708.doInvoke(core.clj:6107)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at fibonacci.core$_main$fn__28.invoke(core.clj:11)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
at clojure.core$apply.invokeStatic(core.clj:646)
at clojure.core$memoize$fn__5708.doInvoke(core.clj:6107)
at clojure.lang.RestFn.invoke(RestFn.java:408)
at fibonacci.core$_main$fn__28.invoke(core.clj:11)
at clojure.lang.AFn.applyToHelper(AFn.java:154)
at clojure.lang.AFn.applyTo(AFn.java:144)
How can I fix this and output the fib value at position 400 without any errors?
Your m-fib creates a tree that grows exponentially with the input.
Source: http://www.sicpdistilled.com/section/1.2.2/
The space used by m-fib grows so large that it blows up the stack (StackOverflow) since all intermediate values are kept on the stack, before they are added.
Your function would work fine it is used tail-recursion. This means that intermediate results are calculated every time and put on the stack. Using tail-recursion you don't need space on the stack, since only the intermediate result is stored.
I did't see a way to implement in your structure since it basically uses the stack size to calculate the answer (it spreads out to a large amount of base cases where N are 0 and 1). Here's a tail recursive other way as an example.
Some languages (like Scheme) do automatic tail-call optimization. Clojure also has limited support for tail recursion (because of limitations of the JVM) as long as you use loop and recur (as also said in the answer by zabeltech.
In Clojure we can only use tail recursion in the tail position (last expression). This can, for example, be done by creating an iterative tail-recursive function where we pass the intermediate results along:
(defn m-fib [n]
(m-fib-iter 1 0 n))
(def m-fib-iter
(memoize
(fn [a b count]
(if (= count 0)
b
(recur (+' a b) a (- count 1)))))) ; note the +'
(fib 248) ;; => 3016128079338728432528443992613633888712980904400501N
Here a and b are updated every recurring round.
This answer was based on the Clojure text based on the Structure and Interpretation of Computer programs at SICP distilled. I only added memoization as you did as well, so that previously calculated answers are stored.
See that site for more information.
#zabeltech is right, you reach the limit of the recursion.
In your case the best thing you can do without changing your code is to precall your fib function with some values lower than your desired index, to get memoized data later, instead of recursively calculating it:
user> (m-fib 100N)
354224848179261915075N
user> (m-fib 248N)
3016128079338728432528443992613633888712980904400501N
you can see that the 100th fib doesn't cause stackoverflow, and then 248th also doesn't (because it doesn't go down to 0 recursively, it only goes down to 100, and 100th value (and below) are already memoized)
Also, clojure has some much more elegant approaches to generating fibs, like this for example:
user> (def fibs (lazy-cat [0 1N] (map + fibs (rest fibs))))
#'user/fibs
user> (nth fibs 248)
3016128079338728432528443992613633888712980904400501N
I get a stack overflow error when I try to advance the memoization by a thousand or so (YMMV). This is because the recursion spins up a stack until it finds the largest memoized argument-> value.
Shlomi's comment refers to solutions that yield a sequence. If we want to preserve constant time access, we can build our own lazy vector. I've worked out a way to do this, whereby you supply
an initial sequence of values and
a function that generates the next value from the preceding however
many.
You get a function that
encloses a vector inside an atom that grows to reach any offered
argument.
And never forgets what it has done.
The function-making function is ...
(defn mem-vector [f inits]
(let [mem (atom (vec inits))]
(fn [n]
(let [content- #mem, size- (count content-)]
(loop [content content-, size size-]
(if (> size n)
(do
(if (> size size-) (reset! mem content))
(content n))
(recur (conj content (apply f (subvec content (- size (count inits))))) (inc size))))))))
To generate fibonacci numbers,
the initial values are 0 and 1 and
the function that generates the next value from the previous two is
+'.
So a lazy vector of all the fibonacci numbers is
(def fibs (mem-vector +' [0 1]))
For example
(fibs 100000)
;2597406934722172416615503402127591541488048538651769658472477070395253454351127368626555677283671674475463758722307443211163839947387509103096569738218830449305228763853133492135302679278956701051276578271635608073050532200243233114383986516137827238124777453778337299916214634050054669860390862750996639366409211890125271960172105060300350586894028558103675117658251368377438684936413457338834365158775425371912410500332195991330062204363035213756525421823998690848556374080179251761629391754963458558616300762819916081109836526352995440694284206571046044903805647136346033000520852277707554446794723709030979019014860432846819857961015951001850608264919234587313399150133919932363102301864172536477136266475080133982431231703431452964181790051187957316766834979901682011849907756686456845066287392485603914047605199550066288826345877189410680370091879365001733011710028310473947456256091444932821374855573864080579813028266640270354294412104919995803131876805899186513425175959911520563155337703996941035518275274919959802257507902037798103089922984996304496255814045517000250299764322193462165366210841876745428298261398234478366581588040819003307382939500082132009374715485131027220817305432264866949630987914714362925554252624043999615326979876807510646819068792118299167964409178271868561702918102212679267401362650499784968843680975254700131004574186406448299485872551744746695651879126916993244564817673322257149314967763345846623830333820239702436859478287641875788572910710133700300094229333597292779191409212804901545976262791057055248158884051779418192905216769576608748815567860128818354354292307397810154785701328438612728620176653953444993001980062953893698550072328665131718113588661353747268458543254898113717660519461693791688442534259478126310388952047956594380715301911253964847112638900713362856910155145342332944128435722099628674611942095166100230974070996553190050815866991144544264788287264284501725332048648319457892039984893823636745618220375097348566847433887249049337031633826571760729778891798913667325190623247118037280173921572390822769228077292456662750538337500692607721059361942126892030256744356537800831830637593334502350256972906515285327194367756015666039916404882563967693079290502951488693413799125174856667074717514938979038653338139534684837808612673755438382110844897653836848318258836339917310455850905663846202501463131183108742907729262215943020429159474030610183981685506695026197376150857176119947587572212987205312060791864980361596092339594104118635168854883911918517906151156275293615849000872150192226511785315089251027528045151238603792184692121533829287136924321527332714157478829590260157195485316444794546750285840236000238344790520345108033282013803880708980734832620122795263360677366987578332625485944906021917368867786241120562109836985019729017715780112040458649153935115783499546100636635745448508241888279067531359950519206222976015376529797308588164873117308237059828489404487403932053592935976454165560795472477862029969232956138971989467942218727360512336559521133108778758228879597580320459608479024506385194174312616377510459921102486879496341706862092908893068525234805692599833377510390101316617812305114571932706629167125446512151746802548190358351688971707570677865618800822034683632101813026232996027599403579997774046244952114531588370357904483293150007246173417355805567832153454341170020258560809166294198637401514569572272836921963229511187762530753402594781448204657460288485500062806934811398276016855584079542162057543557291510641537592939022884356120792643705560062367986544382464373946972471945996555795505838034825597839682776084731530251788951718630722761103630509360074262261717363058613291544024695432904616258691774630578507674937487992329181750163484068813465534370997589353607405172909412697657593295156818624747127636468836551757018353417274662607306510451195762866349922848678780591085118985653555434958761664016447588028633629704046289097067736256584300235314749461233912068632146637087844699210427541569410912246568571204717241133378489816764096924981633421176857150311671040068175303192115415611958042570658693127276213710697472226029655524611053715554532499750843275200199214301910505362996007042963297805103066650638786268157658772683745128976850796366371059380911225428835839194121154773759981301921650952140133306070987313732926518169226845063443954056729812031546392324981793780469103793422169495229100793029949237507299325063050942813902793084134473061411643355614764093104425918481363930542369378976520526456347648318272633371512112030629233889286487949209737847861884868260804647319539200840398308008803869049557419756219293922110825766397681361044490024720948340326796768837621396744075713887292863079821849314343879778088737958896840946143415927131757836511457828935581859902923534388888846587452130838137779443636119762839036894595760120316502279857901545344747352706972851454599861422902737291131463782045516225447535356773622793648545035710208644541208984235038908770223039849380214734809687433336225449150117411751570704561050895274000206380497967960402617818664481248547269630823473377245543390519841308769781276565916764229022948181763075710255793365008152286383634493138089971785087070863632205869018938377766063006066757732427272929247421295265000706646722730009956124191409138984675224955790729398495608750456694217771551107346630456603944136235888443676215273928597072287937355966723924613827468703217858459948257514745406436460997059316120596841560473234396652457231650317792833860590388360417691428732735703986803342604670071717363573091122981306903286137122597937096605775172964528263757434075792282180744352908669606854021718597891166333863858589736209114248432178645039479195424208191626088571069110433994801473013100869848866430721216762473119618190737820766582968280796079482259549036328266578006994856825300536436674822534603705134503603152154296943991866236857638062351209884448741138600171173647632126029961408561925599707566827866778732377419444462275399909291044697716476151118672327238679208133367306181944849396607123345271856520253643621964198782752978813060080313141817069314468221189275784978281094367751540710106350553798003842219045508482239386993296926659221112742698133062300073465628498093636693049446801628553712633412620378491919498600097200836727876650786886306933418995225768314390832484886340318940194161036979843833346608676709431643653538430912157815543512852077720858098902099586449602479491970687230765687109234380719509824814473157813780080639358418756655098501321882852840184981407690738507369535377711880388528935347600930338598691608289335421147722936561907276264603726027239320991187820407067412272258120766729040071924237930330972132364184093956102995971291799828290009539147382437802779051112030954582532888721146170133440385939654047806199333224547317803407340902512130217279595753863158148810392952475410943880555098382627633127606718126171022011356181800775400227516734144169216424973175621363128588281978005788832454534581522434937268133433997710512532081478345067139835038332901313945986481820272322043341930929011907832896569222878337497354301561722829115627329468814853281922100752373626827643152685735493223028018101449649009015529248638338885664893002250974343601200814365153625369199446709711126951966725780061891215440222487564601554632812091945824653557432047644212650790655208208337976071465127508320487165271577472325887275761128357592132553934446289433258105028633583669291828566894736223508250294964065798630809614341696830467595174355313224362664207197608459024263017473392225291248366316428006552870975051997504913009859468071013602336440164400179188610853230764991714372054467823597211760465153200163085336319351589645890681722372812310320271897917951272799656053694032111242846590994556380215461316106267521633805664394318881268199494005537068697621855231858921100963441012933535733918459668197539834284696822889460076352031688922002021931318369757556962061115774305826305535862015637891246031220672933992617378379625150999935403648731423208873977968908908369996292995391977217796533421249291978383751460062054967341662833487341011097770535898066498136011395571584328308713940582535274056081011503907941688079197212933148303072638678631411038443128215994936824342998188719768637604496342597524256886188688978980888315865076262604856465004322896856149255063968811404400429503894245872382233543101078691517328333604779262727765686076177705616874050257743749983775830143856135427273838589774133526949165483929721519554793578923866762502745370104660909382449626626935321303744538892479216161188889702077910448563199514826630802879549546453583866307344423753319712279158861707289652090149848305435983200771326653407290662016775706409690183771201306823245333477966660525325490873601961480378241566071271650383582257289215708209369510995890132859490724306183325755201208090007175022022949742801823445413711916298449914722254196594682221468260644961839254249670903104007581488857971672246322887016438403908463856731164308169537326790303114583680575021119639905615169154708510459700542098571797318015564741406172334145847111268547929892443001391468289103679179216978616582489007322033591376706527676521307143985302760988478056216994659655461379174985659739227379416726495377801992098355427866179123126699374730777730569324430166839333011554515542656864937492128687049121754245967831132969248492466744261999033972825674873460201150442228780466124320183016108232183908654771042398228531316559685688005226571474428823317539456543881928624432662503345388199590085105211383124491861802624432195540433985722841341254409411771722156867086291742124053110620522842986199273629406208834754853645128123279609097213953775360023076765694208219943034648783348544492713539450224591334374664937701655605763384697062918725745426505879414630176639760457474311081556747091652708748125267159913793240527304613693961169892589808311906322510777928562071999459487700611801002296132304588294558440952496611158342804908643860880796440557763691857743754025896855927252514563404385217825890599553954627451385454452916761042969267970893580056234501918571489030418495767400819359973218711957496357095967825171096264752068890806407651445893132870767454169607107931692704285168093413311046353506242209810363216771910420786162184213763938194625697286781413636389620123976910465418956806197323148414224550071617215851321302030684176087215892702098879108938081045903397276547326416916845445627600759561367103584575649094430692452532085003091068783157561519847567569191284784654692558665111557913461272425336083635131342183905177154511228464455136016013513228948543271504760839307556100908786096663870612278690274831819331606701484957163004705262228238406266818448788374548131994380387613830128859885264201992286188208499588640888521352501457615396482647451025902530743172956899636499615707551855837165935367125448515089362904567736630035562457374779100987992499146967224041481601289530944015488942613783140087804311431741858071826185149051138744831358439067228949408258286021650288927228387426432786168690381960530155894459451808735197246008221529343980828254126128257157209350985382800738560472910941184006084485235377833503306861977724501886364070344973366473100602018128792886991861824418453968994777259482169137133647470453172979809245844361129618997595696240971845564020511432589591844724920942930301651488713079802102379065536525154780298059407529440513145807551537794861635879901158192019808879694967187448224156836463534326160242632934761634458163890163805123894184523973421841496889262398489648642093409816681494771155177009562669029850101513537599801272501241971119871526593747484778935488777815192931171431167444773882941064615028751327709474504763922874890662989841540259350834035142035136168819248238998027706666916342133424312054507359388616687691188185776118135771332483965209882085982391298606386822804754362408956522921410859852037330544625953261340234864689275060526893755148403298542086991221052597005628576707702567695300978970046408920009852106980295419699802138053295798159478289934443245491565327845223840551240445208226435420656313310702940722371552770504263482073984454889589248861397657079145414427653584572951329719091947694411910966797474262675590953832039169673494261360032263077428684105040061351052194413778158095005714526846009810352109249040027958050736436961021241137739717164869525493114805040126568351268829598413983222676377804500626507241731757395219796890754825199329259649801627068665658030178877405615167159731927320479376247375505855052839660294566992522173600874081212014209071041937598571721431338017425141582491824710905084715977249417049320254165239323233258851588893337097136310892571531417761978326033750109026284066415801371359356529278088456305951770081443994114674291850360748852366654744869928083230516815711602911836374147958492100860528981469547750812338896943152861021202736747049903930417035171342126923486700566627506229058636911882228903170510305406882096970875545329369434063981297696478031825451642178347347716471058423238594580183052756213910186997604305844068665712346869679456044155742100039179758348979935882751881524675930878928159243492197545387668305684668420775409821781247053354523194797398953320175988640281058825557698004397120538312459428957377696001857497335249965013509368925958021863811725906506436882127156815751021712900765992750370228283963962915973251173418586721023497317765969454283625519371556009143680329311962842546628403142444370648432390374906410811300792848955767243481200090309888457270907750873638873299642555050473812528975962934822878917619920725138309388288292510416837622758204081918933603653875284116785703720989718832986921927816629675844580174911809119663048187434155067790863948831489241504300476704527971283482211522202837062857314244107823792513645086677566622804977211397140621664116324756784216612961477109018826094677377686406176721484293894976671380122788941309026553511096118347012565197540807095384060916863936906673786627209429434264260402902158317345003727462588992622049877121178405563348492490326003508569099382392777297498413565614830788262363322368380709822346012274241379036473451735925215754757160934270935192901723954921426490691115271523338109124042812102893738488167358953934508930697715522989199698903885883275409044300321986834003470271220020159699371690650330547577095398748580670024491045504890061727189168031394528036165633941571334637222550477547460756055024108764382121688848916940371258901948490685379722244562009483819491532724502276218589169507405794983759821006604481996519360110261576947176202571702048684914616894068404140833587562118319210838005632144562018941505945780025318747471911604840677997765414830622179069330853875129298983009580277554145435058768984944179136535891620098725222049055183554603706533183176716110738009786625247488691476077664470147193074476302411660335671765564874440577990531996271632972009109449249216456030618827772947750764777446452586328919159107444252320082918209518021083700353881330983215894608680127954224752071924134648334963915094813097541433244209299930751481077919002346128122330161799429930618800533414550633932139339646861616416955220216447995417243171165744471364197733204899365074767844149929548073025856442942381787641506492878361767978677158510784235702640213388018875601989234056868423215585628508645525258377010620532224244987990625263484010774322488172558602233302076399933854152015343847725442917895130637050320444917797752370871958277976799686113626532291118629631164685159934660693460557545956063155830033697634000276685151293843638886090828376141157732003527565158745906567025439437931104838571313294490604926582363108949535090082673154497226396648088618041573977888472892174618974189721700770009862449653759012727015227634510874906948012210684952063002519011655963580552429180205586904259685261047412834518466736938580027700252965356366721619883672428226933950325930390994583168665542234654857020875504617520521853721567282679903418135520602999895366470106557900532129541336924472492212436324523042895188461779122338069674233980694887270587503389228395095135209123109258159006960395156367736067109050566299603571876423247920752836160805597697778756476767210521222327184821484446631261487584226092608875764331731023263768864822594691211032367737558122133470556805958008310127481673962019583598023967414489867276845869819376783757167936723213081586191045995058970991064686919463448038574143829629547131372173669836184558144505748676124322451519943362182916191468026091121793001864788050061351603144350076189213441602488091741051232290357179205497927970924502479940842696158818442616163780044759478212240873204124421169199805572649118243661921835714762891425805771871743688000324113008704819373962295017143090098476927237498875938639942530595331607891618810863505982444578942799346514915952884869757488025823353571677864826828051140885429732788197765736966005727700162592404301688659946862983717270595809808730901820120931003430058796552694788049809205484305467611034654748067290674399763612592434637719995843862812391985470202414880076880818848087892391591369463293113276849329777201646641727587259122354784480813433328050087758855264686119576962172239308693795757165821852416204341972383989932734803429262340722338155102209101262949249742423271698842023297303260161790575673111235465890298298313115123607606773968998153812286999642014609852579793691246016346088762321286205634215901479188632194659637483482564291616278532948239313229440231043277288768139550213348266388687453259281587854503890991561949632478855035090289390973718988003999026132015872678637873095678109625311008054489418857983565902063680699643165033912029944327726770869305240718416592070096139286401966725750087012218149733133695809600369751764951350040285926249203398111014953227533621844500744331562434532484217986108346261345897591234839970751854223281677187215956827243245910829019886390369784542622566912542747056097567984857136623679023878478161201477982939080513150258174523773529510165296934562786122241150783587755373348372764439838082000667214740034466322776918936967612878983488942094688102308427036452854504966759697318836044496702853190637396916357980928865719935397723495486787180416401415281489443785036291071517805285857583987711145474240156416477194116391354935466755593592608849200546384685403028080936417250583653368093407225310820844723570226809826951426162451204040711501448747856199922814664565893938488028643822313849852328452360667045805113679663751039248163336173274547275775636810977344539275827560597425160705468689657794530521602315939865780974801515414987097778078705357058008472376892422189750312758527140173117621279898744958406199843913365680297721208751934988504499713914285158032324823021340630312586072624541637765234505522051086318285359658520708173392709566445011404055106579055037417780393351658360904543047721422281816832539613634982525215232257690920254216409657452618066051777901592902884240599998882753691957540116954696152270401280857579766154722192925655963991820948894642657512288766330302133746367449217449351637104725732980832812726468187759356584218383594702792013663907689741738962252575782663990809792647011407580367850599381887184560094695833270775126181282015391041773950918244137561999937819240362469558235924171478702779448443108751901807414110290370706052085162975798361754251041642244867577350756338018895379263183389855955956527857227926155524494739363665533904528656215464288343162282921123290451842212532888101415884061619939195042230059898349966569463580186816717074818823215848647734386780911564660755175385552224428524049468033692299989300783900020690121517740696428573930196910500988278523053797637940257968953295112436166778910585557213381789089945453947915927374958600268237844486872037243488834616856290097850532497036933361942439802882364323553808208003875741710969289725499878566253048867033095150518452126944989251596392079421452606508516052325614861938282489838000815085351564642761700832096483117944401971780149213345335903336672376719229722069970766055482452247416927774637522135201716231722137632445699154022395494158227418930589911746931773776518735850032318014432883916374243795854695691221774098948611515564046609565094538115520921863711518684562543275047870530006998423140180169421109105925493596116719457630962328831271268328501760321771680400249657674186927113215573270049935709942324416387089242427584407651215572676037924765341808984312676941110313165951429479377670698881249643421933287404390485538222160837088907598277390184204138197811025854537088586701450623578513960109987476052535450100439353062072439709976445146790993381448994644609780957731953604938734950026860564555693224229691815630293922487606470873431166384205442489628760213650246991893040112513103835085621908060270866604873585849001704200923929789193938125116798421788115209259130435572321635660895603514383883939018953166274355609970015699780289236362349895374653428746875N
It ran out of heap space when I tried 1000000 :(.
A more readable example is ...
(map fibs (range 20))
;(0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597 2584 4181)
The point is that this function gives you constant time access to an already found value. You don't have to run down a sequence.
Edited to correct an error whereby the buffer (correctly) expanded and (wrongly) contracted with each offered argument.
this is probably because you reached the maximum recursion limit. In order to go around this one could rewrite the function to be tail recursive and then use the loop-recur pattern.
Newbie Clojure question. What are the pros and cons of the following two ways to implement/represent the Fibonacci sequence? (In particular, is there anything to completely rule out one or the other as a bad idea.)
(ns clxp.fib
(:gen-class))
; On the one hand, it seems more natural in code to have a function that
; returns 'a' Fibonacci sequence.
(defn fib-1
"Returns an infinite sequence of Fibonnaci numbers."
[]
(map first (iterate (fn [[a b]] [b (+ a b)]) [0 1])))
; > (take 10 (fib-1))
; (0 1 1 2 3 5 8 13 21 34)
; On the other hand, it seems more mathematically natural to define 'the'
; Fibonacci sequence once, and just refer to it.
(def fib-2
"The infinite sequence of Fibonnaci numbers."
(map first (iterate (fn [[a b]] [b (+ a b)]) [0 1])))
; > (take 10 fib-2)
; (0 1 1 2 3 5 8 13 21 34)
a) What are the pros and cons of these two approaches to defining an infinite sequence? (I am aware that this is a somewhat special case in that this particular sequence requires no args to be provided - unlike, say, an infinite sequence of multiples of 'n', which I believe would require the first approach, in order to specify the value of 'n'.)
b) Is there any over-arching reason to prefer one of these implementations to the other? (Memory consumption, applicability when used as an argument, etc.)
fib-2 is preferable in favor of time performance if its elements are looked up multiple times, because in a lazy seq they only need to be calculated one time.
Due to the global binding the seq is unlikely to ever become garbage collected, so if your program will step through a million fibonaccis to make some calculation once, even more so if it doesn't need to hold the seqs head, invoking fib-1 in a local context is preferable in favor of space performance.
It depends on your usage, and how critical it is not to have to recalculate fib seq multiple times. However, from my experiments below, I had issues with the def when using long sequences.
If you're going to be referring to a lot of elements, then you'll need to watch out for head retention, as Leon mentioned.
This can be illustrated as follows (these are extending a couple of examples from Clojure Programming):
(let [[t d] (split-with #(< % 10) (take 1e6 (fib-1)))]
[(count d) (count t)])
=> OutOfMemoryError Java heap space
(let [[t d] (split-with #(< % 10) (take 1e6 (fib-1)))]
[(count t) (count d)])
=> [7 999993]
Note, i had to change your implementation to use the initial vector [0 1N] to avoid ArithmeticException integer overflow when taking large sequences of fib numbers.
Interesting, changing to using fib-2 instead yields same OOM error for holding head case, but the non-head holding version breaks:
(let [[t d] (split-with #(< % 10) (take 1e6 fib-2))]
[(count t) (count d)])
=> [7 270036]
The latter figure should be 999993.
The reason for the OOM in both cases is as stated in Clojure Programming:
Since the last reference of t occurs before the processing of d, no
reference to the head of the range sequence is kept, and no memory
issues arise.
I am trying to write a sieve to generate prime numbers for a Project Euler problem.
The code looks like this:
(defn sieve [n]
(reduce (fn [memo f]
(remove #(and (= 0 (rem % f))
(not= % f))
memo))
(range 2 (+ 1 n))
(range 2 (+ 1 n))))
Until 500000 it runs very fast, under 1 sec, from 600000 and up it starts seg faulting
and crashing with memory errors.
I imagine it has something to do with remove and lazyness, i searched a bit, tried using (doall (remove ...)) instead of (remove) but it becomes incredibly slow...
I am a bit at a loss with this, thanks in advance for any help...
Segfault? That sounds scary! I assume you mean a stack overflow error.
When I run the function I start to get stack overflow errors at about 1000. So why the stack overflows? It has to do with laziness. The fix is to wrap the call to remove in a doall.
Reduce will iterate through each element in the sequence given as the third argument and keep a state along the way. This state is initialized as the rage of integers from 2 to n+1. The state is updated at each iteration using remove. However, since remove is lazy it won't actually do anything. Remove will return an object that can generate a sequence on demand, based on the sequence it was given. I will try to explain this thorugh an example:
(reduce (fn [xs x] (remove #{x} xs)) coll (range 4))
The above expression will return a sequence of the elements of coll, but with numbers 0 to 3 filtered out. To explain what happens at run time I will invent a new notation: I will write the runtime value of (remove #{x} xs) as «I.O.U. a seq like xs but with x removed». The value of the reduce expression is then:
«I.O.U. a seq like
«I.O.U. a seq like
«I.O.U. a seq like
«I.O.U. a seq like 'coll' but with 0 removed»
but with 1 removed»
but with 2 removed»
but with 3 removed»
Each call to remove adds a wrapping "I.O.U.". This causes problems when you finally try to access the first value of the resulting seq (the outermost "I.O.U." value). When a lazy-seq object is traversed, it is first checked whether its value has been calculated. If the value is already done, the value is simply returned. If it's not, then it's calculated, stored, and returned.
The problem arises when one lazy-seq ("I.O.U." value) needs to force another lazy-seq to be able to perform its work, because one stack frame is needed per lazy-seq realization. (A stack frame is needed in order to remember where to return to when the second lazy-seq is done.) If you have 1000 nested "I.O.U." values, you need 1000 stack frames to realize them (assuming all of them were unrealized initiallly).
The solution is to use the doall function from the Clojure standard library. It takes a seq and causes it to be fully realized. If you wrap the call to remove in a doall, the reduce state will always contain a fully realized seq between each iteration, and no cascade of "I.O.U." values will build up. Instead of saving all computations for later, you do them incrementally.
I'm not surprised - you're working with a huge list there.
I think you might need a fundamentally different algorithm. For example; to test if a number n is prime, you need to check if it can be divided by any prime <= the square root of n. Using this knowledge we can start building a list of primes by testing numbers in sequential order, adding each new prime to the list.
This is a slow algorithm but can be sped up by using a "wheel" that skips obvious non-primes (e.g. numbers that are divisible 2 or 3).
This is all of the top of my head, so apologies for any inaccuracies.
In principal, prime calculation is better suited to associateive structures like sets than iterative structures like lists. A few other StackOverflowers have contributed useful clojure answers:
The answer to this quiestion,
this question: why is this prime sieve implementation slower,
this question on fast prime generation in lisp
in general I like to break this sort of problem into individual steps so here is my answer using sets:
(defn next-not-in-sieve [sieve start]
(first (drop-while #(sieve %) (range 1 (inc start)))))
(defn add-multiples [sieve max n] (into sieve (range n (inc max) n)))
(defn primes [max sieve answer]
(let [next (next-not-in-sieve sieve max)]
(if (nil? next) answer
(recur max (add-multiples sieve max next) (conj answer next)))))
it runs sufficiently quickly for basic use, the point here is to learn Clojure, not find primes quickly of course :)
user> (time (def foo (sieve 60000)))
"Elapsed time: 63167.82765 msecs"
user> (time (def foo (primes 60000 #{1} [])))
"Elapsed time: 33272.157235 msecs"
And what would be the point of learning Clojure if we didn't make it into a lazy seq of primes:
(defn primes [max sieve]
(if-let [next (next-not-in-sieve sieve max)]
(lazy-seq (cons next (primes max (add-multiples sieve max next))))))
and check the time:
(time (def foo (doall (primes 60000 #{1}))))
"Elapsed time: 33714.880871 msecs"
and of course for thous who want some back ground check out the wikipedia page on prime sieves
I am trying to write a simple sieve function to calculate prime numbers in clojure. I've seen this question about writing an efficient sieve function, but I am not to that point yet. Right now I am just trying to write a very simple (and slow) sieve. Here is what I have come up with:
(defn sieve [potentials primes]
(if-let [p (first potentials)]
(recur (filter #(not= (mod % p) 0) potentials) (conj primes p))
primes))
For small ranges it works fine, but causes a stack overflow for large ranges:
user=> (sieve (range 2 30) [])
[2 3 5 7 11 13 17 19 23 29]
user=> (sieve (range 2 15000) [])
java.lang.StackOverflowError (NO_SOURCE_FILE:0)
I thought that by using recur this would be a non-stack-consuming looping construct? What am I missing?
You're being hit by filter's laziness. Change (filter ...) to (doall (filter ...)) in your recur form and the problem should go away.
A more in-depth explanation:
The call to filter returns a lazy seq, which materialises actual elements of the filtered seq as required. As written, your code stacks filter upon filter upon filter..., adding one more level of filtering at each iteration; at some point this blows up. The solution is to force the whole result at each iteration so that the next one will do its filtering on a fully realised seq and return a fully realised seq instead of adding an extra layer of lazy seq processing; that's what doall does.
Algorithmically the problem is that you continue filtering when there's no more purpose to it. Stopping as early as possible achieves quadratic reduction in recursion depth (sqrt(n) vs. n):
(defn sieve [potentials primes]
(if-let [p (first potentials)]
(if (> (* p p) (last potentials))
(concat primes potentials)
(recur (filter (fn [n] (not= (mod n p) 0)) potentials)
(conj primes p)))
primes))
Runs OK for 16,000 (performing just 30 iterations instead of 1862), and for 160,000 too, on ideone. Even runs 5% faster without the doall.