When run, the following creates a bunch of classes that gets loaded
(dotimes [i 1000000]
(eval `(fn [] (+ ~i 10))))
It's possible to inspect the objects by connecting to visualvm:
At some point, the vm decides to unload the objects but it's rather arbitrary. Is there a way of unloading these objects when there are 100k loaded instead of at 500k?
Consider why you want to get rid of them. Just lying around they're not doing anything except consuming memory, so I imagine you would like to reduce your memory footprint. So, tell the JVM that. I'm not an expert on JVM flags or performance, but my understanding is that in modern JVMs, loaded classes are GCed in a way similar to ordinary objects, but stored in a separate area called MetaSpace. If you know that your program never needs very many classes loaded at once you can limit that space by setting JVM option -XX:MaxMetaspaceSize.
As per #amalloy's suggestion. Setting -XX:MaxMetaspaceSize=256m produces this (which is the required effect)
Related
Say I'm debugging code where one or more of the functions involved is defined with the help of memoize. I'll edit some code, reload the file in the REPL, and try out the new code. But if the bug is still there I always question whether it's because I haven't fixed the bug or because memoize has cached buggy results.
So, is there some way short of restarting the REPL that I can use to make absolutely sure that memoize has lost its memory?
(Note that eliminating calls to memoize during REPL sessions is both tedious and sometimes even unpractical, because the performance of the function might rely heavily on memoization.)
memoize never, under any circumstances, empties its cache. Its storage is permanent. If you have a new function you wish to use, you must replace your memoized function by re-memoizing the underlying function, and use only the new version of the function, not the old one. This way, your function calls will be passed through to the new underlying function, and the memory used to cache results of the old function will become eligible for garbage collection because nothing points at it.
You may say, Well gosh this is a pain, why is memoize so inflexible! The answer is, memoize is a very blunt instrument, not well suited to almost any production usages. Anytime you memoize a function whose set of possible inputs is not limited, you introduce a memory leak. If your function depends on a cache for its performance, you should think more about a more flexible caching policy than "cache everything forever", and use a library designed for such use cases.
What are tradeoffs of representing state using a single atom and a hashmap vs multiple refs?
For example:
(def start (atom {:location "Chicago" :employer "John"}))
vs
(def location (ref "Chicago"))
(def employer (ref "John"))
Many thanks
The single atom version is better and has less tradeoffs. Given that you don't want to change the employer and the location uncoordinated, your win is that you don't have to create a dosync block to change either location or employer or both. Using the atom, you can simply (swap! start assoc :location "baz").
A big tradeoff of using multiple refs is that all transactions to refs will be tried in parallel and the first one who is ready wins, the others will be restarted. While that is also true for atoms, having more refs for all entries requires more monitoring, grouping (for dosync blocks) etc. behind the scenes. To have less restarts, it makes sense to group the information in a hash-map. Depending on whether coordinated change is required, put it in a ref or an atom.
Multiple Refs allow for more concurrency, since all writes to an Atom are linearized. The STM allows for many parallel transactions to commit when there are no conflicting writes / ensures (and additionally it provides commute which allows one to make certain writes which would normally cause a conflict to not do so).
Additionally, the STM cooperates with Agents -- actions sent to Agents from within a transaction will be performed if and only if the transaction commits. This allows one to cause side effects from inside a transaction safely. Atoms offer no similar facility.
The trade-off is the STM's overhead is larger than an Atom's, plus there is the possibility of certain anomalies occurring (write skew, see the Wikipedia page on snapshot isolation). Additionally, it's possible to achieve great concurrency with the STM while having serious problems with obtaining a snapshot of the entire system; in this connection, see Christophe Grand's excellent blog post and his megaref library.
In many scenarios people find that just storing all state in a single Atom is enough and that's definitely a simpler, more lightweight approach.
I don't think you should be thinking about trade-offs between atoms vs. refs, since they're used for different situations.
You should use an atom when you want to change a single thing atomically.
refs use STM and involve many different things being changed simultaneously, in a transaction.
In your particular situation you should be answering the question about the thing you're changing.
Is it a single thing you want/can change in one step
Are different things you want/need to change transactionally
If you switch the old database for the new database and change everything together to change a single field, and so you say your database is an atom, you're abusing the mechanism.
Hope the distinction helps, for your example I would use an atom none the less.
There's a good summary here with motivations behind each strategy.
Disclaimer: I apologize for the verbosity of this question (I think it's an interesting problem, though!), yet I cannot figure out how to more concisely word it.
I have done hours of research as to the apparently myriad of ways in which to solve the problem of accessing multi-GB files in a 32-bit process on 64-bit Windows 7, ranging from /LARGEADDRESSAWARE to VirtualAllocEx AWE. I am somewhat comfortable in writing a multi-view memory-mapped system in Windows (CreateFileMapping, MapViewOfFile, etc.), yet can't quite escape the feeling that there is a more elegant solution to this problem. Also, I'm quite aware of Boost's interprocess and iostream templates, although they appear to be rather lightweight, requiring a similar amount of effort to writing a system utilizing only Windows API calls (not to mention the fact that I already have a memory-mapped architecture semi-implemented using Windows API calls).
I'm attempting to process large datasets. The program depends on pre-compiled 32-bit libraries, which is why, for the moment, the program itself is also running in a 32-bit process, even though the system is 64-bit, with a 64-bit OS. I know there are ways in which I could add wrapper libraries around this, yet, seeing as it's part of a larger codebase, it would indeed be a bit of an undertaking. I set the binary headers to allow for /LARGEADDRESSAWARE (at the expense of decreasing my kernel space?), such that I get up to around 2-3 GB of addressable memory per process, give or take (depending on heap fragmentation, etc.).
Here's the issue: the datasets are 4+GB, and have DSP algorithms run upon them that require essentially random access across the file. A pointer to the object generated from the file is handled in C#, yet the file itself is loaded into memory (with this partial memory-mapped system) in C++ (it's P/Invoked). Thus, I believe the solution is unfortunately not as simple as simply adjusting the windowing to access the portion of the file I need to access, as essentially I want to still have the entire file abstracted into a single pointer, from which I can call methods to access data almost anywhere in the file.
Apparently, most memory mapped architectures rely upon splitting the singular process into multiple processes.. so, for example, I'd access a 6 GB file with 3x processes, each holding a 2 GB window to the file. I would then need to add a significant amount of logic to pull and recombine data from across these different windows/processes. VirtualAllocEx apparently provides a method of increasing the virtual address space, but I'm still not entirely sure if this is the best way of going about it.
But, let's say I want this program to function just as "easily" as a singular 64-bit proccess on a 64-bit system. Assume that I don't care about thrashing, I just want to be able to manipulate a large file on the system, even if only, say, 500 MB were loaded into physical RAM at any one time. Is there any way to obtain this functionality without having to write a somewhat ridiculous, manual memory system by hand? Or, is there some better way than what I have found through thusfar combing SO and the internet?
This lends itself to a secondary question: is there a way of limiting how much physical RAM would be used by this process? For example, what if I wanted to limit the process to only having 500 MB loaded into physical RAM at any one time (whilst keeping the multi-GB file paged on disk)?
I'm sorry for the long question, but I feel as though it's a decent summary of what appear to be many questions (with only partial answers) that I've found on SO and the net at large. I'm hoping that this can be an area wherein a definitive answer (or at least some pros/cons) can be fleshed out, and we can all learn something valuable in the process!
You could write an accessor class which you give it a base address and a length. It returns data or throws exception (or however else you want to inform of error conditions) if error conditions arise (out of bounds, etc).
Then, any time you need to read from the file, the accessor object can use SetFilePointerEx() before calling ReadFile(). You can then pass the accessor class to the constructor of whatever objects you create when you read the file. The objects then use the accessor class to read the data from the file. Then it returns the data to the object's constructor which parses it into object data.
If, later down the line, you're able to compile to 64-bit, you can just change (or extend) the accessor class to read from memory instead.
As for limiting the amount of RAM used by the process.. that's mostly a matter of making sure that
A) you don't have memory leaks (especially obscene ones) and
B) destroying objects you don't need at the very moment. Even if you will need it later down the line but the data won't change... just destroy the object. Then recreate it later when you do need it, allowing it to re-read the data from the file.
I am looking for a way to store several gb's of data in memory. The data is loaded into a tree structure. I want to be able to access this data through my main function, but I'm not interested in reloading the data into the tree every time I run the program. What is the best way to do this? Should I create a separate program for loading the data and then call it from the main function, or are there better alternatives?
thanks
Mads
I'd say the best alternative would be using a database - which would be then your "separate program for loading the data".
If you are using a POSIX compliant system, then take a look into mmap.
I think Windows has another function to memory map a file.
You could probably solve this using shared memory, to have one process that it long-lived build the tree and expose the address for it, and then other processes that start up can get hold of that same memory for querying. Note that you will need to make sure the tree is up to being read by multiple simultaneous processes, in that case. If the reads are really just pure reads, then that should be easy enough.
You should look into a technique called a Memory mapped file.
I think the best solution is to configure a cache server and put data there.
Look into Ehcache:
Ehcache is an open source, standards-based cache used to boost
performance, offload the database and simplify scalability. Ehcache is
robust, proven and full-featured and this has made it the most
widely-used Java-based cache.
It's written in Java, but should support any language you choose:
The Cache Server has two apis: RESTful resource oriented, and SOAP.
Both support clients in any programming language.
You must be running a 64 bit system to use more than 4 GB's of memory. If you build the tree and set it as a global variable, you can access the tree and data from any function in the program. I suggest you perhaps try an alternative method that requires less memory consumption. If you post what type of program, and what type of tree you're doing, I can perhaps give you some help in finding an alternative method.
Since you don't want to keep reloading the data...file storage and databases are out of question, but several gigs of memory seem like such a hefty price.
Also note that on Windows systems, you can access the memory of another program using ReadProcessMemory(), all you need is a pointer to use for the location of the memory.
You may alternatively implement the data loader as an executable program and the main program as a dll loaded and unloaded on demand. That way you can keep the data in the memory and be able to modify the processing code w/o reloading all the data or doing cross-process memory sharing.
Also, if you can operate on the raw data from the disk w/o making any preprocessing of it (e.g. putting it in a tree, manipulating pointers to its internals), you may want to memory-map the data and avoid loading unused portions of it.
I have an application that, in order to reload plugins, requires them with the :reload option whenever they are to be reloaded. I've noticed that this is building up memory about 2-3 megs at a time when I do it. I'm curious as to what could cause this sort of thing to happen. Is data from previous reloads being kept in memory? Is there a way to totally reload a namespace?
EDIT: It's also relevant to mention that each of these plugins that gets reloaded makes new defmethods for a multimethod in another namespace (that never gets reloaded). Maybe the methods are being kept in memory when it's reloaded?
Clojure defers memory management to the JVM. While I don't know clojure's codebase deeply, it probably just reassigns the vars with the reloaded code - which will leave the old objects around until the JVM runs the garbage collector.
You can hint to the JVM that you want the GC to run using (System/gc), but it's generally not recommended to use.
Alternatively, if you know the contraints of your system, you can tinker with the JVM memory flags to encourage the GC to run more frequently (ie - use a lower heap size).
But if you have a system that's not really memory constrained, saving a few mbs doesn't matter much.
As it turns out, I didn't test it long enough. The memory will only grow to a certain level, and then it'll stop and eventually go back down quite a bit.
Boys and girls: test your code before whining about bugs.