lein uberjar results in "Method code too large!" - clojure

Question

I have a clojure project that runs fine using lein run, but lein uberjar results in
java.lang.RuntimeException: Method code too large!, compiling:(some_file.clj:1:1)
for the same project. I could trace it down to some_file.clj using a large vector of more than 2000 entries that is defined something like this:
(def x
[[1 :qwre :asdf]
[2 :zxcv :fafa]
...
])
When removing enough entries from that vector, lein uberjar compiles without issues. How can I make uberjar do its job, leaving all entries in the vector?
N.B. When changing x to a constant a la (def ^:const x, lein run will also throw a Method too large! error. And btw, that error occurs in the place where x is used, so just defining the constant is fine, if it is not being used.

Answer 1

There is a 64kb limit on the size of the method in Java. It seems that in your case the method that creates the big vector literal exceeds this limit.
Actually, you can check it by yourself, using a fancy library called clj-java-decompiler. Here's a short example using boot:
(set-env!
:dependencies '[[com.clojure-goes-fast/clj-java-decompiler "0.1.0"]])
(require '[clj-java-decompiler.core :as d])
(d/decompile-form
{}
[[1 :qwre :asdf]
[2 :zxcv :fafa]
[3 :zxcv :fafa]])
;; ==> prints:
;;
;; // Decompiling class: cjd__init
;; import clojure.lang.*;
;;
;; public class cjd__init
;; {
;; public static final AFn const__10;
;;
;; public static void load() {
;; const__10;
;; }
;;
;; public static void __init0() {
;; const__10 = (AFn)Tuple.create((Object)Tuple.create((Object)1L, (Object)RT.keyword((String)null, "qwre"), (Object)RT.keyword((String)null, "asdf")), (Object)Tuple.create((Object)2L, (Object)RT.keyword((String)null, "zxcv"), (Object)RT.keyword((String)null, "fafa")), (Object)Tuple.create((Object)3L, (Object)RT.keyword((String)null, "zxcv"), (Object)RT.keyword((String)null, "fafa")));
;; }
;;
;; static {
;; __init0();
;; Compiler.pushNSandLoader(RT.classForName("cjd__init").getClassLoader());
;; try {
;; load();
;; Var.popThreadBindings();
;; }
;; finally {
;; Var.popThreadBindings();
;; }
;; }
;; }
;;
As you can see, there's a method __init0 that creates the actual Java object for the vector literal. Given enough elements in the vector, size of the method can easily exceed the 64kb limit.
I think, in your case, the easiest solution to this problem will be to put your vector to the file and then slurp+read this file. Something, like this:
file vector.edn:
[[1 :qwre :asdf]
[2 :zxcv :fafa]
...
]
and then in your code:
(def x (clojure.edn/read-string (slurp "vector.edn"))

Answer 2

It turns out ahead-of-time-compilation was the difference between lein run and lein uberjar, as my project.clj contains the (default) line
:profiles {:uberjar {:aot :all}})
when I remove the :aot :all, uberjar finishes without complaining - but also without aot-compiling. So I guess I need to turn off aot, or figure out to limit it to exclude that vector.
But I'd like to be able to have this - albeit huge - vector "literal", and still compile everything. But perhaps factoring that huge vector out into a data file, which is read in at startup is not too bad an idea either. Then I'd be able to leave the :aot :all setting as is.

Answer 3

would it work in your case top wrap the value in a call to delay to cause it to be computed when it's first used?
there are limits on how big a method can be in a java classfile, and each top level form in Clojure (usually) produces a class.
To fix this problem it's useful to arrange to have the constants generated and stored either:
in memory by calculating them at runtime
in the resources directory where they can be read at runtime
don't generate the class file by not compiling it in advance, this generates the data at program start.
if you use a future or make a memoized function to get the data you will compute it at program start and store it in memory rather than the class file.
if you put it in the resources directory it will not be subject to the size limit, and can still be computed at compile time.
if you disable AOT compilation, then the class limit will never be hit becuase it will be computed at load time when the program starts.