I recently ran into an issue where including a new dependency in my project broke the functionality of another, unrelated one.
The two in question:
[amazonica "0.3.132"]
[abengoa/clj-stripe "1.0.4"]
Including the above stripe breaks the amazon one, and throws the following error for s3 operations
ERROR org/apache/http/conn/SchemePortResolver
java.lang.NoClassDefFoundError: org/apache/http/conn/SchemePortResolver
at com.amazonaws.http.apache.client.impl.ApacheHttpClientFactory.<init>(ApacheHttpClientFactory.java:41)
Removing the stripe library fixes the Amazon one, so I've come to suspect there's some sort of conflict there. My question, is how do I figure out what is causing the issue, and fix it.
I found a similar issue on the repo: https://github.com/mcohen01/amazonica/issues/294
Which was fixed with
[clj-facebook-graph "0.4.0" :exclusions [clj-http]]
But I'd like to know how I'd be able to find out what is in the Stripe library that is causing the issue, and how I can go in and fix it without breaking the library. I'm also interested in more general ideas on how to do this so should I encounter it again I can fix the issue.
If you're using Leiningen, you can use lein deps :tree to print helpful debugging information, and before it prints the dependency tree it will print suggested exclusions to avoid these types of issues:
$ lein deps :tree
...
[com.taoensso/faraday "1.9.0"] -> [com.taoensso/encore "2.67.2"] -> [com.taoensso/truss "1.3.3"]
overrides
[amazonica "0.3.112" :exclusions [com.amazonaws/amazon-kinesis-client com.amazonaws/aws-java-sdk com.amazonaws/dynamodb-streams-kinesis-adapter]] -> [com.taoensso/nippy "2.12.2"] -> [com.taoensso/encore "2.68.0"] -> [com.taoensso/truss "1.3.4"]
Consider using these exclusions:
[amazonica "0.3.112" :exclusions [com.amazonaws/amazon-kinesis-client com.taoensso/truss com.amazonaws/dynamodb-streams-kinesis-adapter com.amazonaws/aws-java-sdk]]
...
In my experience there's still some trial and error involved in excluding the precise transitive conflicts. I've had similar issues with AWS SDK, Jackson, etc.
As it is with problems like that, there is no real silver bullet. The best start to find problems in a leiningen project is to run the deps task. E.g.:
lein deps :tree
Which e.g. in a random project results in the following:
Possibly confusing dependencies found:
[org.clojure/clojurescript "1.9.293"] -> [org.clojure/tools.reader "1.0.0-beta3"]
overrides
[cljfmt "0.6.0" :exclusions [org.clojure/clojure]] -> [rewrite-cljs "0.4.4"] -> [org.clojure/tools.reader "1.0.5"]
and
[cljfmt "0.6.0" :exclusions [org.clojure/clojure]] -> [org.clojure/tools.reader "1.2.2"]
Consider using these exclusions:
[cljfmt "0.6.0" :exclusions [org.clojure/clojure org.clojure/tools.reader]]
Those are hints to start from. After that initial block you will see the whole tree of transitive deps. E.g.:
[cljfmt "0.6.0" :exclusions [[org.clojure/clojure]]]
[com.googlecode.java-diff-utils/diffutils "1.3.0"]
[org.clojure/tools.cli "0.3.7"]
[rewrite-clj "0.6.0"]
[rewrite-cljs "0.4.4"]
And you can try to investigate more from there.
Related
I'm calling some Clojure (1.8) code in a larger Java project that I don't entirely have control over. Recently some code has been added to the parent that's causing conflicts with the HTTP client clj-http.
From my Clojure project, here's the output of lein deps :tree|grep http:
[clj-http "3.7.0"]
[org.apache.httpcomponents/httpasyncclient "4.1.3" :exclusions [[org.clojure/clojure]]]
[org.apache.httpcomponents/httpcore-nio "4.4.6"]
[org.apache.httpcomponents/httpclient "4.5.3" :exclusions [[org.clojure/clojure]]]
[org.apache.httpcomponents/httpcore "4.4.6" :exclusions [[org.clojure/clojure]]]
[org.apache.httpcomponents/httpmime "4.5.3" :exclusions [[org.clojure/clojure]]]
Running mvn dependency:tree|grep http for the other project returns the following (note that this project is also a child of the parent):
[INFO] +- org.apache.httpcomponents:httpclient:jar:4.1.1:compile
[INFO] | +- org.apache.httpcomponents:httpcore:jar:4.1:compile
Is there a way I can ignore conflicts originating outside of my Clojure code, in lein (or otherwise)?
it's necessary to slog through the conflicts and fix them. There are two approaches:
Exclusions:
Exclude each dependency that is reported in lein deps tree from each of the upstream dependencies where it's provided. This gives you complete control when you need to make sure a transitive dependency is not included. It can though ... "be a bit of a bother" ;-)
Managed Dependencies:
You can also add a managed-dependencies section to your project.clj where you specify the exact version of libraries that will be used, regardless of the resolution of other transitive dependencies. This is a somewhat more blunt instrurment, and one i tend to turn to.
A random example from a project:
:managed-dependencies [[http-kit "2.3.0"]
[org.clojure/clojure "1.9.0"]
[ring/ring-core "1.6.0"]
[ring/ring-jetty-adapter "1.6.0"]
... lots more ...]
I want to use some of the latest features being built frequently at https://github.com/Netflix/RxJava in Clojure, but am having some difficulties getting Leiningen to reference the local .jar that I build. The last version of rxjava released to Clojars was 0.9.0, which I can successfully reach with the following in my projects.clj
:dependencies [[org.clojure/clojure "1.5.1"]
[com.netflix.rxjava/rxjava-clojure "0.9.0"]]
Now, I do a successful build of the current rxjava sources, which produces the following files
/Users/rebcabin/Documents/RxJava/language-adaptors/rxjava-clojure/build/libs/rxjava-clojure-0.9.1-SNAPSHOT-javadoc.jar
/Users/rebcabin/Documents/RxJava/language-adaptors/rxjava-clojure/build/libs/rxjava-clojure-0.9.1-SNAPSHOT-sources.jar
/Users/rebcabin/Documents/RxJava/language-adaptors/rxjava-clojure/build/libs/rxjava-clojure-0.9.1-SNAPSHOT.jar
In my Clojure project directory, I do the following
mvn deploy:deploy-file \
-DgroupId=local \
-DartifactId=rxjava-clojure \
-Dversion=0.9.1-SNAPSHOT \
-Dpackaging=jar \
-Dfile=/Users/rebcabin/Documents/RxJava/language-adaptors/rxjava-clojure/build/libs/rxjava-clojure-0.9.1-SNAPSHOT.jar \
-Durl=file:maven_repository
following the instructions given here: https://gist.github.com/stuartsierra/3062743 (see the bottom) via http://www.pgrs.net/2011/10/30/using-local-jars-with-leiningen/ and leiningen - how to add dependencies for local jars?.
That reports success and produces the following:
$ find maven_repository/
maven_repository/
maven_repository//local
maven_repository//local/rxjava-clojure
maven_repository//local/rxjava-clojure/0.9.1-SNAPSHOT
maven_repository//local/rxjava-clojure/0.9.1-SNAPSHOT/maven-metadata.xml
maven_repository//local/rxjava-clojure/0.9.1-SNAPSHOT/maven-metadata.xml.md5
maven_repository//local/rxjava-clojure/0.9.1-SNAPSHOT/maven-metadata.xml.sha1
maven_repository//local/rxjava-clojure/0.9.1-SNAPSHOT/rxjava-clojure-0.9.1-20130628.172154-1.jar
maven_repository//local/rxjava-clojure/0.9.1-SNAPSHOT/rxjava-clojure-0.9.1-20130628.172154-1.jar.md5
maven_repository//local/rxjava-clojure/0.9.1-SNAPSHOT/rxjava-clojure-0.9.1-20130628.172154-1.jar.sha1
maven_repository//local/rxjava-clojure/0.9.1-SNAPSHOT/rxjava-clojure-0.9.1-20130628.172154-1.pom
maven_repository//local/rxjava-clojure/0.9.1-SNAPSHOT/rxjava-clojure-0.9.1-20130628.172154-1.pom.md5
maven_repository//local/rxjava-clojure/0.9.1-SNAPSHOT/rxjava-clojure-0.9.1-20130628.172154-1.pom.sha1
maven_repository//local/rxjava-clojure/maven-metadata.xml
maven_repository//local/rxjava-clojure/maven-metadata.xml.md5
maven_repository//local/rxjava-clojure/maven-metadata.xml.sha1
I now fix my projects.clj file to contain the following:
:dependencies [[org.clojure/clojure "1.5.1"]
[com.netflix.rxjava/rxjava-clojure "0.9.1"]]
:repositories {"local" ~(str (.toURI (java.io.File. "maven_repository")))}
but lein deps fails to find the local repo
$ lein deps
Could not find artifact com.netflix.rxjava:rxjava-clojure:jar:0.9.1 in central (http://repo1.maven.org/maven2/)
Could not find artifact com.netflix.rxjava:rxjava-clojure:jar:0.9.1 in clojars (https://clojars.org/repo/)
Could not find artifact com.netflix.rxjava:rxjava-clojure:jar:0.9.1 in local (file:/Users/rebcabin/Documents/ClojureProjects/rxjava/expt1/maven_repository/)
This could be due to a typo in :dependencies or network issues.
The following guesses in project.clj also do not work:
:dependencies [[org.clojure/clojure "1.5.1"]
[com.netflix.rxjava/rxjava-clojure "0.9.1-SNAPSHOT"]]
:repositories {"local" ~(str (.toURI (java.io.File. "maven_repository")))}
:dependencies [[org.clojure/clojure "1.5.1"]
[com.netflix.rxjava/rxjava-clojure "0.9.1--20130628.172154-1"]]
:repositories {"local" ~(str (.toURI (java.io.File. "maven_repository")))}
any ideas how to proceed, please & thanks?
(I also answered on the mailing list where you asked this)
The groupId in your deploy-file command is incorrect. It should be com.netflix.rxjava. Then in your project.clj, use [com.netflix.rxjava/rxjava-clojure "0.9.1-SNAPSHOT"] for your dependency. Also note that you'll need to do the same with rxjava.core.
from the rxjava project directory run
mvn install
to install the jars to your system's local maven repo.
from your Clojure project's directory add 0.9.1-SNAPSHOT as a dependency and then
lein deps :tree
and make sure you see the correct version used.
I am using lein 2.0
If a main application
(defproject rexfer "1.0.1-SNAPSHOT"
:description "Filters standard report from Patriot Properties' AssessPro."
:dependencies [[org.clojure/clojure "1.4.0"]
[org.clojure/tools.cli "0.2.2"]
[org.clojure/data.csv "0.1.2"]
[rexfer-csv "1.0.0-SNAPSHOT"]
[util "1.0.7-SNAPSHOT"]]
:omit-source true
:main rexfer.core)
depends on a library (util 1.07-SNAPSHOT)
(defproject util "1.0.7-SNAPSHOT"
:description "A general purpose Clojure library"
:dependencies [[org.clojure/clojure "1.4.0"]
[clojure-csv/clojure-csv "1.3.2"]
[org.clojure/data.csv "0.1.2"]]
:aot [util.core]
:omit-source true)
is the reference fetching a .jar file or a standalone .jar?
Another way to ask this question is should I be building standalone jars for libraries like
util "1.0.7-SNAPSHOT"
that require a main application to use them?
You should deploy the jar produced by lein jar rather than lein uberjar. You can also safely expect all reasonable libraries to be deployed in this way.
This is roughly because of the following:
correctly prepared jars contain metadata on their dependencies, so users will know to fetch the dependencies along with your jar;
it often happens that a project uses several libraries which share common dependencies -- it would be wasteful to have each shared dependency replicated in several jars;
the user may have good reason to override some of your dependency choices (which happens all the time, incidentally; replacing a transitive dependency with a different version or excluding a transitive dependency is common) -- with a regular jar they will be able to do so easily, while it would be a pain with an überjar.
Of course if the coordinates of some of your dependencies do happen to point at überjars, that's what's going to end up being pulled in by Leiningen. I've never come across a library misdeployed like that though.
If you don't want to run lein uberjar then the answer to your second question would be yes. These jars need to be available on the class path for the main application to run. The job of lein uberjar is to include all the dependencies and dependencies of dependencies etc.
Can anyone tell me some working dependencies to get up and running with zeromq and clojure?
I've tried several but leiningen isn't able to fetch them:
(Could not find artifact org.zmq:zmq:jar:2.1.0 in central (http://repo1.maven.org/maven2))
[org.zmq/zmq "2.1.0"]
[org.zmq/jzmq "1.0.0"]
I have compiled jzmq (/usr/local/share/java/jzmq.jar) and added this to my project.clj:
:native-path "/usr/local/lib"
There's a 2.0-SNAPSHOT here:
Lein should already have the clojars repo loaded.
What I propose is a mixture of what's already been proposed, but for the sake of completeness and hopefully to give a final answer I give it a try.
Since the dependency is not in the online repos I would include the jar(s) in the project's directory structure itself, e.g. in the directory repository and keep it in the source control system as the other files of the project. It's an important part of the project and without the dependency it won't run.
In this directory I would save the jar with the help of Maven Install plugin.
mvn install:install-file \
-Dfile=/usr/local/share/java/jzmq.jar \
-DgroupId=org.zeromq \
-DartifactId=jzmq \
-Dversion=2.1.0 \
-Dpackaging=jar \
-DlocalRepositoryPath=repository
When the jar file gets copied to the local repository, you define it and the dependency in project.clj as follows:
(defproject clojure-interal-repo-test "0.1.0-SNAPSHOT"
:dependencies [[org.clojure/clojure "1.4.0"]
[org.zeromq/jzmq "2.1.0"]]
:repositories [["zeromq-repository" {:url "file:repository"
:snapshots false
:checksum :ignore
:update :never}]])
Within the project, run lein2 deps :tree to verify its correctness.
$ lein2 deps :tree
Retrieving org/zeromq/jzmq/2.1.0/jzmq-2.1.0.jar (4k) from file:repository/
[org.clojure/clojure "1.4.0"]
[org.zeromq/jzmq "2.1.0"]
Please note that the 4k above is the size of a fake file I created to test it out.
Read the document Repeatability in Leiningen's wiki should you need a bit more.
The only way I could get it to work was to use jeromq.
[org.zeromq/jeromq "0.3.2"]
Jeromq is native Java.
You can create a Maven local repository, install the compiled library into your local repo, and then, add the following to your project.clj
:repositories {"local" ~(str (.toURI (java.io.File. "your_local_repository_path")))}
Similar question, I have answered earlier here
Unfortunately, ZeroMQ is not present in public repository, at least at the last time I checked (a month ago I think). So you have to install the jar manually:
mvn install:install-file -Dfile=/usr/local/share/java/jzmq.jar -DgroupId=org.zeromq \
-DartifactId=jzmq -Dversion=2.1.0 -Dpackaging=jar
Then you can use the artifact as [org.zeromq/jzmq "2.1.0"] in your project.clj.
This is not a problem question, but a use of tools -- leiningen -- question.
Is there an advantage to creating hierarchical lein projects under one main project, and, if so, what is that advantage?
If I create a project using lein new bene-cmp, which is a Clojure "main" program
(defproject bene-cmp "1.0.4-SNAPSHOT"
:description "This is the main benetrak/GIC comparison program."
:dependencies [[org.clojure/clojure "1.4.0"]
[org.clojure/tools.cli "0.1.0"]
[clojure-csv/clojure-csv "1.3.2"]
[org.clojure/data.csv "0.1.2"]
[bene-csv "1.0.4-SNAPSHOT"]
[util "1.0.4-SNAPSHOT"]]
:omit-source true
:main bene-cmp.core)
and I want to write some Clojure scripts that will sanitize input files that will eventually be fed into bene-cmp by a bash script, should these be projects at the same level as bene-cmp, or under the bene-cmp project, and why?
Thanks.
I always look at the release cycle of the projects to determine if they are subprojects or "same level" projects. If you are always going to release the bene-sanitizers at the same time that bene-cmp, then they are subprojects or even just namespaces in the bene-cmp.