In Clojure, if I want to start a test cluster using the hbase testing utility, I have to annotate my dependencies with:
[org.apache.hbase/hbase "0.92.2" :classifier "tests" :scope "test"]
First of all, I have no idea what this means. According to leiningens sample project.clj
;; Dependencies are listed as [group-id/name version]; in addition
;; to keywords supported by Pomegranate, you can use :native-prefix
;; to specify a prefix. This prefix is used to extract natives in
;; jars that don't adhere to the default "<os>/<arch>/" layout that
;; Leiningen expects.
Question 1: What does that mean?
Question 2: If I upgrade the version:
[org.apache.hbase/hbase "0.94.6.1" :classifier "tests" :scope "test"]
Then I receive a ClassNotFoundException
Exception in thread "main" java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration
Whats going on here and how do I fix it?
any key / value pairs added to a dependency declaration are used as arguments to the clojure pomegranate library
the keys recognized are listed in the source here: in the source to pomagranate (for future reference that is a link to the function resolve-artifacts*)
the maven pom docs may also be helpful
:scope describes the conditions where the dependency is used, so :scope "test" would seem to indicate that the dependency is only pulled in while testing
:classifier seems to indicate the an extra element distinguishing versions
I speculate that the dependency resolution for the newer hbase version may have a poorly configured pom which is not declaring its dependencies properly. Try finding the info for org.apache.hadoop.hbase.HBaseConfiguration and requiring the package manually.
Leinigen uses the Maven dependency mechanism. Read that link to understand that different scopes.
The "classifier" is a token that is part of the dependency coordinates, so that a group of Jars / zip files / etc. can be part of the same logic release, but declared as distinct dependencies in your pom.xml. So in this case "tests" is a distinct HBase artifact from 0.94.6.1 that contains the tests.
You can see this in action by pointing your browser to the Maven "Central" repo location for that version of HBase:
http://repo1.maven.org/maven2/org/apache/hbase/hbase/0.94.6.1/
You can search the maven "central" repo here:
https://repository.apache.org/
or
http://mvnrepository.com/
On ClassNotFOundException - agreed with noisesmith. Your best bet is to find the dependency (jar) that contains that class and explicitly add it to your project dependency configuration.
Usually I do a google search for the classname and "jar" i.e.
https://www.google.com/search?q=jar+org.apache.hadoop.hbase.HBaseConfiguration
Related
Currently in our project, we're using io.xapix/paos clojure library to communicate with a SOAP WS. The library brings in few unlicensed transitive dependencies with it. So, we want an alternative to the paos library.
List of actions performed till now:
Tried to exclude the transitive dependency, but it didn't go well, met ClassNotFoundException :(
Tried to check clj-soap, but that seems to me not maintained anymore
Can anyone from the community can help us finding an alternative to paos.
I was able to solve deps only with the locally downloaded jar + extra wsdl4j-fixed. It works but it is a workaround, not an elegant solution.
{:mvn/repos
{"enonic" {:url "https://repo.enonic.com/public/"}}
:deps
{io.xapix/paos {:mvn/version "0.2.5"}
wsrf/wsrf-xbeans {:local/root "wsrf-xbeans-1.0.jar"}
wsdl4j/wsdl4j {:mvn/version "1.6.2-fixed"}}}
I am using lein repl without a project so there is no project.clj.
I am running Leiningen 2.8.1 on Java 1.8.0_191 OpenJDK 64-Bit Server VM.
When I require a Clojure dependency that I assume should just work - like clojure.data.json - I notice that it is not in my .m2 directory. Is that why I am getting a FileNotFoundException Could not locate clojure/data/json__init.class or clojure/data/js
on.clj on classpath? I can't find my other Clojure dependencies there either so I don't know where they reside and if this dependancy should be in .m2 or not.
I understand the error message but without knowing its location or even knowing how to properly add it to the CLASSPATH for the REPL to see it, I remain stuck.
Is this a dependency that I still need to install? If so, how do I install it without going through a project?
I don't understand the JVM as I am new to it, so add a little extra information in your answer.
I have looked at this, this, this, this and this. I don't know if I am overlooking anything so your help will really be appreciated.
I am using lein run without a project so there is no project.clj.
If you're using Leiningen, this'll be much easier if you create a project.clj file that declares your dependencies. Leiningen will read project.clj and handle fetching any missing dependencies to your local Maven repository, and add them to your classpath when you start your REPL/application. (lein run doesn't work for me in a directory without a project.clj; I get an error: No :main namespace specified in project.clj.. Did you mean lein repl?)
When I require a Clojure dependency that I assume should just work - like clojure.data.json - I notice that it is not in my .m2 directory.
clojure.data.json doesn't ship with Clojure — it's a separate dependency that must be fetched and added to your classpath in order to use it. The classpath tells the JVM where to look when it loads class files. Leiningen will do both of these things for you if you declare the dependency in project.clj:
:dependencies [[org.clojure/clojure "1.10.0"]
[org.clojure/data.json "0.2.6"]]
You can also use the lein deps command if you only want to fetch dependencies.
You can create a new/blank Leiningen project with lein new project_name_goes_here. It will have a project.clj with a few boilerplate entries and a :dependencies key where you can declare dependencies.
I understand the error message but without knowing its location or even knowing how to properly add it to the CLASSPATH for the REPL to see it, I remain stuck. Is this a dependency that I still need to install? If so, how do I install it without going through a project?
You could manually download it from the internet, then manually add its path to your classpath, but if you're already using Leiningen it's much easier to add a line to a project.clj file and have Leiningen handle this for you.
If using a project.clj file w/Leiningen isn't an option, there are other ways to use Clojure and resolve dependencies/build a classpath at runtime. Boot accommodates this workflow, you can use Leiningen like this with a little added effort, as well as the newer tools.deps tooling. There are examples of each in this ClojureVerse thread, but note that some of these approaches are doing essentially the same thing as declaring the dependency in a file — instead declaring them as CLI arguments.
For example, using Clojure CLI tooling:
$ clj -Sdeps "{:deps {org.clojure/data.json {:mvn/version \"0.2.6\"}}}"
Clojure 1.9.0
user=> (require '[clojure.data.json :as json])
nil
user=> (json/write-str {:foo "bar"})
"{\"foo\":\"bar\"}"
user=> (System/getProperty "java.class.path")
"src:
/Users/me/.m2/repository/org/clojure/clojure/1.9.0/clojure-1.9.0.jar:
/Users/me/.m2/repository/org/clojure/data.json/0.2.6/data.json-0.2.6.jar:
/Users/me/.m2/repository/org/clojure/spec.alpha/0.1.143/spec.alpha-0.1.143.jar:
/Users/me/.m2/repository/org/clojure/core.specs.alpha/0.1.24/core.specs.alpha-0.1.24.jar"
You could create a deps.edn file containing {:deps {org.clojure/data.json {:mvn/version \"0.2.6\"}}} in the same directory, and clj would read that, resolve the dependencies if necessary, and build the classpath accordingly.
This is a great opportunity to use lein try. Once you add it to your ~/.lein/profiles.clj, you'd simply run: lein try org.clojure/data.json and you'll be greeted with a running REPL with that dependency just a require away.
Trying to use Clojure Deps and CLI, I was surprised to find out that the following all worked for using clojure.data.json.
Maven dependency:
{:deps {org.clojure/data.json {:mvn/version "0.2.6"}}}
Git dependency with same key:
{:deps {org.clojure/data.json {:git/url "https://github.com/clojure/data.json.git"
:sha "13e9d244678be7b235bb24a10310f9d147ea088d"}}}
Git dependency with random key:
{:deps {lol/this-works {:git/url "https://github.com/clojure/data.json.git"
:sha "13e9d244678be7b235bb24a10310f9d147ea088d"}}}
With Maven and Clojars dependencies, :deps keys identify the artifact. When using a git sha instead, the name doesn't seem to matter.
What do Clojure deps.edn :deps keys mean?
How should I choose my :deps keys?
Resources I've read but may contain what I'm after:
Clojure Deps and CLI Guide
Clojure Deps and CLI Reference
It would seem that this is a result of how the git "extension" is handled in the tooling when compared to other "extensions" like maven. All the relevant code for this can be found here. I'll also make it clear that I haven't read this code in depth, and so my knowledge of this code therefore is not deep.
If you look at the way the lib is treated in the maven extension for example, it appears that it is actually checking maven to see that the artifact exists by the name supplied, which you can see in more than one place, but also including in the multimethod definition of ext/canonicalize :mvn
In the git extension code, the lib is given a different treatment, which you can see in the multimethod definition of ext/canonicalize :git
I don't want to get too deep into the realm of conjecture here, but I would guess that if this was an intentional design decision, it probably has something to do with the notion of the address of a git repo being the SoT for those kind of dependencies (even though the address/name of the repo can change... danger!), whereas in Maven names are registered first-class citizens.
And to try to more directly answer your two questions... What do the :deps keys mean? The simple answer is it depends on what kind of dep it is! When using git, it can be whatever, and when using Maven for example it must reference a known package. How should I choose my dep keys? This has the danger of being subjective, however, I would recommend tending to use any dep that has a reliable immutable repository of packages behind it, and only when needed use a dep like github. This is because github dependencies can change their address/name, or simply vanish into thin air (deleted repository).
I'm new to Clojure and Leiningen, and I've determined that some of what I'll want to use is located in clojure.contrib.generic.math-functions. I found API information for that at http://richhickey.github.com/clojure-contrib/branch-1.1.x/math-api.html, but I can't find anything that helps me figure out what I should put into my project.clj file for that dependency.
I have tried [clojure.contrib.generic.math-functions "1.1"], [clojure.contrib.generic.math-functions "1.1.x"], and [clojure.contrib.generic.math-functions "1.1.0"]. For each of those, I get something like...
...
Caused by: org.apache.maven.artifact.resolver.MultipleArtifactsNotFoundException: Missing:
----------
1) clojure.contrib.generic.math-functions:clojure.contrib.generic.math-functions:jar:1.1
All clojure-contrib namespaces are shipped within a single jar file, for which the dependency has to be listed like:
[org.clojure/clojure-contrib "1.2.0"]
Please note that there are different versions available of that artifact. The 1.2.0 is the current stable release.
In order to use functions coming from the math-functions namespace in your clojure code, you need to either require or use such namespace, usually done within the ns form at the beginning of your source file:
(ns my.namespace
(:use [clojure.contrib.generic.math-functions]))
Have a look here to see the differences between use and require.
The next version of Leiningen will have a search task for precisely this purpose. It will search Clojars, Maven Central, and any other repositories your project has listed, provided they offer up downloadable indices. It's already implemented, so if you run Leiningen from git you can use it.
Also, the Leiningen tutorial covers this. Type "lein help tutorial".
You can generally find what you need at clojars.org - it's the default repository for leiningen. The current stable release of Clojure is 1.2.0, so you'd have this in your leiningen project.clj:
[org.clojure/clojure "1.2.0"]
[org.clojure/clojure-contrib "1.2.0"]
To use the generic math functions in your clojure, require or use it in your namespace declaration at the top of your source file:
(ns your-namespace
(:use [clojure.contrib.generic.math-functions :as mathf]))
This allows you to refer to the functions in that namespace like this:
(mathf/abs -10) ;; => 10
:use-ing namespaces with :as is the preferred way to use functions from other namespaces in your code. require is ok, but you'd have to prefix your functions with the entire namespace (e.g. clojure.contrib.generic.math-functions/abs) so that's not practical. Using a namespace without :as allows you to use these functions without any prefix at all (e.g. abs), but you're more likely to get namespace clashes and it might be difficult to see where functions come from, especially if you :use many libraries.
You can browse all libraries available from the default leiningen repository by checking out http://clojars.org/repo/. The structure of clojure-contrib will change when 1.3.0 is out, so you'll have to include the specific contrib library if you're using version 1.3.0-alpha-xx:
[org.clojure.contrib/generic "1.3.0-alpha4"]
Now that the clojure.contrib has been broken up, the math functions are in something called math.numeric-tower. The lein dependency is specified like this:
[org.clojure/math.numeric-tower "0.0.1"]
You can use or require as seems appropriate, for example
(use '[clojure.math.numeric-tower])
Say I made a change to a Clojure library (eg. added a parameter to the request-token in clj-oauth) and want to use that changed library in my project. What's the best way to do this, short of compiling the new library as a JAR and copying that to my project lib?
I want to be able to tweak the library and my project at the same time (preferably in the REPL). If I were doing this in Ruby, I would download and 'require' the gem, then reopen that class in my own project source and add or override the methods as needed.
You can hack directly at the REPL. Suppose you've got incanter on your classpath.
Start a REPL. The first thing we need to do is bring the incanter classes into it.
user> (require 'incanter.core)
nil
Now we can see the function incanter.core/matrix?
user> (incanter.core/matrix? 2)
false
We can look at the original source code:
user> (require 'clojure.repl)
nil
user> (clojure.repl/source incanter.core/matrix?)
(defn matrix?
" Test if obj is 'derived' incanter.Matrix."
([obj] (is-matrix obj)))
nil
Let's go and screw it up:
First change to the incanter.core namespace:
user> (in-ns 'incanter.core)
#<Namespace incanter.core>
Then we can redefine it, using the old source code as a crib:
incanter.core> (defn matrix? [obj] "hello")
#'incanter.core/matrix?
Unit test:
incanter.core> (matrix? 2)
"hello"
Switch back to the user namespace:
incanter.core> (in-ns 'user)
#<Namespace user>
Try it out:
user> (matrix? 2)
; Evaluation aborted.
There is no definition of user/matrix. We redefined it in the incanter.core namespace.
user> (incanter.core/matrix? 2)
"hello"
For experimenting at the repl, it's ok just to change source files and re-compile the single file (C-C C-k in emacs), or if you're in the right namespace, just re-evaluate the definition.
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
Now, if we want to make our valuable change permanent and available to other projects, it depends on how everything is set up.
I use maven for dependency management, so it would be a question of modifying the source file, and then re-running the build process for the library to compile the new version and install it into the local maven repository.
With a maven project, that should be as simple as
$ mvn install
A note about version numbers:
If you do make permanent modifications and use dependency management to coordinate the differences, then you should change the version number of your library, from maybe 1.2.0 to 1.2.0-johnshack-SNAPSHOT, or something that is unlikely to collide with the real thing when you want to use an unperverted version in another project. You wouldn't want a modified version finding its way into projects where it isn't welcome.
Then you modify your own project files to make sure that you use the hacked version where you want to, and the next time you start your repl, it should pull in the last hack that you installed.
You will need to reinstall again every time you want your changes to make their way into the repository, but that's actually probably a good thing.
Unfortunately, (and it was at this point that I started to wish that I had chosen a different example) Incanter turns out to be a leiningen project which is split into sub-modules in an ad-hoc scripty sort of way, so we need to figure out how it expects to be installed. The figuring out turned out to be quite hard, although the answer is easy. Leiningen sets my hair on fire.
You can get incanter's source here:
$ git clone http://github.com/liebke/incanter.git
and the relevant source file is:
~/incanter/modules/incanter-core/src/incanter/core.clj
Modify it to break the matrix? function, and then it turns out that what you have to do is:
Change the version numbers in both the top level project.clj, and also in the submodule project.clj.
Then you run lein install in the incanter-core directory, and then again in the top-level directory, and you have to do it in that order. I don't quite understand why.
At the moment all this seems needlessly complicated. I'm (fairly) sure that it will settle down as the tools mature.
If you're using (or wouldn't mind using) cake, check out the subproject dependencies section of the README. I think it might be exactly what you're looking for.
You upload it to clojars under a different name depend on that.