Trying to use Clojure Deps and CLI, I was surprised to find out that the following all worked for using clojure.data.json.
Maven dependency:
{:deps {org.clojure/data.json {:mvn/version "0.2.6"}}}
Git dependency with same key:
{:deps {org.clojure/data.json {:git/url "https://github.com/clojure/data.json.git"
:sha "13e9d244678be7b235bb24a10310f9d147ea088d"}}}
Git dependency with random key:
{:deps {lol/this-works {:git/url "https://github.com/clojure/data.json.git"
:sha "13e9d244678be7b235bb24a10310f9d147ea088d"}}}
With Maven and Clojars dependencies, :deps keys identify the artifact. When using a git sha instead, the name doesn't seem to matter.
What do Clojure deps.edn :deps keys mean?
How should I choose my :deps keys?
Resources I've read but may contain what I'm after:
Clojure Deps and CLI Guide
Clojure Deps and CLI Reference
It would seem that this is a result of how the git "extension" is handled in the tooling when compared to other "extensions" like maven. All the relevant code for this can be found here. I'll also make it clear that I haven't read this code in depth, and so my knowledge of this code therefore is not deep.
If you look at the way the lib is treated in the maven extension for example, it appears that it is actually checking maven to see that the artifact exists by the name supplied, which you can see in more than one place, but also including in the multimethod definition of ext/canonicalize :mvn
In the git extension code, the lib is given a different treatment, which you can see in the multimethod definition of ext/canonicalize :git
I don't want to get too deep into the realm of conjecture here, but I would guess that if this was an intentional design decision, it probably has something to do with the notion of the address of a git repo being the SoT for those kind of dependencies (even though the address/name of the repo can change... danger!), whereas in Maven names are registered first-class citizens.
And to try to more directly answer your two questions... What do the :deps keys mean? The simple answer is it depends on what kind of dep it is! When using git, it can be whatever, and when using Maven for example it must reference a known package. How should I choose my dep keys? This has the danger of being subjective, however, I would recommend tending to use any dep that has a reliable immutable repository of packages behind it, and only when needed use a dep like github. This is because github dependencies can change their address/name, or simply vanish into thin air (deleted repository).
Related
spago docs state:
packages.dhall: this file is meant to contain the totality of the packages available to your project (that is, any package you might want to import).
In practice it pulls in the official package-set as a base, and you are then able to add any package that might not be in the package set, or override existing ones.
spago.dhall: this is your project configuration. It includes the above package set, the list of your dependencies, the source paths that will be used to build, and any other project-wide setting that spago will use. (my emphasis)
Why do both files have the notion/concept of dependencies? Example: packages.dhall and spago.dhall from the ebook.
spago.dhall dependencies can be found in the project .spago folder. But I cannot locate the ones from packages.dhall. Others are common like aff. A different perspective:
[...] what you choose is a "snapshot", which is a collection of certain versions of all available packages that are guaranteed to compile and work together.
The snapshot is defined in your packages.dhall file, and then you specify the specific packages that you want to use in spago.dhall. The version for each package comes from the snapshot.
That sounds, like spago.dhall is an excerpt of packages from packages.dhall.The note about versions is a bit confusing, as there aren't version specifiers in both files.
So, why two files? What is the mental model for someone coming from npm ecosystem with package.json (which might be present as well)?
The mental model is that of a Haskell developer, which is what most PureScript developers used to be, and many still are. :-)
But more seriously, the mental model is having multiple "projects" in a "solution", which is the model of Haskell's de-facto standard package manager, Stack. In Haskell this situation is very common, in PureScript - much less so, but still not unheard of.
In a situation like this it's usually beneficial to have all the "projects" to share a common set of packages, which are all guaranteed to be "compatible" with each other, which simply means that they all compile together and their tests pass. In Haskell Stack this common set of packages is defined in stack.yaml. In Spago - it's packages.dhall.
Once you have this common base set of packages established, each individual project may pick and choose the particular packages that it uses. In Haskell Stack this is specified either in package.yaml or in <project-name>.cabal (the latter being phased out). In Spago - it's spago.dhall.
But of course, when you have just the one project, having both packages.dhall to establish the "base set" of packages and then, separately, spago.dhall to pick some particular packages from that set - may seem a bit redundant. And indeed, it's possible to do without the packages.dhall file completely: just specify the URL of the package set directly in spago.dhall, as the value of the packages property:
{ name = "my-project"
, dependencies = [ ... ]
, license = "..."
, packages = https://github.com/purescript/package-sets/releases/download/psc-0.13.8-20201223/packages.dhall
, repository = "..."
, sources = [ "src/**/*.purs" ]
}
This will work, but there is one important caveat: hashing. When the URL of the package set is specified in packages.dhall, running spago install will compute a hash of that package set and put it inside packages.dhall, right next to the URL. Here's what mine looks like:
let upstream =
https://github.com/purescript/package-sets/releases/download/psc-0.13.8-20201222/packages.dhall sha256:620d0e4090cf1216b3bcbe7dd070b981a9f5578c38e810bbd71ece1794bfe13b
Then, if maintainers of the package set become evil and change the contents of that file, Spago will be able to notice that, recompute the hash, and reinstall the packages.
If you put the URL directly in spago.dhall, this doesn't happen, and you're left with the slight possibility of your dependencies getting out of sync.
Now to address this point separately:
Why do both files have the notion/concept of dependencies? Example: packages.dhall and spago.dhall from the ebook.
If you look closer at the examples you linked, you'll see that these are not the same dependencies. The ones in spago.dhall are dependencies of your package - the one where spago.dhall lives.
But dependencies in packages.dhall are dependencies of the test-unit package, which is being added to the package set as an override, presumably because we want to use the special version stackless-default, which isn't present in the official package set. When you override a package like this, you can override any fields specified in that package's own spago.dhall, and in this case we're overriding dependencies, repo, and version.
I am using lein repl without a project so there is no project.clj.
I am running Leiningen 2.8.1 on Java 1.8.0_191 OpenJDK 64-Bit Server VM.
When I require a Clojure dependency that I assume should just work - like clojure.data.json - I notice that it is not in my .m2 directory. Is that why I am getting a FileNotFoundException Could not locate clojure/data/json__init.class or clojure/data/js
on.clj on classpath? I can't find my other Clojure dependencies there either so I don't know where they reside and if this dependancy should be in .m2 or not.
I understand the error message but without knowing its location or even knowing how to properly add it to the CLASSPATH for the REPL to see it, I remain stuck.
Is this a dependency that I still need to install? If so, how do I install it without going through a project?
I don't understand the JVM as I am new to it, so add a little extra information in your answer.
I have looked at this, this, this, this and this. I don't know if I am overlooking anything so your help will really be appreciated.
I am using lein run without a project so there is no project.clj.
If you're using Leiningen, this'll be much easier if you create a project.clj file that declares your dependencies. Leiningen will read project.clj and handle fetching any missing dependencies to your local Maven repository, and add them to your classpath when you start your REPL/application. (lein run doesn't work for me in a directory without a project.clj; I get an error: No :main namespace specified in project.clj.. Did you mean lein repl?)
When I require a Clojure dependency that I assume should just work - like clojure.data.json - I notice that it is not in my .m2 directory.
clojure.data.json doesn't ship with Clojure — it's a separate dependency that must be fetched and added to your classpath in order to use it. The classpath tells the JVM where to look when it loads class files. Leiningen will do both of these things for you if you declare the dependency in project.clj:
:dependencies [[org.clojure/clojure "1.10.0"]
[org.clojure/data.json "0.2.6"]]
You can also use the lein deps command if you only want to fetch dependencies.
You can create a new/blank Leiningen project with lein new project_name_goes_here. It will have a project.clj with a few boilerplate entries and a :dependencies key where you can declare dependencies.
I understand the error message but without knowing its location or even knowing how to properly add it to the CLASSPATH for the REPL to see it, I remain stuck. Is this a dependency that I still need to install? If so, how do I install it without going through a project?
You could manually download it from the internet, then manually add its path to your classpath, but if you're already using Leiningen it's much easier to add a line to a project.clj file and have Leiningen handle this for you.
If using a project.clj file w/Leiningen isn't an option, there are other ways to use Clojure and resolve dependencies/build a classpath at runtime. Boot accommodates this workflow, you can use Leiningen like this with a little added effort, as well as the newer tools.deps tooling. There are examples of each in this ClojureVerse thread, but note that some of these approaches are doing essentially the same thing as declaring the dependency in a file — instead declaring them as CLI arguments.
For example, using Clojure CLI tooling:
$ clj -Sdeps "{:deps {org.clojure/data.json {:mvn/version \"0.2.6\"}}}"
Clojure 1.9.0
user=> (require '[clojure.data.json :as json])
nil
user=> (json/write-str {:foo "bar"})
"{\"foo\":\"bar\"}"
user=> (System/getProperty "java.class.path")
"src:
/Users/me/.m2/repository/org/clojure/clojure/1.9.0/clojure-1.9.0.jar:
/Users/me/.m2/repository/org/clojure/data.json/0.2.6/data.json-0.2.6.jar:
/Users/me/.m2/repository/org/clojure/spec.alpha/0.1.143/spec.alpha-0.1.143.jar:
/Users/me/.m2/repository/org/clojure/core.specs.alpha/0.1.24/core.specs.alpha-0.1.24.jar"
You could create a deps.edn file containing {:deps {org.clojure/data.json {:mvn/version \"0.2.6\"}}} in the same directory, and clj would read that, resolve the dependencies if necessary, and build the classpath accordingly.
This is a great opportunity to use lein try. Once you add it to your ~/.lein/profiles.clj, you'd simply run: lein try org.clojure/data.json and you'll be greeted with a running REPL with that dependency just a require away.
In Clojure, if I want to start a test cluster using the hbase testing utility, I have to annotate my dependencies with:
[org.apache.hbase/hbase "0.92.2" :classifier "tests" :scope "test"]
First of all, I have no idea what this means. According to leiningens sample project.clj
;; Dependencies are listed as [group-id/name version]; in addition
;; to keywords supported by Pomegranate, you can use :native-prefix
;; to specify a prefix. This prefix is used to extract natives in
;; jars that don't adhere to the default "<os>/<arch>/" layout that
;; Leiningen expects.
Question 1: What does that mean?
Question 2: If I upgrade the version:
[org.apache.hbase/hbase "0.94.6.1" :classifier "tests" :scope "test"]
Then I receive a ClassNotFoundException
Exception in thread "main" java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration
Whats going on here and how do I fix it?
any key / value pairs added to a dependency declaration are used as arguments to the clojure pomegranate library
the keys recognized are listed in the source here: in the source to pomagranate (for future reference that is a link to the function resolve-artifacts*)
the maven pom docs may also be helpful
:scope describes the conditions where the dependency is used, so :scope "test" would seem to indicate that the dependency is only pulled in while testing
:classifier seems to indicate the an extra element distinguishing versions
I speculate that the dependency resolution for the newer hbase version may have a poorly configured pom which is not declaring its dependencies properly. Try finding the info for org.apache.hadoop.hbase.HBaseConfiguration and requiring the package manually.
Leinigen uses the Maven dependency mechanism. Read that link to understand that different scopes.
The "classifier" is a token that is part of the dependency coordinates, so that a group of Jars / zip files / etc. can be part of the same logic release, but declared as distinct dependencies in your pom.xml. So in this case "tests" is a distinct HBase artifact from 0.94.6.1 that contains the tests.
You can see this in action by pointing your browser to the Maven "Central" repo location for that version of HBase:
http://repo1.maven.org/maven2/org/apache/hbase/hbase/0.94.6.1/
You can search the maven "central" repo here:
https://repository.apache.org/
or
http://mvnrepository.com/
On ClassNotFOundException - agreed with noisesmith. Your best bet is to find the dependency (jar) that contains that class and explicitly add it to your project dependency configuration.
Usually I do a google search for the classname and "jar" i.e.
https://www.google.com/search?q=jar+org.apache.hadoop.hbase.HBaseConfiguration
I'm new to Clojure and Leiningen, and I've determined that some of what I'll want to use is located in clojure.contrib.generic.math-functions. I found API information for that at http://richhickey.github.com/clojure-contrib/branch-1.1.x/math-api.html, but I can't find anything that helps me figure out what I should put into my project.clj file for that dependency.
I have tried [clojure.contrib.generic.math-functions "1.1"], [clojure.contrib.generic.math-functions "1.1.x"], and [clojure.contrib.generic.math-functions "1.1.0"]. For each of those, I get something like...
...
Caused by: org.apache.maven.artifact.resolver.MultipleArtifactsNotFoundException: Missing:
----------
1) clojure.contrib.generic.math-functions:clojure.contrib.generic.math-functions:jar:1.1
All clojure-contrib namespaces are shipped within a single jar file, for which the dependency has to be listed like:
[org.clojure/clojure-contrib "1.2.0"]
Please note that there are different versions available of that artifact. The 1.2.0 is the current stable release.
In order to use functions coming from the math-functions namespace in your clojure code, you need to either require or use such namespace, usually done within the ns form at the beginning of your source file:
(ns my.namespace
(:use [clojure.contrib.generic.math-functions]))
Have a look here to see the differences between use and require.
The next version of Leiningen will have a search task for precisely this purpose. It will search Clojars, Maven Central, and any other repositories your project has listed, provided they offer up downloadable indices. It's already implemented, so if you run Leiningen from git you can use it.
Also, the Leiningen tutorial covers this. Type "lein help tutorial".
You can generally find what you need at clojars.org - it's the default repository for leiningen. The current stable release of Clojure is 1.2.0, so you'd have this in your leiningen project.clj:
[org.clojure/clojure "1.2.0"]
[org.clojure/clojure-contrib "1.2.0"]
To use the generic math functions in your clojure, require or use it in your namespace declaration at the top of your source file:
(ns your-namespace
(:use [clojure.contrib.generic.math-functions :as mathf]))
This allows you to refer to the functions in that namespace like this:
(mathf/abs -10) ;; => 10
:use-ing namespaces with :as is the preferred way to use functions from other namespaces in your code. require is ok, but you'd have to prefix your functions with the entire namespace (e.g. clojure.contrib.generic.math-functions/abs) so that's not practical. Using a namespace without :as allows you to use these functions without any prefix at all (e.g. abs), but you're more likely to get namespace clashes and it might be difficult to see where functions come from, especially if you :use many libraries.
You can browse all libraries available from the default leiningen repository by checking out http://clojars.org/repo/. The structure of clojure-contrib will change when 1.3.0 is out, so you'll have to include the specific contrib library if you're using version 1.3.0-alpha-xx:
[org.clojure.contrib/generic "1.3.0-alpha4"]
Now that the clojure.contrib has been broken up, the math functions are in something called math.numeric-tower. The lein dependency is specified like this:
[org.clojure/math.numeric-tower "0.0.1"]
You can use or require as seems appropriate, for example
(use '[clojure.math.numeric-tower])
Say I made a change to a Clojure library (eg. added a parameter to the request-token in clj-oauth) and want to use that changed library in my project. What's the best way to do this, short of compiling the new library as a JAR and copying that to my project lib?
I want to be able to tweak the library and my project at the same time (preferably in the REPL). If I were doing this in Ruby, I would download and 'require' the gem, then reopen that class in my own project source and add or override the methods as needed.
You can hack directly at the REPL. Suppose you've got incanter on your classpath.
Start a REPL. The first thing we need to do is bring the incanter classes into it.
user> (require 'incanter.core)
nil
Now we can see the function incanter.core/matrix?
user> (incanter.core/matrix? 2)
false
We can look at the original source code:
user> (require 'clojure.repl)
nil
user> (clojure.repl/source incanter.core/matrix?)
(defn matrix?
" Test if obj is 'derived' incanter.Matrix."
([obj] (is-matrix obj)))
nil
Let's go and screw it up:
First change to the incanter.core namespace:
user> (in-ns 'incanter.core)
#<Namespace incanter.core>
Then we can redefine it, using the old source code as a crib:
incanter.core> (defn matrix? [obj] "hello")
#'incanter.core/matrix?
Unit test:
incanter.core> (matrix? 2)
"hello"
Switch back to the user namespace:
incanter.core> (in-ns 'user)
#<Namespace user>
Try it out:
user> (matrix? 2)
; Evaluation aborted.
There is no definition of user/matrix. We redefined it in the incanter.core namespace.
user> (incanter.core/matrix? 2)
"hello"
For experimenting at the repl, it's ok just to change source files and re-compile the single file (C-C C-k in emacs), or if you're in the right namespace, just re-evaluate the definition.
;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
Now, if we want to make our valuable change permanent and available to other projects, it depends on how everything is set up.
I use maven for dependency management, so it would be a question of modifying the source file, and then re-running the build process for the library to compile the new version and install it into the local maven repository.
With a maven project, that should be as simple as
$ mvn install
A note about version numbers:
If you do make permanent modifications and use dependency management to coordinate the differences, then you should change the version number of your library, from maybe 1.2.0 to 1.2.0-johnshack-SNAPSHOT, or something that is unlikely to collide with the real thing when you want to use an unperverted version in another project. You wouldn't want a modified version finding its way into projects where it isn't welcome.
Then you modify your own project files to make sure that you use the hacked version where you want to, and the next time you start your repl, it should pull in the last hack that you installed.
You will need to reinstall again every time you want your changes to make their way into the repository, but that's actually probably a good thing.
Unfortunately, (and it was at this point that I started to wish that I had chosen a different example) Incanter turns out to be a leiningen project which is split into sub-modules in an ad-hoc scripty sort of way, so we need to figure out how it expects to be installed. The figuring out turned out to be quite hard, although the answer is easy. Leiningen sets my hair on fire.
You can get incanter's source here:
$ git clone http://github.com/liebke/incanter.git
and the relevant source file is:
~/incanter/modules/incanter-core/src/incanter/core.clj
Modify it to break the matrix? function, and then it turns out that what you have to do is:
Change the version numbers in both the top level project.clj, and also in the submodule project.clj.
Then you run lein install in the incanter-core directory, and then again in the top-level directory, and you have to do it in that order. I don't quite understand why.
At the moment all this seems needlessly complicated. I'm (fairly) sure that it will settle down as the tools mature.
If you're using (or wouldn't mind using) cake, check out the subproject dependencies section of the README. I think it might be exactly what you're looking for.
You upload it to clojars under a different name depend on that.