Are there any libraries to perform FTP transfer with clojure, idiomatic to clojure, or is it necessary to use a java library such as apache commons?
Thanks
It is not necessary to use java library and you can roll complete FTP implementation in Clojure but that would be like re-inventing the wheel and not a feasible thing to do. What you can do is probably write a more functional wrapper over the Java library and then use that wrapper in your clojure code so that everything seems seamless and that't how many of the existing Java libraries are being used in Clojure.
You can use https://github.com/miner/clj-ftp either by invoking few convenience functions or by opening a client and invoking multiple commands with it.
The full API is documented in GitHub at https://github.com/miner/clj-ftp/blob/master/src/miner/ftp.clj .
Contents of project.clj
(defproject my-sweet-project "0.5.0"
:dependencies [[com.velisco/clj-ftp "0.3.0"]
; Other deps
]
; ...
)
Invoking a single FTP command
This will open new FTP connection for each command so it should be used for invoking a single command only. See the full API for complete list of these convenience functions.
(ns my-sweet-name.space
(:require [miner.ftp :as ftp]))
(defn list-files-from-ftp-server []
"Here we list contents of a directory with a convenience function"
(let [ftp-url "ftp://username:password#my.ftp.server.host:port/path/to/stuff"]
(ftp/list-files ftp-url)))
Invoking multiple commands with same connection
This will open FTP connection and invoke arbitrary amount of commands with it. This should be used when multiple commands should be invoked. The FTP connection will be automatically closed. Again check the full API for complete list of functions.
(ns my-sweet-name.space
(:require [miner.ftp :as ftp]))
(defn list-and-download-files []
"Here we list and download contents of a directory"
(let [ftp-url "ftp://username:password#my.ftp.server.host:port/path/to/stuff"]
(ftp/with-ftp [ftp-client ftp-url]
; client-file-names is used to list contents of the ftp-url
; client-get is used to download a file
(doseq [file-name (ftp/client-file-names ftp-client)]
(let [local-file-name (str "/download-path/" file-name)]
(ftp/client-get ftp-client file-name local-file-name))))))
https://github.com/miner/clj-ftp is a wrapper over Apache Commons Net.
Related
I'm currently writing a restful API and need to receive whole files as string.
I didn't write the actual functions which parse those files and must somehow feed these strings into functions which except a path.
So, what I want to find/build is a function which would solve this Problem:
(slurp (INSERT-MAGIC-HERE "The content of my file."))
EDIT:
While both answers seemed to work, the most reliable thing I found was to use "char-array". This prevents any error about the stream being closed which I got quite often.
slurp uses a very flexible mechanism to figure out how to understand its input arguments: it certainly doesn't insist that they be a file. For example, it will accept a java.io.Reader. And it is easy to build a Reader from a String, simply by constructing a StringReader. So,
(slurp (java.io.StringReader. "The content of my file."))
If you really need to write to the file system, a handy way is to use java.io.File/createTempFile. See
https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/io/File.html
Another, easier option is to use tupelo.string/string->stream from the Tupelo Library. It was built for just this purpose:
(ns tst.demo.core
(:use demo.core tupelo.core tupelo.test)
(:require
[tupelo.core :as t]
[tupelo.string :as ts] ))
(dotest
(is= "Hello World!"
(slurp
(ts/string->stream "Hello World!"))) )
Add the following to your project.clj file:
[tupelo "0.9.152"]
(defn my-func [opts]
(assoc opts :something :else))
What i want to be able to do, is serialize a reference to the function (maybe via #'my-func ?) to a string in such a way that i can upon deserializing it, invoke it with args.
How does this work?
Edit-- Why This is Not a Duplicate
The other question asked how to serialize a function body-- the entire function code. I am not asking how to do that. I am asking how to serialize a reference.
Imagine a cluster of servers all running the same jar, attached to a MQ. The MQ pubs in fn-reference and fn-args for functions in the jar, and the server in the cluster runs it and acks it. That's what i'm trying to do-- not pass function bodies around.
In some ways, this is like building a "serverless" engine in clojure.
Weirdly, a commit for serializing var identity was just added to Clojure yesterday: https://github.com/clojure/clojure/commit/a26dfc1390c53ca10dba750b8d5e6b93e846c067
So as of the latest master snapshot version, you can serialize a Var (like #'clojure.core/conj) and deserialize it on another JVM with access to the same loaded code, and invoke it.
(import [java.io File FileOutputStream FileInputStream ObjectOutputStream ObjectInputStream])
(defn write-obj [o f]
(let [oos (ObjectOutputStream. (FileOutputStream. (File. f)))]
(.writeObject oos o)
(.close oos)))
(defn read-obj [f]
(let [ois (ObjectInputStream. (FileInputStream. (File. f)))
o (.readObject ois)]
(.close ois)
o))
;; in one JVM
(write-obj #'clojure.core/conj "var.ser")
;; in another JVM
(read-obj "var.ser")
As suggested on the comments, if you can just serialize a keyword label for the function and store/retrieve that, you are finished.
If you need to transmit the function from one place to another, you essentially need to send the function source code as a string and then have it compiled via eval on the other end. This is what Datomic does when a Database Function is stored in the DB and automatically run by Datomic for any new additions/changes to the DB (these can perform automatic data validation, for example). See:
http://docs.datomic.com/database-functions.html
http://docs.datomic.com/clojure/index.html#datomic.api/function
As similar technique is used in the book Clojure in Action (1st Edition) for the distributed compute engine example using RabbitMQ.
I have a rabbitMQ connection that seems to be started at compile time (when I type lein compile) and then blocks the building of my project. Here are more details on the problem. Let us say this is the clojure file bla_test.clj
(import (com.rabbitmq.client ConnectionFactory Connection Channel QueueingConsumer))
;; And then we have to translate the equivalent java hello world program using
;; Clojure's excellent interop.
;; It feels very strange writing this sort of ceremony-oriented imperative code
;; in Clojure:
;; Make a connection factory on the local host
(def connection-factory
(doto (ConnectionFactory.)
(.setHost "localhost")))
;; and get it to make you a connection
(def connection (.newConnection connection-factory))
;; get that to make you a channel
(def channel (. connection createChannel))
;;HERE I WOULD LIKE TO USE THE SAME CONNECTION AND THE SAME CHANNEL INSTANCE AS OFTEN AS
;; I LIKE
(dotimes [ i 10 ]
(. channel basicPublish "" "hello" nil (. (format "Hello World! (%d)" i) getBytes)))
The clojure file above is part of a bigger clojure program that I build using lein. My problem is that when I compile with "lein compile", a connection is done because of the line (def connection (.newConnection connection-factory)) and then the compilation is stopped! How can I avoid this? Is there a way to compile without building connection? How can I manage to use the same instance of channel over several calls coming from external components?
Any help would be appreciated.
Regards,
Horace
The Clojure compiler must evaluate all top-level forms, because it can be required to run arbitrary code when expanding calls to macros.
The usual solution to issues like the one you describe is to define a top-level Var holding an object of a dereferenceable type, for example an atom or a promise, and have an initialization function provide the value at runtime. (You could also use a delay and specify the value inline; this is less flexible, since it makes it more difficult to use a different value for testing etc.)
I used to like to include all of clojure.contrib, and require all the libraries. This makes find-doc useful as a discovery tool.
Nowadays (clojure 1.4) clojure.contrib is split into many sub-libraries. And that rather spoils my scheme, and it also means that I am constantly having to restart the JVM every time I need a new library.
So I'm busy constructing a project.clj file with many lines:
[org.clojure/algo.generic "0.0.6"]
....
[org.clojure/data.xml "0.0.4"]
....
So that I can get leiningen to put every clojure contrib library on the classpath, whether I need them or not.
And I reckon that this is going to be a spectacular pain in the neck, what with the version numbers, and all.
And I wonder if anyone has a better way to do the same thing?
EDIT: Thinking about it, if there's a web page somewhere that has a list of library names and current versions, I can turn that into a project file fairly easily.
You could use pomegranate if you just want to run it in the REPL (which seems like it would be the only appropriate use case, right?). You can have it look up the latest versions using the Maven Central API. I think this is better than maintaining some sort of dependencies project, generated or otherwise.
(require '[cemerick.pomegranate :refer [add-dependencies]])
(add-dependencies
:coordinates '[[clj-http "0.5.8"]]
:repositories {"clojars" "http://clojars.org/repo"})
(require '[clj-http.client :as client])
;; contrib project names from https://github.com/clojure
(def contrib ["tools.nrepl" "tools.trace" "tools.namespace" "tools.macro"
"test.generative" "math.numeric-tower" "core.match" "core.logic"
"data.priority-map" "core.contracts" "tools.cli" "java.jmx"
"java.jdbc" "java.classpath" "data.xml" "data.json" "core.unify"
"core.incubator" "core.cache" "algo.monads" "data.generators"
"core.memoize" "math.combinatorics" "java.data" "tools.logging"
"data.zip" "data.csv" "algo.generic" "data.codec"
"data.finger-tree"])
(defn add-contrib-dependencies
"look up the latest version of every contrib project in maven central,
and add them as dependencies using pomegranate."
[project-names]
(add-dependencies
:coordinates
(map (juxt
(comp symbol (partial format "org.clojure/%s"))
(fn [proj]
(Thread/sleep 100)
(-> "http://search.maven.org/solrsearch/select?q=%s&rows=1&wt=json"
(format proj)
(client/get {:as :json})
:body :response :docs first :latestVersion)))
project-names)))
Now you can just invoke this function on the list of project names:
user=> (add-contrib-dependencies contrib)
{[org.clojure/data.zip "0.1.1"] nil,
[org.clojure/java.classpath "0.2.0"] nil,
[org.clojure/core.cache "0.6.2"] nil, ...}
UPDATE: as suggested earlier, I had made this answer into a library. It can be used either as nREPL middleware or invoked manually from a running REPL session. The code can be found at https://github.com/rplevy/contrib-repl, where usage instructions can also be found.
I feel your pain. It should be helpful http://dev.clojure.org/display/community/Where+Did+Clojure.Contrib+Go
I am using emacs and swank-clojure. How do I resolve the below scenario -
I have added a new dependency to project.clj.
I run lein deps in a shell to get the new dep.
I have an existing slime session that is open and want to use a function from the new dep.
How do I get the existing slime session to load the new dependency ?
Thanks,
Murtaza
You should have a look at pomegranate which is designed to provide similar if not identical capabilities as the one you describe.
As pointed out by #gergek in the comments, leiningen2 ships with pomegranate, so that if you're on lein2 you have to just fire something like the following in the slime REPL:
(require '[cemerick.pomegranate :as p])
(p/add-dependencies :coordinates '[[org.clojure/core.logic "0.7.5"]])
Have a look at add-dependencies docs for more info on how to use it.