I need to read an geometry as a WKB string in Clojure, for that I try to use clojure/java.jdbc
(require '[clojure.java.jdbc :as j])
(->> (j/query db "select SDO_UTIL.TO_WKBGEOMETRY(geometry) wkb from t where idf = 1")
(map #(-> % :wkb .getBinaryStream .readAllBytes)))
unfortunately I got:
Exception thrown: java.sql.SQLRecoverableException (Closed Connection)
getDBAccess - (BLOB.java:1122)
getBinaryStream - (BLOB.java:265)
invoke0 - (NativeMethodAccessorImpl.java:-2)
invoke - (NativeMethodAccessorImpl.java:62)
invoke - (DelegatingMethodAccessorImpl.java:43)
invoke - (Method.java:498)
invokeMatchingMethod - (Reflector.java:93)
invokeNoArgInstanceMember - (Reflector.java:313)
eval16213/fn - user - (form-init2938139155321903837.clj:3)
map/fn - clojure.core - (core.clj:2646)
so I could read the number bytes in blob using oracle.sql.Blob/length
(->> (j/query db "select SDO_UTIL.TO_WKBGEOMETRY(geometry) wkb t where idf = 1")
(map #(-> % :wkb .length))
)
(42241)
this looks like what I call "the lazy bug".
the connection is opened and the query is sent then it returns a sequence, containing some code, that will fetch the responses from the DB when the next part of the program reads from the sequence. in this case the DB connection is closed before that happens.
Fortunatly it's easy to fix, just put a doall around the map.
(require '[clojure.java.jdbc :as j])
(->> (j/query db
"select SDO_UTIL.TO_WKBGEOMETRY(geometry)
wkb t where idf = 1")
(map #(-> % :wkb .getBinaryStream .readAllBytes))
doall)
this code works for me, suggested by #Arthur doall hasn't helped in my case,
what I've added is a transaction:
(require '[clojure.java.jdbc :as j])
(j/with-db-transaction [t-con db ]
(->> (j/query t-con
"select SDO_UTIL.TO_WKBGEOMETRY(geometry)
wkb from t where idf = 1 ")
(map #(-> % :wkb .getBinaryStream))
(map #(java.util.Scanner. %))
(map #(.next %))
doall
)
)
Related
I'm building a datomic schema and have the following at the foot of my clj file which defines and transacts schema and initial data. The functions being called below each call d/transact.
(defn recreate-database []
"To recreate db after running delete-database in bin/repl"
(pt1-transact-schema)
(pt1-transact-data)
(pt2-transact-schema)
(pt2-transact-data)
(pt3-transact-schema)
(pt3-transact-data))
By default we only see the return value of the last form, but I'd like to see, or save, the result of each of the six function calls.
Wondering what a nice way to do this is.
Thought of something like (map (comp println eval) [functions]), but that's not right.
there is also a nice functional composition function called juxt:
user> ((juxt + - * /) 1 2)
;;=> [3 -1 2 1/2]
user> ((juxt (constantly 1) (constantly 2) (constantly 3)))
;;=> [1 2 3]
or in your case:
(def recreate-database (juxt pt1-transact-schema
pt1-transact-data
pt2-transact-schema
pt2-transact-data
pt3-transact-schema
pt3-transact-data))
You could try this:
(defn recreate-database []
"To recreate db after running delete-database in bin/repl"
(mapv #(%) [pt1-transact-schema
pt1-transact-data
pt2-transact-schema
pt2-transact-data
pt3-transact-schema
pt3-transact-data]))
The expression #(%) is a shorthand notation for a lambda function that takes one argument, representing a function, and calls that function. If you find it more readable, you can replace that expression by (fn [f] (f)).
With datomic, all you need is a connection and a list of tx-data. Then you can use map to return the transact result on each step (i.e. each tx-data):
(defn recreate-database [conn & tx-data]
(->> tx-data
(map (partial d/transact conn))
doall))
I am using taoensso.carmine redis client and want to achieve the following: given sequence s, get all its elements that aren't exist in redis. (I mean for which redis's EXISTS command return false)
At first I thought to do the following:
(wcar conn
(remove #(car/exists %) s))
but it returns sequence of car/exists responses rather than filtering my sequence by them
(remove #(wcar conn (car exists %)) s)
Does the job but takes a lot of time because no-pipeling and using new connection each time.
So I end up with some tangled map manipulation below, but I believe there should be simplier way to achieve it. How?
(let [s (range 1 100)
existance (wcar conn
(doall
(for [i s]
(car/exists i))))
existance-map (zipmap s existance)]
(mapv first (remove (fn [[k v]] (= v 1)) existance-map)))
Your remove function is lazy, so it won't do anything. You also can't do data manipulation inside the wcar macro so I'd so something like this:
(let [keys ["exists" "not-existing"]]
(zipmap keys
(mapv pos?
(car/wcar redis-db
(mapv (fn [key]
(car/exists key))
keys)))))
Could you reexamine you're first solution? I don't know what wcar does, but this example shows that you're on the right track:
> (remove #(odd? %) (range 9))
(0 2 4 6 8)
The anonymous function #(odd? %) returns either true or false results which are used to determine which numbers to keep. However, it is the original numbers that are returned by (remove...), not true/false.
Consider this snippet in carmine
(wcar* (car/set "counter" 1) ;; expect to be number counter=1
(let [id (car/get "counter")] ;; expect to have id=1
(println id))) ;; [nil [[SET counter 1] [GET counter]]]
What I am doing wrong here? Is there a way to use let inside wcar* macro?
You can nest wcar forms which gives you access to the return values inside wcar:
(wcar*
(car/set "counter" 1)
(let [id (wcar*
(car/get "counter"))]
(println id)
id))
I'm having problems with the datomic tutorial at the "Revisiting the past" section http://datomic.com/company/resources/tutorial.html
For the two queries below:
query = "[:find ?c :where [?c :community/name]]";
db_asOf_schema = conn.db().asOf(schema_tx_date);
System.out.println(Peer.q(query, db_asOf_schema).size()); // 0
db_since_data = conn.db().since(data_tx_date);
System.out.println(Peer.q(query, db_since_data).size()); // 0
I have tried these commands in clojure, but cannot get them working as described in the tutorial:
(since (db conn) (java.util.Date.) )
;; It should return 0 but returns the whole database instead
(def ts (q '[:find ?when :where [?tx :db/txInstant ?when]] (db conn)))
(count (since (db conn) (ffirst (reverse (sort ts))))))
;; returns 13, but should return 0
(count (as-of (db conn) (ffirst (sort ts)))))
;; returns 13, but should return 0
I'm not too sure is this is the right behaviour, is there anything I'm doing wrong?
If you are working through the Seattle tutorial in Clojure, probably the most important single thing to know is that working Clojure code is included in the Datomic distribution. The filename is samples/seattle/getting-started.clj, and you can simply follow along at the REPL.
Two observations on the Clojure code in your question:
The since function is documented to return a database value, not a number, so the behavior you are seeing is as expected. In order to see what is in the database, you need to issue a query.
Databases do not have any documented semantics for the Clojure count function, so you should not call count on them. Again, if you want to see what is in the database, you need to issue a query, e.g.
;; Find all transaction times, sort them in reverse order
(def tx-instants (reverse (sort (q '[:find ?when :where [_ :db/txInstant ?when]]
(db conn)))))
;; pull out two most recent transactions, most recent loaded
;; seed data, second most recent loaded schema
(def data-tx-date (ffirst tx-instants))
(def schema-tx-date (first (second tx-instants)))
;; make query to find all communities
(def communities-query '[:find ?c :where [?c :community/name]])
;; find all communities as of schema transaction
(let [db-asof-schema (-> conn db (d/as-of schema-tx-date))]
(println (count (seq (q communities-query db-asof-schema)))))
;; find all communities as of seed data transaction
(let [db-asof-data (-> conn db (d/as-of data-tx-date))]
(println (count (seq (q communities-query db-asof-data)))))
;; find all communities since seed data transaction
(let [db-since-data (-> conn db (d/since data-tx-date))]
(println (count (seq (q communities-query db-since-data)))))
Hope this helps. There is also a Datomic google group if you have more questions.
I have a URL checker that I use in Perl. I was wondering how something like this would be done in Clojure. I have a file with thousands of URLs and I'd like the output file to contain the URL (minus http://, https://) and a simple :1 for valid and :0 for false. Ideally, I could check each site concurrently, considering that this is one of Clojure's strengths.
Input
http://www.google.com
http://www.cnn.com
http://www.msnbc.com
http://www.abadurlisnotgood.com
Output
www.google.com:1
www.cnn.com:1
www.msnbc.com:1
www.abadurlisnotgood.com:0
I assume by "valid URL" you mean HTTP response 200. This might work. It requires clojure-contrib. Change map to pmap to attempt to make it parallel, like Arthur Ulfeldt mentioned.
(use '(clojure.contrib duck-streams
java-utils
str-utils))
(import '(java.net URL
URLConnection
HttpURLConnection
UnknownHostException))
(defn check-url [url]
(str (re-sub #"^(?i)http:/+" "" url)
":"
(try
(let [c (cast HttpURLConnection
(.openConnection (URL. url)))]
(if (= 200 (.getResponseCode c))
1
0))
(catch UnknownHostException _
0))))
(defn check-urls-from-file [filename]
(doseq [line (map check-url
(read-lines (as-file filename)))]
(println line)))
Given your example as input:
user> (check-urls-from-file "urls.txt")
www.google.com:1
www.cnn.com:1
www.msnbc.com:1
www.abadurlisnotgood.com:0
Write a small function that appends a ":1" or ":0" to a url and then use pmap to apply it in parallel to all the urls.
(defn check-a-url [url] .... )
(pmap #(if (check-a-url %) (str url ":1") (str url ":0")))
Clojure now has a as-url function in clojure.java.io:
(as-url "http://google.com") ;;=> #object[java.net.URL 0x5dedf9bd "http://google.com"]
(str (as-url "http://google.com")) ;;=> "http://google.com"
(as-url "notanurl") ;; java.net.MalformedURLException
Based on that we could write a function like so:
(defn check-url
"checks if the url is well formed"
[url]
(str (clojure.string/replace-first url #"(http://|https://)" "")
":"
(try (as-url url) ;; built-in, does not perform an actual request, and does very little validation
1
(catch Exception e 0))))
(defn check-urls-from-file
"from Brian Carper answer"
[filename]
(doseq [line (map check-url (read-lines (as-file filename)))]
(println line)))
Instead of pmap, I used agents with send-off in conjunction with the above solution. I think this is better when there is blocking I/O. I believe pmap has limited concurrency too. Here's what I have so far. I wonder how this will scale with thousands of URLs.
(use '(clojure.contrib duck-streams
java-utils
str-utils))
(import '(java.net URL
URLConnection
HttpURLConnection
UnknownHostException))
(defn check-url [url]
(str (re-sub #"^(?i)http:/+" "" url)
":"
(try
(let [c (cast HttpURLConnection
(.openConnection (URL. url)))]
(if (= 200 (.getResponseCode c))
1
0))
(catch UnknownHostException _
0))))
(def urls (read-lines "urls.txt"))
(def agents (for [url urls] (agent url)))
(doseq [agent agents]
(send-off agent check-url))
(apply await agents)
(def x '())
(doseq [url (filter deref agents)]
(def x (cons #url x)))
(prn x)
(shutdown-agents)