A while back, Chris Granger posted this middleware to get JSON hashes to appear in the defpage params under an umbrella "backbone" element.
(defn backbone [handler]
(fn [req]
(let [neue (if (= "application/json" (get-in req [:headers "content-type"]))
(update-in req [:params] assoc :backbone (json/parse-string (slurp (:body req)) true))
req)]
(handler neue))))
How could I modify this code to have the JSON elements appear as top-level params in defpage; i.e. get rid of the :backbone umbrella?
There are two things you can do. One option is to replace the value of :params with the map returned after parsing the JSON. In order to do that, just associate the new map to the :params key.
(assoc req [:params] (json/parse-string (slurp (:body req)) true))
The other option (as suggested by #dAni) is to merge the values of the parsed JSON into so that existing values in the :params map are not overridden. The reason why you need to use partial instead of just using merge here is because the final map is the merged result of the maps from left-to-right. Your solution works if you want the values from the JSON map to take precedence.
(update-in req [:params]
(partial merge (json/parse-string (slurp (:body req)) true)))
Got it. assoc just works for one element so you have to put everything under the :backbone umbrella. To push all the JSON elements into the params, you have to use merge. So change the 4th line to:
(update-in req [:params] merge (json/parse-string (slurp (:body req)) true))
If you don't mind pulling in another dependency, you can use the ring-middleware-format library.
Instructions:
Add [ring-middleware-format "0.1.1"] to your project.clj
and then in your server.clj, add the following code:
Code:
(:require [ring.middleware.format-params :as format-params])
(server/add-middleware format-params/wrap-json-params)
(defn -main [& m]
; Start the server...
)
Now any incoming JSON will be available to use just like form POSTdata.
Related
I have many JSON objects, and I am trying to filter those objects by the date. These objects are being parsed from several JSON files using Cheshire.core, meaning that the JSON objects are in a collection. The date is being passed in in the following format "YYYY-MM-DD" (eg. 2015-01-10). I have tried using the filter and contains? functions to do this, but I am having no luck so far. How can I filter these JSON objects by my chosen date?
Current Clojure code:
(def filter-by-date?
(fn [orders-data date-chosen]
(contains? (get (get orders-data :date) :date) date-chosen)))
(prn (filter (filter-by-date? orders-data "2017-12-25")))
Example JSON object:
{
"id":"05d8d404-b3f6-46d1-a0f9-dbdab7e0261f",
"date":{
"date":"2015-01-10T19:11:41.000Z"
},
"total":{
"GBP":57.45
}
}
JSON after parsing with Cheshire:
[({:id "05d8d404-b3f6-46d1-a0f9-dbdab7e0261f",
:date {:date "2015-01-10T19:11:41.000Z"},
:total {:GBP 57.45}}) ({:id "325bd04-b3f6-46d1-a0f9-dbdab7e0261f",
:date {:date "2015-02-23T10:15:14.000Z"},
:total {:GBP 32.90}})]
First, I'm going to assume you've parsed the JSON first into something like this:
(def parsed-JSON {:id "05d8d404-b3f6-46d1-a0f9-dbdab7e0261f",
:date {:date "2015-01-10T19:11:41.000Z"},
:total {:GBP 57.45}})
The main problem is the fact that the date as stored in the JSON contains time information, so you aren't going to be able to check it directly using equality.
You can get around this by using clojure.string/starts-with? to check for prefixes. I'm using s/ here as an alias for clojure.string:
(defn filter-by-date [date jsons]
(filter #(s/starts-with? (get-in % [:date :date]) date)
jsons))
You were close, but I made a few changes:
You can't use contains? like that. From the docs of contains?: Returns true if key is present in the given collection, otherwise returns false. It can't be used to check for substrings; it's used to test for the presence of a key in a collection.
Use -in postfix versions to access nested structures instead of using multiple calls. I'm using (get-in ...) here instead of (get (get ...)).
You're using (def ... (fn [])) which makes things more complicated than they need to be. This is essentially what defn does, although defn also adds some more stuff as well.
To address the new information, you can just flatten the nested sequences containing the JSONs first:
(->> nested-json-colls ; The data at the bottom of the question
(flatten)
(filter-by-date "2015-01-10"))
#!/usr/bin/env boot
(defn deps [new-deps]
(merge-env! :dependencies new-deps))
(deps '[[org.clojure/clojure "1.9.0"]
[cheshire "5.8.0"]])
(require '[cheshire.core :as json]
'[clojure.string :as str])
(def orders-data-str
"[{
\"id\":\"987654\",
\"date\":{
\"date\":\"2015-01-10T19:11:41.000Z\"
},
\"total\":{
\"GBP\":57.45
}
},
{
\"id\":\"123456\",
\"date\":{
\"date\":\"2016-01-10T19:11:41.000Z\"
},
\"total\":{
\"GBP\":23.15
}
}]")
(def orders (json/parse-string orders-data-str true))
(def ret (filter #(clojure.string/includes? (get-in % [:date :date]) "2015-01-") orders))
(println ret) ; ({:id 987654, :date {:date 2015-01-10T19:11:41.000Z}, :total {:GBP 57.45}})
You can convert the date string to Date object using any DateTime library like joda-time and then do a proper filter if required.
clj-time has functions for parsing strings and comparing date-time objects. So you could do something like:
(ns filter-by-time-example
(:require [clj-time.coerce :as tc]
[clj-time.core :as t]))
(def objs [{"id" nil
"date" {"date" "2015-01-12T19:11:41.000Z"}
"total" nil}
{"id" "05d8d404-b3f6-46d1-a0f9-dbdab7e0261f"
"date" {"date" "2015-01-10T19:11:41.000Z"}
"total" {"GBP" :57.45}}
{"id" nil
"date" {"date" "2015-01-11T19:11:41.000Z"}
"total" nil}])
(defn filter-by-day
[objs y m d]
(let [start (t/date-time y m d)
end (t/plus start (t/days 1))]
(filter #(->> (get-in % ["date" "date"])
tc/from-string
(t/within? start end)) objs)))
(clojure.pprint/pprint (filter-by-day objs 2015 1 10)) ;; Returns second obj
If you're going to repeatedly do this (e.g. for multiple days) you could parse all dates in your collection into date-time objects with
(map #(update-in % ["date" "date"] tc/from-string) objs)
and then just work with that collection to avoid repeating the parsing step.
(ns filter-by-time-example
(:require [clj-time.format :as f]
[clj-time.core :as t]
[cheshire.core :as cheshire]))
(->> json-coll
(map (fn [json] (cheshire/parse-string json true)))
(map (fn [record] (assoc record :dt-date (f/format (get-in record [:date :date])))))
(filter (fn [record] (t/after? (tf/format "2017-12-25") (:dt-date record))))
(map (fn [record] (dissoc record :dt-date))))
Maybe something like this? You might need to change the filter for your usecase but as :dt-time is now a jodo.DateTime you can leverage all the clj-time predicates.
Attempting to write a function which takes a url, api-key and an optional map of query parameters.
This code works if there are no query params:
(defn get-api
[url api-key & query]
(let [api-map {:apikey api-key}]
(let [params (if query
(merge query api-map)
api-map)]
(.log js/console (pr-str params))
(GET url {:params params}))))
(get-api "some/url" "some-key") ;works just fine
(get-api "some/url" "some-key" {:id 123}) ;ends up producing ({:apikey "some-key"} {:id 123})
What am I missing to produce a single map with the values I need?
The destructuring [x & y] guarantees that y will be a sequence of all args after the first. If you just want to have a single optional argument, use multiple arities instead. Also, your if is not needed - nil (or an empty map, as I use below) will merge just fine.
(defn get-api
([url api-key] (get-api url api-key {}))
([url api-key query]
(let [params (merge query {:api-key api-key})]
...)))
I have one defresource, that is supposed to take POST-requests, validate request body in :malformed-decision, save the body to database in :post!-decision and return the saved body in :handle-created.
(defn parse-project [context] (json/read-str
(slurp (get-in context [:request :body]))
:key-fn keyword))
(defresource add-new-project
:malformed? (fn[ctx] (not (project-is-valid (parse-project ctx))))
:handle-malformed (fn [_] (generate-string (str "Malformed json!")))
...
:post! (fn [ctx] (save-to-db (parse-project ctx))
:handle-created (fn [ctx] (... parse-project ...))
So my code reads three times the ByteArrayInputStream(that comes from :request :body) with slurp-function. First time works but second time when slurp is called, nil is passed as a param and comes java.io.EOFException: JSON error. I think the reader starts reading where it was left last time.
How could I read the body of the request three times? Or is there nice way to save the result of reading to variable and pass that to other liberator-decisions?
The context can be updated by the outcome of each decision and action functions. You can parse the project once in malformed? and return a map with the parsed project which will be merged into the context so it is available to the following decisions and actions. For example:
(defresource add-new-project
:malformed? (fn[ctx] (let [project (parse-project ctx)]
(when (project-is-valid project)
{:project project})))
:handle-malformed (fn [_] (generate-string (str "Malformed json!")))
:post! (fn [ctx] (save-to-db (:project ctx)))
:handle-created (fn [ctx] (do-something (:project ctx))))
If the project is valid, :malformed? returns the {:project project} map which will be merged into the context to be used in the next decisions and actions.
If the project is not valid, it will return nil so execution continues in :handle-malformed.
For more info on liberator's execution model, see https://clojure-liberator.github.io/liberator/doc/execution-model.html
hopefully this is something simple for the more experienced out there. I am using clj-http and trying to pass the command line arg int it (to take a URL). I am an absolute Clojure beginer but I have managed to pass the args through to a ptintln which works.
(ns foo.core
(:require [clj-http.client :as client]))
(defn -main
[& args]
(def url (str args))
(println url)
(def resp (client/get url))
(def headers (:headers resp))
(def server (headers "server"))
(println server))
Error message
Ants-MacBook-Pro:target ant$ lein run "http://www.bbc.com"
("http://www.bbc.com")
Exception in thread "main" java.net.MalformedURLException: no protocol: ("http://www.bbc.com")
This works
(def resp (client/get "http://www.bbc.com"))
thanks in advance.
args is a list, which means that calling str on it returns the representation of the list, complete with parentheses and inner quotes, as you can see in your error trace:
(println (str '("http://www.bbc.com")))
;; prints ("http://www.bbc.com")
Of course, URLs don't start with parentheses and quotes, which is why the JVM tells you your URL is malformed.
What you really want to pass to get is not the string representation of your argument list, but your first argument:
(let [url (first args)]
(client/get url)) ;; Should work!
In addition, you should never use def calls within functions -- they create or rebind vars at the toplevel of your namespace, which don't want.
What you should be using instead is let forms, which create local variables (like url in my example). For more information on let, look at http://clojure.org/special_forms.
I'd probably structure your code like so:
(defn -main
[& args]
(let [url (first args)
resp (client/get url)
server (get-in resp [:headers "server"])]
(println url)
(println server)))
If I have the request "size=3&mean=1&sd=3&type=pdf&distr=normal" what's the idiomatic way of writing the function (defn request->map [request] ...) that takes this request and
returns a map {:size 3, :mean 1, :sd 3, :type pdf, :distr normal}
Here is my attempt (using clojure.walk and clojure.string):
(defn request-to-map
[request]
(keywordize-keys
(apply hash-map
(split request #"(&|=)"))))
I am interested in how others would solve this problem.
Using form-decode and keywordize-keys:
(use 'ring.util.codec)
(use 'clojure.walk)
(keywordize-keys (form-decode "hello=world&foo=bar"))
{:foo "bar", :hello "world"}
Assuming you want to parse HTTP request query parameters, why not use ring? ring.middleware.params contains what you want.
The function for parameter extraction goes like this:
(defn- parse-params
"Parse parameters from a string into a map."
[^String param-string encoding]
(reduce
(fn [param-map encoded-param]
(if-let [[_ key val] (re-matches #"([^=]+)=(.*)" encoded-param)]
(assoc-param param-map
(codec/url-decode key encoding)
(codec/url-decode (or val "") encoding))
param-map))
{}
(string/split param-string #"&")))
You can do this easily with a number of Java libraries. I'd be hesitant to try to roll my own parser unless I read the URI specs carefully and made sure I wasn't missing any edge cases (e.g. params appearing in the query twice with different values). This uses jetty-util:
(import '[org.eclipse.jetty.util UrlEncoded MultiMap])
(defn parse-query-string [query]
(let [params (MultiMap.)]
(UrlEncoded/decodeTo query params "UTF-8")
(into {} params)))
user> (parse-query-string "size=3&mean=1&sd=3&type=pdf&distr=normal")
{"sd" "3", "mean" "1", "distr" "normal", "type" "pdf", "size" "3"}
Can also use this library for both clojure and clojurescript: https://github.com/cemerick/url
user=> (-> "a=1&b=2&c=3" cemerick.url/query->map clojure.walk/keywordize-keys)
{:a "1", :b "2", :c "3"}
Yours looks fine. I tend to overuse regexes, so I would have solved it as
(defn request-to-keywords [req]
(into {} (for [[_ k v] (re-seq #"([^&=]+)=([^&]+)" req)]
[(keyword k) v])))
(request-to-keywords "size=1&test=3NA=G")
{:size "1", :test "3NA=G"}
Edit: try to stay away from clojure.walk though. I don't think it's officially deprecated, but it's not very well maintained. (I use it plenty too, though, so don't feel too bad).
I came across this question when constructing my own site and the answer can be a bit different, and easier, if you are passing parameters internally.
Using Secretary to handle routing: https://github.com/gf3/secretary
Parameters are automatically extracted to a map in :query-params when a route match is found. The example given in the documentation:
(defroute "/users/:id" [id query-params]
(js/console.log (str "User: " id))
(js/console.log (pr-str query-params)))
(defroute #"/users/(\d+)" [id {:keys [query-params]}]
(js/console.log (str "User: " id))
(js/console.log (pr-str query-params)))
;; In both instances...
(secretary/dispach! "/users/10?action=delete")
;; ... will log
;; User: 10
;; "{:action \"delete\"}"
You can use ring.middleware.params. Here's an example with aleph:
user=> (require '[aleph.http :as http])
user=> (defn my-handler [req] (println "params:" (:params req)))
user=> (def server (http/start-server (wrap-params my-handler)))
wrap-params creates an entry in the request object called :params. If you want the query parameters as keywords, you can use ring.middleware.keyword-params. Be sure to wrap with wrap-params first:
user=> (require '[ring.middleware.params :refer [wrap-params]])
user=> (require '[ring.middleware.keyword-params :refer [wrap-keyword-params])
user=> (def server
(http/start-server (wrap-keyword-params (wrap-params my-handler))))
However, be mindful that this includes a dependency on ring.