I made service endpoint api for getting single object by id and it works as expected. I tested it with Postman and in handler function. I use cljs-ajax library for asynchronous client. I cant change the state of Reagent atom when I get response. Here is the code:
(ns businesspartners.core
(:require [reagent.core :as r]
[ajax.core :refer [GET POST]]
[clojure.string :as string]))
(def business-partner (r/atom nil))
(defn get-partner-by-id [id]
(GET "/api/get-partner-by-id"
{:headers {"Accept" "application/transit+json"}
:params {:id id}
:handler #(reset! business-partner (:business-partner %))}))
When I tried to access business-partner atom I got nil value for that atom. I can't figure out why because another method is almost the same except it get's list of business partners and works fine.
When I change the get-partner-by-id function:
(defn get-partner-by-id [id]
(GET "/api/get-partner-by-id"
{:headers {"Accept" "application/transit+json"}
:params {:id id}
:handler (fn [arg]
(println :handler-arg arg)
(reset! business-partner (:business-partner arg))
(println "Business partner from handler: " #business-partner))}))
Output in the browser console:
:handler-arg {:_id 5e7ad2c84b5c2d44583e8ecd,
:address Main Street,
:email nenmit#gmail.com,
:phone 555888,
:name Nen Mit}
Business partner from handler: nil
So, as you can see, I have my object in handler as desired, but when I try to reset my atom nothing happens. That's the core of the problem I think. Thank you Alan.
When in doubt, use debug print statements. Make your handler look like this:
:handler (fn [arg]
(println :handler-arg arg)
(reset! business-partner (:business-partner arg)))
You may also want to use clojure.pprint/pprint to pretty-print the output, or also add (type arg) to the output.
You may also want to initialize the atom to a specific value like
:bp-default so you can see if the nil you observe is the original one or if it is being reset to nil.
Update
So it is clear the key :business-partner does not exist in the map you are receiveing. This is what you must debug.
Trying to pull a non-existent key out of a map always returns nil. You could also use the 3-arg version of get to make this explicit. Convert
(:business-partner arg) => (get arg :business-partner ::not-found)
and you'll see the keyword ::not-found appear in your atom, verifying what is occurring.
In order to catch these problems early, I nearly always use a simple function grab from the Tupelo library like so:
(:business-partner arg) => (grab :business-partner arg)
The grab function will throw an exception if the expected key is not found. This provides early-warning of problems so you can track them down faster.
Another hint: next time use prn instead of println and it will retain double-quotes on string output like:
"Main Street"
Related
I have a very simple question about using Prismatic/schema to validate functions. I have a schema for a map that has a single key, the value of which is a function that takes a Bar schema as its single argument and returns anything (used for side effects):
(require '[schema.core :as s])
(def Bar {:baz s/Int})
(def Action :???)
(def Foo {:action Action})
The question is, how do I define Action? I've tried this:
(require '[schema.macros :as sm])
(def Action (sm/=> s/Any Bar))
This looks promising, but I can't get it to fail validation:
(s/explain Action)
;=> (=> Any {:baz Int})
;; This should fail
(s/validate Foo {:action :anything-goes})
;=> {:action :anything-goes}
What am I doing wrong here?
I read the docs and the tests in core_test, but I can't figure out how to do this.
I've found this: https://github.com/Prismatic/schema/blob/a21cc0113ed497f6410c55d92d9088bd710f0b47/src/cljx/schema/core.cljx#L888
So it would be something like:
(def Action (s/make-fn-schema s/Any [[Bar]]))
Although, the documentation does say this:
Currently function schemas are purely descriptive; they validate against any function, regardless of actual input and output types
I'm trying to build a simple rest api, using liberator and monger :
(ns tm.place.endpoint
(:require [liberator.core :refer [defresource]]
[monger [collection :as mc] [query :as mq]]
[clojure.data [json :as json]]
monger.json))
(defresource entity-place-resource [id]
:available-media-types ["application/json"]
:allowed-methods [:get :put :delete]
:exists? (if-let [place (mc/find-map-by-id "place" id)] {:place place})
:handle-ok :place
; :handle-created (fn [ctx]
; (let [body (-> ctx :request :body)]
; (assoc body :_id id)))
:handle-not-found []
:handle-not-acceptable "Should accept application/json"
:handle-method-not-allowed (json/write-str
{:message "Can only handle get, put and delete"})
; :handle-not-implemented
; :handle-no-content
:put! (fn [ctx]
(let [body (-> ctx :request :body)]
(mc/update-by-id "place" id (assoc body :_id id))))
:delete! (mc/remove-by-id "place" id)
:can-put-to-missing? false)
I'm using advanced rest client to see if it works. If the method is :get or :delete, it does perfectly what I want (It first check if a document is exists, and then do the appropriate action). However, if the method is :put, it simply spits out http 201 created, which I think the request is success, but the corresponding document is deleted, not updated.
If I comment out the :delete! line, the :put! is work as expected, so I'm guessing the culprit is the :delete! line, but I have no idea why because since I'm using :put method, I'm assuming :delete! should remain untouched. Any idea why?
As you have defined your delete! as a bare function it's going to get evaluated when you don't expect. The pattern you have followed for put! needs to be applied, where you return a fn that does the work rather than act directly.
http://clojure-liberator.github.io/liberator/faq.html
Due to the exact details of the defresource macro expansion, the form used as the value is evaluated at unexpected times.
hopefully this is something simple for the more experienced out there. I am using clj-http and trying to pass the command line arg int it (to take a URL). I am an absolute Clojure beginer but I have managed to pass the args through to a ptintln which works.
(ns foo.core
(:require [clj-http.client :as client]))
(defn -main
[& args]
(def url (str args))
(println url)
(def resp (client/get url))
(def headers (:headers resp))
(def server (headers "server"))
(println server))
Error message
Ants-MacBook-Pro:target ant$ lein run "http://www.bbc.com"
("http://www.bbc.com")
Exception in thread "main" java.net.MalformedURLException: no protocol: ("http://www.bbc.com")
This works
(def resp (client/get "http://www.bbc.com"))
thanks in advance.
args is a list, which means that calling str on it returns the representation of the list, complete with parentheses and inner quotes, as you can see in your error trace:
(println (str '("http://www.bbc.com")))
;; prints ("http://www.bbc.com")
Of course, URLs don't start with parentheses and quotes, which is why the JVM tells you your URL is malformed.
What you really want to pass to get is not the string representation of your argument list, but your first argument:
(let [url (first args)]
(client/get url)) ;; Should work!
In addition, you should never use def calls within functions -- they create or rebind vars at the toplevel of your namespace, which don't want.
What you should be using instead is let forms, which create local variables (like url in my example). For more information on let, look at http://clojure.org/special_forms.
I'd probably structure your code like so:
(defn -main
[& args]
(let [url (first args)
resp (client/get url)
server (get-in resp [:headers "server"])]
(println url)
(println server)))
I tried to create function to scrape and tags from HTML page, whose URL I provide to a function, and this works as it should. I get sequence of <h3> and <table> elements, when I try to use select function to extract only table or h3 tags from resulting sequence,
I get (), or if I try to map those tags I get (nil nil nil ...).
Could you please help me to resolve this issue, or explain me what am I doing wrong?
Here is the code:
(ns Test2
(:require [net.cgrand.enlive-html :as html])
(:require [clojure.string :as string]))
(defn get-page
"Gets the html page from passed url"
[url]
(html/html-resource (java.net.URL. url)))
(defn h3+table
"returns sequence of <h3> and <table> tags"
[url]
(html/select (get-page url)
{[:div#wrap :div#middle :div#content :div#prospekt :div#prospekt_container :h3]
[:div#wrap :div#middle :div#content :div#prospekt :div#prospekt_container :table]}
))
(def url "http://www.belex.rs/trgovanje/prospekt/VZAS/show")
This line gives me headache :
(html/select (h3+table url) [:table])
Could you please tell me what am I doing wrong?
Just to clarify my question: is it possible to use enlive's select function to extract only table tags from result of (h3+table url) ?
As #Julien pointed out, you will probably have to work with the deeply nested tree structure that you get from applying (html/select raw-html selectors) on the raw html. It seems like you try to apply html/select multiple times, but this doesn't work. html/select parses html into a clojure datastructure, so you can't apply it on that datastructure again.
I found that parsing the website was actually a little involved, but I thought that it might be a nice use case for multimethods, so I hacked something together, maybe this will get you started:
(The code is ugly here, you can also checkout this gist)
(ns tutorial.scrape1
(:require [net.cgrand.enlive-html :as html]))
(def *url* "http://www.belex.rs/trgovanje/prospekt/VZAS/show")
(defn get-page [url]
(html/html-resource (java.net.URL. url)))
(defn content->string [content]
(cond
(nil? content) ""
(string? content) content
(map? content) (content->string (:content content))
(coll? content) (apply str (map content->string content))
:else (str content)))
(derive clojure.lang.PersistentStructMap ::Map)
(derive clojure.lang.PersistentArrayMap ::Map)
(derive java.lang.String ::String)
(derive clojure.lang.ISeq ::Collection)
(derive clojure.lang.PersistentList ::Collection)
(derive clojure.lang.LazySeq ::Collection)
(defn tag-type [node]
(case (:tag node)
:tr ::CompoundNode
:table ::CompoundNode
:th ::TerminalNode
:td ::TerminalNode
:h3 ::TerminalNode
:tbody ::IgnoreNode
::IgnoreNode))
(defmulti parse-node
(fn [node]
(let [cls (class node)] [cls (if (isa? cls ::Map) (tag-type node) nil)])))
(defmethod parse-node [::Map ::TerminalNode] [node]
(content->string (:content node)))
(defmethod parse-node [::Map ::CompoundNode] [node]
(map parse-node (:content node)))
(defmethod parse-node [::Map ::IgnoreNode] [node]
(parse-node (:content node)))
(defmethod parse-node [::String nil] [node]
node)
(defmethod parse-node [::Collection nil] [node]
(map parse-node node))
(defn h3+table [url]
(let [ws-content (get-page url)
h3s+tables (html/select ws-content #{[:div#prospekt_container :h3]
[:div#prospekt_container :table]})]
(for [node h3s+tables] (parse-node node))))
A few words on what's going on:
content->string takes a data structure and collects its content into a string and returns that so you can apply this to content that may still contain nested subtags (like <br/>) that you want to ignore.
The derive statements establish an ad hoc hierarchy which we will later use in the multi-method parse-node. This is handy because we never quite know which data structures we're going to encounter and we could easily add more cases later on.
The tag-type function is actually a hack that mimics the hierarchy statements - AFAIK you can't create a hierarchy out of non-namespace qualified keywords, so I did it like this.
The multi-method parse-node dispatches on the class of the node and if the node is a map additionally on the tag-type.
Now all we have to do is define the appropriate methods: If we're at a terminal node we convert the contents to a string, otherwise we either recur on the content or map the parse-node function on the collection we're dealing with. The method for ::String is actually not even used, but I left it in for safety.
The h3+table function is pretty much what you had before, I simplified the selectors a bit and put them into a set, not sure if putting them into a map as you did works as intended.
Happy scraping!
Your question is hard to understand, but I think your last line should simply be
(h3+table url)
This will return a deeply nested data structure containing scraped HTML that you can then dive into with the usual Clojure sequence APIs. Good luck.
If I have the request "size=3&mean=1&sd=3&type=pdf&distr=normal" what's the idiomatic way of writing the function (defn request->map [request] ...) that takes this request and
returns a map {:size 3, :mean 1, :sd 3, :type pdf, :distr normal}
Here is my attempt (using clojure.walk and clojure.string):
(defn request-to-map
[request]
(keywordize-keys
(apply hash-map
(split request #"(&|=)"))))
I am interested in how others would solve this problem.
Using form-decode and keywordize-keys:
(use 'ring.util.codec)
(use 'clojure.walk)
(keywordize-keys (form-decode "hello=world&foo=bar"))
{:foo "bar", :hello "world"}
Assuming you want to parse HTTP request query parameters, why not use ring? ring.middleware.params contains what you want.
The function for parameter extraction goes like this:
(defn- parse-params
"Parse parameters from a string into a map."
[^String param-string encoding]
(reduce
(fn [param-map encoded-param]
(if-let [[_ key val] (re-matches #"([^=]+)=(.*)" encoded-param)]
(assoc-param param-map
(codec/url-decode key encoding)
(codec/url-decode (or val "") encoding))
param-map))
{}
(string/split param-string #"&")))
You can do this easily with a number of Java libraries. I'd be hesitant to try to roll my own parser unless I read the URI specs carefully and made sure I wasn't missing any edge cases (e.g. params appearing in the query twice with different values). This uses jetty-util:
(import '[org.eclipse.jetty.util UrlEncoded MultiMap])
(defn parse-query-string [query]
(let [params (MultiMap.)]
(UrlEncoded/decodeTo query params "UTF-8")
(into {} params)))
user> (parse-query-string "size=3&mean=1&sd=3&type=pdf&distr=normal")
{"sd" "3", "mean" "1", "distr" "normal", "type" "pdf", "size" "3"}
Can also use this library for both clojure and clojurescript: https://github.com/cemerick/url
user=> (-> "a=1&b=2&c=3" cemerick.url/query->map clojure.walk/keywordize-keys)
{:a "1", :b "2", :c "3"}
Yours looks fine. I tend to overuse regexes, so I would have solved it as
(defn request-to-keywords [req]
(into {} (for [[_ k v] (re-seq #"([^&=]+)=([^&]+)" req)]
[(keyword k) v])))
(request-to-keywords "size=1&test=3NA=G")
{:size "1", :test "3NA=G"}
Edit: try to stay away from clojure.walk though. I don't think it's officially deprecated, but it's not very well maintained. (I use it plenty too, though, so don't feel too bad).
I came across this question when constructing my own site and the answer can be a bit different, and easier, if you are passing parameters internally.
Using Secretary to handle routing: https://github.com/gf3/secretary
Parameters are automatically extracted to a map in :query-params when a route match is found. The example given in the documentation:
(defroute "/users/:id" [id query-params]
(js/console.log (str "User: " id))
(js/console.log (pr-str query-params)))
(defroute #"/users/(\d+)" [id {:keys [query-params]}]
(js/console.log (str "User: " id))
(js/console.log (pr-str query-params)))
;; In both instances...
(secretary/dispach! "/users/10?action=delete")
;; ... will log
;; User: 10
;; "{:action \"delete\"}"
You can use ring.middleware.params. Here's an example with aleph:
user=> (require '[aleph.http :as http])
user=> (defn my-handler [req] (println "params:" (:params req)))
user=> (def server (http/start-server (wrap-params my-handler)))
wrap-params creates an entry in the request object called :params. If you want the query parameters as keywords, you can use ring.middleware.keyword-params. Be sure to wrap with wrap-params first:
user=> (require '[ring.middleware.params :refer [wrap-params]])
user=> (require '[ring.middleware.keyword-params :refer [wrap-keyword-params])
user=> (def server
(http/start-server (wrap-keyword-params (wrap-params my-handler))))
However, be mindful that this includes a dependency on ring.