I see that when a web browser connects to a HTTP server the following lines are sent:
GET / HTTP/1.1
Content-Length: 13
Hello, world!
I want to write a program that takes an InputStream, reads these lines and returns a Ring request map according to the ring specification.
Could you please suggest a Clojure library for this purpose? I took a quick look at web server source codes with ring support (for example http-kit) but with no success so far.
If you are looking for a library to transform textual HTTP input into a Ring request map, search no further than Ring, the premiere library for both dealing with HTTP and producing Ring request maps.
This surely can be achieved with http-kit library. But there are 2 problems:
The class org.httpkit.server.ClojureRing is private. You have to access its method via reflection.
:remote-addr value is read from InetSocketAddress object assigned to request, or you need to to have X-Forwarded-For header in your request string.
Here is working demo:
(let [request "POST / HTTP/1.1\nContent-Length: 13\nX-Forwarded-For: 127.0.0.1\n\nHello, world!"
bytes (java.nio.ByteBuffer/wrap (.getBytes request))
decoder (org.httpkit.server.HttpDecoder.
8388608 4096 ProxyProtocolOption/DISABLED)
method (.getMethod org.httpkit.server.ClojureRing
"buildRequestMap"
(into-array [org.httpkit.server.HttpRequest]))]
(.setAccessible method true)
(.invoke method nil (object-array [(.decode decoder bytes)])))
=>
{:remote-addr "127.0.0.1",
:headers {"content-length" "13", "x-forwarded-for" "127.0.0.1"},
:async-channel nil,
:server-port 80,
:content-length 13,
:websocket? false,
:content-type nil,
:character-encoding "utf8",
:uri "/",
:server-name nil,
:query-string nil,
:body #object[org.httpkit.BytesInputStream 0x4f078b2 "BytesInputStream[len=13]"],
:scheme :http,
:request-method :post}
You may wish to review the Clojure Cookbook, online here, or even better in print form.
Web development with Clojure is also a good book.
I also have a "Hello World" example using the Pedestal library here.
Here is a sample request map using Pedestal:
request =>
{:protocol "HTTP/1.1",
:async-supported? true,
:remote-addr "127.0.0.1",
:servlet-response
#object[io.pedestal.test$test_servlet_response$reify__34946 0x3e71aa38 "io.pedestal.test$test_servlet_response$reify__34946#3e71aa38"],
:servlet
#object[io.pedestal.http.servlet.FnServlet 0x7168112e "io.pedestal.http.servlet.FnServlet#7168112e"],
:headers {"content-length" "0", "content-type" ""},
:server-port -1,
:servlet-request
#object[io.pedestal.test$test_servlet_request$reify__34934 0x3422eca "io.pedestal.test$test_servlet_request$reify__34934#3422eca"],
:content-length 0,
:content-type "",
:path-info "/echo/abcdef/12345",
:character-encoding "UTF-8",
:url-for #<Delay#5190186c: :not-delivered>,
:uri "/echo/abcdef/12345",
:server-name nil,
:query-string nil,
:path-params {:list-id "abcdef", :item-id "12345"},
:body
#object[io.pedestal.test.proxy$javax.servlet.ServletInputStream$ff19274a 0x2aff7cc4 "io.pedestal.test.proxy$javax.servlet.ServletInputStream$ff19274a#2aff7cc4"],
:scheme nil,
:request-method :get,
:context-path ""}
Update - There is almost no parsing required. You can generate a ring response with a very simple map:
(defn respond-hello [request] ; we ignore the request
{:status 200 :body "Hello, world!"})
Not sure what you are really after here.....? If you really want to do parsing from the ground up, the best option is Instaparse.
Related
I have come back to clojure after moderately dabbling with it about 10+ years ago, and so I might be doing something silly here.
I am trying to write a simple API with compojure and ring server, and right now I've isolated my problem to just a few lines. I have a route and a handler, and I've wrapped my handler with wrap-json-body as is suggested in ring-json documentation.
My handler.clj is like so:
(defroutes app-routes
(PUT "/item/:id" {{id :id} :params body :body} (str id ", " body))
(route/not-found "Not Found"))
(def app
(-> app-routes
(middleware/wrap-json-body)
(middleware/wrap-json-response)))
This should be simple enough, and I am able to return clojure data as json OK. Problem is when I'm trying to read PUT request body json.
$ curl -XPUT -H "Content-type: application/json" -d '{ "id": 32, "name": "pad" }' 'localhost:3001/item/5'
5, {"id" 32, "name" "pad"}
I would expect body to be populated with {:id 32 :name "pad"}
Here's the whole request object:
; (PUT "/item/:id" body (str id body))
$ curl -XPUT -H "Content-type: application/json" -d '{ "id": 32, "name": "pad" }'
{:ssl-client-cert nil, :protocol "HTTP/1.1", :remote-addr "0:0:0:0:0:0:0:1", :params {:id "5"}, :route-params {:id "5"}, :headers {"user-agent" "curl/7.58.0", "host" "localhost:3001", "accept" "*/*", "content-length" "27", "content-type" "application/json"}, :server-port 3001, :content-length 27, :compojure/route [:put "/item/:id"], :content-type "application/json", :character-encoding "UTF-8", :uri "/item/5", :server-name "localhost", :query-string nil, :body {"id" 32, "name" "pad"}, :scheme :http, :request-method :put}
I've tweaked and changed it to a few things but I can't seem to get :body to be populated with keyword-ed clojure data.
What am I doing wrong please?
ps: If you'd like to see a working example of this problem, I've uploaded it to github
Use the keywords? parameter in wrap-json-body:
(middleware/wrap-json-body app-routes {:keywords? true})
Alternatively, since ring-clojure version 0.5.1, you can specify a custom key-fn to convert the keys of maps:
(middleware/wrap-json-body app-routes {:key-fn keyword})
There also exists a "wrap-keyword-params" ring middleware which is usually placed at the end of the middleware chain and it converts all the keys in :params gathered up to that point into keywords.
Where I work at, we usually use the following pattern:
(def app
(-> handler
wrap-keyword-params ;; <--- converts any string keys in the :params map to keywords
wrap-nested-params ;; <--- parses nested params
wrap-params ;; <--- parses query-params into :query-params and adds parsed request body props into :params
wrap-json-params ;; <--- parses body of a request and adds ":params" and ":json-params" keys to "request" argument (in the following format: {"key" "val"})
wrap-json-response) ;; <--- converts response with body that is a map or a vector to JSON)
I'm sending the following response in clojure ring:
(res/set-cookie
(res/redirect (env :some-url))
"some-id"
(->
req
foo-ns/bar
:id
)
{:max-age (* 30 24 60 60 1000) :path "/"})
And on printing this response I get:
{:status 302, :headers {"Location" "http://localhost:5000"}, :body "", :cookies {"some-id" {:value "1341313515135490454", :max-age 2592000000, :path "/"}}}
But on the client side, the cookie isn't set, which I can see in the console. What am I doing wrong?
It looks like you're using ring.response/set-cookie to set the cookie. That will set the cookie attributes under :cookies in your response map. Before returning the response to the browser, you need to encode those cookies into a Set-Cookie header that the browser can understand. To do this, add the ring.middleware.cookies/wrap-cookies middleware to your middleware stack.
You should expect your response to look something like:
{:status 302
:body ""
:headers {"Location" "http://localhost:5000"
"Set-Cookie" "some-id=1341313515135490454; max-age=2592000000; path=/"}}
I'm using the last available lacinia version: "0.36.0-alpha-3" with Luminus (Ring+reitit), but this version asks for a specific header:
$ curl 'http://localhost:3000/api/graphql' -X POST --data "{test_by_id(id: 5) { title } }" -H 'Content-Type: application/graphql'
that request works fine, but without "'Content-Type: application/graphql'" the request wouldn't work. So I need to define my re-graph init vector like:
[::re-graph/init
{:ws-url nil
:http-url "http://localhost:3000/api/graphql"
:http-parameters {:with-credentials? false
:headers {"Content-Type" "application/graphql"}
}
:ws-reconnect-timeout nil
:resume-subscriptions? false
:connection-init-payload {}}]
but putting that header makes re-graph unable to work properly:
{"errors":[{"message":"Failed to parse GraphQL query.","extensions":{"errors":[{"locations":[{"line":1,"column":null}],"message":"mismatched input '\"query\"' expecting {'query', 'mutation', 'subscription',
it looks like re-graph sends and receives data using "application/json" header, so lacinia asks for some type of header but re-graph can't work with that option.
I had the same problem, and I think I got a solution for it. re-frame requests follows the Apollo Specification, as stated by #aarkerio. Here is the code to keep the original endpoint working with the origina specification, and allow it to respond to re-frame requests. This will make the endpoint respond to Graphiql request (from your http://localhost:3000/graphiql route), and re-graph ones. Any comments or corrections are welcomed.
Replace the original function set on the /graphql route on src/clj/mem_learning/routes/services.clj:
["/graphql" {:post graphql-call}
Add the graphql-call function on that same file:
(defn graphql-call [req]
(let [body (:body-params req)
content-type (keyword (get-in req [:headers "content-type"]))]
(case content-type
:application/json (ok (graphql/execute-request-re-graph body))
:application/graphql (ok (graphql/execute-request (-> req :body slurp))))))
add the execute-request-re-graph to the src/clj/mem_learning/routes/services/graphql.clj file:
(defn execute-request-re-graph
"execute request with re-graph/apollo format"
[{:keys [variables query context]}]
(lacinia/execute compiled-schema query variables context)))
ANSWER:
It looks that Luminus creates a middleware configuration:
(defn service-routes []
["/api"
{:coercion spec-coercion/coercion
:muuntaja formats/instance
:swagger {:id ::api}
:middleware [;; query-params & form-params
parameters/parameters-middleware
;; content-negotiation
muuntaja/format-negotiate-middleware
;; encoding response body
muuntaja/format-response-middleware
;; exception handling
exception/exception-middleware
;; decoding request body
muuntaja/format-request-middleware
;; coercing response bodys
coercion/coerce-response-middleware
;; coercing request parameters
coercion/coerce-request-middleware
;; multipart
multipart/multipart-middleware
]}
commenting the line "muuntaja/format-negotiate-middleware" makes the "application/json" call possible.
SECOND UPDATE (four hours later)
Ok, that muuntaja middleware thing was not the problem at all, the real problem is that curl send the data with the format:
{ test_by_id(id: 7, archived: false) { title } }
meanwhile re-graph uses:
{"query":"query { test_by_id(id: 7, archived: false) { title } }","variables":null}
this is a normal java string btw not a data structure, so we need to do some changes, first a new function:
(defn graphql-call [req]
(let [body (-> req :body slurp)
full-query (json/read-str body :key-fn keyword)
_ (log/info (str ">>> **** full-query >>>>> " full-query))]
(ok (graphql/execute-request full-query))))
we set the function:
["/graphql" {:post graphql-call}]
and in my_app.routes.services.graphql file:
(defn execute-request [{:keys [variables query context]}]
(json/write-str (lacinia/execute compiled-schema query variables context)))
and now re-graph works!
(also now I can send and use variables in GraphQL)
It's necessary to set:
:http-parameters {:with-credentials? false
:oauth-token "ah4rdSecr3t"
:headers {"Content-Type" "application/graphql"}
btw. Also, maybe it's better:
(lacinia/execute compiled-schema query variables context)
than:
(json/write-str (lacinia/execute compiled-schema query variables context))
because it interferes with re-graph importing the data already as a native ClojureScript map.
I have POSTed data to a Pedestal endpoint "/my-post. I have routed that end point as such:
[[["/" {:get landing} ^:interceptors [(body-params/body-params) ...]
["/my-post {:post mypost-handler}
....
So to my mind this means that the body-params interceptor will fire for /my-post too.
In mypost-handler I have:
(defn mypost-handler
[request]
****HOW TO ACCESS THEN FORM DATA HERE ****
)
How do I now access the form data here? I can see from printing the request that I have a #object[org.eclipse.jetty.sever.HttpInputOverHTTP..] which will clearly need further processing before it is useful to me.
(I must say, the documentation for Pedestal is pretty sketchy at best...)
Something like this should work. Note the body-params interceptor on the mypost-handler route
(defn mypost-handler
[{:keys [headers params json-params path-params] :as request}]
;; json-params is the posted json, so
;; (:name json-params) will be the value (i.e. John) of name property of the posted json {"name": "John"}
;; handle request
{:status 200
:body "ok"})
(defroutes routes
[[["/mypost-handler" {:post mypost-handler}
^:interceptors [(body-params/body-params)]
]
]])
The mypost-handler is acting as a Ring handler, i. e. it should accept a Ring request map and return a Ring response map. Thus, you can expect a typical Ring request structure:
(defn mypost-handler
[{:keys [headers params json-params path-params] :as request}]
;; handle request
{:status 200
:body "ok"})
Here's more relevant info on defining such handlers in your route tables.
i am trying to retrieve a website with Clojure and the clj-http library.
I wanted to start slow with a simple example:
(:body (client/get (str "http://www.google.com") {:as :clojure}))
As far as i understand the behaviour of the library, this call should return the body of the website but all it returns is <!doctype.
When i try to call
(:body (client/get (str "http://www.google.com") {:as :json}))
i even get an exception:
com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60))
I can not imagine the library to be broken but also i am not able to see an obvious error in my call. Has anybody of you experienced this behaviour?
To get the http response body as a string you can use the following:
(:body (client/get "http://www.google.com"))
The :as entry in the {:as :clojure} options is output-coercion, and is trying to convert the HTML body, from google.com, into a Clojure data structure. This will fail unless the response body actually contains Clojure code.
If you are trying to parse the HTML response, you might need to look into an additional library, like Enlive.
The problem is you're querying a URL that is not returning the data type you're coercing the result to.
For instance if you try with http://ip.jsontest.com/ this url which returns a proper json:
(require '[clj-http.client :as client])
(client/get "http://ip.jsontest.com/" {:as :json})
=> {:trace-redirects ["http://ip.jsontest.com/"], :request-time 1153,
:status 200,
:headers {"access-control-allow-origin" "*", "content-type" "application/json; charset=ISO-8859-1", "date" "Tue, 22 Oct 2013 19:50:36 GMT", "server" "Google Frontend", "cache-control" "private", "alternate-protocol" "80:quic,80:quic", "connection" "close"}, :body {:ip "186.54.233.167"}}
Response is properly parsed.
Checking the response body you can easily see there's a json indeed there:
(:body (client/get "http://ip.jsontest.com/"))
=> "{\"ip\": \"186.54.233.167\"}\n"