I am new to clojure and I am trying to make a simple API with 3 endpoints.
I am trying to implement an endpoint which will take each row of a query and put it to a json.
So I have an SQLite database. These are my entries for example:
{:timestamp 2020-09-11 14:29:30, :lat 36.0, :long 36.0, :user michav}
{:timestamp 2020-09-11 14:31:47, :lat 36.0, :long 36.0, :user michav}
So I want a json response like the below:
{:get
:status 200
:body {:timestamp "2020-09-11 14:29:30"
:lat 36.0
:long 36.0
:user "michav"}
{:timestamp "2020-09-11 14:31:47"
:lat 36.0
:long 36.0
:user "michav"}
}
}
Here is my code that I am trying to fix it in order to get the above result.
(def db
{:classname "org.sqlite.JDBC"
:subprotocol "sqlite"
:subname "db/database.db"
})
(defn getparameter [req pname] (get (:params req) pname))
(defn output
"execute query and return lazy sequence"
[]
(query db ["select * from traffic_users"]))
(defn print-result-set
"prints the result set in tabular form"
[result-set]
(doseq [row result-set]
(println row)))
(defn request-example [req]
(response {:get {:status 200
:body (-> (doseq [row output]
{:timestamp (getparameter row :timestamp) :lat (getparameter row :lat) :long (getparameter row :long) :user (getparameter row :user)} ))
}}))
(defroutes my_routes
(GET "/request" [] request-example)
(route/resources "/"))
(def app (-> #'my_routes wrap-cookies wrap-keyword-params wrap-params wrap-json-response))
The error I encounter is the following :
java.lang.IllegalArgumentException: Don't know how to create ISeq from: mybank.core$output
RT.java:557 clojure.lang.RT.seqFrom
RT.java:537 clojure.lang.RT.seq
core.clj:137 clojure.core/seq
core.clj:137 clojure.core/seq
core.clj:65 mybank.core/request-example[fn]
core.clj:65 mybank.core/request-example
core.clj:63 mybank.core/request-example
response.clj:47 compojure.response/eval1399[fn]
response.clj:7 compojure.response/eval1321[fn]
core.clj:158 compojure.core/wrap-response[fn]
core.clj:128 compojure.core/wrap-route-middleware[fn]
core.clj:137 compojure.core/wrap-route-info[fn]
core.clj:146 compojure.core/wrap-route-matches[fn]
core.clj:185 compojure.core/routing[fn]
core.clj:2701 clojure.core/some
core.clj:2692 clojure.core/some
core.clj:185 compojure.core/routing
core.clj:182 compojure.core/routing
RestFn.java:139 clojure.lang.RestFn.applyTo
core.clj:667 clojure.core/apply
core.clj:660 clojure.core/apply
core.clj:192 compojure.core/routes[fn]
Var.java:384 clojure.lang.Var.invoke
cookies.clj:171 ring.middleware.cookies/wrap-cookies[fn]
keyword_params.clj:32 ring.middleware.keyword-params/wrap-keyword-params[fn]
params.clj:57 ring.middleware.params/wrap-params[fn]
json.clj:42 ring.middleware.json/wrap-json-response[fn]
Var.java:384 clojure.lang.Var.invoke
reload.clj:18 ring.middleware.reload/wrap-reload[fn]
stacktrace.clj:17 ring.middleware.stacktrace/wrap-stacktrace-log[fn]
stacktrace.clj:80 ring.middleware.stacktrace/wrap-stacktrace-web[fn]
jetty.clj:27 ring.adapter.jetty/proxy-handler[fn]
(Unknown Source) ring.adapter.jetty.proxy$org.eclipse.jetty.server.handler.AbstractHandler$ff19274a.handle
HandlerWrapper.java:127 org.eclipse.jetty.server.handler.HandlerWrapper.handle
Server.java:500 org.eclipse.jetty.server.Server.handle
HttpChannel.java:386 org.eclipse.jetty.server.HttpChannel.lambda$handle$1
HttpChannel.java:562 org.eclipse.jetty.server.HttpChannel.dispatch
HttpChannel.java:378 org.eclipse.jetty.server.HttpChannel.handle
HttpConnection.java:270 org.eclipse.jetty.server.HttpConnection.onFillable
AbstractConnection.java:311 org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded
FillInterest.java:103 org.eclipse.jetty.io.FillInterest.fillable
ChannelEndPoint.java:117 org.eclipse.jetty.io.ChannelEndPoint$2.run
EatWhatYouKill.java:336 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask
EatWhatYouKill.java:313 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce
EatWhatYouKill.java:171 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce
EatWhatYouKill.java:135 org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produce
QueuedThreadPool.java:806 org.eclipse.jetty.util.thread.QueuedThreadPool.runJob
QueuedThreadPool.java:938 org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run
(Unknown Source) java.lang.Thread.run
Your mistake is on this line:
(doseq [row output] ...
The error message
Don't know how to create ISeq from: mybank.core$output
gives the clue. The variable output is the function, not a sequence like doseq expects. You meant to call the output function, so you need to use parentheses to create a function call like:
(doseq [row (output)] ...
Update
You need to include an external library to convert between EDN data and a JSON string. My favorite way is my own library Tupelo Clojure. Use it like this:
(ns tst.demo.core
(:use tupelo.core tupelo.test))
(let [data [{:timestamp "2020-09-11 14:29:30", :lat 36.0, :long 36.0, :user "michav"}
{:timestamp "2020-09-11 14:31:47", :lat 36.0, :long 36.0, :user "michav"}]]
(println (edn->json data)))
with result:
[{"timestamp":"2020-09-11 14:29:30","lat":36.0,"long":36.0,"user":"michav"},
{"timestamp":"2020-09-11 14:31:47","lat":36.0,"long":36.0,"user":"michav"}]
You will need a line like this in your project.clj:
[tupelo "20.08.27"]
Please also see this template project for an easy way to get started. Enjoy!
Related
(:require [utils.base64 :as base64])
(:import [com.google.api.client.googleapis.auth.oauth2 GoogleCredential]
[com.google.api.services.indexing.v3 Indexing]
[com.google.api.services.indexing.v3.model UrlNotification]
[com.google.api.client.http HttpTransport]
[com.google.api.client.json.gson GsonFactory]
[java.io ByteArrayInputStream]
[java.util Base64]
[com.google.api.services.indexing.v3 IndexingScopes]))
;; Set the private key and client email
(def private-key "")
(def client-email "my-client-email")
(def encoded-private-key (base64/encode-string private-key))
;; Set the private key and client email
(def private-key-json '{
"type": "service_account",
"client_email": "' client-email '",
"private_key": "' encoded-private-key '"
}')
;; Create the input stream from the JSON string
(def input-stream (ByteArrayInputStream. (.getBytes private-key-json)))
;; Create the OAuth 2.0 credentials
(def credentials (doto (GoogleCredential/fromStream input-stream)
(.createScoped (java.util.Collections/singleton IndexingScopes/INDEXING))))
;; Set up the HTTP transport and JSON factory
(def http-transport (HttpTransport/newTrustedTransport))
(def json-factory (GsonFactory/getDefaultInstance))
;; Set up the Indexing API client
(def indexing-client (doto (Indexing/Builder. http-transport json-factory credentials)
(.setApplicationName "My Indexing App")
(.build)))
;; Publish a URL notification
(def url-notification (UrlNotification. "https://aaa.com" "URL_UPDATED"))
(.execute (indexing-client/urlNotifications) (publish url-notification))
I'm trying to use private-key-json but the format is not valid. What is the best way to pass data? I referred https://developers.google.com/search/apis/indexing-api/v3/prereqs#examples.
Here I'm trying not to upload json file which we get once we create a service account and make use of only mandatory fields like client_email, private_key and type fields.
Modified code:
(:require [cheshire.core :as json]
[sketches.utils.base64 :as base64]
[clojure.string :as str])
(:import [com.google.api.client.googleapis.auth.oauth2 GoogleCredential]
[com.google.api.services.indexing.v3 Indexing$Builder]
[com.google.api.services.indexing.v3.model UrlNotification]
[com.google.api.client.googleapis.javanet GoogleNetHttpTransport]
[com.google.api.client.http HttpTransport]
[com.google.api.client.json.gson GsonFactory]
[java.io ByteArrayInputStream]
[com.google.api.services.indexing.v3 IndexingScopes]))
(def creds-json {:type "service_account",
:client_email "test-indexing-api",
:client_id "117578194507835125202",
:private_key_id "964321e5fc29980944da518887116ea50dfb7803",
:private_key ""})
(defn string->stream
([s] (string->stream s "UTF-8"))
([s encoding]
(-> s
(.getBytes encoding)
(ByteArrayInputStream.))))
(defn authorize [cred]
(-> cred
(json/encode)
(string->stream)
(GoogleCredential/fromStream)
(.createScoped [(IndexingScopes/INDEXING)])))
(defn valid-private-key [private-key]
(try
(when private-key
(->> (str/trim private-key)
(re-matches #"^-----BEGIN PRIVATE KEY-----[\s|\S]*-----END PRIVATE KEY-----[\\|n]*")
(some?)))
(catch Exception e
false)))
(def encoded-private-key (base64/encode-string private-key))
(def private-creds-json-string (json/generate-string creds-json))
(def input-stream (ByteArrayInputStream. (.getBytes private-creds-json-string)))
(defn build-analytics-client [creds]
(-> (Indexing$Builder. (GoogleNetHttpTransport/newTrustedTransport) (GsonFactory/getDefaultInstance) (authorize creds))
(.build)))
(defn url-notification []
(-> (UrlNotification.)
(.setType "URL_update")
(.setUrl "http://ace.madrid.quintype.io")))
(defn realtime-data [creds]
(-> (build-analytics-client creds)
(.urlNotifications)
(.publish (url-notification))
(.execute)))
```
But I get this error `Execution error (NoSuchMethodError) at com.google.api.client.http.ConsumingInputStream/close (ConsumingInputStream.java:40).
com.google.common.io.ByteStreams.exhaust(Ljava/io/InputStream;)J`
I'm trying to adapt the code here https://github.com/danownsthisspace/shorturl/blob/main/src/shorturl/core.clj with this:
(ns todo.core
(:require [clojure.pprint :as pprint]
[muuntaja.core :as m]
[reitit.ring :as ring]
[reitit.ring.middleware.muuntaja :as muuntaja]
[ring.adapter.jetty :as ring-jetty]
[ring.util.response :as r]
[todo.db :as db]))
(defn todo-items-save [req]
(clojure.pprint/pprint req)
(let [title (get-in req [:body-params :title])
content (get-in req [:body-params :content])]
(r/response (str "foooo" title))))
(def app
(ring/ring-handler
(ring/router
[["/"
["" {:handler (fn [req] {:body "hello" :status 200})}]]
["/api/todo" {:post {:handler todo-items-save}
:get (fn [req]
(let [todos db/get-todos]
(r/response todos)))}]
{:data {:muuntaja m/instance :middleware [muuntaja/format-middleware]}}])))
(defn start []
(ring-jetty/run-jetty #'app {:port 3002 :join? false}))
(def server (start))
(.stop server)
but I'm seeing that the body-params values are null. I was wondering why that is when I'm making a post request with the body {"content":"only a test", "title":"second"}. Thank you.
The problem was, that the muuntaja config was passed as part of the
routes instead of argument to ring/router
(def app
(ring/ring-handler
(ring/router
[["/"
["" {:handler (fn [req] {:body "hello" :status 200})}]]
["/api/todo" {:post {:handler todo-items-save}
:get (fn [req]
(let [todos db/get-todos]
(r/response todos)))}]]
; XXX this must be the second argument to `ring/router`
{:data {:muuntaja m/instance
:middleware [muuntaja/format-middleware]}})))
I have the following client-side request:
(let [form-data (doto
(js/FormData.)
(.append "filename" file))]
(ajax-request
{:uri "/some/uri"
:method :post
:body form-data
:format (raw-request-format)
:response-format (raw-response-format)})
)
And on the server (with liberator):
(defresource some-resource [_]
:allowed-methods [:get :post]
:available-media-types ["application/json"]
:exists? (fn [ctx]
(prn "form-data " (-> ctx :request :body slurp)) ;prints
(prn "form-data " (-> ctx :request :body slurp cheshire/decode)) ;doesn't print
))
At the server, I do get the print of the form-data like so:
"form-data " "------WebKitFormBoundaryI9CA2zKhFT0Stmx7\r\nContent-Disposition: form-data; name=\"new-name\"\r\n\r\n{:uploaded-file #object[File [object File]], :name \"somename\", :another \"1234\"}\r\n------WebKitFormBoundaryI9CA2zKhFT0Stmx7--\r\n"
But the next prn doesn't print anything where my goal is to extract the :uploaded-file. What am I doing wrong?
How do I make this test pass:
(ns imp-rest.parser-test-rest
(:require [clojure.test :refer :all])
(:require [ring.mock.request :as mock] )
(:require [imp-rest.web :as w]))
(deftest test-parser-rest
(testing "put settings"
(w/app
(mock/request :put "/settings/coordinateName" "FOO" ))
(let [response (w/app (mock/request :get "/settings"))]
(println response )
(is (= (get (:body response) :coordinateName) "FOO")))))
it fails with:
FAIL in (test-parser-rest) (parser_test_rest.clj:30)
put settings
expected: (= (get (:body response) :coordinateName) "FOO")
actual: (not (= nil "FOO"))
Here's my handler:
(ns imp-rest.web
(:use compojure.core)
(:use ring.middleware.json-params)
(:require [clj-json.core :as json])
(:require [ring.util.response :as response])
(:require [compojure.route :as route])
(:require [imp-rest.settings :as s]))
(defn json-response [data & [status]]
{:status (or status 200)
:headers {"Content-Type" "application/json"}
:body (json/generate-string data)})
(defroutes handler
(GET "/settings" []
(json-response (s/get-settings)))
(GET "/settings/:id" [id]
(json-response (s/get-setting id)))
(PUT "/settings" [id value]
(json-response (s/put-setting id value)))
(route/not-found "Page not found") )
(def app
(-> handler
wrap-json-params))
which exposes this map (of settings):
(ns imp-rest.settings)
(def settings
(atom
{:coordinateName nil
:burnin nil
:nslices nil
:mrsd nil
}))
(defn get-settings []
#settings)
(defn get-setting [id]
(#settings (keyword id)))
(defn put-setting [id value]
(swap! settings assoc (keyword id) value)
value)
and the entry point:
(ns imp-rest.core
(:use ring.adapter.jetty)
(:require [imp-rest.web :as web]))
(defn -main
"Entry point"
[& args]
(do
(run-jetty #'web/app {:port 8080})
);END;do
);END: main
Now when I 'lein run' I can make a (working) request like this:
curl -X PUT -H "Content-Type: application/json" \
-d '{"id" : "coordinateName", "value" : "FOO"}' \
http://localhost:8080/settings
which is what I try to mock with the test. Any help appreciated.
If you want to have :id in your PUT /settings/:id route accepting body in format {"value": "..."}, you need to change your routes definition:
(defroutes handler
(GET "/settings" []
(json-response (s/get-settings)))
(GET "/settings/:id" [id]
(json-response (s/get-setting id)))
(PUT "/settings/:id" [id value]
(json-response (s/put-setting id value)))
(route/not-found "Page not found"))
And change how you call your PUT endpoint in the test:
(w/app
(-> (mock/request
:put
"/settings/coordinateName"
(json/generate-string {:value "FOO"}))
(mock/content-type "application/json")))
What was changed?
:id in your PUT URL route definition (/settings -> /settings/:id)
Your PUT request didn't send a correct request and content type.
If you want to have a PUT /settings route expecting {"id": "...", "value": "..."} request body, then you need to change how you create a mock request:
(w/app
(-> (mock/request
:put
"/settings"
(json/generate-string {:id "coordinateName" :value "FOO"}))
(mock/content-type "application/json"))
Your curl request specifies the parameters as JSON in the body of the PUT request, but your mock request tries to use URL parameters.
There are two options to resolve this:
compojure can automatically translate parameters, but only when the relevant middleware is present -- you have wrap-json-params added to your handler, but you're missing wrap-params. The answer from Piotrek Bzdyl amounts to making these params explicit in the compojure routes.
Alternatively, you can add the ID/value pair as JSON in the body of the mock request using request.mock.body.
I am trying to use Clojure manifold library, and in order to understand it, I need wanted to convert a core.async channel into a manifold stream.
I would like to create the equivalent the following using a core.async channel :
(require '[manifold.stream :as s])
(s/periodically 100 #(str " ok "))
;; Here is what I tried, it fails with an error 500
(let [ch (chan)]
(go-loop []
(>! ch " ok ")
(<! (timeout 100))
(recur))
(s/->source ch))
I am trying to feed a core.async channel into yada. The first code sample, using manifold.stream/periodic works, not the others using core.async. I tried on yada 1.0.0 and 1.1.0-SNAPSHOT.
Using manifold.stream/periodic works :
(def get-stream
(yada (fn [ctx]
(-> (:response ctx)
(assoc :status 202)
(assoc :body (s/periodically 1000 #(str (System/currentTimeMillis) " ")))))
{:representations [{:media-type "application/json"
:charset "UTF-8"}
{:media-type "application/edn"
:charset "UTF-8"}]}))
Using manifold.stream/->source returns an error 500 :
(def get-stream
(yada (fn [ctx]
(-> (:response ctx)
(assoc :status 202)
;; Similar to this : https://github.com/juxt/yada/blob/94f3ee93de155a8513b27e0508608691ed556a55/dev/src/yada/dev/async.clj
(assoc :body (let [ch (chan)]
(go-loop []
(>! ch " ok ")
(<! (timeout 100))
(recur))
(s/->source ch)))))
{:representations [{:media-type "application/json"
:charset "UTF-8"}
{:media-type "application/edn"
:charset "UTF-8"}]}))
;; Error on the page :
;; 500: Unknown
;; Error on GET
;; #error {
;; :cause "No implementation of method: :to-body of protocol: #'yada.body/MessageBody found for class: manifold.stream.async.CoreAsyncSource"
;; :via
;; [{:type clojure.lang.ExceptionInfo
;; :message "Error on GET"
;; :data {:response #yada.response.Response{:representation {:media-type #yada.media-type[application/json;q=1.0], :charset #yada.charset.CharsetMap{:alias "UTF-8", :quality 1.0}}, :vary #{:media-type}}, :resource #function[backend.routes.examples.ts/fn--57734]}
;; :at [clojure.core$ex_info invoke "core.clj" 4593]}
;; {:type java.lang.IllegalArgumentException
;; :message "No implementation of method: :to-body of protocol: #'yada.body/MessageBody found for class: manifold.stream.async.CoreAsyncSource"
;; :at [clojure.core$_cache_protocol_fn invoke "core_deftype.clj" 554]}]
;; :trace
Third attempt, with a core.async channel (different error 500) :
(def get-stream
(yada (fn [ctx]
(-> (:response ctx)
(assoc :status 202)
(assoc :body (chan)))
{:representations [{:media-type "application/json"
:charset "UTF-8"}
{:media-type "application/edn"
:charset "UTF-8"}]}))
;; Error on the page :
;; 500: Unknown
;; Error on GET
;; #error {
;; :cause "No implementation of method: :to-body of protocol: #'yada.body/MessageBody found for class: clojure.core.async.impl.channels.ManyToManyChannel"
;; :via
;; [{:type clojure.lang.ExceptionInfo
;; :message "Error on GET"
;; :data {:response #yada.response.Response{:representation {:media-type #yada.media-type[application/json;q=1.0], :charset #yada.charset.CharsetMap{:alias "UTF-8", :quality 1.0}}, :vary #{:media-type}}, :resource #function[backend.routes.api.subscribe/subscribe$fn--64130]}
;; :at [clojure.core$ex_info invoke "core.clj" 4593]}
;; {:type java.lang.IllegalArgumentException
;; :message "No implementation of method: :to-body of protocol: #'yada.body/MessageBody found for class: clojure.core.async.impl.channels.ManyToManyChannel"
;; :at [clojure.core$_cache_protocol_fn invoke "core_deftype.clj" 554]}]
;; :trace
The key error is this:
"No implementation of method: :to-body of protocol:
#'yada.body/MessageBody
found for class: manifold.stream.async.CoreAsyncSource"
It reveals that the yada version you are using is trying to coerce a body type it doesn't understand. More recent versions of yada are more permissive about what you can send through to the web-server, and allow anything through that it doesn't know how to transform.