Clojure - use a core.async channel with Yada/Aleph - clojure

I am trying to use Clojure manifold library, and in order to understand it, I need wanted to convert a core.async channel into a manifold stream.
I would like to create the equivalent the following using a core.async channel :
(require '[manifold.stream :as s])
(s/periodically 100 #(str " ok "))
;; Here is what I tried, it fails with an error 500
(let [ch (chan)]
(go-loop []
(>! ch " ok ")
(<! (timeout 100))
(recur))
(s/->source ch))
I am trying to feed a core.async channel into yada. The first code sample, using manifold.stream/periodic works, not the others using core.async. I tried on yada 1.0.0 and 1.1.0-SNAPSHOT.
Using manifold.stream/periodic works :
(def get-stream
(yada (fn [ctx]
(-> (:response ctx)
(assoc :status 202)
(assoc :body (s/periodically 1000 #(str (System/currentTimeMillis) " ")))))
{:representations [{:media-type "application/json"
:charset "UTF-8"}
{:media-type "application/edn"
:charset "UTF-8"}]}))
Using manifold.stream/->source returns an error 500 :
(def get-stream
(yada (fn [ctx]
(-> (:response ctx)
(assoc :status 202)
;; Similar to this : https://github.com/juxt/yada/blob/94f3ee93de155a8513b27e0508608691ed556a55/dev/src/yada/dev/async.clj
(assoc :body (let [ch (chan)]
(go-loop []
(>! ch " ok ")
(<! (timeout 100))
(recur))
(s/->source ch)))))
{:representations [{:media-type "application/json"
:charset "UTF-8"}
{:media-type "application/edn"
:charset "UTF-8"}]}))
;; Error on the page :
;; 500: Unknown
;; Error on GET
;; #error {
;; :cause "No implementation of method: :to-body of protocol: #'yada.body/MessageBody found for class: manifold.stream.async.CoreAsyncSource"
;; :via
;; [{:type clojure.lang.ExceptionInfo
;; :message "Error on GET"
;; :data {:response #yada.response.Response{:representation {:media-type #yada.media-type[application/json;q=1.0], :charset #yada.charset.CharsetMap{:alias "UTF-8", :quality 1.0}}, :vary #{:media-type}}, :resource #function[backend.routes.examples.ts/fn--57734]}
;; :at [clojure.core$ex_info invoke "core.clj" 4593]}
;; {:type java.lang.IllegalArgumentException
;; :message "No implementation of method: :to-body of protocol: #'yada.body/MessageBody found for class: manifold.stream.async.CoreAsyncSource"
;; :at [clojure.core$_cache_protocol_fn invoke "core_deftype.clj" 554]}]
;; :trace
Third attempt, with a core.async channel (different error 500) :
(def get-stream
(yada (fn [ctx]
(-> (:response ctx)
(assoc :status 202)
(assoc :body (chan)))
{:representations [{:media-type "application/json"
:charset "UTF-8"}
{:media-type "application/edn"
:charset "UTF-8"}]}))
;; Error on the page :
;; 500: Unknown
;; Error on GET
;; #error {
;; :cause "No implementation of method: :to-body of protocol: #'yada.body/MessageBody found for class: clojure.core.async.impl.channels.ManyToManyChannel"
;; :via
;; [{:type clojure.lang.ExceptionInfo
;; :message "Error on GET"
;; :data {:response #yada.response.Response{:representation {:media-type #yada.media-type[application/json;q=1.0], :charset #yada.charset.CharsetMap{:alias "UTF-8", :quality 1.0}}, :vary #{:media-type}}, :resource #function[backend.routes.api.subscribe/subscribe$fn--64130]}
;; :at [clojure.core$ex_info invoke "core.clj" 4593]}
;; {:type java.lang.IllegalArgumentException
;; :message "No implementation of method: :to-body of protocol: #'yada.body/MessageBody found for class: clojure.core.async.impl.channels.ManyToManyChannel"
;; :at [clojure.core$_cache_protocol_fn invoke "core_deftype.clj" 554]}]
;; :trace

The key error is this:
"No implementation of method: :to-body of protocol:
#'yada.body/MessageBody
found for class: manifold.stream.async.CoreAsyncSource"
It reveals that the yada version you are using is trying to coerce a body type it doesn't understand. More recent versions of yada are more permissive about what you can send through to the web-server, and allow anything through that it doesn't know how to transform.

Related

Google Indexing Api in clojure

(:require [utils.base64 :as base64])
(:import [com.google.api.client.googleapis.auth.oauth2 GoogleCredential]
[com.google.api.services.indexing.v3 Indexing]
[com.google.api.services.indexing.v3.model UrlNotification]
[com.google.api.client.http HttpTransport]
[com.google.api.client.json.gson GsonFactory]
[java.io ByteArrayInputStream]
[java.util Base64]
[com.google.api.services.indexing.v3 IndexingScopes]))
;; Set the private key and client email
(def private-key "")
(def client-email "my-client-email")
(def encoded-private-key (base64/encode-string private-key))
;; Set the private key and client email
(def private-key-json '{
"type": "service_account",
"client_email": "' client-email '",
"private_key": "' encoded-private-key '"
}')
;; Create the input stream from the JSON string
(def input-stream (ByteArrayInputStream. (.getBytes private-key-json)))
;; Create the OAuth 2.0 credentials
(def credentials (doto (GoogleCredential/fromStream input-stream)
(.createScoped (java.util.Collections/singleton IndexingScopes/INDEXING))))
;; Set up the HTTP transport and JSON factory
(def http-transport (HttpTransport/newTrustedTransport))
(def json-factory (GsonFactory/getDefaultInstance))
;; Set up the Indexing API client
(def indexing-client (doto (Indexing/Builder. http-transport json-factory credentials)
(.setApplicationName "My Indexing App")
(.build)))
;; Publish a URL notification
(def url-notification (UrlNotification. "https://aaa.com" "URL_UPDATED"))
(.execute (indexing-client/urlNotifications) (publish url-notification))
I'm trying to use private-key-json but the format is not valid. What is the best way to pass data? I referred https://developers.google.com/search/apis/indexing-api/v3/prereqs#examples.
Here I'm trying not to upload json file which we get once we create a service account and make use of only mandatory fields like client_email, private_key and type fields.
Modified code:
(:require [cheshire.core :as json]
[sketches.utils.base64 :as base64]
[clojure.string :as str])
(:import [com.google.api.client.googleapis.auth.oauth2 GoogleCredential]
[com.google.api.services.indexing.v3 Indexing$Builder]
[com.google.api.services.indexing.v3.model UrlNotification]
[com.google.api.client.googleapis.javanet GoogleNetHttpTransport]
[com.google.api.client.http HttpTransport]
[com.google.api.client.json.gson GsonFactory]
[java.io ByteArrayInputStream]
[com.google.api.services.indexing.v3 IndexingScopes]))
(def creds-json {:type "service_account",
:client_email "test-indexing-api",
:client_id "117578194507835125202",
:private_key_id "964321e5fc29980944da518887116ea50dfb7803",
:private_key ""})
(defn string->stream
([s] (string->stream s "UTF-8"))
([s encoding]
(-> s
(.getBytes encoding)
(ByteArrayInputStream.))))
(defn authorize [cred]
(-> cred
(json/encode)
(string->stream)
(GoogleCredential/fromStream)
(.createScoped [(IndexingScopes/INDEXING)])))
(defn valid-private-key [private-key]
(try
(when private-key
(->> (str/trim private-key)
(re-matches #"^-----BEGIN PRIVATE KEY-----[\s|\S]*-----END PRIVATE KEY-----[\\|n]*")
(some?)))
(catch Exception e
false)))
(def encoded-private-key (base64/encode-string private-key))
(def private-creds-json-string (json/generate-string creds-json))
(def input-stream (ByteArrayInputStream. (.getBytes private-creds-json-string)))
(defn build-analytics-client [creds]
(-> (Indexing$Builder. (GoogleNetHttpTransport/newTrustedTransport) (GsonFactory/getDefaultInstance) (authorize creds))
(.build)))
(defn url-notification []
(-> (UrlNotification.)
(.setType "URL_update")
(.setUrl "http://ace.madrid.quintype.io")))
(defn realtime-data [creds]
(-> (build-analytics-client creds)
(.urlNotifications)
(.publish (url-notification))
(.execute)))
```
But I get this error `Execution error (NoSuchMethodError) at com.google.api.client.http.ConsumingInputStream/close (ConsumingInputStream.java:40).
com.google.common.io.ByteStreams.exhaust(Ljava/io/InputStream;)J`

How to save the file in the server?

I'm receiving the following request in the server that I'm extracting using (-> req :params):
{"_parts":[["video",{"_data":{"size":2971246,"blobId":"D002459C-47C5-4403-ABC6-A2DE6A46230A","offset":0,"type":"video/quicktime","name":"DCDE604A-954F-4B49-A1F9-1BCC2C2F37BC.mov","__collector":null}}],["key","VAL"]]}
It contains a file "video" with a name and blobId. However, I want to access the file's data and save it to a file. So far, I've tried the following:
(defn upload-shot-video [req]
(prn "uploading video")
(prn "video is! " (-> req :multipart-params))
(prn "video is " (-> req :params))
(prn "video before is " (slurp (-> req :body)))
(.reset (-> req :body))
(prn "req full" (-> req))
(prn "video after is " (-> req :body))
(prn "video is! " (-> req :multipart-params))
(clojure.java.io/copy (-> req :body) (clojure.java.io/file "./resources/public/video.mov"))
(let [filename (str (rand-str 100) ".mov")]
(s3/put-object
:bucket-name "humboi-videos"
:key filename
:file "./resources/public/video.mov"
:access-control-list {:grant-permission ["AllUsers" "Read"]})
(db/add-video {:name (-> req :params :name)
:uri (str "https://humboi-videos.s3-us-west-1.amazonaws.com/" filename)}))
(r/response {:res "okay!"}))
In which I'm trying to save the (-> req :body) into the file (which is an inputstream). This must be incorrect. What's the correct way to save this file that the server has received into a file, by saving the data into a file on the server? How to extract this data from the request?
If you are using Ring, you need to use wrap-multipart-params middleware.
(ns controller
(:require [ring.middleware.params :refer [wrap-params]]
[ring.middleware.multipart-params :refer [wrap-multipart-params]])
(defn upload-shot-video [req]
(let [uploaded-file (-> req :params "file" :tempfile) ;; here is a java.io.File instance of your file
(save-file uploaded-file)
{:status 201 :body "Upload complete"}))
(def app
(-> upload-shot-video
wrap-params
wrap-multipart-params))

Pedestal early termination is not working

http://pedestal.io/reference/servlet-interceptor Says this
Before invoking the :enter functions, the servlet interceptor sets up a "terminator" predicate on the context. It terminates the interceptor chain when the context map returned by an interceptor has a response map attached.
My server has this:
(ns wp-server.server
(:gen-class) ; for -main method in uberjar
(:require
[io.pedestal.http :as server]
[io.pedestal.http.route :as route]
[wp-server.service :as service]
[wp-server.datomic :as datomic]))
(defonce runnable-service (atom nil))
(defn -main
"The entry-point for 'lein run'"
[& args]
(println "/nConnecting to datomic...")
(datomic/connect!)
(println "\nCreating your server...")
(reset! runnable-service (server/create-servlet service/service))
(server/start runnable-service))
(defn create-runnable-dev-service
"The entry-point for 'lein run-dev'"
[& args]
(println "\nConnecting to [DEV] database...")
(datomic/connect-dev!)
(println "\nCreating your [DEV] server...")
(-> service/service
(merge {:env :dev
::server/join? false
::server/routes #(route/expand-routes (deref #'service/routes))
::server/allowed-origins {:creds true :allowed-origins (constantly true)}
::server/secure-headers {:content-security-policy-settings {:object-src "none"}}})
server/default-interceptors
server/dev-interceptors
server/create-servlet))
(defn start-dev []
(when-not #runnable-service
(reset! runnable-service (create-runnable-dev-service)))
(server/start #runnable-service))
(defn stop-dev []
(server/stop #runnable-service))
(defn restart-dev []
(stop-dev)
(start-dev))
My service looks like this:
(ns wp-server.service
(:require
[datomic.api :as d]
[io.pedestal.http :as http]
[io.pedestal.http.body-params :as body-params]
[io.pedestal.http.route :as route]
[ring.util.response :as ring-resp]
[wp-server.datomic :as datomic]
[wp-common.client :as wp-common-client]
[wp-common.core :as wp-common-core]
[clojure.spec.alpha :as s]
[ring.util.response :as ring-response]))
(defn about-page
[request]
(ring-resp/response (format "Clojure %s - served from %s"
(clojure-version)
(route/url-for ::about-page))))
(def home-page
{:name :home-page
:enter (fn [context]
(prn "TWO")
(assoc context :response {:status 200 :body "Hello, world!"}))})
(defn db-test-page
[{:keys [:database]}]
(ring-resp/response
(prn-str
(d/q '[:find ?text
:where
[?e :advisor/first-name ?text]]
database))))
(def common-interceptors [datomic/db-interceptor (body-params/body-params) http/html-body])
(defn create-spec-validator
[spec]
{:name :validate-spec
:enter
(fn [{{:keys [:edn-params]} :request :as context}]
(prn "ONE")
(if-let [explained (s/explain-data spec edn-params)]
(assoc context
:response {:status 400 :body explained})))})
(def routes #{
;["/" :get (conj common-interceptors `home-page)]
["/clients" :post (conj common-interceptors (create-spec-validator ::wp-common-client/schema) home-page)]
["/db-test" :get (conj common-interceptors `db-test-page)]
["/about" :get (conj common-interceptors `about-page)]})
(def service {:env :prod
::http/routes routes
::http/type :jetty
::http/port 5000
::http/container-options {:h2c? true
:h2? false
:ssl? false}})
When sending a request to localhost:5000/clients with a body that does not pass spec, the create-spec-validator interceptor adds a response to the context. I have confirmed this by logging the context in the home-page interceptor. I would expect the home-page interceptor to be skipped as per documentation. This does not happen. Instead the :enter function of the home-page interceptor is called and the response overwritten.
Why isn't the home-page interceptor being skipped when the create-spec-validator prior to it is returning context with a response?
If you track down the termination code, it invokes
(defn response?
"True if the supplied value is a valid response map."
{:added "1.1"}
[resp]
(and (map? resp)
(integer? (:status resp))
(map? (:headers resp))))
to test if there is a valid response-map in :response. Try setting an empty map in :headers in your response: it should terminate then.
Ideally, of course, you'll set Content-Type to be something meaningful.

Compojure - map query parameters with string keys

I know I can map the query string as a keyworkd map.
(defroutes my-routes
(GET "/" {params :query-params} params))
But is there a way do the same with a string keyed map?
(Using Compojure or Ring)
The point here is not to iterate over the map or use a function, but create it with string keys by default.
{ :a "b" } -> {"a" "b"}
Compojure 1.5.1 does not parse any query strings by default (by not using any middleware). However, this might have been different in earlier versions.
(require '[compojure.core :refer :all])
(require '[clojure.pprint :refer [pprint]])
(defroutes handler
(GET "/" x
(with-out-str (pprint x)))) ;; just a way to receive a pretty printed string response
$ curl localhost:3000/?a=b
{:ssl-client-cert nil,
:protocol "HTTP/1.1",
:remote-addr "127.0.0.1",
:params {}, ;; EMPTY!
:route-params {},
:headers
{"user-agent" "curl/7.47.1", "accept" "*/*", "host" "localhost:3000"},
:server-port 3000,
:content-length nil,
:compojure/route [:get "/"],
:content-type nil,
:character-encoding nil,
:uri "/",
:server-name "localhost",
:query-string "a=b", ;; UNPARSED QUERY STRING
:body
#object[org.eclipse.jetty.server.HttpInputOverHTTP 0x6756d3a3 "HttpInputOverHTTP#6756d3a3"],
:scheme :http,
:request-method :get}
Ring offers the ring.params.wrap-params middleware, which parses the query string and creates a hashmap of it under the params-key:
(defroutes handler
(wrap-params (GET "/" x
(prn-str (:params x)))))
$ curl localhost:3000/?a=55
{"a" "55"}
Additionaly ring.params.wrap-params can be used:
(defroutes handler
(wrap-params (wrap-keyword-params (GET "/" x
(prn-str (:params x))))))
$ curl localhost:3000/?a=55
{:a "55"}
Not sure about compojure, but you can undo it yourself:
(use 'clojure.walk)
(stringify-keys {:a 1 :b {:c {:d 2}}})
;=> {"a" 1, "b" {"c" {"d" 2}}}
https://clojuredocs.org/clojure.walk/stringify-keys

How do I mock a PUT request using ring.mock.request

How do I make this test pass:
(ns imp-rest.parser-test-rest
(:require [clojure.test :refer :all])
(:require [ring.mock.request :as mock] )
(:require [imp-rest.web :as w]))
(deftest test-parser-rest
(testing "put settings"
(w/app
(mock/request :put "/settings/coordinateName" "FOO" ))
(let [response (w/app (mock/request :get "/settings"))]
(println response )
(is (= (get (:body response) :coordinateName) "FOO")))))
it fails with:
FAIL in (test-parser-rest) (parser_test_rest.clj:30)
put settings
expected: (= (get (:body response) :coordinateName) "FOO")
actual: (not (= nil "FOO"))
Here's my handler:
(ns imp-rest.web
(:use compojure.core)
(:use ring.middleware.json-params)
(:require [clj-json.core :as json])
(:require [ring.util.response :as response])
(:require [compojure.route :as route])
(:require [imp-rest.settings :as s]))
(defn json-response [data & [status]]
{:status (or status 200)
:headers {"Content-Type" "application/json"}
:body (json/generate-string data)})
(defroutes handler
(GET "/settings" []
(json-response (s/get-settings)))
(GET "/settings/:id" [id]
(json-response (s/get-setting id)))
(PUT "/settings" [id value]
(json-response (s/put-setting id value)))
(route/not-found "Page not found") )
(def app
(-> handler
wrap-json-params))
which exposes this map (of settings):
(ns imp-rest.settings)
(def settings
(atom
{:coordinateName nil
:burnin nil
:nslices nil
:mrsd nil
}))
(defn get-settings []
#settings)
(defn get-setting [id]
(#settings (keyword id)))
(defn put-setting [id value]
(swap! settings assoc (keyword id) value)
value)
and the entry point:
(ns imp-rest.core
(:use ring.adapter.jetty)
(:require [imp-rest.web :as web]))
(defn -main
"Entry point"
[& args]
(do
(run-jetty #'web/app {:port 8080})
);END;do
);END: main
Now when I 'lein run' I can make a (working) request like this:
curl -X PUT -H "Content-Type: application/json" \
-d '{"id" : "coordinateName", "value" : "FOO"}' \
http://localhost:8080/settings
which is what I try to mock with the test. Any help appreciated.
If you want to have :id in your PUT /settings/:id route accepting body in format {"value": "..."}, you need to change your routes definition:
(defroutes handler
(GET "/settings" []
(json-response (s/get-settings)))
(GET "/settings/:id" [id]
(json-response (s/get-setting id)))
(PUT "/settings/:id" [id value]
(json-response (s/put-setting id value)))
(route/not-found "Page not found"))
And change how you call your PUT endpoint in the test:
(w/app
(-> (mock/request
:put
"/settings/coordinateName"
(json/generate-string {:value "FOO"}))
(mock/content-type "application/json")))
What was changed?
:id in your PUT URL route definition (/settings -> /settings/:id)
Your PUT request didn't send a correct request and content type.
If you want to have a PUT /settings route expecting {"id": "...", "value": "..."} request body, then you need to change how you create a mock request:
(w/app
(-> (mock/request
:put
"/settings"
(json/generate-string {:id "coordinateName" :value "FOO"}))
(mock/content-type "application/json"))
Your curl request specifies the parameters as JSON in the body of the PUT request, but your mock request tries to use URL parameters.
There are two options to resolve this:
compojure can automatically translate parameters, but only when the relevant middleware is present -- you have wrap-json-params added to your handler, but you're missing wrap-params. The answer from Piotrek Bzdyl amounts to making these params explicit in the compojure routes.
Alternatively, you can add the ID/value pair as JSON in the body of the mock request using request.mock.body.