I want to change certain key's in a large map in clojure.
These key's can be present at any level in the map but will always be within a required-key
I was looking at using camel-snake-kebab library but need it to change only a given set of keys in the required-key map. It doesn't matter if the change is made in json or the map
(def my-map {:allow_kebab_or-snake {:required-key {:must_be_kebab ""}}
:allow_kebab_or-snake2 {:optional-key {:required-key {:must_be_kebab ""}}}})
currently using /walk/postwalk-replace but fear it may change keys not nested within the :required-key map
(walk/postwalk-replace {:must_be_kebab :must-be-kebab} my-map))
ummmm.. could you clarify: do you want to change the keys of the map?! or their associated values?
off-topic: your map above is not correct (having two identical keys :allow_kebab_or_snake - i-m assuming you're just underlining the point and not showing the actual example :))
postwalk-replace WILL replace any occurrence of the key with the value.
so if you know the exact map struct you could first select your sub-struct with get-in and then use postwalk-replace :
(walk/postwalk-replace {:must_be_kebab :mus-be-kebab}
(get-in my-map [:allow_kebab_or_snake :required-key]))
But then you'll have to assoc this into your initial map.
You should also consider the walk function and construct your own particular algorithm if the interleaved DS is too complex.
Here is a solution. Since you need to control when the conversion does/doesn't occur, you can't just use postwalk. You need to implement your own recursion and change the context from non-convert -> convert when your condition is found.
(ns tst.clj.core
(:use clj.core clojure.test tupelo.test)
(:require
[clojure.string :as str]
[clojure.pprint :refer [pprint]]
[tupelo.core :as t]
[tupelo.string :as ts]
))
(t/refer-tupelo)
(t/print-versions)
(def my-map
{:allow_kebab_or-snake {:required-key {:must_be_kebab ""}}
:allow_kebab_or-snake2 {:optional-key {:required-key {:must_be_kebab ""}}}})
(defn children->kabob? [kw]
(= kw :required-key))
(defn proc-child-maps
[ctx map-arg]
(apply t/glue
(for [curr-key (keys map-arg)]
(let [curr-val (grab curr-key map-arg)
new-ctx (if (children->kabob? curr-key)
(assoc ctx :snake->kabob true)
ctx)
out-key (if (grab :snake->kabob ctx)
(ts/kw-snake->kabob curr-key)
curr-key)
out-val (if (map? curr-val)
(proc-child-maps new-ctx curr-val)
curr-val)]
{out-key out-val}))))
(defn nested-keys->snake
[arg]
(let [ctx {:snake->kabob false}]
(if (map? arg)
(proc-child-maps ctx arg)
arg)))
The final result is shown in the unit test:
(is= (nested-keys->snake my-map)
{:allow_kebab_or-snake
{:required-key
{:must-be-kebab ""}},
:allow_kebab_or-snake2
{:optional-key
{:required-key
{:must-be-kebab ""}}}} ))
For this solution I used some of the convenience functions in the Tupelo library.
Just a left of field suggestion which may or may not work. This is a problem that can come up when dealing with SQL databases because the '-' is seen as a reserved word and cannot be used in identifiers. However, it is common to use '-' in keywords when using clojure. Many abstraction layers used when working with SQL in clojure take maps as arguments/bindings for prepared statements etc.
Ideally, what is needed is another layer of abstraction which converts between kebab and snake case as needed depending on the direction you are going i.e. to sql or from sql. The advantage of this aproach is your not walking through maps making conversions - you do the conversion 'on the fly" when it is needed.
Have a look at https://pupeno.com/2015/10/23/automatically-converting-case-between-sql-and-clojure/
Related
Is there a library to calculate some sort of a SHA for persistent data structures?
(sha (pr-str <datastructure>)) does not work because sometimes the order of keys are not the same when printed.
While it isn't a cryptographic function, clojure.core/hash-unordered-coll will give you a consistent hash value as long as the collections have the same contents, and maybe you can leverage on that:
user=> (hash-unordered-coll (sorted-map :b 2 :a 1))
161871944
user=> (hash-unordered-coll {:b 2, :a 1})
161871944
user=> (hash-unordered-coll [[:b 2] [:a 1]])
161871944
See https://clojuredocs.org/clojure.core/hash-unordered-coll
It really depends on what you want it for. For the simplest use cases, clojure.core/hash is fine. But since "data structure" is a much more complicated input format than "sequence of bytes", there's no obvious universal concept of a fingerprint - you have to decide what features it needs.
I found via Google search the following question and discussion of whether there is a cryptographically strong way to combine crypto-strong hash values of elements of an unordered set, into a crypto-strong hash for the entire set, ignoring order. One answer claims that sorting the hash values of the elements into one string of bits, then calculating a crypto-strong hash on that string, should be strong. XORing or adding the hashes of the elements together is not. I did not read all responses, so there may be better approaches known: https://crypto.stackexchange.com/questions/54544/how-to-to-calculate-the-hash-of-an-unordered-set
The function tupelo.lexical/compare-generic implements a comparitor that is safe to use across different types. You could combine this with sorted-map-by and sorted-set-by in order to convert all maps/sets into stable versions that always print in the same order. Then the technique of (sha (pr-str XXX)) would work.
The above logic is already available in the function tupelo.core/unlazy. The function tupelo.misc/str->sha also does what it says on the tin. So now, the final solution becomes:
(ns demo.core
(:require
[tupelo.core :as t]
[tupelo.misc :as tm] ))
(tm/str->sha (pr-str (t/unlazy XXX)))
where XXX is any Clojure collection. Demo code:
(ns tst.demo.core
(:use tupelo.core tupelo.test)
(:require
[tupelo.core :as t]
[tupelo.misc :as tm]))
(dotest
(let [stuff {:hello "there"
1 [2 3 4]
"gooodbye" #{"cruel" :world}
'forever ['and "ever" :and #{"ever" 'more}]}
stuff-str (pr-str (t/unlazy stuff))
stuff-sha (tm/str->sha (pr-str (t/unlazy stuff)))]
(is= stuff-str
"{:hello \"there\", forever [and \"ever\" :and #{more \"ever\"}], 1 [2 3 4], \"gooodbye\" #{:world \"cruel\"}}")
(is= stuff-sha "af3ade069e7a33139f5ee1fd1d35fd82807e3b1c")))
I'm trying to figure out how to do custom coercion with compojure-api and spec. By reading the docs and code I have been able to do coercion on the input (the body) but am unable to do coercion on the response body.
Specifically, I have a custom type, a timestamp, that is represented as a long within my app but for the web API I want to consume and return ISO timestamps (no, I don't want to use Joda internally).
The following is what I have that works for input coercion but I have been unable to properly coerce the response.
(ns foo
(:require [clj-time.core :as t]
[clj-time.coerce :as t.c]
[spec-tools.conform :as conform]
[spec-tools.core :as st]))
(def timestamp (st/create-spec
{:spec pos-int?
:form `pos-int?
:json-schema/default "2017-10-12T05:04:57.585Z"
:type :timestamp}))
(defn json-timestamp->long [_ val]
(t.c/to-long val))
(def custom-json-conforming
(st/type-conforming
(merge
conform/json-type-conforming
{:timestamp json-timestamp->long}
conform/strip-extra-keys-type-conforming)))
(def custom-coercion
(-> compojure.api.coercion.spec/default-options
(assoc-in [:body :formats "application/json"] custom-json-
conforming)
compojure.api.coercion.spec/create-coercion))
;; how do I use this for the response coercion?
(defn timestamp->json-string [_ val]
(t.c/to-string val))
;; I've tried the following but it doesn't work:
#_(def custom-coercion
(-> compojure.api.coercion.spec/default-options
(assoc-in [:body :formats "application/json"] custom-json-
conforming)
(assoc-in [:response :formats "application/json"]
(st/type-conforming
{:timestamp timestamp->json-string}))
compojure.api.coercion.spec/create-coercion))
Problem is that Spec Conforming is a one-way transformation pipeline:
s/conform (and because of that st/conform) does both transform and validate for the result. Your response coercion first converts the integer into date string and the validates it against the original spec predicate, which is pos-int? and it fails on that.
To support two-way transformations, you need to define the end result as either of the possible formats: e.g. change your predicate to something like #(or (pos-int? %) (string? %)) and it should work.
Or you can have two different Spec Records, one for input (timestamp-long with pos-int? predicate) and another for outputs (timestamp-string with string? predicate). But one needs to remember to use correct ones for request & responses.
CLJ-2251 could possible help if there was and extra :transform mode (not written in the issue yet), which would do conforming without validating the end results.
Normally, return transformations are done by the format encoder, but they usually dispatch on value types. For example Cheshire just sees an Long and has no clue that it should be written as date string.
Maybe people on the #clojure-spec slack could help. Would also like to know how to build this kind of two-way transformation with spec.
Although the below example seems a bit strange, it's because I'm trying to reduce a fairly large problem I've got at present to a minimal example. I'm struggling to work out how to call into multimethods when they're sitting behind a couple of abstraction layers and the defmulti and corresponding defmethods are defined in multiple namespaces. I really feel like I'm missing something obvious here...
Suppose I've got the following scenario:
I purchase stuff from a variety of suppliers, via their own proprietary interfaces
I want to implement a common interface to talk to each of those suppliers
I want to be able to purchase different items from different suppliers
Using Clojure, the recommended ways of implementing a common interface would be via protocols or multimethods. In this case, as I'm switching based on the value of the supplier, I think the best way to handle the situation I'm describing below is via multimethods (but I could be wrong).
My multimethod definitions would look something like this, which defines a common interface I want to use to talk to every supplier's APIs:
(ns myapp.suppliers.interface)
(defmulti purchase-item :supplier)
(defmulti get-item-price :supplier)
For each supplier, I probably want something like:
(ns myapp.suppliers.supplier1
(:require [myapp.suppliers.interface :as supplier-api]))
(defmethod purchase-item :supplier1 [item quantity] ...)
(defmethod get-item-price :supplier1 [item] ...)
and
(ns myapp.suppliers.supplier2
(:require [myapp.suppliers.interface :as supplier-api]))
(defmethod purchase-item :supplier2 [item quantity] ...)
(defmethod get-item-price :supplier2 [item] ...)
So far, no problem
Now to my code which calls these abstracted methods, which I assume looks something like:
(ns myapp.suppliers.api
(:require [myapp.suppliers.supplier1 :as supplier1]
[myapp.suppliers.supplier2 :as supplier2])
(defn buy-something
[supplier item quantity]
(purchase-item [supplier item quantity])
(defn price-something
[supplier item]
(get-item-price [supplier item])
This is starting to look a bit ... ugly. Every time I implement a new supplier's API, I'll need to change myapp.suppliers.api to :require that new supplier's methods and recompile.
Now I'm working at the next level up, and I want to buy a widget from supplier2.
(ns myapp.core
(:require [myapp.suppliers.api :as supplier])
(def buy-widget-from-supplier2
(buy-something :supplier2 widget 1)
This can't work, because :supplier2 hasn't been defined anywhere in this namespace.
Is there a more elegant way to write this code? In particular, in myapp.core, how can I buy-something from :supplier2?
Initial notes
It's hard to tell if you mixed up some things in the process of simplifying the example, or if they weren't quite right out of the gate. For an example of what I'm referring to, consider purchase-item, though the issues are similar for get-item-price:
The defmulti call is a single-argument function
The defmethod calls each take two arguments
The call in buy-something passes a vector to purchase-item, but looking up the :supplier keyword in a vector will always return nil
Your concerns
Every time I implement a new supplier's API, I'll need to change myapp.suppliers.api to :require that new supplier's methods and recompile.
If you require the myapp.suppliers.interface namespace myapp.suppliers.api, the problem can be avoided
This can't work, because :supplier2 hasn't been defined anywhere in this namespace.
Simply put, this will work. :)
Is there a more elegant way to write this code? In particular, in myapp.core, how can I buy-something from :supplier2?
Certainly, but this solution is going to make some assumption based on the ambiguities in the Initial notes.
Without straying too far from your original design, here's a fully-working example of how I interpret what you were trying to achieve:
myapp.suppliers.interface
(ns myapp.suppliers.interface)
(defmulti purchase-item (fn [supplier item quantity] supplier))
myapp.suppliers.supplier1
(ns myapp.suppliers.supplier1
(:require [myapp.suppliers.interface :as supplier-api]))
(defmethod supplier-api/purchase-item :supplier1 [supplier item quantity]
(format "Purchasing %dx %s from %s" quantity (str item) (str supplier)))
myapp.suppliers.supplier2
(ns myapp.suppliers.supplier2
(:require [myapp.suppliers.interface :as supplier-api]))
(defmethod supplier-api/purchase-item :supplier2 [supplier item quantity]
(format "Purchasing %dx %s from %s" quantity (str item) (str supplier)))
myapp.suppliers.api
(ns myapp.suppliers.api
(:require [myapp.suppliers.interface :as interface]))
(defn buy-something [supplier item quantity]
(interface/purchase-item supplier item quantity))
myapp.core
(ns myapp.core
(:require [myapp.suppliers.api :as supplier]))
(def widget {:id 1234 :name "Monchkin"})
(supplier/buy-something :supplier1 widget 15)
;;=> "Purchasing 15x {:id 1234, :name \"Monchkin\"} from :supplier1"
(supplier/buy-something :supplier2 widget 3)
;;=> "Purchasing 3x {:id 1234, :name \"Monchkin\"} from :supplier2"
As you can see, the supplier/buy-something calls propagate to the appropriate multimethod implementations. Hopefully this helps you get where you were trying to go.
According to Om Next's documentation:
query->ast
(om.next/query->ast '[(:foo {:bar 1})])
Given a query expression return the AST.
ast->query
(om.next/ast->query ast)
Given a query expression AST, unparse it into a query expression.
Question: Why would one need these functions? That is, why would one need to directly manipulate a query abstract syntax tree (which I'm assuming are clojure maps that represent a query tree, along with some meta data) in om next?
There are some scenarios where you need to manipulate the query ast directly. In remote parsing mode, the parser expects your read functions to return either {:remote-name true } or a (possibly modified) {:remote-name AST-node} (which comes in as :ast in env). Most often you'll have to modify the AST to restructure it or add some data.
Example 1:
You have a query: [{:widget {:list [:name :created]}}]
The :widget part is pure UI related, your server doesn't need to know it exists, it only cares/knows about the :list.
Basically you'll have to modify the AST in the parser:
(defmethod read :list
[{:keys [ast query state]} key _ ]
(let [st #state]
{:value (om/db->tree query (get st key) st)
:remote (assoc ast :query-root true)}))
If you use om/process-rootsin your send function, it'll pick up the :query-root out of the ast and rewrite the query from [{:widget {:list [:name :created]}}] to [{:list [:name :created]}].
Example 2:
Another example would be when you want to mutate something at a remote:
(defmethod mutate 'item/update
[{:keys [state ast]} key {:keys [id title]}]
{:remote (assoc ast :params {:data {:id id :title title })})
Here you need to explicitly tell Om to include the data you want to send in the AST. At your remote you then pick apart :data to update the title at the given id
Most of the time you won't use the functions you described in your questions directly. The env available in every method of the parser has the ast in it.
Something I stumbled on, while trying to use Compassus:
Let's say you have a complex union/join query that includes parametric sub-queries. Something like this:
`[({:foo/info
{:foo/header [:foo-id :name]
:foo/details [:id :description :title]}} {:foo-id ~'?foo-id
:foo-desc ~'?foo-desc})]
Now let's say you want to set parameters so on the server you can parse it with om/parser and see those params as 3rd argument of read dispatch. Of course it's possible to write a function that would find all necessary parameters in the query and set the values. That's not easy though, and as I said - imagine your queries can be quite complex.
So what you can do - is to modify ast, ast includes :children :params key. So let's say the actual values for :foo-id and :foo-desc are in the state atom under :route-params key:
(defn set-ast-params [children params]
"traverses given vector of `children' in an AST and sets `params`"
(mapv
(fn [c]
(let [ks (clojure.set/intersection (-> params keys set)
(-> c :params keys set))]
(update-in c [:params] #(merge % (select-keys params (vec ks))))))
children))
(defmethod readf :foo/info
[{:keys [state query ast] :as env} k params]
(let [{:keys [route-params] :as st} #state
ast' (-> ast
(update :children #(set-ast-params % route-params))
om/ast->query
om.next.impl.parser/expr->ast)]
{:value (get st k)
:remote ast'}))
So basically you are:
- grabbing ast
- modifying it with actual values
you think maybe you can send it to server right then. Alas, no! Not yet. Thing is - when you do {:remote ast}, Om takes :query part of the ast, composes ast out of it and then sends it to the server. So you actually need to: turn your modified ast into query and then convert it back to ast again.
Notes:
set-ast-params function in this example would only work for the first level (if you have nested parametrized queries - it won't work),
make it recursive - it's not difficult
there are two different ways to turn ast to query and vice-versa:
(om/ast->query) ;; retrieves query from ast and sets the params based
;; of `:params` key of the ast, BUT. it modifies the query,
;; if you have a join query it takes only the first item in it. e.g. :
[({:foo/foo [:id]
:bar/bar [:id]} {:id ~'?id})]
;; will lose its `:bar` part
(om.next.impl.parser/ast->expr) ;; retrieves query from an ast,
;; but doesn't set query params based on `:params` keys of the ast.
;; there are also
(om/query->ast) ;; and
(om.next.impl.parser/expr->ast)
I have a set of functions that all have the same first parameter.
(defn get-file [dir filename] ...)
(defn write-file [dir filename] ...)
I'd like to partially apply all of them at once, basically. Seems like I could wrap them all in a function like this:
(defn get-fns [dir]
{:get-file (fn [filename] ...)
:write-file (fn [filename] ...)})
But that seems like accessing the functions would be kind of annoying.
(let [fns (get-fns dir-name)]
((fns :get-file) filename)))
I suppose I could use a mutable var as well, but that doesn't seem very, well, functional. Is there a canonical/idiomatic way to do this?
Let's think it through: you've got a list of things, and you want to apply something to each of them, so map. What do you want to apply to them? You want to partially apply the first argument, so partial. Then you want to have a unique reference to each of those values, so use deconstruction on the list.
Adding all that up, assuming dir, get-file, and write-file are all defined, you'd do
(let [[get-file-here write-file-here] (map #(partial % dir) [get-file write-file])]
...)
Here's a full example
(let [[add-to-3 sub-from-3] (map #(partial % 3) [+ -])]
(prn (add-to-3 2)) ; 5
(prn (sub-from-3 5))) ; -2
So we want a series of partial applications that we can use in a local binding context.
(defn get-fns
[dir]
{:get-f (partial get-file dir)
:write-f (partial write-file dir)})
Then, in a local binding, we can use them
(let [{:keys [get-f write-f]} (get-fns dir)]
(get-f file-name))
(you are of course free to use your original keywords, by changing the keywords I make it unambiguous that get-f is coming from get-fns and is not the globally bound var).