I have a spec definition that validates contents of incoming data. Since the data is a map of fields, I use spec/keys for validating it. For example:
(def person-data {:name "Jon Doe", :age 30})
(s/def ::name string?)
(s/def ::age pos-int?)
(s/def ::person-info (s/keys :req-un [::name ::age])
...
;validate data via spec and make sure no additional keys are included
(s/valid? ::person-spec some-input)
But an additional need I have is to make sure incoming data only contains the keys I want. (in this case :name and :age keys only. For that, I do something like:
(def permitted-keys [:age :name])
(select-keys some-input permitted-keys)
, ensuring only those keys get filtered in.
Is there a way I can reuse some code between my spec definition for the map structure (s/keys) and this additional step I take for filtering the allowed keys (permitted-keys)?
Perhaps by either extracting the list of keys from the s/keys definition, or by passing an existing vector of keys to s/keys?
I'd recommend looking at this macro as it covers all the bases and provides a generator, but here's another take that assumes your maps only used unnamespaced keys:
(defmacro keys-strict
[& args]
(let [{:keys [req opt req-un opt-un]} args
ks (into #{} (->> (concat req opt req-un opt-un)
(map #(keyword (name %)))))] ;; strip namespaces from keywords
`(s/and (s/keys ~#args) (s/map-of ~ks any?))))
The only trick here for reusing the same source of truth for the keys is that your key specs will be namespaced but your map keys will not. You could do the same without a macro, you just s/and your s/keys spec with a s/map-of spec or some other spec that restricts the keys allowed.
Is there a way I can reuse some code between my spec definition for the map structure (s/keys) and this additional step I take for filtering the allowed keys (permitted-keys)?
Yes, this is handled in the example above by splicing args onto the s/keys call, and done similarly in this more complete macro here.
Note: There may be situations where you really need to restrict what keys you'll accept, but I think it's generally advised to define map specs that are open for later extension.
Related
I need to validate the shape of clojure maps that have been converted from json strings. The json strings are messages of a protocol I'm implementing.
For this I'm trying out clojure.spec.alpha.
I'm using s/keys. Multiple messages in my
protocol have the same key names, but differently shaped values attached to those keys, so they cannot be validated by the same spec.
An example:
;; Here status should have the same key name, but their shape
;; is different. But definining another spec forces me to register it with a
;; different keyword, which breaks the "should have name 'status'" requirement.
(s/def ::a-message
(s/keys :req [::status]))
(s/def ::another-message
(s/keys :req [::status]))
I think I could define the :status spec in different namespaces, but it seems overkill to me.
After all it's just different messages in the same protocol and i just have a couple of clashes.
Is there a way for (s/keys) to separate the name of the key whose presence is being checked
from the name of the spec that is validating it?
In spec, qualified keywords are used to create global semantics (via the spec) whose name is the qualified keyword. If you use the same qualified keyword with different semantics, I'd say you should change your code to use different qualifiers :ex1/status and :ex2/status for different semantics.
If you are using unqualified keywords (not uncommon when coming from JSON), you can use s/keys and :req-un to map different specs to the same unqualified keyword in different parts of your data.
(s/def :ex1/status string?)
(s/def :ex2/status int?)
(s/def ::a-message (s/keys :req-un [:ex1/status]))
(s/def ::another-message (s/keys :req-un [:ex2/status]))
(s/valid? ::a-message {:status "abc"}) ;; true
(s/valid? ::another-message {:status 100}) ;; true
I want to create a clojure spec for a map that has rules about the presence of particular keys.
The map must have a :type and can have either :default or :value but not both. I tried:
(s/def ::propertyDef
(s/keys :req [::type (s/or ::default ::value) ] :opt [::description ::required]))
but I got
CompilerException java.lang.AssertionError: Assert failed:
spec/or expects k1 p1 k2 p2..., where ks are keywords
(c/and (even? (count key-pred-forms)) (every? keyword? keys)),
compiling:(C:\Users\MartinRoberts\AppData\Local\Temp\form-init4830956164341520551.clj:1:22)
but the or gave me an error as it is in the wrong format. I have to admit to not really understanding in the documentation for s/or.
First: you are using s/or to specify either a ::default or a ::value in your list of required keys. s/or requires :label spec pairs, and you are giving only the specs themselves, which is the cause of the error.
To solve, simply use or instead:
(s/def ::propertyDef (s/keys :req [::type (or ::default ::value)]
:opt [::description ::required]))
This allows both ::default and ::value to be present in the map, but this is almost always okay. The code which actually uses the map can simply check for the presence of ::value and use that, and if it's not there, then use ::default (or whatever your logic happens to be). This is usually done as such:
(let [myvalue (or (::value mymap) (::default mymap))] ...)
There could be thousands of keys in the map, and it would not affect your ability to extract the keys you need. This is why spec does not provide a built-in way to specify keys that should not be in the map, only ways to specify which keys should be present (namely, :req and :req-un in s/keys). Think of how most http servers work: you can give them nonsensical header keys and values, but they don't refuse to service the request; they just ignore them and return a response.
So, you likely don't need to enforce that only one or the other be present, but if you must, you can define an exclusive or function:
(defn xor
[p q]
(and (or p q)
(not (and p q))))
and then add this as an additional predicate on the spec:
(s/def ::propertyDef (s/and (s/keys :req [::type (or ::default ::value)]
:opt [::description ::required])
#(xor (::default %) (::value %))))
(s/valid? ::propertyDef {::type "type" ::default "default"})
=> true
(s/valid? ::propertyDef {::type "type" ::value "value"})
=> true
(s/valid? ::propertyDef {::type "type" ::default "default" ::value "value"})
=> false
I have a DSL specification which is a sequence as usual (cat). I want to take advantage of spec's parsing (i.e. conforming) to get the AST of an expression that conforms with my DSL. E.g.
user> (s/def ::person (s/cat :person-sym '#{person} :name string? :age number?))
=> :user/person
user> (s/conform ::person '(person "Henry The Sloth" 55))
=> {:person-sym person, :name "Henry The Sloth", :age 55}
Now that it's parsed and I have my AST, I would want to do interesting things with it, so I would want to test it and whatnot. So now I need to write a spec for that AST, and that's basically duplicating everything. Actually it's worse than that because now I have to s/def specs for predicates that I didn't have to before, because as the docs for keys says: "there is no support for inline value specification, by design." / "It is the (enforced) opinion of spec that the specification of values associated with a namespaced keyword, like :my.ns/k, should be registered under that keyword itself..". So duplicating (with omitting the person-sym part):
user> (s/def ::name string?)
=> :user/name
user> (s/def ::age number?)
=> :user/age
user> (s/def ::person-ast (s/keys :req-un [::name ::age]))
:user/person-ast
And now it seems to be compatible:
user> (s/conform ::person-ast (s/conform ::person '(person "Henry The Sloth" 55)))
=> {:person-sym person, :name "Henry The Sloth", :age 55}
In practice, I have more complicated data of course, and I wonder what should I do? AFAIK spec doesn't give me the spec for the AST that it creates (actually personally I would figure that this is something it should do). Any suggestions?
I'd say right now you have two options - one is to do what you're doing and create two sets of specs for the before/after.
The other option is to create a model of your domain in data and generate both specs (I've seen many people are doing something like this).
I have not heard Rich talk about generating the output spec of conformed results so I don't think that is likely in the current roadmap.
I am following the clojure.spec guide. I understand it is possible to declare required and optional attributes when using clojure.spec/keys.
I don't understand what is meant by optional. To me :opt doesn't do anything.
(s/valid? (s/keys :req [:my/a]) {:my/a 1 :my/b 2}) ;=> true
(s/valid? (s/keys :req [:my/a] :opt []) {:my/a 1 :my/b 2}) ;=> true
The guide promises to explain this to me, "We’ll see later where optional attributes can be useful", but I fail to find the explanation. Can I declare forbidden keys? Or somehow declare the set of valid keys to equal the keys in :req and :opt?
This is a very good question, and the clojure.spec API gives the (granted, short and unsatisfying) answer:
The :opt keys serve as documentation and
may be used by the generator.
I do not think you can invalidate a map if it contains an extra (this is what you mean by "forbidden" I think) key using this method. However, you could use this spec to make sure ::bad-key is not present:
(s/def ::m (s/and (s/keys :req [::a]) #(not (contains? % ::bad-key))))
(s/valid? ::m {::a "required!"}) ; => true
(s/valid? ::m {::a "required!" ::b "optional!"}) ; => true
(s/valid? ::m {::a "required!" ::bad-key "no good!"}) ; => false
You could limit the number of keys to exactly the set you want by using this spec:
(s/def ::r (s/and (s/keys :req [::reqd1 ::reqd2]) #(= (count %) 2)))
(s/valid? ::r {::reqd1 "abc" ::reqd2 "xyz"}) ; => true
(s/valid? ::r {::reqd1 "abc" ::reqd2 "xyz" ::extra 123}) ; => false
Still, the best way to handle this IMO, would be to simply ignore that there is a key present that you don't care about.
Hopefully as spec matures, these nice things will be added. Or, maybe they are already there (it is changing rapidly) and I simply don't know about it. This is a very new concept in clojure, so most of us have a lot to learn about it.
UPDATE - December 2016
I just wanted to revisit this 6 months since writing it. It looks like my initial comment about ignoring keys you don't care about is the preferred way to go. In fact, at the clojure/conj conference I attended two weeks ago, Rich's keynote specifically addressed the notion of versioning in all levels of software, from the function level up to the application level. He even specifically mentions this notion of disallowing keys in the talk, which can be found on youtube. He says that it was intentionally designed so that only required keys can be spec'd. Disallowing keys really serves no good purpose, and it should be done with caution.
Regarding the :opt keys, I think the original answer still stands up pretty well--it's documentation, and practically, it allows these optionally specified keys to be generated:
(s/def ::name #{"Bob" "Josh" "Mary" "Susan"})
(s/def ::height-inches (s/int-in 48 90))
(s/def ::person (s/keys :req-un [::name] :opt-un [::height-inches]))
(map first (s/exercise ::person))
; some generated data have :height-inches, some do not
({:name "Susan"}
{:name "Mary", :height-inches 48}
{:name "Bob", :height-inches 49}
{:name "Josh"}
The point about optional keys is that the value will be validated if they appear in the map
According to Om Next's documentation:
query->ast
(om.next/query->ast '[(:foo {:bar 1})])
Given a query expression return the AST.
ast->query
(om.next/ast->query ast)
Given a query expression AST, unparse it into a query expression.
Question: Why would one need these functions? That is, why would one need to directly manipulate a query abstract syntax tree (which I'm assuming are clojure maps that represent a query tree, along with some meta data) in om next?
There are some scenarios where you need to manipulate the query ast directly. In remote parsing mode, the parser expects your read functions to return either {:remote-name true } or a (possibly modified) {:remote-name AST-node} (which comes in as :ast in env). Most often you'll have to modify the AST to restructure it or add some data.
Example 1:
You have a query: [{:widget {:list [:name :created]}}]
The :widget part is pure UI related, your server doesn't need to know it exists, it only cares/knows about the :list.
Basically you'll have to modify the AST in the parser:
(defmethod read :list
[{:keys [ast query state]} key _ ]
(let [st #state]
{:value (om/db->tree query (get st key) st)
:remote (assoc ast :query-root true)}))
If you use om/process-rootsin your send function, it'll pick up the :query-root out of the ast and rewrite the query from [{:widget {:list [:name :created]}}] to [{:list [:name :created]}].
Example 2:
Another example would be when you want to mutate something at a remote:
(defmethod mutate 'item/update
[{:keys [state ast]} key {:keys [id title]}]
{:remote (assoc ast :params {:data {:id id :title title })})
Here you need to explicitly tell Om to include the data you want to send in the AST. At your remote you then pick apart :data to update the title at the given id
Most of the time you won't use the functions you described in your questions directly. The env available in every method of the parser has the ast in it.
Something I stumbled on, while trying to use Compassus:
Let's say you have a complex union/join query that includes parametric sub-queries. Something like this:
`[({:foo/info
{:foo/header [:foo-id :name]
:foo/details [:id :description :title]}} {:foo-id ~'?foo-id
:foo-desc ~'?foo-desc})]
Now let's say you want to set parameters so on the server you can parse it with om/parser and see those params as 3rd argument of read dispatch. Of course it's possible to write a function that would find all necessary parameters in the query and set the values. That's not easy though, and as I said - imagine your queries can be quite complex.
So what you can do - is to modify ast, ast includes :children :params key. So let's say the actual values for :foo-id and :foo-desc are in the state atom under :route-params key:
(defn set-ast-params [children params]
"traverses given vector of `children' in an AST and sets `params`"
(mapv
(fn [c]
(let [ks (clojure.set/intersection (-> params keys set)
(-> c :params keys set))]
(update-in c [:params] #(merge % (select-keys params (vec ks))))))
children))
(defmethod readf :foo/info
[{:keys [state query ast] :as env} k params]
(let [{:keys [route-params] :as st} #state
ast' (-> ast
(update :children #(set-ast-params % route-params))
om/ast->query
om.next.impl.parser/expr->ast)]
{:value (get st k)
:remote ast'}))
So basically you are:
- grabbing ast
- modifying it with actual values
you think maybe you can send it to server right then. Alas, no! Not yet. Thing is - when you do {:remote ast}, Om takes :query part of the ast, composes ast out of it and then sends it to the server. So you actually need to: turn your modified ast into query and then convert it back to ast again.
Notes:
set-ast-params function in this example would only work for the first level (if you have nested parametrized queries - it won't work),
make it recursive - it's not difficult
there are two different ways to turn ast to query and vice-versa:
(om/ast->query) ;; retrieves query from ast and sets the params based
;; of `:params` key of the ast, BUT. it modifies the query,
;; if you have a join query it takes only the first item in it. e.g. :
[({:foo/foo [:id]
:bar/bar [:id]} {:id ~'?id})]
;; will lose its `:bar` part
(om.next.impl.parser/ast->expr) ;; retrieves query from an ast,
;; but doesn't set query params based on `:params` keys of the ast.
;; there are also
(om/query->ast) ;; and
(om.next.impl.parser/expr->ast)