I've enjoyed working with clojure.spec; it has helped uncover data errors closer to the cause. Currently I am using it to validate a response to a web server request, but I am having difficulty with the syntax for the clojure.spec operation that would allow two different map structure responses.
In my data, there are two possible responses from the web server request:
{:assignment "1232123"} and
{:no-more-assignments true}
I could use multi-spec, but that seems verbose for something that could be as simple as having one spec for each case and defining the spec as:
(s/def ::response
(s/or ::case-1 ::case-2))
Is there some syntax that I am overlooking or will I need to use multi-spec?
You can use or and and with keys specs:
(s/def ::assignment string?)
(s/def ::no-more-assignments boolean?)
(s/def ::response
(s/keys :req-un [(or ::assignment ::no-more-assignments)]))
(s/explain ::response {:assignment "123"})
;; Success!
(s/explain ::response {:foo true})
;; val: {:foo true} fails spec: :sandbox.so/response predicate: (or (contains? % :assignment) (contains? % :no-more-assignments))
Related
I'm learning Clojure, all by myself and I've been working on a simple toy project to create a Kakebo (japanese budgeting tool) for me to learn. First I will work on a CLI, then an API.
Since I'm just begining, I've been able to "grok" specs, which seems to be a great tool in clojure for validation. So, my questions are:
People test their own written specs?
I tested mine like the following code. Advice on get this better?
As I understand, there are ways to automatically test functions with generative testing, but for the bare bones specs, is this sort of test a good practice?
Specs file:
(ns kakebo.specs
(:require [clojure.spec.alpha :as s]))
(s/def ::entry-type #{:income :expense})
(s/def ::expense-type #{:fixed :basic :leisure :culture :extras})
(s/def ::income-type #{:salary :investment :reimbursement})
(s/def ::category-type (s/or ::expense-type ::income-type))
(s/def ::money (s/and double? #(> % 0.0)))
(s/def ::date (java.util.Date.))
(s/def ::item string?)
(s/def ::vendor (s/nilable string?))
(s/def ::entry (s/keys :req [::entry-type ::date ::item ::category-type ::vendor ::money]))
Tests file:
(ns kakebo.specs-test
(:require [midje.sweet :refer :all]
[clojure.spec.alpha :as s]
[kakebo.specs :refer :all]))
(facts "money"
(fact "bigger than zero"
(s/valid? :kakebo.specs/money 100.0) => true
(s/valid? :kakebo.specs/money -10.0) => false)
(fact "must be double"
(s/valid? :kakebo.specs/money "foo") => false
(s/valid? :kakebo.specs/money 1) => false))
(facts "entry types"
(fact "valid types"
(s/valid? :kakebo.specs/entry-type :income) => true
(s/valid? :kakebo.specs/entry-type :expense) => true
(s/valid? :kakebo.specs/entry-type :fixed) => false))
(facts "expense types"
(fact "valid types"
(s/valid? :kakebo.specs/expense-type :fixed) => true))
As a last last question, why can't I access the specs if I try the following import:
(ns specs-test
(:require [kakebo.specs :as ks]))
(fact "my-fact" (s/valid? :ks/money 100.0) => true)
I personally would not write tests at that are tightly coupled to the code whether I'm using spec or not. That's almost a test for each line of code - which can be hard to maintain.
There are a couple of what look to be mistakes in the specs:
;; this will not work, you probably meant to say the category type
;; is the union of the expense and income types
(s/def ::category-type (s/or ::expense-type ::income-type))
;; this will not work, you probably meant to check if that the value
;; is an instance of the Date class
(s/def ::date (java.util.Date.))
You can really get a lot out of spec by composing the atomic specs you have there into higher level specs that do the heavy lifting in your application. I would test these higher level specs, but often they may be behind regular functions and the specs may not be exposed at all.
For example, you've defined entry as a composition of other specs:
(s/def ::entry (s/keys :req [::entry-type ::date ::item ::category-type ::vendor ::money]))
This works for verifying all required data is present and for generating tests that use this data, but there are some transitive dependencies within the data such as :expense can not be of type :salary so we can add this to the entry spec:
;; decomplecting the entry types
(def income-entry? #{:income})
(def expense-entry? #{:expense})
(s/def ::entry-type (clojure.set/union expense-entry? income-entry?))
;; decomplecting the category types
(def expense-type? #{:fixed :basic :leisure :culture :extras})
(def income-type? #{:salary :investment :reimbursement})
(s/def ::category-type (clojure.set/union expense-type? income-type?))
(s/def ::money (s/and double? #(> % 0.0)))
(s/def ::date (partial instance? java.util.Date))
(s/def ::item string?)
(s/def ::vendor (s/nilable string?))
(s/def ::expense
(s/cat ::entry-type expense-entry?
::category-type expense-type?))
(s/def ::income
(s/cat ::entry-type income-entry?
::category-type income-type?))
(defn expense-or-income? [m]
(let [data (map m [::entry-type ::category-type])]
(or (s/valid? ::expense data)
(s/valid? ::income data))))
(s/def ::entry
(s/and
expense-or-income?
(s/keys :req [::entry-type ::date ::item
::category-type ::vendor ::money])))
Depending on the app or even the context you may have different specs that describe the same data. Above I combined expense and income into entry which may be good for output to a report or spreadsheet but in another area of the app you may want to keep them completely separate for data validation purposes; which is really where I use spec the most - at the boundaries of the system such as user input, database calls, etc.
Most of the tests I have for specs are in the area of validating data going into the application. The only time I test single specs is if they have business logic in them and not just data type information.
Say for a minimal example, I've got a map with the following fields.
{:name
:password
:confirm-password}
and I've written the following specs for this shape.
(s/def ::name string?)
;; password is a string and between 8 - 255 characters
(s/def ::password (s/and string? #(<= 8 (count %) 255))
;; How to write (s/def ::confirm-password)
(s/def ::sign-up-form (s/keys :req-un [::name
::password
::confirm-password])
How would I go about writing a ::confirm-password spec to check whether the two values are equal? i.e. I need access to that other field (password) to get to it.
One thing I tried was to write the spec on the sign-up-form to get access to the keys to make sure they were the same and that kind of works but the problem with that is I lose the path specificity. Basically the spec/problem that get's generated points towards the sign-up form rather than the ::confirm-password which I would like ideally.
You can s/and another predicate with your s/keys spec to check equality between the two keys' values:
(s/def ::sign-up-form
(s/and
(s/keys :req-un [::name
::password
::confirm-password])
#(= (:password %) (:confirm-password %))))
This anonymous function predicate receives the entire conformed map output of the s/keys spec.
(s/explain ::sign-up-form
{:name "Taylor"
:password "weak pass"
:confirm-password "weak pass!"})
;; val: {:name "Taylor", :password "weak pass", :confirm-password "weak pass!"}
;; fails spec: :sandbox.so/sign-up-form predicate:
;; (= (:password %) (:confirm-password %))
I want to create a clojure spec for a map that has rules about the presence of particular keys.
The map must have a :type and can have either :default or :value but not both. I tried:
(s/def ::propertyDef
(s/keys :req [::type (s/or ::default ::value) ] :opt [::description ::required]))
but I got
CompilerException java.lang.AssertionError: Assert failed:
spec/or expects k1 p1 k2 p2..., where ks are keywords
(c/and (even? (count key-pred-forms)) (every? keyword? keys)),
compiling:(C:\Users\MartinRoberts\AppData\Local\Temp\form-init4830956164341520551.clj:1:22)
but the or gave me an error as it is in the wrong format. I have to admit to not really understanding in the documentation for s/or.
First: you are using s/or to specify either a ::default or a ::value in your list of required keys. s/or requires :label spec pairs, and you are giving only the specs themselves, which is the cause of the error.
To solve, simply use or instead:
(s/def ::propertyDef (s/keys :req [::type (or ::default ::value)]
:opt [::description ::required]))
This allows both ::default and ::value to be present in the map, but this is almost always okay. The code which actually uses the map can simply check for the presence of ::value and use that, and if it's not there, then use ::default (or whatever your logic happens to be). This is usually done as such:
(let [myvalue (or (::value mymap) (::default mymap))] ...)
There could be thousands of keys in the map, and it would not affect your ability to extract the keys you need. This is why spec does not provide a built-in way to specify keys that should not be in the map, only ways to specify which keys should be present (namely, :req and :req-un in s/keys). Think of how most http servers work: you can give them nonsensical header keys and values, but they don't refuse to service the request; they just ignore them and return a response.
So, you likely don't need to enforce that only one or the other be present, but if you must, you can define an exclusive or function:
(defn xor
[p q]
(and (or p q)
(not (and p q))))
and then add this as an additional predicate on the spec:
(s/def ::propertyDef (s/and (s/keys :req [::type (or ::default ::value)]
:opt [::description ::required])
#(xor (::default %) (::value %))))
(s/valid? ::propertyDef {::type "type" ::default "default"})
=> true
(s/valid? ::propertyDef {::type "type" ::value "value"})
=> true
(s/valid? ::propertyDef {::type "type" ::default "default" ::value "value"})
=> false
I have a DSL specification which is a sequence as usual (cat). I want to take advantage of spec's parsing (i.e. conforming) to get the AST of an expression that conforms with my DSL. E.g.
user> (s/def ::person (s/cat :person-sym '#{person} :name string? :age number?))
=> :user/person
user> (s/conform ::person '(person "Henry The Sloth" 55))
=> {:person-sym person, :name "Henry The Sloth", :age 55}
Now that it's parsed and I have my AST, I would want to do interesting things with it, so I would want to test it and whatnot. So now I need to write a spec for that AST, and that's basically duplicating everything. Actually it's worse than that because now I have to s/def specs for predicates that I didn't have to before, because as the docs for keys says: "there is no support for inline value specification, by design." / "It is the (enforced) opinion of spec that the specification of values associated with a namespaced keyword, like :my.ns/k, should be registered under that keyword itself..". So duplicating (with omitting the person-sym part):
user> (s/def ::name string?)
=> :user/name
user> (s/def ::age number?)
=> :user/age
user> (s/def ::person-ast (s/keys :req-un [::name ::age]))
:user/person-ast
And now it seems to be compatible:
user> (s/conform ::person-ast (s/conform ::person '(person "Henry The Sloth" 55)))
=> {:person-sym person, :name "Henry The Sloth", :age 55}
In practice, I have more complicated data of course, and I wonder what should I do? AFAIK spec doesn't give me the spec for the AST that it creates (actually personally I would figure that this is something it should do). Any suggestions?
I'd say right now you have two options - one is to do what you're doing and create two sets of specs for the before/after.
The other option is to create a model of your domain in data and generate both specs (I've seen many people are doing something like this).
I have not heard Rich talk about generating the output spec of conformed results so I don't think that is likely in the current roadmap.
I am following the clojure.spec guide. I understand it is possible to declare required and optional attributes when using clojure.spec/keys.
I don't understand what is meant by optional. To me :opt doesn't do anything.
(s/valid? (s/keys :req [:my/a]) {:my/a 1 :my/b 2}) ;=> true
(s/valid? (s/keys :req [:my/a] :opt []) {:my/a 1 :my/b 2}) ;=> true
The guide promises to explain this to me, "We’ll see later where optional attributes can be useful", but I fail to find the explanation. Can I declare forbidden keys? Or somehow declare the set of valid keys to equal the keys in :req and :opt?
This is a very good question, and the clojure.spec API gives the (granted, short and unsatisfying) answer:
The :opt keys serve as documentation and
may be used by the generator.
I do not think you can invalidate a map if it contains an extra (this is what you mean by "forbidden" I think) key using this method. However, you could use this spec to make sure ::bad-key is not present:
(s/def ::m (s/and (s/keys :req [::a]) #(not (contains? % ::bad-key))))
(s/valid? ::m {::a "required!"}) ; => true
(s/valid? ::m {::a "required!" ::b "optional!"}) ; => true
(s/valid? ::m {::a "required!" ::bad-key "no good!"}) ; => false
You could limit the number of keys to exactly the set you want by using this spec:
(s/def ::r (s/and (s/keys :req [::reqd1 ::reqd2]) #(= (count %) 2)))
(s/valid? ::r {::reqd1 "abc" ::reqd2 "xyz"}) ; => true
(s/valid? ::r {::reqd1 "abc" ::reqd2 "xyz" ::extra 123}) ; => false
Still, the best way to handle this IMO, would be to simply ignore that there is a key present that you don't care about.
Hopefully as spec matures, these nice things will be added. Or, maybe they are already there (it is changing rapidly) and I simply don't know about it. This is a very new concept in clojure, so most of us have a lot to learn about it.
UPDATE - December 2016
I just wanted to revisit this 6 months since writing it. It looks like my initial comment about ignoring keys you don't care about is the preferred way to go. In fact, at the clojure/conj conference I attended two weeks ago, Rich's keynote specifically addressed the notion of versioning in all levels of software, from the function level up to the application level. He even specifically mentions this notion of disallowing keys in the talk, which can be found on youtube. He says that it was intentionally designed so that only required keys can be spec'd. Disallowing keys really serves no good purpose, and it should be done with caution.
Regarding the :opt keys, I think the original answer still stands up pretty well--it's documentation, and practically, it allows these optionally specified keys to be generated:
(s/def ::name #{"Bob" "Josh" "Mary" "Susan"})
(s/def ::height-inches (s/int-in 48 90))
(s/def ::person (s/keys :req-un [::name] :opt-un [::height-inches]))
(map first (s/exercise ::person))
; some generated data have :height-inches, some do not
({:name "Susan"}
{:name "Mary", :height-inches 48}
{:name "Bob", :height-inches 49}
{:name "Josh"}
The point about optional keys is that the value will be validated if they appear in the map