I'm learning Clojure, all by myself and I've been working on a simple toy project to create a Kakebo (japanese budgeting tool) for me to learn. First I will work on a CLI, then an API.
Since I'm just begining, I've been able to "grok" specs, which seems to be a great tool in clojure for validation. So, my questions are:
People test their own written specs?
I tested mine like the following code. Advice on get this better?
As I understand, there are ways to automatically test functions with generative testing, but for the bare bones specs, is this sort of test a good practice?
Specs file:
(ns kakebo.specs
(:require [clojure.spec.alpha :as s]))
(s/def ::entry-type #{:income :expense})
(s/def ::expense-type #{:fixed :basic :leisure :culture :extras})
(s/def ::income-type #{:salary :investment :reimbursement})
(s/def ::category-type (s/or ::expense-type ::income-type))
(s/def ::money (s/and double? #(> % 0.0)))
(s/def ::date (java.util.Date.))
(s/def ::item string?)
(s/def ::vendor (s/nilable string?))
(s/def ::entry (s/keys :req [::entry-type ::date ::item ::category-type ::vendor ::money]))
Tests file:
(ns kakebo.specs-test
(:require [midje.sweet :refer :all]
[clojure.spec.alpha :as s]
[kakebo.specs :refer :all]))
(facts "money"
(fact "bigger than zero"
(s/valid? :kakebo.specs/money 100.0) => true
(s/valid? :kakebo.specs/money -10.0) => false)
(fact "must be double"
(s/valid? :kakebo.specs/money "foo") => false
(s/valid? :kakebo.specs/money 1) => false))
(facts "entry types"
(fact "valid types"
(s/valid? :kakebo.specs/entry-type :income) => true
(s/valid? :kakebo.specs/entry-type :expense) => true
(s/valid? :kakebo.specs/entry-type :fixed) => false))
(facts "expense types"
(fact "valid types"
(s/valid? :kakebo.specs/expense-type :fixed) => true))
As a last last question, why can't I access the specs if I try the following import:
(ns specs-test
(:require [kakebo.specs :as ks]))
(fact "my-fact" (s/valid? :ks/money 100.0) => true)
I personally would not write tests at that are tightly coupled to the code whether I'm using spec or not. That's almost a test for each line of code - which can be hard to maintain.
There are a couple of what look to be mistakes in the specs:
;; this will not work, you probably meant to say the category type
;; is the union of the expense and income types
(s/def ::category-type (s/or ::expense-type ::income-type))
;; this will not work, you probably meant to check if that the value
;; is an instance of the Date class
(s/def ::date (java.util.Date.))
You can really get a lot out of spec by composing the atomic specs you have there into higher level specs that do the heavy lifting in your application. I would test these higher level specs, but often they may be behind regular functions and the specs may not be exposed at all.
For example, you've defined entry as a composition of other specs:
(s/def ::entry (s/keys :req [::entry-type ::date ::item ::category-type ::vendor ::money]))
This works for verifying all required data is present and for generating tests that use this data, but there are some transitive dependencies within the data such as :expense can not be of type :salary so we can add this to the entry spec:
;; decomplecting the entry types
(def income-entry? #{:income})
(def expense-entry? #{:expense})
(s/def ::entry-type (clojure.set/union expense-entry? income-entry?))
;; decomplecting the category types
(def expense-type? #{:fixed :basic :leisure :culture :extras})
(def income-type? #{:salary :investment :reimbursement})
(s/def ::category-type (clojure.set/union expense-type? income-type?))
(s/def ::money (s/and double? #(> % 0.0)))
(s/def ::date (partial instance? java.util.Date))
(s/def ::item string?)
(s/def ::vendor (s/nilable string?))
(s/def ::expense
(s/cat ::entry-type expense-entry?
::category-type expense-type?))
(s/def ::income
(s/cat ::entry-type income-entry?
::category-type income-type?))
(defn expense-or-income? [m]
(let [data (map m [::entry-type ::category-type])]
(or (s/valid? ::expense data)
(s/valid? ::income data))))
(s/def ::entry
(s/and
expense-or-income?
(s/keys :req [::entry-type ::date ::item
::category-type ::vendor ::money])))
Depending on the app or even the context you may have different specs that describe the same data. Above I combined expense and income into entry which may be good for output to a report or spreadsheet but in another area of the app you may want to keep them completely separate for data validation purposes; which is really where I use spec the most - at the boundaries of the system such as user input, database calls, etc.
Most of the tests I have for specs are in the area of validating data going into the application. The only time I test single specs is if they have business logic in them and not just data type information.
Related
I am trying to use Clojure Spec to define a data structure containing a java.time.LocalDate element:
(s/def :ex/first-name string?)
(s/def :ex/last-name string?)
(s/def :ex/birth-date (s/valid? inst? (java.time.LocalDate/now)))
(s/def :ex/person
(s/keys :req [:ex/first-name
:ex/last-name
:ex/birth-date]))
(def p1 #:ex{:first-name "Jenny"
:last-name "Barnes"
:birth-date (java.time.LocalDate/parse "1910-03-15")})
(println p1)
produces the following output
#:ex{:first-name Jenny, :last-name Barnes, :birth-date #object[java.time.LocalDate 0x4ed4f9db 1910-03-15]}
However, when I test to see if p1 conforms to the :ex/person spec, it fails:
(s/valid? :ex/person p1)
ClassCastException java.lang.Boolean cannot be cast to clojure.lang.IFn clojure.spec.alpha/spec-impl/reify--1987 (alpha.clj:875)
Looking closer at the Clojure examples for inst?, I see:
(inst? (java.time.Instant/now))
;;=> true
(inst? (java.time.LocalDateTime/now))
;;=> false
However, I don't see an obvious reason as to why that returns false. This seems to be the root of my issue, but I have not found a solution and would like some help.
You're probably looking for instance?- and your example fails, because in:
(s/def :ex/birth-date (s/valid? inst? (java.time.LocalDate/now)))
this part (s/valid? inst? (java.time.LocalDate/now)) should be a function (predicate), not boolean. The full code:
(s/def :ex/first-name string?)
(s/def :ex/last-name string?)
(s/def :ex/birth-date #(instance? java.time.LocalDate %))
(s/def :ex/person
(s/keys :req [:ex/first-name
:ex/last-name
:ex/birth-date]))
(def p1 #:ex{:first-name "Jenny"
:last-name "Barnes"
:birth-date (java.time.LocalDate/parse "1910-03-15")})
(s/valid? :ex/person p1)
=> true
inst? won't work here, because Inst is a protocol, used to extend java.util.Date and java.time.Instant:
(defprotocol Inst
(inst-ms* [inst]))
(extend-protocol Inst
java.util.Date
(inst-ms* [inst] (.getTime ^java.util.Date inst)))
(defn inst?
"Return true if x satisfies Inst"
{:added "1.9"}
[x]
(satisfies? Inst x))
(extend-protocol clojure.core/Inst
java.time.Instant
(inst-ms* [inst] (.toEpochMilli ^java.time.Instant inst)))
And you can use satisfies? to check whether some object satisfies given protocol:
(satisfies? Inst (java.time.LocalDate/parse "1910-03-15"))
=> false
I'd like to use clojure spec to build up a set of type constraints that can be aliased or further constrained by other specs.
For example, I might have many fields that all need to be valid sanitized markdown.
The following example works for validation (s/valid?) but not for generation (gen/generate)
(s/def ::sanitized-markdown string?)
(s/def ::instruction-list #(s/valid? ::sanitized-markdown %)) ;; works
(gen/generate (s/gen ::instruction-list)) ;; fails
However (gen/generate (s/gen ::sanitized-markdown)) does work.
Is there a way to extend ::instruction-list from ::sanitized-markdown so that it preserves all behavior?
You can alias another spec by providing it directly to s/def:
(s/def ::instruction-list ::sanitized-markdown)
You can use s/merge when merging map specs and s/and in other cases.
(s/def ::sanitized-markdown string?)
(s/def ::instruction-list (s/and ::sanitized-markdown #(> (count %) 10)))
(s/valid? ::instruction-list "abcd")
;; false
(s/valid? ::instruction-list "abcdefghijkl")
;; true
(gen/generate (s/gen ::instruction-list))
;; "178wzJW3W3zx2G0GJ1931eEeO"
An example with maps
(s/def ::a string?)
(s/def ::b string?)
(s/def ::c string?)
(s/def ::d string?)
(s/def ::first-map (s/keys :opt [::a ::b]))
(s/def ::second-map (s/keys :opt [::c ::d]))
(s/def ::third-map (s/merge ::first-map ::second-map))
(s/valid? ::third-map {:a "1" :d "2"})
;; true
(gen/generate (s/gen ::third-map))
;; {::b "gvQ7DI1kQ9DxG7C4poeWhk553", ::d "9KIp77974TEqs9HCq", ::c "qeSZA8NcYr7UVpJDsA17K"}
I want to write a clojure spec for a hash-map wherein the value of one of the
keys is constrained to be equal to the sum of the values of two other keys. I
know one way to write a test generator for such a spec by hand:
(ns my-domain)
(require '[clojure.test :refer :all ]
'[clojure.spec.alpha :as s ]
'[clojure.spec.gen.alpha :as gen ]
'[clojure.pprint :refer (pprint) ])
(s/def ::station-id string?)
(s/def ::sim-time (s/double-in :infinite? true, :NaN? false))
(s/def ::reserved-counts (s/and int? #(not (neg? %))))
(s/def ::free-counts (s/and int? #(not (neg? %))))
(def counts-preimage (s/gen (s/keys :req [::station-id
::sim-time
::reserved-counts
::free-counts])))
(pprint (gen/generate
(gen/bind
counts-preimage
#(gen/return
(into % {::total-counts
(+ (::reserved-counts %)
(::free-counts %))})))))
#:my-domain{:station-id "sHN8Ce0tKWSdXmRd4e46fB",
:sim-time -3.4619293212890625,
:reserved-counts 58,
:free-counts 194,
:total-counts 252}
But I haven't figured out how to write a spec for it, let alone a spec that
produces a similar generator. The gist of the problem is that I lack, in the space of specs, a way to
get hold of the "preimage" in the spec, that is, I lack an analogue to bind
from the space of generators. Here is a failed attempt:
(s/def ::counts-partial-hash-map
(s/keys :req [::station-id
::sim-time
::reserved-counts
::free-counts]))
(s/def ::counts-attempted-hash-map
(s/and ::counts-partial-hash-map
#(into % {::total-counts (+ (::reserved-counts %)
(::free-counts %))})))
(pprint (gen/generate (s/gen ::counts-attempted-hash-map)))
#:my-domain{:station-id "ls5qBUoF",
:sim-time ##Inf,
:reserved-counts 56797960,
:free-counts 17}
The generated sample conforms to the spec because #(into % {...}) is truthy,
but the result doesn't contain the new attribute with the key ::total-counts.
I'd be grateful for any guidance.
EDIT: Today I Learned about s/with-gen, which will allow me to attach
my (working) test generator to my "preimage" or "partial" spec. Perhaps
that's the best way forward?
You could use the nat-int? predicate (for which there's a built-in spec, thanks #glts) for the count keys, and add a ::total-counts spec too:
(s/def ::reserved-counts nat-int?)
(s/def ::free-counts nat-int?)
(s/def ::total-counts nat-int?)
(s/def ::counts-partial-hash-map
(s/keys :req [::station-id
::sim-time
::reserved-counts
::free-counts]))
spec for a hash-map wherein the value of one of the keys is constrained to be equal to the sum of the values of two other keys
To add this assertion you can s/and a predicate function with the keys spec (or in this example the merge spec that merges the partial map spec with a ::total-count keys spec):
(s/def ::counts-attempted-hash-map
(s/with-gen
;; keys spec + sum-check predicate
(s/and
(s/merge ::counts-partial-hash-map (s/keys :req [::total-counts]))
#(= (::total-counts %) (+ (::reserved-counts %) (::free-counts %))))
;; custom generator
#(gen/fmap
(fn [m]
(assoc m ::total-counts (+ (::reserved-counts m) (::free-counts m))))
(s/gen ::counts-partial-hash-map))))
This also uses with-gen to associate a custom generator with the spec that sets ::total-count to the sum of the other count keys.
(gen/sample (s/gen ::counts-attempted-hash-map) 1)
=> (#:user{:station-id "", :sim-time 0.5, :reserved-counts 1, :free-counts 1, :total-counts 2})
The generated sample conforms to the spec because #(into % {...}) is truthy, but the result doesn't contain the new attribute with the key ::total-counts.
I'd recommend against using specs to calculate/add ::total-counts to the map. Specs generally shouldn't be used for data transformation.
I'm trying to use clojure.spec and metosin/spec-tools to validate and conform data in my application. After reading spec-tools documentation it was not clear to me how I should wrap my specifications using spec-tools.core/spec so that the conformed data don't have extra keys (it works on a top level map but not on maps on inner structures).
Some code to help clarify the problem:
(ns prodimg.spec
(:require [clojure.spec.alpha :as s]
[spec-tools.core :as st]
[spec-tools.spec :as st.spec]))
(def ^:private not-blank? #(and (string? %)
(not (clojure.string/blank? %))))
(s/def :db/id integer?)
(s/def :model.image/id :db/id)
(s/def :model.image/type not-blank?)
(s/def :model.image/product-id :db/id)
(s/def :model.product/id :db/id)
(s/def :model.product/parent-id (s/nilable :db/id))
(s/def :model.product/name not-blank?)
(s/def :model.product/description string?)
(s/def :model.product/price (s/nilable decimal?))
; ----- request specs -----
; create product
(s/def :req.product.create/images (s/* (s/keys :req-un [:model.image/type])))
(s/def :req.product.create/children
(s/* (s/keys :req-un [:model.product/name :model.product/description]
:opt-un [:model.product/price])))
(s/def :req.product/create
(st/spec (s/keys :req-un [:model.product/name :model.product/description]
:opt-un [:model.product/price
:model.product/parent-id
:req.product.create/images
:req.product.create/children])))
Now suppose I have the following data that I want to validate/conform:
(def data {:name "Product"
:description "Product description"
:price (bigdec "399.49")
:extra-key "something"
:images [{:type "PNG" :extra-key "something else"}])
(st/conform :req.product/create data st/strip-extra-keys-conforming)
; below is the result
; {:name "Product"
:description "Product description"
:price 399.49M
:images [{:type "PNG" :extra-key "something else"}]
I tried changing the :req.product.create/images declaration to include st/spec call wrapping the s/* form, or the s/keys form, or both, but that changes didn't change the result.
Any ideas how I can solve this problem?
Strange, with latest version [metosin/spec-tools "0.5.1"] released 2017-10-31 (so before your post), the only change I had to do was to follow the example in the documentation under section Map conforming, which seems to be one of the attempts you already tried: wrap s/keys with st/spec as follows:
Change
(s/def :req.product.create/images (s/* (s/keys :req-un [:model.image/type])))
to
(s/def :req.product.create/images (s/* (st/spec (s/keys :req-un [:model.image/type]))))
and I get the expected output:
(st/conform :req.product/create data st/strip-extra-keys-conforming)
=>
{:description "Product description",
:images [{:type "PNG"}],
:name "Product",
:price 399.49M}
After writing this answer, I was inspired to try to specify Clojure's destructuring language using spec:
(require '[clojure.spec :as s])
(s/def ::binding (s/or :sym ::sym :assoc ::assoc :seq ::seq))
(s/def ::sym (s/and simple-symbol? (complement #{'&})))
The sequential destructuring part is easy to spec with a regex (so I'm ignoring it here), but I got stuck at associative destructuring. The most basic case is a map from binding forms to key expressions:
(s/def ::mappings (s/map-of ::binding ::s/any :conform-keys true))
But Clojure provides several special keys as well:
(s/def ::as ::sym)
(s/def ::or ::mappings)
(s/def ::ident-vec (s/coll-of ident? :kind vector?))
(s/def ::keys ::ident-vec)
(s/def ::strs ::ident-vec)
(s/def ::syms ::ident-vec)
(s/def ::opts (s/keys :opt-un [::as ::or ::keys ::strs ::syms]))
How can I create an ::assoc spec for maps that could be created by merging together a map that conforms to ::mappings and a map that conforms to ::opts? I know that there's merge:
(s/def ::assoc (s/merge ::opts ::mappings))
But this doesn't work, because merge is basically an analogue of and. I'm looking for something that's analogous to or, but for maps.
You can spec hybrid maps using an s/merge of s/keys and s/every of the map as tuples. Here's a simpler example:
(s/def ::a keyword?)
(s/def ::b string?)
(s/def ::m
(s/merge (s/keys :opt-un [::a ::b])
(s/every (s/or :int (s/tuple int? int?)
:option (s/tuple keyword? any?))
:into {})))
(s/valid? ::m {1 2, 3 4, :a :foo, :b "abc"}) ;; true
This simpler formulation has several benefits over a conformer approach. Most importantly, it states the truth. Additionally, it should generate, conform, and unform without further effort.
You can use s/conformer as an intermediate step in s/and to transform your map to the form that’s easy to validate:
(s/def ::assoc
(s/and
map?
(s/conformer #(array-map
::mappings (dissoc % :as :or :keys :strs :syms)
::opts (select-keys % [:as :or :keys :strs :syms])))
(s/keys :opt [::mappings ::opts])))
That will get you from e.g.
{ key :key
:as name }
to
{ ::mappings { key :key }
::opts { :as name } }