Clojure.spec: field existence based on other field in generator - unit-testing

Let's say we have API to save different type of files' attributes into DB*: text, images, audio and video files. It should be able to get the following fields based on their type:
base properties for all the files:
{"file-type": "text",
"location": "/Documents",
"creation-time": "2020-03-02",
"name": "sometext.txt"}
additional props specific only for some types:
text: only base props
video file: base props + {"duration(s)":123, "resolution":"4k"}
audio file: base props + {"duration(s)":123}
image: base props + {"resolution":"2048x1536"}
as we can see, some fields should be nil depenping on "file-type".
Let's say we need to validate this kind of input, which it can be any type of the described.
So the specs are:
(s/def ::file-type (s/with-gen (s/and string? not-empty) #(s/gen #{"text" "image" "video" "audio"})))
(s/def ::location (s/with-gen (s/and string? not-empty) #(s/gen #{"/Documents"})))
(s/def ::creation-time (s/with-gen (s/and string? not-empty) #(s/gen #{"2020-03-02"}))) ;for simplicity
(s/def ::name (s/with-gen (s/and string? not-empty) #(s/gen #{"sometext.txt" "image.jpg" "video.mp4" "audio.mp3"}))) ;for simplicity
(s/def ::duration (s/and int? not-empty)) ;for simplicity
(s/def ::resolution (s/with-gen (s/and string? not-empty) #(s/gen #{"4k" "2048x1536"}))) ;for simplicity
(s/def ::files-input (s/keys :req-un [::file-type ::location ::creation-time ::extension] :opt-un [::duration ::resolution]))
Let's also say that there is a programmatic validator that checks the right field set was passed for each file type (e.g., "video" has "duration" field, but "text" has not).
The question is: how to generate the full ready-made input with those dependencies for unit-tests (including the dependant fields)?
*(let's leave aside the question if it is a right design for API as this example is not from real life and for demonstration purpose only)

Based on multi-spec doc, we have to add separate method with its own set of fields for each case:
(s/def ::file-type (s/with-gen (s/and string? not-empty) #(s/gen #{"text" "image" "video" "audio"})))
(s/def ::location (s/with-gen (s/and string? not-empty) #(s/gen #{"/Documents"})))
(s/def ::creation-time (s/with-gen (s/and string? not-empty) #(s/gen #{"2020-03-02"}))) ;for simplicity
(s/def ::name (s/with-gen (s/and string? not-empty) #(s/gen #{"sometext.txt" "image.jpg" "video.mp4" "audio.mp3"}))) ;for simplicity
(s/def ::duration pos-int?) ;for simplicity
(s/def ::resolution (s/with-gen (s/and string? not-empty) #(s/gen #{"4k" "2048x1536"}))) ;for simplicity
(s/def ::base-props (s/keys :req-un [::file-type ::location ::creation-time ::name]))
(s/def ::file-type-key (s/with-gen keyword? #(s/gen #{:text :image :video :audio})))
(defmulti file :file-type-key)
(defmethod file :text [_]
(s/merge (s/keys :req-un [::file-type-key]) ::base-props))
(defmethod file :image [_]
(s/merge (s/keys :req-un [::file-type-key ::resolution]) ::base-props))
(defmethod file :video [_]
(s/merge (s/keys :req-un [::file-type-key ::duration ::resolution]) ::base-props))
(defmethod file :audio [_]
(s/merge (s/keys :req-un [::file-type-key ::duration]) ::base-props))
(defmethod file :default [_]
(s/merge (s/keys :req-un [::file-type-key]) ::base-props))
(s/def ::file-input (s/and (s/multi-spec file :file-type-key)
(fn [{:keys [file-type file-type-key]}]
(= file-type-key (keyword file-type)))))
will give generated input for unit-test (checkable by stest/check):
(gen/sample (s/gen ::file-input) 2)
=>
({:file-type-key :text, :file-type "text", :location "/Documents", :creation-time "2020-03-02", :name "sometext.txt"}
{:file-type-key :image,
:resolution "4k",
:file-type "image",
:location "/Documents",
:creation-time "2020-03-02",
:name "sometext.txt"})
we had to add one more field ::file-type-key as a selector (has to be a keyword), but it does not affect tests and can be dissoced easily.

Related

Clojure.Spec derive or alias another spec

I'd like to use clojure spec to build up a set of type constraints that can be aliased or further constrained by other specs.
For example, I might have many fields that all need to be valid sanitized markdown.
The following example works for validation (s/valid?) but not for generation (gen/generate)
(s/def ::sanitized-markdown string?)
(s/def ::instruction-list #(s/valid? ::sanitized-markdown %)) ;; works
(gen/generate (s/gen ::instruction-list)) ;; fails
However (gen/generate (s/gen ::sanitized-markdown)) does work.
Is there a way to extend ::instruction-list from ::sanitized-markdown so that it preserves all behavior?
You can alias another spec by providing it directly to s/def:
(s/def ::instruction-list ::sanitized-markdown)
You can use s/merge when merging map specs and s/and in other cases.
(s/def ::sanitized-markdown string?)
(s/def ::instruction-list (s/and ::sanitized-markdown #(> (count %) 10)))
(s/valid? ::instruction-list "abcd")
;; false
(s/valid? ::instruction-list "abcdefghijkl")
;; true
(gen/generate (s/gen ::instruction-list))
;; "178wzJW3W3zx2G0GJ1931eEeO"
An example with maps
(s/def ::a string?)
(s/def ::b string?)
(s/def ::c string?)
(s/def ::d string?)
(s/def ::first-map (s/keys :opt [::a ::b]))
(s/def ::second-map (s/keys :opt [::c ::d]))
(s/def ::third-map (s/merge ::first-map ::second-map))
(s/valid? ::third-map {:a "1" :d "2"})
;; true
(gen/generate (s/gen ::third-map))
;; {::b "gvQ7DI1kQ9DxG7C4poeWhk553", ::d "9KIp77974TEqs9HCq", ::c "qeSZA8NcYr7UVpJDsA17K"}

Clojure using value from another required key in validation

I'm relatively new to clojure and I'm looking for a way to use the value of one required key in the validation of another. I can do it by creating another map with the two values and passing that, but I was hoping there was a simpler way. Thanks
(s/def ::country string?)
(s/def ::postal-code
;sudo-code
;(if (= ::country "Canda")
;(re-matches #"^[A-Z0-9]{5}$")
;(re-matches #"^[0-9]{5}$"))
)
(s/def ::address
(s/keys :req-un [
::country
::postal-code
::street
::state
]))
Here's a way to do it with multi-spec:
(defmulti country :country)
(defmethod country "Canada" [_]
(s/spec #(re-matches #"^[A-Z0-9]{5}$" (:postal-code %))))
(defmethod country :default [_]
(s/spec #(re-matches #"^[0-9]{5}$" (:postal-code %))))
(s/def ::country string?)
(s/def ::postal-code string?)
(s/def ::address
(s/merge
(s/keys :req-un [::country ::postal-code])
(s/multi-spec country :country)))
(s/explain ::address {:country "USA" :postal-code "A2345"})
;; val: {:country "USA", :postal-code "A2345"} fails spec: :sandbox.so/address at: ["USA"] predicate: (re-matches #"^[0-9]{5}$" (:postal-code %))
(s/explain ::address {:country "Canada" :postal-code "A2345"})
;; Success!
Another option is and-ing another predicate on your keys spec:
(s/def ::address
(s/and
(s/keys :req-un [::country ::postal-code])
#(case (:country %)
"Canada" (re-matches #"^[A-Z0-9]{5}$" (:postal-code %))
(re-matches #"^[0-9]{5}$" (:postal-code %)))))
You might prefer the multi-spec approach because it's open for extension i.e. you can define more defmethods for country later as opposed to keeping all the logic in the and predicate.

How to remove extra keys from internal map using spec-tools

I'm trying to use clojure.spec and metosin/spec-tools to validate and conform data in my application. After reading spec-tools documentation it was not clear to me how I should wrap my specifications using spec-tools.core/spec so that the conformed data don't have extra keys (it works on a top level map but not on maps on inner structures).
Some code to help clarify the problem:
(ns prodimg.spec
(:require [clojure.spec.alpha :as s]
[spec-tools.core :as st]
[spec-tools.spec :as st.spec]))
(def ^:private not-blank? #(and (string? %)
(not (clojure.string/blank? %))))
(s/def :db/id integer?)
(s/def :model.image/id :db/id)
(s/def :model.image/type not-blank?)
(s/def :model.image/product-id :db/id)
(s/def :model.product/id :db/id)
(s/def :model.product/parent-id (s/nilable :db/id))
(s/def :model.product/name not-blank?)
(s/def :model.product/description string?)
(s/def :model.product/price (s/nilable decimal?))
; ----- request specs -----
; create product
(s/def :req.product.create/images (s/* (s/keys :req-un [:model.image/type])))
(s/def :req.product.create/children
(s/* (s/keys :req-un [:model.product/name :model.product/description]
:opt-un [:model.product/price])))
(s/def :req.product/create
(st/spec (s/keys :req-un [:model.product/name :model.product/description]
:opt-un [:model.product/price
:model.product/parent-id
:req.product.create/images
:req.product.create/children])))
Now suppose I have the following data that I want to validate/conform:
(def data {:name "Product"
:description "Product description"
:price (bigdec "399.49")
:extra-key "something"
:images [{:type "PNG" :extra-key "something else"}])
(st/conform :req.product/create data st/strip-extra-keys-conforming)
; below is the result
; {:name "Product"
:description "Product description"
:price 399.49M
:images [{:type "PNG" :extra-key "something else"}]
I tried changing the :req.product.create/images declaration to include st/spec call wrapping the s/* form, or the s/keys form, or both, but that changes didn't change the result.
Any ideas how I can solve this problem?
Strange, with latest version [metosin/spec-tools "0.5.1"] released 2017-10-31 (so before your post), the only change I had to do was to follow the example in the documentation under section Map conforming, which seems to be one of the attempts you already tried: wrap s/keys with st/spec as follows:
Change
(s/def :req.product.create/images (s/* (s/keys :req-un [:model.image/type])))
to
(s/def :req.product.create/images (s/* (st/spec (s/keys :req-un [:model.image/type]))))
and I get the expected output:
(st/conform :req.product/create data st/strip-extra-keys-conforming)
=>
{:description "Product description",
:images [{:type "PNG"}],
:name "Product",
:price 399.49M}

Specify content of a submap based on a field

Maybe my question has already been answered but I am stuck with a submap specification.
Imagine I have two possibilities like that
{:type :a
:spec {:name "a"}}
{:type :b
:spec {:id "b"}}
In short: the :spec keys depends on the type. For the type :a, the :spec must contain the field :name and for type :b the spec must contain the field :id.
I tried this:
(s/def ::type keyword?)
(defmulti input-type ::type)
(defmethod input-type :a
[_]
(s/keys :req-un [::name]))
(defmethod input-type :b
[_]
(s/keys :req-un [::id]))
(s/def ::spec (s/multi input-type ::type))
(s/def ::input (s/keys :req-un [::type ::spec]))
This tells me: no method ([:spec nil]).
I think I see why: maybe type is not acccessible.
So I thought to make a multi-spec of a higher level (based on the whole map).
Problem: I do not know how to define :spec based on :type because they have the same name. Do you know how to perform this?
Thanks
(s/def ::type keyword?)
(s/def ::id string?)
(s/def ::name string?)
(s/def :id/spec (s/keys :req-un [::id]))
(s/def :name/spec (s/keys :req-un [::name]))
To accommodate the two different meanings for your :spec map, we can define those in different namespaces: :id/spec and :name/spec. Note that the non-namespace suffix of these keywords are both spec and our keys specs are using un-namespaced keywords. These are "fake" namespaces here, but you could also define these in other, "real" namespaces in your project.
(defmulti input-type :type)
(defmethod input-type :a [_]
(s/keys :req-un [::type :name/spec]))
(defmethod input-type :b [_]
(s/keys :req-un [::type :id/spec]))
(s/def ::input (s/multi-spec input-type :type))
(s/valid? ::input {:type :a, :spec {:name "a"}})
=> true
You can also get samples of this spec:
(gen/sample (s/gen ::input))
=>
({:type :a, :spec {:name ""}}
{:type :b, :spec {:id "aI"}} ...

How to write a spec where a map has a key which has heterogeneous content based on another key

I'm pretty sure I need a multi-spec, which works. But I am unsure how to say that a key value which is a vector can contain heterogeneous maps.
My is my source data I want to spec:
(def int-attr { :type "int" :validations [{ :min 0 } { :max 100 }] })
(def string-attr { :type "string" :validations [{ :presence true }] })
It is the validations key I am having problems with, depending on the type key, "int" or "string", I want a different spec in the validations key.
I'm pretty sure I have to use a multi-spec. Here is what I have tried:
(defmulti attribute-type :type)
(defmethod attribute-type "string" [a] ::string-validations)
(defmethod attribute-type "int" [a] ::int-validations)
;; how to say this is a vector of int-min-validation, or int-max-validation etc.
;; (s/+ ...) and (s/or ...) maybe?
(s/def ::int-validations (...)
(s/def ::string-validations (...)
;; not sure how to incorporate these...
(s/def ::int-min-validation (s/keys :req-un [::min]))
(s/def ::int-max-validation (s/keys :req-un [::max]))
(s/def ::string-presence-validation (s/keys :req-un [::presence]))
(s/def ::attribute (s/multi-spec attribute-type ::type))
(s/explain ::attribute int-attr)
(s/explain ::attribute string-attr)
Use namespaced keywords to allow same key, but spec the value using two different specs, namely int/validations and string/validations. To allow a vector that contains maps, a good option is to use s/coll-of.
(def int-attr { :type "int" :validations [{ :min 0 } { :max 100 }] })
(def string-attr { :type "string" :validations [{ :presence true }] })
(defmulti attribute-type :type)
(defmethod attribute-type "string" [_]
(s/keys :req-un [::type :string/validations]))
(defmethod attribute-type "int" [_]
(s/keys :req-un [::type :int/validations]))
(s/def :int/validations (s/coll-of :int/validation))
(s/def :string/validations (s/coll-of :string/presence))
(s/def :int/validation (s/keys :opt-un [:int/min :int/max]))
(s/def :int/min number?)
(s/def :int/max number?)
(s/def :string/presence (s/keys :req-un [::presence]))
(s/def ::attribute (s/multi-spec attribute-type ::type))
(s/explain ::attribute int-attr) ;; => Success!
(s/explain ::attribute string-attr) ;; => Success!