I'm trying to use clojure.spec and metosin/spec-tools to validate and conform data in my application. After reading spec-tools documentation it was not clear to me how I should wrap my specifications using spec-tools.core/spec so that the conformed data don't have extra keys (it works on a top level map but not on maps on inner structures).
Some code to help clarify the problem:
(ns prodimg.spec
(:require [clojure.spec.alpha :as s]
[spec-tools.core :as st]
[spec-tools.spec :as st.spec]))
(def ^:private not-blank? #(and (string? %)
(not (clojure.string/blank? %))))
(s/def :db/id integer?)
(s/def :model.image/id :db/id)
(s/def :model.image/type not-blank?)
(s/def :model.image/product-id :db/id)
(s/def :model.product/id :db/id)
(s/def :model.product/parent-id (s/nilable :db/id))
(s/def :model.product/name not-blank?)
(s/def :model.product/description string?)
(s/def :model.product/price (s/nilable decimal?))
; ----- request specs -----
; create product
(s/def :req.product.create/images (s/* (s/keys :req-un [:model.image/type])))
(s/def :req.product.create/children
(s/* (s/keys :req-un [:model.product/name :model.product/description]
:opt-un [:model.product/price])))
(s/def :req.product/create
(st/spec (s/keys :req-un [:model.product/name :model.product/description]
:opt-un [:model.product/price
:model.product/parent-id
:req.product.create/images
:req.product.create/children])))
Now suppose I have the following data that I want to validate/conform:
(def data {:name "Product"
:description "Product description"
:price (bigdec "399.49")
:extra-key "something"
:images [{:type "PNG" :extra-key "something else"}])
(st/conform :req.product/create data st/strip-extra-keys-conforming)
; below is the result
; {:name "Product"
:description "Product description"
:price 399.49M
:images [{:type "PNG" :extra-key "something else"}]
I tried changing the :req.product.create/images declaration to include st/spec call wrapping the s/* form, or the s/keys form, or both, but that changes didn't change the result.
Any ideas how I can solve this problem?
Strange, with latest version [metosin/spec-tools "0.5.1"] released 2017-10-31 (so before your post), the only change I had to do was to follow the example in the documentation under section Map conforming, which seems to be one of the attempts you already tried: wrap s/keys with st/spec as follows:
Change
(s/def :req.product.create/images (s/* (s/keys :req-un [:model.image/type])))
to
(s/def :req.product.create/images (s/* (st/spec (s/keys :req-un [:model.image/type]))))
and I get the expected output:
(st/conform :req.product/create data st/strip-extra-keys-conforming)
=>
{:description "Product description",
:images [{:type "PNG"}],
:name "Product",
:price 399.49M}
Related
Let's say we have API to save different type of files' attributes into DB*: text, images, audio and video files. It should be able to get the following fields based on their type:
base properties for all the files:
{"file-type": "text",
"location": "/Documents",
"creation-time": "2020-03-02",
"name": "sometext.txt"}
additional props specific only for some types:
text: only base props
video file: base props + {"duration(s)":123, "resolution":"4k"}
audio file: base props + {"duration(s)":123}
image: base props + {"resolution":"2048x1536"}
as we can see, some fields should be nil depenping on "file-type".
Let's say we need to validate this kind of input, which it can be any type of the described.
So the specs are:
(s/def ::file-type (s/with-gen (s/and string? not-empty) #(s/gen #{"text" "image" "video" "audio"})))
(s/def ::location (s/with-gen (s/and string? not-empty) #(s/gen #{"/Documents"})))
(s/def ::creation-time (s/with-gen (s/and string? not-empty) #(s/gen #{"2020-03-02"}))) ;for simplicity
(s/def ::name (s/with-gen (s/and string? not-empty) #(s/gen #{"sometext.txt" "image.jpg" "video.mp4" "audio.mp3"}))) ;for simplicity
(s/def ::duration (s/and int? not-empty)) ;for simplicity
(s/def ::resolution (s/with-gen (s/and string? not-empty) #(s/gen #{"4k" "2048x1536"}))) ;for simplicity
(s/def ::files-input (s/keys :req-un [::file-type ::location ::creation-time ::extension] :opt-un [::duration ::resolution]))
Let's also say that there is a programmatic validator that checks the right field set was passed for each file type (e.g., "video" has "duration" field, but "text" has not).
The question is: how to generate the full ready-made input with those dependencies for unit-tests (including the dependant fields)?
*(let's leave aside the question if it is a right design for API as this example is not from real life and for demonstration purpose only)
Based on multi-spec doc, we have to add separate method with its own set of fields for each case:
(s/def ::file-type (s/with-gen (s/and string? not-empty) #(s/gen #{"text" "image" "video" "audio"})))
(s/def ::location (s/with-gen (s/and string? not-empty) #(s/gen #{"/Documents"})))
(s/def ::creation-time (s/with-gen (s/and string? not-empty) #(s/gen #{"2020-03-02"}))) ;for simplicity
(s/def ::name (s/with-gen (s/and string? not-empty) #(s/gen #{"sometext.txt" "image.jpg" "video.mp4" "audio.mp3"}))) ;for simplicity
(s/def ::duration pos-int?) ;for simplicity
(s/def ::resolution (s/with-gen (s/and string? not-empty) #(s/gen #{"4k" "2048x1536"}))) ;for simplicity
(s/def ::base-props (s/keys :req-un [::file-type ::location ::creation-time ::name]))
(s/def ::file-type-key (s/with-gen keyword? #(s/gen #{:text :image :video :audio})))
(defmulti file :file-type-key)
(defmethod file :text [_]
(s/merge (s/keys :req-un [::file-type-key]) ::base-props))
(defmethod file :image [_]
(s/merge (s/keys :req-un [::file-type-key ::resolution]) ::base-props))
(defmethod file :video [_]
(s/merge (s/keys :req-un [::file-type-key ::duration ::resolution]) ::base-props))
(defmethod file :audio [_]
(s/merge (s/keys :req-un [::file-type-key ::duration]) ::base-props))
(defmethod file :default [_]
(s/merge (s/keys :req-un [::file-type-key]) ::base-props))
(s/def ::file-input (s/and (s/multi-spec file :file-type-key)
(fn [{:keys [file-type file-type-key]}]
(= file-type-key (keyword file-type)))))
will give generated input for unit-test (checkable by stest/check):
(gen/sample (s/gen ::file-input) 2)
=>
({:file-type-key :text, :file-type "text", :location "/Documents", :creation-time "2020-03-02", :name "sometext.txt"}
{:file-type-key :image,
:resolution "4k",
:file-type "image",
:location "/Documents",
:creation-time "2020-03-02",
:name "sometext.txt"})
we had to add one more field ::file-type-key as a selector (has to be a keyword), but it does not affect tests and can be dissoced easily.
I'm relatively new to clojure and I'm looking for a way to use the value of one required key in the validation of another. I can do it by creating another map with the two values and passing that, but I was hoping there was a simpler way. Thanks
(s/def ::country string?)
(s/def ::postal-code
;sudo-code
;(if (= ::country "Canda")
;(re-matches #"^[A-Z0-9]{5}$")
;(re-matches #"^[0-9]{5}$"))
)
(s/def ::address
(s/keys :req-un [
::country
::postal-code
::street
::state
]))
Here's a way to do it with multi-spec:
(defmulti country :country)
(defmethod country "Canada" [_]
(s/spec #(re-matches #"^[A-Z0-9]{5}$" (:postal-code %))))
(defmethod country :default [_]
(s/spec #(re-matches #"^[0-9]{5}$" (:postal-code %))))
(s/def ::country string?)
(s/def ::postal-code string?)
(s/def ::address
(s/merge
(s/keys :req-un [::country ::postal-code])
(s/multi-spec country :country)))
(s/explain ::address {:country "USA" :postal-code "A2345"})
;; val: {:country "USA", :postal-code "A2345"} fails spec: :sandbox.so/address at: ["USA"] predicate: (re-matches #"^[0-9]{5}$" (:postal-code %))
(s/explain ::address {:country "Canada" :postal-code "A2345"})
;; Success!
Another option is and-ing another predicate on your keys spec:
(s/def ::address
(s/and
(s/keys :req-un [::country ::postal-code])
#(case (:country %)
"Canada" (re-matches #"^[A-Z0-9]{5}$" (:postal-code %))
(re-matches #"^[0-9]{5}$" (:postal-code %)))))
You might prefer the multi-spec approach because it's open for extension i.e. you can define more defmethods for country later as opposed to keeping all the logic in the and predicate.
Maybe my question has already been answered but I am stuck with a submap specification.
Imagine I have two possibilities like that
{:type :a
:spec {:name "a"}}
{:type :b
:spec {:id "b"}}
In short: the :spec keys depends on the type. For the type :a, the :spec must contain the field :name and for type :b the spec must contain the field :id.
I tried this:
(s/def ::type keyword?)
(defmulti input-type ::type)
(defmethod input-type :a
[_]
(s/keys :req-un [::name]))
(defmethod input-type :b
[_]
(s/keys :req-un [::id]))
(s/def ::spec (s/multi input-type ::type))
(s/def ::input (s/keys :req-un [::type ::spec]))
This tells me: no method ([:spec nil]).
I think I see why: maybe type is not acccessible.
So I thought to make a multi-spec of a higher level (based on the whole map).
Problem: I do not know how to define :spec based on :type because they have the same name. Do you know how to perform this?
Thanks
(s/def ::type keyword?)
(s/def ::id string?)
(s/def ::name string?)
(s/def :id/spec (s/keys :req-un [::id]))
(s/def :name/spec (s/keys :req-un [::name]))
To accommodate the two different meanings for your :spec map, we can define those in different namespaces: :id/spec and :name/spec. Note that the non-namespace suffix of these keywords are both spec and our keys specs are using un-namespaced keywords. These are "fake" namespaces here, but you could also define these in other, "real" namespaces in your project.
(defmulti input-type :type)
(defmethod input-type :a [_]
(s/keys :req-un [::type :name/spec]))
(defmethod input-type :b [_]
(s/keys :req-un [::type :id/spec]))
(s/def ::input (s/multi-spec input-type :type))
(s/valid? ::input {:type :a, :spec {:name "a"}})
=> true
You can also get samples of this spec:
(gen/sample (s/gen ::input))
=>
({:type :a, :spec {:name ""}}
{:type :b, :spec {:id "aI"}} ...
I'm pretty sure I need a multi-spec, which works. But I am unsure how to say that a key value which is a vector can contain heterogeneous maps.
My is my source data I want to spec:
(def int-attr { :type "int" :validations [{ :min 0 } { :max 100 }] })
(def string-attr { :type "string" :validations [{ :presence true }] })
It is the validations key I am having problems with, depending on the type key, "int" or "string", I want a different spec in the validations key.
I'm pretty sure I have to use a multi-spec. Here is what I have tried:
(defmulti attribute-type :type)
(defmethod attribute-type "string" [a] ::string-validations)
(defmethod attribute-type "int" [a] ::int-validations)
;; how to say this is a vector of int-min-validation, or int-max-validation etc.
;; (s/+ ...) and (s/or ...) maybe?
(s/def ::int-validations (...)
(s/def ::string-validations (...)
;; not sure how to incorporate these...
(s/def ::int-min-validation (s/keys :req-un [::min]))
(s/def ::int-max-validation (s/keys :req-un [::max]))
(s/def ::string-presence-validation (s/keys :req-un [::presence]))
(s/def ::attribute (s/multi-spec attribute-type ::type))
(s/explain ::attribute int-attr)
(s/explain ::attribute string-attr)
Use namespaced keywords to allow same key, but spec the value using two different specs, namely int/validations and string/validations. To allow a vector that contains maps, a good option is to use s/coll-of.
(def int-attr { :type "int" :validations [{ :min 0 } { :max 100 }] })
(def string-attr { :type "string" :validations [{ :presence true }] })
(defmulti attribute-type :type)
(defmethod attribute-type "string" [_]
(s/keys :req-un [::type :string/validations]))
(defmethod attribute-type "int" [_]
(s/keys :req-un [::type :int/validations]))
(s/def :int/validations (s/coll-of :int/validation))
(s/def :string/validations (s/coll-of :string/presence))
(s/def :int/validation (s/keys :opt-un [:int/min :int/max]))
(s/def :int/min number?)
(s/def :int/max number?)
(s/def :string/presence (s/keys :req-un [::presence]))
(s/def ::attribute (s/multi-spec attribute-type ::type))
(s/explain ::attribute int-attr) ;; => Success!
(s/explain ::attribute string-attr) ;; => Success!
After writing this answer, I was inspired to try to specify Clojure's destructuring language using spec:
(require '[clojure.spec :as s])
(s/def ::binding (s/or :sym ::sym :assoc ::assoc :seq ::seq))
(s/def ::sym (s/and simple-symbol? (complement #{'&})))
The sequential destructuring part is easy to spec with a regex (so I'm ignoring it here), but I got stuck at associative destructuring. The most basic case is a map from binding forms to key expressions:
(s/def ::mappings (s/map-of ::binding ::s/any :conform-keys true))
But Clojure provides several special keys as well:
(s/def ::as ::sym)
(s/def ::or ::mappings)
(s/def ::ident-vec (s/coll-of ident? :kind vector?))
(s/def ::keys ::ident-vec)
(s/def ::strs ::ident-vec)
(s/def ::syms ::ident-vec)
(s/def ::opts (s/keys :opt-un [::as ::or ::keys ::strs ::syms]))
How can I create an ::assoc spec for maps that could be created by merging together a map that conforms to ::mappings and a map that conforms to ::opts? I know that there's merge:
(s/def ::assoc (s/merge ::opts ::mappings))
But this doesn't work, because merge is basically an analogue of and. I'm looking for something that's analogous to or, but for maps.
You can spec hybrid maps using an s/merge of s/keys and s/every of the map as tuples. Here's a simpler example:
(s/def ::a keyword?)
(s/def ::b string?)
(s/def ::m
(s/merge (s/keys :opt-un [::a ::b])
(s/every (s/or :int (s/tuple int? int?)
:option (s/tuple keyword? any?))
:into {})))
(s/valid? ::m {1 2, 3 4, :a :foo, :b "abc"}) ;; true
This simpler formulation has several benefits over a conformer approach. Most importantly, it states the truth. Additionally, it should generate, conform, and unform without further effort.
You can use s/conformer as an intermediate step in s/and to transform your map to the form that’s easy to validate:
(s/def ::assoc
(s/and
map?
(s/conformer #(array-map
::mappings (dissoc % :as :or :keys :strs :syms)
::opts (select-keys % [:as :or :keys :strs :syms])))
(s/keys :opt [::mappings ::opts])))
That will get you from e.g.
{ key :key
:as name }
to
{ ::mappings { key :key }
::opts { :as name } }