I'm trying Interactive Development with clojure.spec and have a problem with specs for function arguments that cannot change a lot. For instance, if a function receives a file name as argument, I can write the following spec:
(s/def ::file-name string?)
(s/fdef test-fn :args (s/cat :x ::file-name))
If I exercise it:
(s/exercise-fn `test-fn)
The function will be tested with lots of random file names that will fail to read any file. It is possible to limit the file names to a set of valid and invalid file names. That is fine for testing, but it will make the spec specific to the chosen set.
That is a problem not only with file names, but with any complex data structure where even small random changes may render it useless.
What should I do? Any relevant technique or good practice?
That is fine for testing, but it will make the spec specific to the chosen set.
This is where custom generators are useful:
(s/def ::file-name
(s/with-gen string? #(gen/elements #{"good.txt" "bad.txt"})))
(s/fdef test-fn :args (s/cat :x ::file-name))
(Where gen is clojure.test.check.generators or clojure.spec.gen.alpha.)
Now your spec's predicate is still string? but the values generated from this spec will always be from #{"good.txt" "bad.txt"}. You can compose generators in several ways, for example you could make a generator that took from a string set ~50% of the time and generated a purely "random" string for the other ~50%.
FYI clojure.spec.test.alpha/check also takes an opts map that allows you to override/specify generators.
Related
I'm quite new to functional programming and clojure. I like it.
I'd like to know what the community thinks about following fn-naming approach:
Is it a feasible way to go with such naming, or is it for some reason something to avoid?
Example:
(defn coll<spec>?
"Checks wether the given thing is a collection of elements of ::spec-type"
[spec-type thing]
(and (coll? thing)
(every? #(spec/valid? spec-type %)
thing)))
(defn coll<spec>?nil
"Checks if the given thing is a collection of elements of ::spec-type or nil"
[spec-type thing]
(or (nil? thing)
(coll<spec>? spec-type thing)))
now ... somewhere else I'm using partial applications in clojure defrecrod/specs ...
similar to this:
; somewhere inside a defrecord
^{:spec (partial p?/coll<spec>?nil ::widgets)} widgets component-widgets
now, I created this for colls, sets, hash-maps with a generic-like form for specs (internal application of spec/valid? ...) and with a direct appication of predicates and respectively a <p> instead of the <spec>
currently I'm discussing with my colleagues wether this is a decent and meaningful and useful approach - since it is at least valid to name function like this in clojure - or wether the community thinks this is rather a no-go.
I'd very much like your educated opinions on that.
It's a weird naming convention, but a lot of projects use weird naming conventions because their authors believe they help. I imagine if I read your codebase, variable naming conventions would probably not be the thing that surprises me most. This is not an insult: every project has some stuff that surprises me.
But I don't see why you need these specific functions at all. As Sean Corfield says in a comment, there are good spec combinator functions to build fancier specs out of simpler ones. Instead of writing coll<spec>?, you could just combine s/valid? with s/coll-of:
(s/valid? (s/coll-of spec-type) thing)
Likewise you can use s/or and nil? to come up with coll<spec>?nil.
(s/valid? (s/or nil? (s/coll-of spec-type) thing))
This is obviously more to write at each call site than a mere coll<spec>?nil, so you may dismiss my advice. But the point is you can extract functions for defining these specs, and use those specs.
(defn nil-or-coll-of [spec]
(s/or nil? (s/coll-of spec)))
(s/valid? (nil-or-coll-of spec-type) thing)
Importantly, this means you could pass the result of nil-or-coll-of to some other function that expects a spec, perhaps to build a larger spec out of it, or to try to conform it. You can't do that if all your functions have s/valid? baked into them.
Think in terms of composing general functions, not defining a new function from scratch for each thing you want to do.
I've been trying out clojure.spec, and one idea I have for how to use it is to generate the UI for editing an instance of the map I'm specifying. For example, it might generate a web form with a datepicker field for a key that's specified to be a date, that kind of thing.
There is a get-spec method in the library, but it seems like there are no functions that operate on specifications-as-values in the way I need. Is there some way to do things like take a map spec and get back the required keys for that map as a vector? Is this kind of metaprogramming with specifications outside of the intended use case of clojure.spec?
Metaprogramming with specs is definitely within the intended use case of clojure.spec.
We have not yet released (but have written and intend to) specs for spec forms themselves. With these, it is possible to conform a spec form itself and get back a data structure representing the spec which can be used to (for example), grab the required keys from a map spec.
Conforming with a ::spec spec might look something like this:
user=> (s/def ::name string?)
:user/name
user=> (s/def ::m (s/keys :req [::name]))
:user/m
user=> (s/conform ::spec (s/form ::m))
[:form {:s clojure.spec/keys, :args {:req [[:key :user/name]]}}]
You could then pluck the set of keys out of that structure.
I want to write a spec with keys/keys* but being able to inline the value specs, which is not supported by design, and I get the reasoning behind it. However sometimes you do want (or simply have, by legacy or 3rd-party) coupling between keys and values when there is specific context to the map.
I'm still new to spec and this is just my first time integrating it with an existing project and it constantly gives me issues because it is assuming way too much, especially because of the reason mentioned above. E.g. imagine a map that describes a time period and has an until key for a date, and in the same ns there's a map for list processing and there's an until that takes a predicate function. I now need to mess with manually writing fully namespaced keys for namespaces that don't even exist (aliasing is cute but it would have to be duplicated constantly across multiple namespaces/files). Aside from being irritating I feel like it's also error-prone.
And another place where keys/keys* assumes too much is if I even want keywords as my keys. I'm writing a DSL for non-programmers but technical users right now, and bottom line is that I want to spec a map with symbols as keys. That doesn't seem to be supported whatsoever.
Is there something I'm not getting? or is spec really missing essential functionality?
You can spec a map with symbols as keys either with map-of:
(s/def ::sm (s/map-of symbol? any?))
or by spec'ing the map as a collection of entries:
(s/def ::sm (s/every (s/tuple symbol? any?) :kind map? :into {}))
The latter is particularly interesting as instead of a single tuple you can s/or many different kinds of tuples to describe more interesting maps. You can even connect those symbols to other existing specs in this manner:
(s/def ::attr1 int?)
(s/def ::attr2 boolean?)
(s/def ::sm (s/every (s/or :foo (s/tuple #{'foo} ::attr1)
:bar (s/tuple #{'bar} ::attr2))
:kind map? :into {}))
(s/valid? ::sm {'foo 10 'bar true}) ;; => true
I now need to mess with manually writing fully namespaced keys for
namespaces that don't even exist
I've been using this approach as well, and I think I actually like it more than I like making sure that your keywords' namespaces always correspond to real Clojure NS forms. I use keywords like :business-domain-concept/a-name rather than :my-project.util.lists/a-name.
You can make keywords with arbitrary namespaces that don't map to any Clojure NS. For instance, in your until situation, you could define a :date/until spec that describes dates, and a (perhaps there's a better name for this one) :list/until spec that describes your list processing map's field.
It sounds like you're already aware of this arbitrary-keyword-namespace approach - in particular, I buy that it feels error-prone, since you're typing this stuff out by hand and spec doesn't appear to choke if you feed your s/keys a :fate/until by accident. FWIW, though, I think you're currently feeling the pain that namespaced keywords are specifically intended to solve: you're in one Clojure file, you've got two maps with keys named until, and they mean two completely different things.
I'm writing a DSL for non-programmers but technical users right now,
and bottom line is that I want to spec a map with symbols as keys.
I think that map-of is what you want here:
user=> (s/def ::valid-symbols #{'foo 'bar 'baz})
:user/valid-symbols
user=> (s/def ::symbol-map (s/map-of ::valid-symbols int?))
:user/symbol-map
user=> (s/valid? ::symbol-map {'foo 1 'bar 3})
true
user=> (s/valid? ::symbol-map {'foo 1 'quux 3})
false
I have recently watched Rich Hickeys talk at Cojure Conj 2016 and although it was very interesting, I didn't really understand the point in clojure.spec or when you'd use it. It seemed like most of the ideas, such as conform, valid etc, had similar functions in Clojure already.
I have only been learning clojure for around 3 months now so maybe this is due to lack of programming/Clojure experience.
Do clojure.spec and cljs.spec work in similar ways to Clojure and Cljs in that, although they are not 100% the same, they are based on the same underlying principles.
Are you tired of documenting your programs?
Does the prospect of making up yet more tests cause procrastination?
When the boss says "test coverage", do you cower with fear?
Do you forget what your data names mean?
For smooth expression of hard specifications, you need Clojure.Spec!
Clojure.spec gives you a uniform method of documenting, specifying, and automatically testing your programs, and of validating your live data.
It steals virtually every one of its ideas. And it does nothing you can't do for yourself.
But in my - barely informed - opinion, it changes the economy of specification, making it worth while doing properly. A game-changer? - quite possibly.
At the clojure/conj conference last week, probably half of the presentations featured spec in some way, and it's not even out of alpha yet. spec is a major feature of clojure; it is here to stay, and it is powerful.
As an example of its power, take static type checking, hailed as a kind of safety net by so many, and a defining characteristic of so many programming languages. It is incredibly limited in that it's only good at compile time, and it only checks types. spec, on the other hand, validates and conforms any predicate (not just type) for the args, the return, and can also validate relationships between the two. All of this is external to the function's code, separating the logic of the function from being commingled with validation and documentation about the code.
Regarding WORKFLOW:
One archetypal example of the benefits of relationship-checking, versus only type-checking, is a function which computes the substring of a string. Type checking ensures that in (subs s start end) the s is a string and start and end are integers. However, additional checking must be done within the function to ensure that start and end are positive integers, that end is greater than start, and that the resulting substring is no larger than the original string. All of these things can be spec'd out, for example (forgive me if some of this is a bit redundant or maybe even inaccurate):
(s/fdef clojure.core/subs
:args (s/and (s/cat :s string? :start nat-int? :end (s/? nat-int?))
(fn [{:keys [s start end]}]
(if end
(<= 0 start end (count s))
(<= 0 start (count s)))))
:ret string?
:fn (fn [{{:keys [s start end]} :args, substring :ret}]
(and (if end
(= (- end start) (count substring))
(= (- (count s) start) (count substring)))
(<= (count substring) (count s)))))
Call the function with sample data meeting the above args spec:
(s/exercise-fn `subs)
Or run 1000 tests (this may fail a few times, but keep running and it will work--this is due to the built-in generator not being able to satisfy the second part of the :args predicate; a custom generator can be written if needed):
(stest/check `subs)
Or, want to see if your app makes calls to subs that are invalid while it's running in real time? Just run this, and you'll get a spec exception if the function is called and the specs are not met:
(stest/instrument `subs)
We have not integrated this into our work flow yet, and can't in production since it's still alpha, but the first goal is to write specs. I'm putting them in the same namespace but in separate files currently.
I foresee our work flow being to run the tests for spec'd functions using this (found in the clojure spec guide):
(-> (stest/enumerate-namespace 'user) stest/check)
Then, it would be advantageous to turn on instrumenting for all functions, and run the app under load as we normally would test it, and ensure that "real world" data works.
You can also use s/conform to destructure complex data in functions themselves, or use s/valid as pre- and post- conditions for running functions. I'm not too keen on this, as it's overhead in a production system, but it is a possibility.
The sky's the limit, and we've just scratched the surface! Cool things coming in the next months and years with spec!
How have you used metadata in your Clojure program?
I saw one example from Programming Clojure:
(defn shout [#^{:tag String} message] (.toUpperCase message))
;; Clojure casts message to String and then calls the method.
What are some uses? This form of programming is completely new to me.
Docstrings are stored as metadata under the :doc key. This is probably the number 1 most apparent use of metadata.
Return and parameter types can be optionally tagged with metadata to improve performance by avoiding the overhead of reflecting on the types at runtime. These are also known as "type hints." #^String is a type hint.
Storing things "under the hood" for use by the compiler, such as the arglist of a function, the line number where a var has been defined, or whether a var holds a reference to a macro. These are usually automatically added by the compiler and normally don't need to be manipulated directly by the user.
Creating simple testcases as part of a function definition:
(defn #^{:test (fn [] (assert true))} something [] nil)
(test #'something)
If you are reading Programming Clojure, then Chapter 2 provides a good intro to metadata. Figure 2.3 provides a good summary of common metadata.
For diversity some answer, which does not concentrate on interaction with the language itself:
You can also eg. track the source of some data. Unchecked input is marked as :tainted. A validator might check things and then set the status to :clean. Code doing security relevant things might then barf on :tainted and only accept :cleaned input.
Meta Data was extremely useful for me for purposes of typing. I'm talking not just about type hints, but about complete custom type system. Simplest example - overloading of print-method for structs (or any other var):
(defstruct my-struct :foo :bar :baz)
(defn make-my-struct [foo bar baz]
(with-meta (struct-map my-struct :foo foo :bar baz :baz baz)
{:type ::my-struct}))
(defmethod print-method
[my-struct writer]
(print-method ...))
In general, together with Clojure validation capabilities it may increase safety and, at the same time, flexibility of your code very very much (though it will take some more time to do actual coding).
For more ideas on typing see types-api.
metadata is used by the compiler extensively for things like storing the type of an object.
you use this when you give type hints
(defn foo [ #^String stringy] ....
I have used it for things like storing the amount of padding that was added to a number. Its intended for information that is 'orthogonal' to the data and should not be considered when deciding if you values are the same.