clojure - presence of a given set of keys in a clojure map - clojure

There is a map like this
{:buyers [{:name "James" :city "Berlin"} {:name "Jane" :city "Milan"}]
:sellers [{:name "Dustin" :city "Turin" :age "21"} {:name "Mark" :city "Milan"}]}
and I need to check only for :sellers that all the keys :name, :city and :age are present and if one is missing drop
that map all together and have a new structure as below:
{:buyers [{:name "James" :city "Berlin"} {:name "Jane" :city "Milan"}]
:sellers [{:name "Dustin" :city "Turin" :age "21"}]}
I came across validateur and I am trying to use it like:
(:require [validateur.validation :as v])
(def my-map {:buyers [{:name "James" :city "Berlin"} {:name "Jane" :city "Milan"}]
:sellers [{:name "Dustin" :city "Turin" :age "21"} {:name "Dustin" :city "Milan" :age "" } {:city "Rome" :age "22"}]})
(defn valid-seller? [mp]
(let [v-set (v/validation-set
(v/presence-of #{:name :city :age}))]
(fmap vec (v-set mp))))
(map valid-seller? (:sellers my-map))
=> ({} {:age ["can't be blank"]} {:name ["can't be blank"]})
But I do not know how to update my map so missing keys or nil values be dropped

To make the code more readable, I created a new predicate, valid-seller?, and put validation there. You can use any of these versions:
Pure Clojure:
(defn valid-seller? [m]
(every? #(contains? m %) [:name :city :age]))
Spec:
[org.clojure/spec.alpha "0.3.218"], require [clojure.spec.alpha :as spec]
(defn valid-seller? [m]
(spec/valid? (spec/keys :req-un [::name ::city ::age]) m))
Malli (if you also want to test type of values):
[metosin/malli "0.8.9"], require [malli.core :as malli]
(defn valid-seller? [m]
(malli/validate [:map
[:name :string]
[:city :string]
[:age :string]] m))
Then I used this predicate:
(update {:buyers [{:name "James" :city "Berlin"} {:name "Jane" :city "Milan"}]
:sellers [{:name "Dustin" :city "Turin" :age "21"} {:name "Mark" :city "Milan"}]}
:sellers
#(filter valid-seller? %))
=>
{:buyers [{:name "James", :city "Berlin"} {:name "Jane", :city "Milan"}],
:sellers ({:name "Dustin", :city "Turin", :age "21"})}
After your answer, I think you should use Malli, as it also checks the type of values. You can use some? for any non-nil value:
(defn valid-seller? [m]
(malli/validate [:map
[:name some?]
[:city some?]
[:age some?]] m))

(let [data {:buyers [{:name "James" :city "Berlin"} {:name "Jane" :city "Milan"}]
:sellers [{:name "Dustin" :city "Turin" :age "21"} {:name "Mark" :city "Milan"}]}]
(update data :sellers #(filter (every-pred :name :city :age) %)))

Related

Spec: partially overriding generators in a map spec

Assuming I have already defined a spec from which I'd like to generate test data:
(s/def :customer/id uuid?)
(s/def :customer/given-name string?)
(s/def :customer/surname string?)
(s/def :customer/age pos?)
(s/def ::customer
(s/keys
:req-un [:customer/id
:customer/given-name
:customer/surname
:customer/age]))
In generating test data, I'd like to override how ids are generated in order to ensure they're from a smaller pool to encourage collisions:
(defn customer-generator
[id-count]
(gen/let [id-pool (gen/not-empty (gen/vector (s/gen :customer/id) id-count))]
(assoc (s/gen ::customer) :id (gen/element id-pool))))
Is there a way I can simplify this by overriding the :customer/id generator in my test code and then just using (s/gen ::customer)? So, something like the following:
(with-generators [:customer/id (gen/not-empty (gen/vector (s/gen :customer/id) id-count)))]
(s/gen ::customer))
Officially, you can override generators for specs by passing an overrides map to s/gen (See the docstring for more details):
(s/def :customer/id uuid?)
(s/def :customer/given-name string?)
(s/def :customer/surname string?)
(s/def :customer/age nat-int?)
(s/def ::customer
(s/keys
:req-un [:customer/id
:customer/given-name
:customer/surname
:customer/age]))
(def fixed-customer-id (java.util.UUID/randomUUID))
fixed-customer-id
;=> #uuid "c73ff5ea-8702-4066-a31d-bc4cc7015811"
(gen/generate (s/gen ::customer {:customer/id #(s/gen #{fixed-customer-id})}))
;=> {:id #uuid "c73ff5ea-8702-4066-a31d-bc4cc7015811",
; :given-name "1042IKQhd",
; :surname "Uw0AzJzj",
; :age 104}
Alternatively, there is a library for such stuff named genman, which I developed before :)
Using it, you can also write as:
(require '[genman.core :as genman :refer [defgenerator]])
(def fixed-customer-id (java.util.UUID/randomUUID))
(genman/with-gen-group :test
(defgenerator :customer/id
(s/gen #{fixed-customer-id})))
(genman/with-gen-group :test
(gen/generate (genman/gen ::customer)))
Clojure spec uses test.check internally to generate sample values. Here is how test.check can be overridden. Whenever trying to write unit tests with a "fake" function, with-redefs is your friend:
(ns tst.demo.core
(:use tupelo.core tupelo.test)
(:require
[clojure.test.check.generators :as gen]
))
(def id-gen gen/uuid)
(dotest
(newline)
(spyx-pretty (take 3 (gen/sample-seq id-gen)))
(newline)
(with-redefs [id-gen (gen/choose 1 5)]
(spyx-pretty (take 33 (gen/sample-seq id-gen))))
(newline)
)
with result:
-----------------------------------
Clojure 1.10.3 Java 15.0.2
-----------------------------------
Testing tst.demo.core
(take 3 (gen/sample-seq id-gen)) =>
[#uuid "cbfea340-1346-429f-ba68-181e657acba5"
#uuid "7c119cf7-0842-4dd0-a23d-f95b6a68f808"
#uuid "ca35cb86-1385-46ad-8fc2-e05cf7a1220a"]
(take 33 (gen/sample-seq id-gen)) =>
[5 4 3 3 2 2 3 1 2 1 4 1 2 2 4 3 5 2 3 5 3 2 3 2 3 5 5 5 5 1 3 2 2]
Example created
using my favorite template project.
Update
Unfortunately, the above technique does not work for Clojure Spec since (s/def ...) uses a global registery of Spec definitions, and is therefore immune to with-redefs. However, we can overcome this definition by simply redefining the desired spec in the unit test namespace like:
(ns tst.demo.core
(:use tupelo.core tupelo.test)
(:require
[clojure.spec.alpha :as s]
[clojure.spec.gen.alpha :as gen]
))
(s/def :app/id (s/int-in 9 99))
(s/def :app/name string?)
(s/def :app/cust (s/keys :req-un [:app/id :app/name]))
(dotest
(newline)
(spyx-pretty (gen/sample (s/gen :app/cust)))
(newline)
(s/def :app/id (s/int-in 2 5)) ; overwrite the definition of :app/id for testing
(spyx-pretty (gen/sample (s/gen :app/cust)))
(newline))
with result
-----------------------------------
Clojure 1.10.3 Java 15.0.2
-----------------------------------
Testing tst.demo.core
(gen/sample (s/gen :app/cust)) =>
[{:id 10, :name ""}
{:id 9, :name "n"}
{:id 10, :name "fh"}
{:id 9, :name "aI"}
{:id 11, :name "8v5F"}
{:id 10, :name ""}
{:id 10, :name "7"}
{:id 10, :name "3m6Wi"}
{:id 13, :name "OG2Qzfqe"}
{:id 10, :name ""}]
(gen/sample (s/gen :app/cust)) =>
[{:id 3, :name ""}
{:id 3, :name ""}
{:id 2, :name "5e"}
{:id 3, :name ""}
{:id 2, :name "y01C"}
{:id 3, :name "l2"}
{:id 3, :name "c"}
{:id 3, :name "pF"}
{:id 4, :name "0yrxyJ7l"}
{:id 4, :name "40"}]
So, it's a little ugly, but the redefinition of :app/id does the trick, and it only takes effect during unit test runs, leaving the main application unaffected.
user> (def ^:dynamic *idgen* (s/gen uuid?))
#'user/*idgen*
user> (s/def :customer/id (s/with-gen uuid? (fn [] ##'*idgen*)))
:customer/id
user> (s/def :customer/age pos-int?)
:customer/age
user> (s/def ::customer (s/keys :req-un [:customer/id :customer/age]))
:user/customer
user> (gen/sample (s/gen ::customer))
({:id #uuid "d18896f1-6199-42bf-9be3-3d0652583902", :age 1}
{:id #uuid "b6209798-4ffa-4e20-9a76-b3a799a31ec6", :age 2}
{:id #uuid "6f9c6400-8d79-417c-bc62-6b4557f7d162", :age 1}
{:id #uuid "47b71396-1b5f-4cf4-bd80-edf4792300c8", :age 2}
{:id #uuid "808692b9-0698-4fb8-a0c5-3918e42e8f37", :age 2}
{:id #uuid "ba663f0a-7c99-4967-a2df-3ec6cb04f514", :age 1}
{:id #uuid "8521b611-c38c-4ea9-ae84-35c8a2d2ff2f", :age 4}
{:id #uuid "c559d48d-4c50-438f-846c-780cdcdf39d5", :age 3}
{:id #uuid "03c2c114-03a0-4709-b9dc-6d326a17b69d", :age 40}
{:id #uuid "14715a50-81c5-48e4-bffe-e194631bb64b", :age 4})
user> (binding [*idgen* (let [idpool (gen/sample (s/gen :customer/id) 5)] (gen/elements idpool))] (gen/sample (s/gen ::customer)))
({:id #uuid "3e64131d-e7ad-4450-993d-fa651339df1c", :age 2}
{:id #uuid "575b2bef-956d-4c42-bdfa-982c7756a33c", :age 1}
{:id #uuid "575b2bef-956d-4c42-bdfa-982c7756a33c", :age 1}
{:id #uuid "3e64131d-e7ad-4450-993d-fa651339df1c", :age 1}
{:id #uuid "1a2eafed-8242-4229-b432-99edb361569d", :age 3}
{:id #uuid "1a2eafed-8242-4229-b432-99edb361569d", :age 1}
{:id #uuid "05bd521a-26f9-46e0-8b26-f798e0bf0452", :age 3}
{:id #uuid "575b2bef-956d-4c42-bdfa-982c7756a33c", :age 19}
{:id #uuid "31b80714-7ae0-40a0-b932-f7b5f078f2ad", :age 2}
{:id #uuid "05bd521a-26f9-46e0-8b26-f798e0bf0452", :age 5})
user>
A little clumsier than what you wanted, but maybe this is adequate.
You are probably better off using binding rather than with-redefs since binding modifies thread-local bindings, whereas with-redefs changes the root binding.
Since this is for generating bad test data, I'd consider avoiding the use of dynamic vars and binding altogether and just use a different spec that is only local to the test env.

Clojure: Is there a way to remove a key but keep its values

I need some help with a pickle. I have the below map
(def my-list
{:data-0
{:a1 ;;this is the value for :ward-room
{:type "Specialist"
:name "Dr Spongebob"
:illness "dehydration"
:ward-room "a1"}}
:data-1
{:b5
{:type "GP"
:name "Dr Patrick"
:illness "fishy eyes"}
:ward-room "b5"}})
I want to transform this into a map such as below
(def my-new-list
{:data-0
{:type "Specialist"
:name "Dr Spongebob"
:illness "dehydration"
:ward-room "a1"}
:data-1
{:type "GP"
:name "Dr Patrick"
:illness "fishy eyes"}
:ward-room "b5"}})
I couldn't figure out how to do this so I've tried using seq and then flatten but have not had success. Im not too sure how to get started Solving this problem. I would really appreciate a little help
considering corrected structure, with :ward-room having the same nesting level for both entries:
(def my-list
{:data-0
{:a1 ;;this is the value for :ward-room
{:type "Specialist"
:name "Dr Spongebob"
:illness "dehydration"
:ward-room "a1"}}
:data-1
{:b5
{:type "GP"
:name "Dr Patrick"
:illness "fishy eyes"
:ward-room "b5"}}})
there are some options:
first of all you could use reduce-kv, remapping to new vals:
(reduce-kv
(fn [acc k v] (assoc acc k (val (first v))))
{}
my-list)
;; {:data-0
;; {:type "Specialist",
;; :name "Dr Spongebob",
;; :illness "dehydration",
;; :ward-room "a1"},
;; :data-1
;; {:type "GP",
;; :name "Dr Patrick",
;; :illness "fishy eyes",
;; :ward-room "b5"}}
or you could use zipmap to zip keys with new vals:
(zipmap (keys my-list)
(map (comp val first val) my-list))
also there is a nice way to do it with functional composition:
(into {} (map (juxt key (comp val first val))) my-list)
Another approach is to splice the entries for every internal map at the upper level. This would also work for 'malformed' data, like yours:
(def my-list
{:data-0
{:a1 ;;this is the value for :ward-room
{:type "Specialist"
:name "Dr Spongebob"
:illness "dehydration"
:ward-room "a1"}}
:data-1
{:b5
{:type "GP"
:name "Dr Patrick"
:illness "fishy eyes"}
:ward-room "b5"}})
(defn lift-entries [old-entries]
(into {} (mapcat #(if (map? (val %))
(val %)
[%]))
old-entries))
(zipmap (keys my-list) (map lift-entries (vals my-list)))
;; {:data-0
;; {:type "Specialist",
;; :name "Dr Spongebob",
;; :illness "dehydration",
;; :ward-room "a1"},
;; :data-1
;; {:type "GP",
;; :name "Dr Patrick",
;; :illness "fishy eyes",
;; :ward-room "b5"}}
I think your data sample had a cut/paste error. Here is a simple answer:
(def my-data
{:data-0 {:a1 {:type "Specialist"
:name "Dr Spongebob"
:illness "dehydration"
:ward-room "a1"}}
:data-1 {:b5 {:type "GP"
:name "Dr Patrick"
:illness "fishy eyes"
:ward-room "b5"
}}})
(defn delete-intermediate-level
[data]
(into {}
(for [[k1 v1] data]
[k1 (into {}
(for [[k2 v2] v1]
v2))])))
with result:
(delete-intermediate-level my-data) =>
{:data-0
{:type "Specialist",
:name "Dr Spongebob",
:illness "dehydration",
:ward-room "a1"},
:data-1
{:type "GP",
:name "Dr Patrick",
:illness "fishy eyes",
:ward-room "b5"}}
If my-list is supposed to be defined as
(def my-list
{:data-0
{:a1 ;;this is the value for :ward-room
{:type "Specialist"
:name "Dr Spongebob"
:illness "dehydration"
:ward-room "a1"}}
:data-1
{:b5
{:type "GP"
:name "Dr Patrick"
:illness "fishy eyes"
:ward-room "b5"}}})
(that is, if we include the :ward-room key and value in the :b5 map rather than having it floating around on the loose) then the quickest way I can see to do this is:
(zipmap (keys my-list) (map #(first (vals %)) (vals my-list)))
It we wrap the above in a (pprint...) form it returns
{:data-0
{:type "Specialist",
:name "Dr Spongebob",
:illness "dehydration",
:ward-room "a1"},
:data-1
{:type "GP",
:name "Dr Patrick",
:illness "fishy eyes",
:ward-room "b5"}}
EDIT
Or if you prefer it in thread-last form you can use
(->> my-list
(vals)
(map #(first (vals %)))
(zipmap (keys my-list)))

Custom equality in Clojure distinct

In a Clojure program I've an array composed by maps containing peoples' names and emails.
e.g.
[
{ :name "John" :email "john#gmail.com" }
{ :name "Batman" :email "batman#gmail.com" }
{ :name "John Doe" :email "john#gmail.com" }
]
I'd like to remove the duplicate entries considering, for comparison purposes, pairs having the same e-mail to be equals. In the example above the output would be:
[
{ :name "John" :email "john#gmail.com" }
{ :name "Batman" :email "batman#gmail.com" }
]
What's the best way to achieve this in Clojure? Is there a way to let distinct knows what equals function to use?
Thanks.
yet another way to do it, kinda more idiomatic, i guess:
(let [items [{ :name "John" :email "john#gmail.com" }
{ :name "Batman" :email "batman#gmail.com" }
{ :name "John Doe" :email "john#gmail.com" }]]
(map first (vals (group-by :email items))))
output:
({:name "John", :email "john#gmail.com"}
{:name "Batman", :email "batman#gmail.com"})
that is how it works:
(group-by :email items) makes a map, whose keys are emails, and values are groups of records with this email
{"john#gmail.com" [{:name "John", :email "john#gmail.com"}
{:name "John Doe", :email "john#gmail.com"}],
"batman#gmail.com" [{:name "Batman", :email "batman#gmail.com"}]}
then you just need to take its vals (groups of records) and select firsts from them.
And another way is to create a sorted set by email, so it will treat all the records with equal emails as equal records:
(let [items [{ :name "John" :email "john#gmail.com" }
{ :name "Batman" :email "batman#gmail.com" }
{ :name "John Doe" :email "john#gmail.com" }]]
(into (sorted-set-by #(compare (:email %1) (:email %2))) items))
output:
#{{:name "Batman", :email "batman#gmail.com"}
{:name "John", :email "john#gmail.com"}}
don't really know which of them is more idiomatic and has a better performance. But i bet on the first one.
This would do it: https://crossclj.info/fun/medley.core/distinct-by.html.
The function in the link goes through every value lazily and stores everything it's seen. If the value in the coll is already seen, it does not add it.
You could then call this as: (distinct-by #(% :email) maps), where maps is your vector of people-maps.
A distinct-by can easily be implemented as
(defn distinct-by [f coll]
(let [groups (group-by f coll)]
(map #(first (groups %)) (distinct (map f coll)))))
For the example case this can be used like
(distinct-by :email
[{:name "John" :email "john#gmail.com"}
{:name "Batman" :email "batman#gmail.com"}
{:name "John Doe" :email "john#gmail.com"}])

How best to update this tree?

I've got the following tree:
{:start_date "2014-12-07"
:data {
:people [
{:id 1
:projects [{:id 1} {:id 2}]}
{:id 2
:projects [{:id 1} {:id 3}]}
]
}
}
I want to update the people and projects subtrees by adding a :name key-value pair.
Assuming I have these maps to perform the lookup:
(def people {1 "Susan" 2 "John")
(def projects {1 "Foo" 2 "Bar" 3 "Qux")
How could I update the original tree so that I end up with the following?
{:start_date "2014-12-07"
:data {
:people [
{:id 1
:name "Susan"
:projects [{:id 1 :name "Foo"} {:id 2 :name "Bar"}]}
{:id 2
:name "John"
:projects [{:id 1 :name "Foo"} {:id 3 :name "Qux"}]}
]
}
}
I've tried multiple combinations of assoc-in, update-in, get-in and map calls, but haven't been able to figure this out.
I have used letfn to break down the update into easier to understand units.
user> (def tree {:start_date "2014-12-07"
:data {:people [{:id 1
:projects [{:id 1} {:id 2}]}
{:id 2
:projects [{:id 1} {:id 3}]}]}})
#'user/tree
user> (def people {1 "Susan" 2 "John"})
#'user/people
user> (def projects {1 "Foo" 2 "Bar" 3 "Qux"})
#'user/projects
user>
(defn integrate-tree
[tree people projects]
;; letfn is like let, but it creates fn, and allows forward references
(letfn [(update-person [person]
;; -> is the "thread first" macro, the result of each expression
;; becomes the first arg to the next
(-> person
(assoc :name (people (:id person)))
(update-in [:projects] update-projects)))
(update-projects [all-projects]
(mapv
#(assoc % :name (projects (:id %)))
all-projects))]
(update-in tree [:data :people] #(mapv update-person %))))
#'user/integrate-tree
user> (pprint (integrate-tree tree people projects))
{:start_date "2014-12-07",
:data
{:people
[{:projects [{:name "Foo", :id 1} {:name "Bar", :id 2}],
:name "Susan",
:id 1}
{:projects [{:name "Foo", :id 1} {:name "Qux", :id 3}],
:name "John",
:id 2}]}}
nil
Not sure if entirely the best approach:
(defn update-names
[tree people projects]
(reduce
(fn [t [id name]]
(let [person-idx (ffirst (filter #(= (:id (second %)) id)
(map-indexed vector (:people (:data t)))))
temp (assoc-in t [:data :people person-idx :name] name)]
(reduce
(fn [t [id name]]
(let [project-idx (ffirst (filter #(= (:id (second %)) id)
(map-indexed vector (get-in t [:data :people person-idx :projects]))))]
(if project-idx
(assoc-in t [:data :people person-idx :projects project-idx :name] name)
t)))
temp
projects)))
tree
people))
Just call it with your parameters:
(clojure.pprint/pprint (update-names tree people projects))
{:start_date "2014-12-07",
:data
{:people
[{:projects [{:name "Foo", :id 1} {:name "Bar", :id 2}],
:name "Susan",
:id 1}
{:projects [{:name "Foo", :id 1} {:name "Qux", :id 3}],
:name "John",
:id 2}]}}
With nested reduces
Reduce over the people to update corresponding names
For each people, reduce over projects to update corresponding names
The noisesmith solution looks better since doesn't need to find person index or project index for each step.
Naturally you tried to assoc-in or update-in but the problem lies in your tree structure, since the key path to update John name is [:data :people 1 :name], so your assoc-in code would look like:
(assoc-in tree [:data :people 1 :name] "John")
But you need to find John's index in the people vector before you can update it, same things happens with projects inside.

What is the idiomatic way to assoc several keys/values in a nested map in Clojure?

Imagine you have a map like this:
(def person {
:name {
:first-name "John"
:middle-name "Michael"
:last-name "Smith" }})
What is the idiomatic way to change values associated with both :first-name and :last-name in one expression?
(Clarification: Let's say you want to set :first-name to "Bob" and :last-name to "Doe". Let's also say that this map has some other values in it that we want to preserve, so constructing it from scratch is not an option)
Here are a couple of ways.
user> (update-in person [:name] assoc :first-name "Bob" :last-name "Doe")
{:name {:middle-name "Michael", :last-name "Doe", :first-name "Bob"}}
user> (update-in person [:name] merge {:first-name "Bob" :last-name "Doe"})
{:name {:middle-name "Michael", :last-name "Doe", :first-name "Bob"}}
user> (update-in person [:name] into {:first-name "Bob" :last-name "Doe"})
{:name {:middle-name "Michael", :last-name "Doe", :first-name "Bob"}}
user> (-> person
(assoc-in [:name :first-name] "Bob")
(assoc-in [:name :last-name] "Doe"))
{:name {:middle-name "Michael", :last-name "Doe", :first-name "Bob"}}
Edit
update-in does recursive assocs on your map. In this case it's roughly equivalent to:
user> (assoc person :name
(assoc (:name person)
:first-name "Bob"
:last-name "Doe"))
The repetition of keys becomes more and more tedious as you go deeper into a series of nested maps. update-in's recursion lets you avoid repeating keys (e.g. :name) over and over; intermediary results are stored on the stack between recursive calls. Take a look at the source for update-in to see how it's done.
user> (def foo {:bar {:baz {:quux 123}}})
#'user/foo
user> (assoc foo :bar
(assoc (:bar foo) :baz
(assoc (:baz (:bar foo)) :quux
(inc (:quux (:baz (:bar foo)))))))
{:bar {:baz {:quux 124}}}
user> (update-in foo [:bar :baz :quux] inc)
{:bar {:baz {:quux 124}}}
assoc is dynamic (as are update-in, assoc-in, and most other Clojure functions that operate on Clojure data structures). If assoc onto a map, it returns a map. If you assoc onto a vector, it returns a vector. Look at the source for assoc and take a look in in RT.java in the Clojure source for details.
I'm not sure if my case is quite the same but I had list of changes to apply:
(def updates [[[:name :first-name] "Bob"]
[[:name :last-name] "Doe"]])
In that case you can reduce over that list with assoc-in like this:
(reduce #(assoc-in %1 (first %2) (second %2)) person updates)