i have a set of data like this
{
"data": [
{
"target_group_id": "1234",
"target_group_name": "abc",
"targets": [
{
"target_id": "456",
"target_name": "john"
}
]
},
{
"target_group_id": "56789",
"target_group_name": "cdes",
"targets": [
{
"target_id": "0987",
"target_name": "john"
}
]
},
{
"target_group_id": "1234",
"target_group_name": "abc",
"targets": [
{
"target_id": "789",
"target_name": "doe"
}
]
}
]
}
and want to transform by grouping and merging data by target group id so the target within the same target_group_id will be added to the existing target group and changing the key root of data from "data" into "target_groups"
{
"target_groups": [
{
"target_group_id": "1234",
"target_group_name": "abc",
"targets": [
{
"target_id": "456",
"target_name": "john"
},
{
"target_id": "789",
"target_name": "doe"
}
]
},
{
"target_group_id": "56789",
"target_group_name": "cdes",
"targets": [
{
"target_id": "0987",
"target_name": "john"
}
]
}
]
}
is there any effective way to do it with clojure since my original code using php and take a lot of "if-clause" and "foreach"? thanks...
Using just core clojure (with the data.json library).
First, acquire and unwrap our data:
(def data (-> "grouping-and-merging.json"
slurp
clojure.data.json/read-str
(get "data")))
When we address groups of targets, we are going to need to concatenate them. I was doing this inline, but it looks messy in the reduce, so here's a helper function:
(defn concat-targets [acc item]
(update acc "targets" concat (item "targets")))
Then let's do the work!
(def output (->> data
(group-by #(get % "target_group_id"))
vals
(map #(reduce concat-targets %))
(assoc {} "target_groups")
clojure.data.json/write-str))
I'm feeling lucky that I got away with the threading macros working so well, although you'll note I had to switch from pre-threading to post-threading between the two phases. Normally I find myself wanting something like the Tupelo it-> used in Alan's answer.
I also feel like the reduce is cheating slightly - I am assuming that there won't be any subtleties and that just taking any extra keys from the first item will be sufficient.
Another way to do the transformation:
{"target_groups" (map merge-vector (-> "data.json"
slurp
json/read-str
(get "data")
(set/index ["target_group_id" "target_group_name"])
vals))}
;; =>
{"target_groups"
({"target_group_id" "1234",
"target_group_name" "abc",
"targets"
({"target_id" "789", "target_name" "doe"}
{"target_id" "456", "target_name" "john"})}
{"target_group_id" "56789",
"target_group_name" "cdes",
"targets" [{"target_id" "0987", "target_name" "john"}]})}
The intermediary data structure is a sequence of set indexed by group id and group name (like using group-by). I.e.
(-> "data.json"
slurp
json/read-str
(get "data")
(set/index ["target_group_id" "target_group_name"])
vals)
;; =>
(#{{"target_group_id" "1234",
"target_group_name" "abc",
"targets" [{"target_id" "789", "target_name" "doe"}]}
{"target_group_id" "1234",
"target_group_name" "abc",
"targets" [{"target_id" "456", "target_name" "john"}]}}
#{{"target_group_id" "56789",
"target_group_name" "cdes",
"targets" [{"target_id" "0987", "target_name" "john"}]}})
The targets (which is a vector) are then concat together with merge-vector:
(def merge-vector
(partial apply
merge-with
(fn [& xs] (if (every? vector? xs) (apply concat xs) (last xs)))))
Here is how I would approach it:
(ns tst.demo.core
(:use demo.core tupelo.core tupelo.test)
(:require
[clojure.string :as str]
[tupelo.string :as ts]
[tupelo.core :as t]))
(def data-json
"{ 'data': [
{ 'target_group_id': '1234',
'target_group_name': 'abc',
'targets': [
{ 'target_id': '456',
'target_name': 'john' }
]
},
{ 'target_group_id': '56789',
'target_group_name': 'cdes',
'targets': [
{ 'target_id': '0987',
'target_name': 'john' }
]
},
{
'target_group_id': '1234',
'target_group_name': 'abc',
'targets': [
{ 'target_id': '789',
'target_name': 'doe' }
]
}
]
} " )
with transformation:
(dotest
(let [data-edn (t/json->edn
(ts/quotes->double data-json))
d2 (t/it-> data-edn
(:data it) ; unnest from :data key
(group-by :target_group_id it ) )
d3 (t/forv [[tgt-id entries] d2]
{:tgt-group-id tgt-id
:tgt-group-name (:target_group_name (first entries))
:targets-all (mapv :targets entries)}) ]
and results/tests:
(is= data-edn
{:data
[{:target_group_id "1234",
:target_group_name "abc",
:targets [{:target_id "456", :target_name "john"}]}
{:target_group_id "56789",
:target_group_name "cdes",
:targets [{:target_id "0987", :target_name "john"}]}
{:target_group_id "1234",
:target_group_name "abc",
:targets [{:target_id "789", :target_name "doe"}]}]})
(is= d2
{"1234"
[{:target_group_id "1234",
:target_group_name "abc",
:targets [{:target_id "456", :target_name "john"}]}
{:target_group_id "1234",
:target_group_name "abc",
:targets [{:target_id "789", :target_name "doe"}]}],
"56789"
[{:target_group_id "56789",
:target_group_name "cdes",
:targets [{:target_id "0987", :target_name "john"}]}]})
(is= d3
[{:tgt-group-id "1234",
:tgt-group-name "abc",
:targets-all [[{:target_id "456", :target_name "john"}]
[{:target_id "789", :target_name "doe"}]]}
{:tgt-group-id "56789",
:tgt-group-name "cdes",
:targets-all [[{:target_id "0987", :target_name "john"}]]}]) ))
Related
a={
"car": "Nissan",
"from": [
{
"Japan1": "People1",
"make_type": "allow",
"driver": {
"id": "12345",
"name": "user1",
"type": "user"
}
},
{
"Japan1": "People1",
"make_type": "allow",
"driver": {
"id": "98765",
"name": "user2",
"type": "user"
}
}
]}
Objective: want to remove "name" and "type" inside "driver"
I have tried a lot of method, if use print the name or type, just put into a new var level by level.
But how can I delete it?
Thanks
You can use del:
for f in a["from"]:
del f["driver"]["name"]
del f["driver"]["type"]
print(a)
Prints:
{
"car": "Nissan",
"from": [
{"Japan1": "Peop1", "make_type": "plastic", "driver": {"id": "12345"}}
],
}
Or:
for f in a["from"]:
f["driver"] = {
k: v for k, v in f["driver"].items() if k not in {"name", "type"}
}
I need to store values from response data (below) in variable.
Response Body:
{
"data": {
"packages": [
{
"object": "AAA",
"id": "BBB",
"code": "123",
"name": "Test8",
"description": "Test111",
"fee": "130.00 bath",
"productSeq": "3"
}, {
"object": "AAA",
"id": "CCC",
"code": "456",
"name": "Test9",
"description": "Test222",
"fee": "80.00 bath",
"productSeq": "2"
}, {
"object": "AAA",
"id": "CCC",
"code": "789",
"name": "Test10",
"description":"Test111",
"fee": "70.00 bath",
"productSeq": "1"
}
]
},
"resultCode": "20000",
"resultDesc": "Success",
"developerMessage": "Success"
}
If the id value of an object in the response is "CCC", I need to store that data like this, in a variable:
{
"object": "AAA",
"id": "CCC",
"code": "456",
"name": "Test9",
"description": "Test222",
"fee": "80.00 bath",
"productSeq": "2"
}, {
"object": "AAA",
"id": "CCC",
"code": "789",
"name": "Test10",
"description":"Test111",
"fee": "70.00 bath",
"productSeq": "1"
}
Can you help me create the script that would do that?
You can do it like this: -
var body = JSON.parse(responseBody)
var objArray = [];
for (i=0; i<body.data.packages.length;i++){
if (body.data.packages[i].id === "CCC"){
objArray.push(body.data.packages[i]);
}
}
postman.setEnvironmentVariable("varName", objArray)
I am trying to sort the below map( Clojure) on the basis of "col_nm" field, but unable to do so.
{:Mawb {:user_val "3", :col_nm "1"},
:HawbDate {:user_val "", :col_nm "3"},
:EtlBatchID {:user_val "1", :col_nm "2"}}
The output should be:
{:Mawb {:user_val "3", :col_nm "1"},
:EtlBatchID {:user_val "1", :col_nm "2"},
:HawbDate {:user_val "", :col_nm "3"} }
Can anyone help me, thanks in advance.
Try this one:
(def m {:Mawb {:user_val "3", :col_nm "1"},
:HawbDate {:user_val "", :col_nm "3"},
:EtlBatchID {:user_val "1", :col_nm "2"}})
(sort-by (comp :col_nm second) m)
=> ([:Mawb {:user_val "3", :col_nm "1"}]
[:EtlBatchID {:user_val "1", :col_nm "2"}]
[:HawbDate {:user_val "", :col_nm "3"}])
I'm trying to clean up some JSON data in Clojure. Some values in the JSON document are encapsulated in objects with associated (and no longer needed) metadata. I start with a JSON document like:
{ "household": {
"address": {
"street": { "value": "123 Fire Ln", "foo": "bar1" },
"zip": { "value": "01234", "foo": "bar2" }
},
"persons": [
{
"id": "0001",
"name": { "value": "John Smith", "foo": "bar3" }
},
{
"id": "0002",
"name": { "value": "Jane Smith", "foo": "bar4" }
}
]
} }
Using Cheshire I parse this JSON and get the following data structure:
{ "household" {
"address" {
"street" {"value" "123 Fire Ln", "foo" "bar1"},
"zip" {"value" "01234", "foo" "bar2"}
},
"persons" [
{"id" "0001", "name" {"value" "John Smith", "foo" "bar3"}}
{"id" "0002", "name" {"value" "Jane Smith", "foo" "bar4"}}
]
} }
My goal is to "collapse" those nested maps with a "value" key, drop the "foo" assoc, and assign the value to the map key one level higher (e.g., "street", "zip", "name"). The resulting data structure would look like:
{ "household" {
"address" {
"street" "123 Fire Ln",
"zip" "01234"
},
"persons" [
{"id" "0001", "name" "John Smith"}
{"id" "0002", "name" "Jane Smith"}
]
} }
Any help here would be wonderful, thanks!
Sounds like a job for clojure.walk/postwalk!
(defn collapse [obj]
(postwalk (fn [obj]
(or (and (map? obj)
(get obj "value"))
obj))
obj))
You can actually shorten this substantially because get is willing to work on non-map objects (it just returns nil), but I think it's a lot more clear what is going on in the first version.
(defn collapse [obj]
(postwalk #(get % "value" %) obj))
I've a vector of maps like this:
[{:categoryid 1, :categoryname "foo" }
{:categoryid 2, :categoryname "bar" }
{:categoryid 3, :categoryname "baz" }]
and would like to generate a map of maps like this for searching by categoryname
{"foo" {:categoryid 1, :categoryname "foo" },
"bar" {:categoryid 2, :categoryname "bar" },
"baz" {:categoryid 3, :categoryname "baz" }}
How can I do this?
Another way: (into {} (map (juxt :categoryname identity) [...]))
(reduce (fn [m {catname :categoryname :as input}]
(assoc m catname input))
{}
[{:categoryid 1, :categoryname "foo" }
{:categoryid 2, :categoryname "bar" }
{:categoryid 3, :categoryname "baz" }])
Better yet,
(#(zipmap (map :categoryname %) %)
[{:categoryid 1, :categoryname "foo" }
{:categoryid 2, :categoryname "bar" }
{:categoryid 3, :categoryname "baz" }])
(ns code.groupby
(:use clojure.contrib.seq-utils))
(def vector-of-maps [ {:categoryid 1, :categoryname "foo" }
{:categoryid 2, :categoryname "bar" }
{:categoryid 3, :categoryname "baz" } ])
(group-by :categoryname vector-of-maps)
Gives you a map of Vectors of maps
{"bar" [{:categoryid 2, :categoryname "bar"}],
"baz" [{:categoryid 3, :categoryname "baz"}],
"foo" [{:categoryid 1, :categoryname "foo"}]}
(which I now realise isn't what you wanted...sorry)
I haven't tested it in Clojure, but in ClojureScript this variant appears to be faster than others:
(reduce #(assoc %1 (:categoryname %2) %2) {} [...])