Clojure: How to collapse nested maps with a specific key? - clojure

I'm trying to clean up some JSON data in Clojure. Some values in the JSON document are encapsulated in objects with associated (and no longer needed) metadata. I start with a JSON document like:
{ "household": {
"address": {
"street": { "value": "123 Fire Ln", "foo": "bar1" },
"zip": { "value": "01234", "foo": "bar2" }
},
"persons": [
{
"id": "0001",
"name": { "value": "John Smith", "foo": "bar3" }
},
{
"id": "0002",
"name": { "value": "Jane Smith", "foo": "bar4" }
}
]
} }
Using Cheshire I parse this JSON and get the following data structure:
{ "household" {
"address" {
"street" {"value" "123 Fire Ln", "foo" "bar1"},
"zip" {"value" "01234", "foo" "bar2"}
},
"persons" [
{"id" "0001", "name" {"value" "John Smith", "foo" "bar3"}}
{"id" "0002", "name" {"value" "Jane Smith", "foo" "bar4"}}
]
} }
My goal is to "collapse" those nested maps with a "value" key, drop the "foo" assoc, and assign the value to the map key one level higher (e.g., "street", "zip", "name"). The resulting data structure would look like:
{ "household" {
"address" {
"street" "123 Fire Ln",
"zip" "01234"
},
"persons" [
{"id" "0001", "name" "John Smith"}
{"id" "0002", "name" "Jane Smith"}
]
} }
Any help here would be wonderful, thanks!

Sounds like a job for clojure.walk/postwalk!
(defn collapse [obj]
(postwalk (fn [obj]
(or (and (map? obj)
(get obj "value"))
obj))
obj))
You can actually shorten this substantially because get is willing to work on non-map objects (it just returns nil), but I think it's a lot more clear what is going on in the first version.
(defn collapse [obj]
(postwalk #(get % "value" %) obj))

Related

Sorting a list into smaller lists according to a characteristic in dart

Sorry for my poor explanation, I just started learning dart.
With a mock service and a json file I created a set amount of items
Example:
{
"items": [
{
"id": "01",
"type": "a"
},
{
"id": "02",
"type": "b"
},
{
"id": "03",
"type": "c"
}
]
}
when creating the list on the service it creates a single list like this:
if (json['items'] != null) {
final itemList = <ItemList>[];
json['items'].forEach((v) {
itemlistList.add(ItemList.fromJson(v));
});
return ItemList;
} else {
return [];
}
is there a way to, form the create list step to already separate them into 3 different lists for the type a, b, and c items? and if now, where and how would I divide this itemlist into 3 based on the type characteristic of each item?
Using groupBy, as suggested in this extremely similar question: Flutter/Dart how to groupBy list of maps?
import "package:collection/collection.dart";
main(List<String> args) {
var data = [
{"id": "01", "type": "a"},
{"id": "02", "type": "b"},
{"id": "03", "type": "c"},
{"id": "04", "type": "a"},
{"id": "05", "type": "a"},
{"id": "06", "type": "b"},
];
var newMap = groupBy(data, (Map obj) => obj["type"]);
print(newMap);
}

Grouping and merging by value using clojure?

i have a set of data like this
{
"data": [
{
"target_group_id": "1234",
"target_group_name": "abc",
"targets": [
{
"target_id": "456",
"target_name": "john"
}
]
},
{
"target_group_id": "56789",
"target_group_name": "cdes",
"targets": [
{
"target_id": "0987",
"target_name": "john"
}
]
},
{
"target_group_id": "1234",
"target_group_name": "abc",
"targets": [
{
"target_id": "789",
"target_name": "doe"
}
]
}
]
}
and want to transform by grouping and merging data by target group id so the target within the same target_group_id will be added to the existing target group and changing the key root of data from "data" into "target_groups"
{
"target_groups": [
{
"target_group_id": "1234",
"target_group_name": "abc",
"targets": [
{
"target_id": "456",
"target_name": "john"
},
{
"target_id": "789",
"target_name": "doe"
}
]
},
{
"target_group_id": "56789",
"target_group_name": "cdes",
"targets": [
{
"target_id": "0987",
"target_name": "john"
}
]
}
]
}
is there any effective way to do it with clojure since my original code using php and take a lot of "if-clause" and "foreach"? thanks...
Using just core clojure (with the data.json library).
First, acquire and unwrap our data:
(def data (-> "grouping-and-merging.json"
slurp
clojure.data.json/read-str
(get "data")))
When we address groups of targets, we are going to need to concatenate them. I was doing this inline, but it looks messy in the reduce, so here's a helper function:
(defn concat-targets [acc item]
(update acc "targets" concat (item "targets")))
Then let's do the work!
(def output (->> data
(group-by #(get % "target_group_id"))
vals
(map #(reduce concat-targets %))
(assoc {} "target_groups")
clojure.data.json/write-str))
I'm feeling lucky that I got away with the threading macros working so well, although you'll note I had to switch from pre-threading to post-threading between the two phases. Normally I find myself wanting something like the Tupelo it-> used in Alan's answer.
I also feel like the reduce is cheating slightly - I am assuming that there won't be any subtleties and that just taking any extra keys from the first item will be sufficient.
Another way to do the transformation:
{"target_groups" (map merge-vector (-> "data.json"
slurp
json/read-str
(get "data")
(set/index ["target_group_id" "target_group_name"])
vals))}
;; =>
{"target_groups"
({"target_group_id" "1234",
"target_group_name" "abc",
"targets"
({"target_id" "789", "target_name" "doe"}
{"target_id" "456", "target_name" "john"})}
{"target_group_id" "56789",
"target_group_name" "cdes",
"targets" [{"target_id" "0987", "target_name" "john"}]})}
The intermediary data structure is a sequence of set indexed by group id and group name (like using group-by). I.e.
(-> "data.json"
slurp
json/read-str
(get "data")
(set/index ["target_group_id" "target_group_name"])
vals)
;; =>
(#{{"target_group_id" "1234",
"target_group_name" "abc",
"targets" [{"target_id" "789", "target_name" "doe"}]}
{"target_group_id" "1234",
"target_group_name" "abc",
"targets" [{"target_id" "456", "target_name" "john"}]}}
#{{"target_group_id" "56789",
"target_group_name" "cdes",
"targets" [{"target_id" "0987", "target_name" "john"}]}})
The targets (which is a vector) are then concat together with merge-vector:
(def merge-vector
(partial apply
merge-with
(fn [& xs] (if (every? vector? xs) (apply concat xs) (last xs)))))
Here is how I would approach it:
(ns tst.demo.core
(:use demo.core tupelo.core tupelo.test)
(:require
[clojure.string :as str]
[tupelo.string :as ts]
[tupelo.core :as t]))
(def data-json
"{ 'data': [
{ 'target_group_id': '1234',
'target_group_name': 'abc',
'targets': [
{ 'target_id': '456',
'target_name': 'john' }
]
},
{ 'target_group_id': '56789',
'target_group_name': 'cdes',
'targets': [
{ 'target_id': '0987',
'target_name': 'john' }
]
},
{
'target_group_id': '1234',
'target_group_name': 'abc',
'targets': [
{ 'target_id': '789',
'target_name': 'doe' }
]
}
]
} " )
with transformation:
(dotest
(let [data-edn (t/json->edn
(ts/quotes->double data-json))
d2 (t/it-> data-edn
(:data it) ; unnest from :data key
(group-by :target_group_id it ) )
d3 (t/forv [[tgt-id entries] d2]
{:tgt-group-id tgt-id
:tgt-group-name (:target_group_name (first entries))
:targets-all (mapv :targets entries)}) ]
and results/tests:
(is= data-edn
{:data
[{:target_group_id "1234",
:target_group_name "abc",
:targets [{:target_id "456", :target_name "john"}]}
{:target_group_id "56789",
:target_group_name "cdes",
:targets [{:target_id "0987", :target_name "john"}]}
{:target_group_id "1234",
:target_group_name "abc",
:targets [{:target_id "789", :target_name "doe"}]}]})
(is= d2
{"1234"
[{:target_group_id "1234",
:target_group_name "abc",
:targets [{:target_id "456", :target_name "john"}]}
{:target_group_id "1234",
:target_group_name "abc",
:targets [{:target_id "789", :target_name "doe"}]}],
"56789"
[{:target_group_id "56789",
:target_group_name "cdes",
:targets [{:target_id "0987", :target_name "john"}]}]})
(is= d3
[{:tgt-group-id "1234",
:tgt-group-name "abc",
:targets-all [[{:target_id "456", :target_name "john"}]
[{:target_id "789", :target_name "doe"}]]}
{:tgt-group-id "56789",
:tgt-group-name "cdes",
:targets-all [[{:target_id "0987", :target_name "john"}]]}]) ))

How do I make an User required JSON

I have a JSON file, in that three objects are available, In that 2nd and 3rd objects does not have some fields which I actually needed. In missing fields, I need to add my own values. I will provide my code below
I tried this So far:
with open("final.json") as data1:
a = json.load(data1)
final = []
for item in a:
d = {}
d["AppName"]= item["name"]
d["AppId"] = item["id"]
d["Health"] = item["health"]
d["place1"] = item["cities"][0]["place1"]
d["place2"] = item["cities"][0]["place2"]
print(final)
Error: I am getting Key Error
My Input JSON file has similar data:
[{
"name": "python",
"id": 1234,
"health": "Active",
"cities": {
"place1": "us",
"place2": "newyork"
}
},
{
"name": "java",
"id": 2345,
"health": "Active"
}, {
"name": "python",
"id": 1234
}
]
I am expecting output:
[{
"name": "python",
"id": 1234,
"health": "Active",
"cities": {
"place1": "us",
"place2": "newyork"
}
},
{
"name": "java",
"id": 2345,
"health": "Null",
"cities": {
"place1": "0",
"place2": "0"
}
}, {
"name": "python",
"id": 1234,
"health": "Null",
"cities": {
"place1": "0",
"place2": "0"
}
}
]
I see two issues with the code that you have posted.
First, you are referring to the 'cities' field in you input JSON as if it is a list when it is, in fact, an object.
Second, to handle JSON containing objects which may be missing certain fields, you should use the Python dictionary get method. This method takes a key and an optional value to return if the key is not found (default is None).
for item in a:
d = {}
d["AppName"]= item["name"]
d["AppId"] = item["id"]
d["Health"] = item.get("health", "Null")
d["place1"] = item.get("cities", {}).get("place1", "0")
d["place2"] = item.get("cities", {}).get("place2", "0")

How can I reference query string list/array params in an Amazon API Gateway mapping template?

I am trying to create a template that maps an array of ids from a query string param to a json template. I can't seem to figure out how to do it. This is what I have so far:
{
"raw_ids_param_value": "$input.params('ids')",
"mapped_ids": [
#foreach($id in $input.params('ids'))
{
"id": "$id",
"foo": "bar"
}
#if($foreach.hasNext),#end
#end
]
}
When I make the request, you can see that the ids param is available, but it does nothing when I try to iterate over it:
GET /endpoint?ids=1,2,3
{
"raw_ids_param_value": "1,2,3",
"mapped_ids": [
]
}
What am I doing wrong?
Here is one solution: use .split(',') to split the string into an array.
{
"mapped_ids": [
#foreach($id in $input.params('ids').split(','))
{
"id": "$id",
"foo": "bar"
}
#if($foreach.hasNext),#end
#end
]
}
Result:
GET /endpoint?ids=1,2,3
{
"mapped_ids": [
{
"id": "1",
"foo": "bar"
},
{
"id": "2",
"foo": "bar"
},
{
"id": "3",
"foo": "bar"
}
]
}

CouchDB exclude from view based on list of regex expressions

Whats the best approach for excluding documents from a view based on a list of regex expressions. For example I want to exclude anything where doc.issue.name contains a value that matches a list of regex expressions.
e.g. exclusion list: [/foo/, /bar/]
{
"_id": "1",
"issue": {
"name": "foo"
}
{
"_id": "2",
"issue": {
"name": "bar"
}
{
"_id": "3",
"issue": {
"name": "fred"
}
So based on the documents above, just return the document where doc.issue.name = "fred"
OK so to answer my own question here in case anybody else needs to do this type of thing!
Based on the following documents:
{
"_id": "1",
"issue": {
"name": "foo"
}
{
"_id": "2",
"issue": {
"name": "bar"
}
{
"_id": "3",
"issue": {
"name": "fred"
}
This map function:
function(doc) {
var reg_exps = [/foo/g, /bar/g];
for (r in reg_exps){
if (doc.name.match(reg_exps[r])){
return;
}
}
emit(doc.name, 1);
}
Will only return the document with the name of "fred"