Removing ids from Postman Collection with bash script - sed and regex - regex

Question

I'm trying to solve an issue with Postman Collections.
Test scripts added to collection generates additional field "id".
Id field change after each export of the Collection to file.
Due to this fact PRs with changes in Postman Collections are very hard to read.
I want to solve that issue with git pre commit hook and bash script which will remove all id's from script object of collection.
There are three possible locations of the id in scripts object:
First element of object
"script":{
"id": "83d9076e-64c7-47fa-9b50-b7635718c925",
"exec": [
"console.log(\"foo\");"
],
"type": "text/javascript"
}
Middle of object
"script":{
"exec": [
"console.log(\"foo\");"
],
"id": "83d9076e-64c7-47fa-9b50-b7635718c925",
"type": "text/javascript"
}
End of object
"script":{
"exec": [
"console.log(\"foo\");"
],
"type": "text/javascript",
"id": "83d9076e-64c7-47fa-9b50-b7635718c925"
}
From regex point of view case 1 and 2 are the same:
.*"id": "[a-f0-9-]*",
Case 3 is different and regex which handles this option is:
,\n.*"id": "[a-f0-9-]*",
As I mentioned before, I want to use this regexp in bash script:
postmanClean.sh
#!/bin/bash
COLLECTION_FILES=$(find . -type f -name "*postman_collection.json")
for POSTMAN_COLLECTION in ${COLLECTION_FILES}
do
echo "Harmonizing Postman $POSTMAN_COLLECTION"
sed -i -e 's/.*"id": "[a-f0-9-]*"\,//' ${POSTMAN_COLLECTION} # Remove test/script ID
sed -i -e 's/\,\n.*"id": "[a-f0-9-]*"//' ${POSTMAN_COLLECTION} # Remove test/script ID
done
Above solution is incorrect. I tried different options, but this regexp are not working.
How properly build this request to make them work with sed command?
Collection file:
demo.postman_collection.json
{
"info": {
"_postman_id": "258b2fe2-5768-47f8-9e82-70971bab6bbd",
"name": "demo",
"schema": "https://schema.getpostman.com/json/collection/v2.1.0/collection.json"
},
"item": [
{
"name": "One",
"item": [
{
"name": "Demo 1",
"event": [
{
"listen": "test",
"script": {
"id": "83d9076e-64c7-47fa-9b50-b7635718c925",
"exec": [
"console.log(\"foo\");"
],
"type": "text/javascript"
}
}
],
"protocolProfileBehavior": {
"disableBodyPruning": true
},
"request": {
"method": "GET",
"header": [],
"body": {
"mode": "raw",
"raw": "foo"
},
"url": {
"raw": "https://postman-echo.com/delay/1",
"protocol": "https",
"host": [
"postman-echo",
"com"
],
"path": [
"delay",
"1"
]
}
},
"response": []
}
],
"protocolProfileBehavior": {}
},
{
"name": "Two",
"item": [
{
"name": "Demo 2",
"event": [
{
"listen": "test",
"script": {
"exec": [
"console.log(\"bar\");"
],
"type": "text/javascript",
"id": "facb28f7-c54d-46e2-adb2-4c929fd1edd3"
}
}
],
"protocolProfileBehavior": {
"disableBodyPruning": true
},
"request": {
"method": "GET",
"header": [],
"body": {
"mode": "raw",
"raw": "bar"
},
"url": {
"raw": "https://postman-echo.com/delay/2",
"protocol": "https",
"host": [
"postman-echo",
"com"
],
"path": [
"delay",
"2"
]
}
},
"response": []
},
{
"name": "Demo 3",
"event": [
{
"listen": "test",
"script": {
"exec": [
"console.log(\"foobar\");"
],
"id": "facb28f7-c54d-46e2-adb2-4c929fd1edd3",
"type": "text/javascript"
}
}
],
"protocolProfileBehavior": {
"disableBodyPruning": true
},
"request": {
"method": "GET",
"header": [],
"body": {
"mode": "raw",
"raw": "bar"
},
"url": {
"raw": "https://postman-echo.com/delay/3",
"protocol": "https",
"host": [
"postman-echo",
"com"
],
"path": [
"delay",
"3"
]
}
},
"response": []
}
],
"protocolProfileBehavior": {}
}
],
"protocolProfileBehavior": {}
}

Answer 1

I think jq is the right tool for this job and the solution will be as simple as walk(del(.id?)). here a rewrite of your script using jq:
#!/bin/bash
COLLECTION_FILES=$(find . -type f -name "*postman_collection.json")
for f in ${COLLECTION_FILES}
do
echo "Harmonizing Postman $f"
jq --indent 4 'walk(del(.id?))' "$f" > "$f.tmp" && mv "$f.tmp" "$f"
done
and a demo (please note how jq takes care of removing the extra , from "type": "text/javascript", which will otherwise invalidate the json):
$ cp demo.postman_collection.json demo.postman_collection.json.bak
$ ./postmanClean.sh
Harmonizing Postman ./demo.postman_collection.json
$ diff demo.postman_collection.json.bak demo.postman_collection.json
17d16
< "id": "83d9076e-64c7-47fa-9b50-b7635718c925",
65,66c64
< "type": "text/javascript",
< "id": "facb28f7-c54d-46e2-adb2-4c929fd1edd3"
---
> "type": "text/javascript"
104d101
< "id": "facb28f7-c54d-46e2-adb2-4c929fd1edd3",
$

Answer 2

You don't need to distinguish the two patterns, as you can use sed to just match any line that contains the "id": "..." pattern and then use it to delete the entire line where it matched using the d command. So you do not need to care about the newlines, whitespace or whether the trailing comma is there or not.
Executed on your example
sed -i '/"id": "[a-f0-9-]*"/d' demo.postman_collection.json
removes all the id lines (except the "_postman_id" of course).