Remove whitespace after +44 and quotation marks around lat/long in JSON string with regex - regex

In the example snippet below I have some JSON which needs to be edited (over 1400 entries). I need to achieve 2 things:
In this example line: "phone": "+44 2079693900", I need to remove the whitespace between +44 and 2079693900 but for all records. Resulting in: "+442079693900"
For latitude and longitude I need to get rid of the double quotes around the numbers, as the API I am using only accepts these values as floats.
Example: "latitude": "51.51736", needs to be: "latitude": 51.51736
I am most familiar with Ruby, and have done some parsing of JSON with this in the past, but I thought Regex would be the best tool to use for this kind of basic data cleaning task. I have referred to regex101.com and regular-expressions.info but I'm pretty stuck at this point. Thanks in advance!
[
{
"id": "101756",
"name": "1 Lombard Street
"email": "reception#1lombardstreet.com",
"website": "http://www.1lombardstreet.com",
"location": {
"latitude": "51.5129",
"longitude": "-0.089",
"address": {
"line1": "1 Lombard Street",
"line2": "",
"line3": "",
"postcode": "EC3V 9AA",
"city": "London",
"country": "UK"
}
}
},
{
"id": "105371",
"name": "108 Brasserie",
"phone": "+44 2079693900",
"email": "enquiries#108marylebonelane.com",
"website": "http://www.108brasserie.com",
"location": {
"latitude": "51.51795",
"longitude": "-0.15079",
"address": {
"line1": "108 Marylebone Lane",
"line2": "",
"line3": "",
"postcode": "W1U 2QE",
"city": "London",
"country": "UK"
}
}
},
{
"id": "108701",
"name": "1901 Restaurant",
"phone": "+44 2076187000",
"email": "london.restres#andaz.com",
"website": "http://www.andazdining.com",
"location": {
"latitude": "51.51736",
"longitude": "-0.08123",
"address": {
"line1": "Andaz Hotel",
"line2": "40 Liverpool Street",
"line3": "",
"postcode": "EC2M 7QN",
"city": "London",
"country": "UK"
}
}
},
{
"id": "102190",
"name": "2 Bridge Place",
"phone": "+44 2078028555",
"email": "fb#dtlondonvictoria.com",
"website": "http://crimsonhotels.comdoubletreelondonvictoriadiningpre-theatre-dining",
"location": {
"latitude": "51.49396",
"longitude": "-0.14343",
"address": {
"line1": "2 Bridge Place",
"line2": "Victoria",
"line3": "",
"postcode": "SW1V 1QA",
"city": "London",
"country": "UK"
}
}
},
{
"id": "102063",
"name": "2 Veneti",
"phone": "+44 2076370789",
"email": "2veneti#btconnect.com",
"website": "http://www.2veneti.com",
"location": {
"latitude": "51.5168",
"longitude": "-0.14673",
"address": {
"line1": "10 Wigmore Street",
"line2": "",
"line3": "",
"postcode": "W1U 2RD",
"city": "London",
"country": "UK"
}
}
},

You can use the following regex:
("phone":\s*"\+44)\s+|("(?:latitude|longitude)":\s*)"([^"]+)"
With the following replacement:
$1$2$3
The idea is to capture what we want and not capture what we do not, and then use backreferences to restore the substrings we want to keep.
Regex explanation:
The pattern contains 2 alternatives joined with | alternation operator:
("phone":\s*"\+44)\s+:
("phone":\s*"\+44) - the 1st capturing group matching literal "phone": + optional whitespace, then +44 literally
\s+ - 1 or more whitespaces that we'll remove
("(?:latitude|longitude)":\s*)"([^"]+)":
("(?:latitude|longitude)":\s*) - the second capturing group matching "latitude": or "longitude": and 0 or more whitespace characters
" - Literal " that we'll drop
([^"]+) - the third capturing group matching 1 or more characters other than " (we'll keep that)
" - again, a literal " that we'll drop.
See demo

Related

"Buffalo" query fails to return "Buffalo Exchange" from /discover endpoint

NOTE: this question is specifically for support staff of the HERE Developer API because they ask freemium users to post support questions on Stack Overflow rather than trying to contact them directly. If you're not a member of their staff and you're unable to help or if the question is unclear to you, don't worry about it. :)
For some reason the /discover endpoint doesn't return the "Buffalo Exchange" place that's at my specified coordinates, but only returns 2 localities that are much further away. This is the query that I'm using: https://discover.search.hereapi.com/v1/discover?at=34.003975%2C-118.484823&q=Buffalo&limit=20&apiKey=<insert API KEY>. These are the results I currently receive:
{
"items": [
{
"title": "Buffalo, NY, United States",
"id": "here:cm:namedplace:21018816",
"resultType": "locality",
"localityType": "city",
"address": {
"label": "Buffalo, NY, United States",
"countryCode": "USA",
"countryName": "United States",
"stateCode": "NY",
"state": "New York",
"county": "Erie",
"city": "Buffalo",
"postalCode": "14202"
},
"position": {
"lat": 42.88544,
"lng": -78.87846
},
"distance": 3551940,
"mapView": {
"west": -78.9168,
"south": 42.82603,
"east": -78.79492,
"north": 42.96651
}
},
{
"title": "Buffalo City, Eastern Cape, South Africa",
"id": "here:cm:namedplace:23402337",
"resultType": "locality",
"localityType": "city",
"address": {
"label": "Buffalo City, Eastern Cape, South Africa",
"countryCode": "ZAF",
"countryName": "South Africa",
"state": "Eastern Cape",
"county": "Buffalo City",
"city": "Buffalo City",
"postalCode": "5201"
},
"position": {
"lat": -33.0148,
"lng": 27.9038
},
"distance": 16910944,
"mapView": {
"west": 27.15745,
"south": -33.28749,
"east": 28.08053,
"north": -32.674
}
}
]
}
You can see that for both places the resultType is "locality".
Now compare that to the first result of a similar query that searches for the term "Exchange" instead of "Buffalo". All other query params are the same. This is the URL: https://discover.search.hereapi.com/v1/discover?at=34.003975%2C-118.484823&q=Exchange&limit=20&apiKey=<insert API KEY>, and this is how the results begin (not shown fully because there are many results):
{
"items": [
{
"title": "Buffalo Exchange",
"id": "here:pds:place:8403fv6k-b15f290ec4f409deea99318f7388bbd6",
"resultType": "place",
"address": {
"label": "Buffalo Exchange, 2449 Main St, Santa Monica, CA 90405, United States",
"countryCode": "USA",
"countryName": "United States",
"stateCode": "CA",
"state": "California",
"county": "Los Angeles",
"city": "Santa Monica",
"district": "City of Santa Monica",
"street": "Main St",
"postalCode": "90405",
"houseNumber": "2449"
},
"position": {
"lat": 34.00342,
"lng": -118.48483
},
"access": [
{
"lat": 34.00331,
"lng": -118.48493
}
],
"distance": 61,
"categories": [
{
"id": "600-6800-0090",
"name": "Women's Apparel",
"primary": true
},
{
"id": "600-6800-0000",
"name": "Clothing & Accessories"
},
{
"id": "600-6800-0089",
"name": "Men's Apparel"
},
{
"id": "600-6900-0251",
"name": "Used/Second-hand Merchandise Stores"
}
],
"references": [
{
"supplier": {
"id": "core"
},
"id": "1211447153"
},
{
"supplier": {
"id": "yelp"
},
"id": "5PzeN6hGLBPmJpCJ2ZmfCQ"
}
],
"contacts": [
{
"phone": [
{
"value": "+13103147300"
},
{
"value": "+13103924301",
"categories": [
{
"id": "600-6800-0000"
}
]
}
],
"fax": [
{
"value": "(520) 622-7015",
"categories": [
{
"id": "600-6800-0000"
}
]
}
],
"www": [
{
"value": "http://www.buffaloexchange.com",
"categories": [
{
"id": "600-6800-0000"
},
{
"id": "600-6900-0251"
}
]
}
],
"email": [
{
"value": "contact#bufex.com",
"categories": [
{
"id": "600-6800-0000"
}
]
}
]
}
],
"openingHours": [
{
"categories": [
{
"id": "600-6800-0000"
}
],
"text": [
"Mon-Sun: 11:00 - 20:00"
],
"isOpen": false,
"structured": [
{
"start": "T110000",
"duration": "PT09H00M",
"recurrence": "FREQ:DAILY;BYDAY:MO,TU,WE,TH,FR,SA,SU"
}
]
},
{
"categories": [
{
"id": "600-6800-0090"
},
{
"id": "600-6900-0251"
}
],
"text": [
"Mon-Sat: 11:00 - 20:00",
"Sun: 11:00 - 19:00"
],
"isOpen": false,
"structured": [
{
"start": "T110000",
"duration": "PT09H00M",
"recurrence": "FREQ:DAILY;BYDAY:MO,TU,WE,TH,FR,SA"
},
{
"start": "T110000",
"duration": "PT08H00M",
"recurrence": "FREQ:DAILY;BYDAY:SU"
}
]
}
]
},
...
}
You can see that the first result has the name "Buffalo Exchange" and the "resultType" is "place". This is the result I want. The question is why does this result fail to show up when the search query is "Buffalo"? Of course with the /discover endpoint I can't specify the category IDs I want to search, that's only available via the /browse endpoint. But with the /browse endpoint I can't specify a specific search term like "Buffalo" or "Exchange".
Update: this problem also happens with the "Bison" query in Alberta, Canada. The query for this is https://discover.search.hereapi.com/v1/discover?at=56.745531%2C-111.351341&q=Exchange&limit=20&apiKey=<insert API KEY>. This query yields only 10 results, and only 4 of them have resultType of "place".
We recommend application developers to use both Autosuggest and Discover, you get the nearby Buffalo Exchange because Autosuggest considers that the query is incomplete. Note that Discover considers the query to be complete.
As HERE Geocoding and Search is meant to provide relevant responses to user queries.
Geocoding Search Api (Autosuggest): https://developer.here.com/documentation/geocoding-search-api/dev_guide/topics/endpoint-autosuggest-brief.html
Geocoding Search Api (Discover): https://developer.here.com/documentation/geocoding-search-api/dev_guide/topics/endpoint-discover-brief.html
The above-mentioned user guide can help modify the query for accurate results.

Querying data on S3 Object

Trying to query data on json file using S3-Select
{
"groups_id":
{
"307225":
{
"created_at": "2015-02-10T17:24:15-08:00",
"updated_at": "2017-09-06T17:25:22-07:00",
"name": "Company 1",
"company": true,
"contact_name": "User 1",
"email": "",
"phone_number": "",
"address": "",
"website": "",
"notes": "Testing",
"id": "307225"
},
"1058565":
{
"created_at": "2017-04-02T23:44:10-07:00",
"updated_at": "2017-07-18T17:39:21-07:00",
"name": "Company 3",
"company": true,
"contact_name": "User 1",
"email": "",
"phone_number": "",
"address": "",
"website": "",
"notes": null,
"id": "1058565"
}
}
}
Can someone help to get the desired output using s3 select, based on the condition as WHERE contact_name='User 1'.
{"id": "307225", "name": "Company 1"},
{"id": "1058565","name": "Company 3"}
below are the queries we had tried,
Select s.groups_id['307225'].id from s3object s
Select s.groups_id['1058565'].id from s3object s
in the above queries, we had hardcoded the group_id and we are able to fetch below
{
"id": "1058565"
}
but in our case groupid is dynamic so i am not understanding. how to handle that?

How to Implement Partial Auth on Authorize.net API [duplicate]

Okay I am setting up Partial Payments via the Authorize.net API in order to enable multiple cards to be used to cover a single balance/charge.
I'm assuming thier Partial Auth feature covers my use case, but in testing, there is an issue I can show you using the API live console here: https://developer.authorize.net/api/reference/index.html#payment-transactions-charge-a-credit-card
Go to the link above and authorize partial payments with the request I edited below and you will notice when you press submit you get a splitTenderId:
{
"createTransactionRequest": {
"merchantAuthentication": {
"name": "5KP3u95bQpv",
"transactionKey": "346HZ32z3fP4hTG2"
},
"refId": "123456",
"transactionRequest": {
"transactionType": "authCaptureTransaction",
"amount": "462.25",
"payment": {
"creditCard": {
"cardNumber": "5424000000000015",
"expirationDate": "2020-12",
"cardCode": "999"
}
},
"lineItems": {
"lineItem": {
"itemId": "1",
"name": "vase",
"description": "Cannes logo",
"quantity": "18",
"unitPrice": "45.00"
}
},
"tax": {
"amount": "4.26",
"name": "level2 tax name",
"description": "level2 tax"
},
"duty": {
"amount": "8.55",
"name": "duty name",
"description": "duty description"
},
"shipping": {
"amount": "4.26",
"name": "level2 tax name",
"description": "level2 tax"
},
"poNumber": "456654",
"customer": {
"id": "99999456654"
},
"billTo": {
"firstName": "Ellen",
"lastName": "Johnson",
"company": "Souveniropolis",
"address": "14 Main Street",
"city": "Pecan Springs",
"state": "TX",
"zip": "44628",
"country": "USA"
},
"shipTo": {
"firstName": "China",
"lastName": "Bayles",
"company": "Thyme for Tea",
"address": "12 Main Street",
"city": "Pecan Springs",
"state": "TX",
"zip": "44628",
"country": "USA"
},
"customerIP": "192.168.1.1",
"transactionSettings": {
"setting": [
{
"settingName": "emailCustomer",
"settingValue": "true"
}, {
"settingName": "allowPartialAuth",
"settingValue": "true"
},
]
},
"userFields": {
"userField": [
{
"name": "MerchantDefinedFieldName1",
"value": "MerchantDefinedFieldValue1"
},
{
"name": "favorite_color",
"value": "blue"
}
]
}
}
}
}
This is only successful because the amount is 462.25 as the docs say to use for testing, if I use any other (real) amount the splitTenderId is not there.
Here is a new example request, you can post this again on the link above:
{
"createTransactionRequest": {
"merchantAuthentication": {
"name": "5KP3u95bQpv",
"transactionKey": "346HZ32z3fP4hTG2"
},
"refId": "123456",
"transactionRequest": {
"transactionType": "authCaptureTransaction",
"amount": "462",
"payment": {
"creditCard": {
"cardNumber": "5424000000000015",
"expirationDate": "2020-12",
"cardCode": "999"
}
},
"lineItems": {
"lineItem": {
"itemId": "1",
"name": "vase",
"description": "Cannes logo",
"quantity": "18",
"unitPrice": "45.00"
}
},
"tax": {
"amount": "4.26",
"name": "level2 tax name",
"description": "level2 tax"
},
"duty": {
"amount": "8.55",
"name": "duty name",
"description": "duty description"
},
"shipping": {
"amount": "4.26",
"name": "level2 tax name",
"description": "level2 tax"
},
"poNumber": "456654",
"customer": {
"id": "99999456654"
},
"billTo": {
"firstName": "Ellen",
"lastName": "Johnson",
"company": "Souveniropolis",
"address": "14 Main Street",
"city": "Pecan Springs",
"state": "TX",
"zip": "44628",
"country": "USA"
},
"shipTo": {
"firstName": "China",
"lastName": "Bayles",
"company": "Thyme for Tea",
"address": "12 Main Street",
"city": "Pecan Springs",
"state": "TX",
"zip": "44628",
"country": "USA"
},
"customerIP": "192.168.1.1",
"transactionSettings": {
"setting": [
{
"settingName": "emailCustomer",
"settingValue": "true"
}, {
"settingName": "allowPartialAuth",
"settingValue": "true"
},
]
},
"userFields": {
"userField": [
{
"name": "MerchantDefinedFieldName1",
"value": "MerchantDefinedFieldValue1"
},
{
"name": "favorite_color",
"value": "blue"
}
]
}
}
}
}
And with the 462 amount entered this is no longer a partial auth payment and I no longer get a splitTenderId.
Can someone please help me figure out what is going on?

How to solve the error of The element 'createCustomerProfileRequest' in namespace while adding the customer mutiple payment

I want to add the multiple payment profiles of a single customer in authorize.net. After a long searching I reached a post posted on the community of the authorize.net see link. In this link there a xml formated data in the accepted solution I convert it and send it through the postman then I'm facing the error of
"The element 'createCustomerProfileRequest' in namespace 'AnetApi/xml/v1/schema/AnetApiSchema.xsd' has invalid child element '_xmlns' in namespace 'AnetApi/xml/v1/schema/AnetApiSchema.xsd'."
The json I'm sending is :-
{
"createCustomerProfileRequest": {
"merchantAuthentication": {
"name": "name",
"transactionKey": "transactionKey"
},
"refId": "1361101257555",
"profile": {
"merchantCustomerId": "2CLINC056",
"description": "hiiiiii.",
"email": "",
"paymentProfiles": [
{
"customerType": "individual",
"billTo": {
"firstName": "Joe",
"lastName": "Test",
"company": "CompanyA",
"address": "hello",
"city": "Bangalore",
"state": "Delhi",
"zip": "560078",
"country": "IN",
"phoneNumber": "415-555-1212",
"faxNumber": "415-555-1313"
},
"payment": {
"creditCard": {
"cardNumber": "370000000000002",
"expirationDate": "2029-12"
}
}
},
{
"customerType": "individual",
"billTo": {
"firstName": "Joe",
"lastName": "Test",
"company": "CompanyA",
"address": "vel",
"city": "Chennai",
"state": "AK",
"zip": "560089",
"country": "US",
"phoneNumber": "415-555-1212",
"faxNumber": "415-555-1313"
},
"payment": {
"creditCard": {
"cardNumber": "38000000000006",
"expirationDate": "2029-12"
}
}
}
]
},
"validationMode": "testMode",
"_xmlns": "AnetApi/xml/v1/schema/AnetApiSchema.xsd"
}
}
Developer link
I think your issue is not with the payment profiles but with the JSON itself. You have an invalid element in there and I think it is this line:
"_xmlns": "AnetApi/xml/v1/schema/AnetApiSchema.xsd"
Otherwise your request looks fine as I was able to create multiple payment profiles using similar JSON. The only difference between the two was my JSON didn't have that line.
Request JSON:
{
"createCustomerProfileRequest": {
"merchantAuthentication": {
"name": "xxxxxxxx",
"transactionKey": "xxxxxxxxxxxxxxxx"
},
"profile": {
"merchantCustomerId": "52353345",
"email": "user#example.com",
"paymentProfiles": [
{
"customerType": "individual",
"billTo": {
"firstName": "John",
"lastName": "Smith",
"address": "12345 Main Street",
"city": "Townsville",
"state": "NJ",
"zip": "12345",
"phoneNumber": "800-555-1234"
},
"payment": {
"creditCard": {
"cardNumber": "5555555555554444",
"expirationDate": "2023-08"
}
}
},
{
"customerType": "individual",
"billTo": {
"firstName": "John",
"lastName": "Smithberg",
"address": "42 Main Street",
"city": "Townsville",
"state": "NJ",
"zip": "12345",
"phoneNumber": "800-555-1234"
},
"payment": {
"creditCard": {
"cardNumber": "5105105105105100",
"expirationDate": "2023-09"
}
}
}
],
"shipToList": {
"firstName": "John",
"lastName": "Smith",
"address": "12345 Main Street",
"city": "Townsville",
"state": "NJ",
"zip": "12345",
"phoneNumber": "800-555-1234"
}
},
"validationMode": "liveMode"
}
}
Response JSON:
{
"customerProfileId": "1506322353",
"customerPaymentProfileIdList": [
"1505667207",
"1505667208"
],
"customerShippingAddressIdList": [
"1505655763"
],
"validationDirectResponseList": [
"1,1,1,This transaction has been approved.,A2FD5O,Y,40023515435,none,Test transaction for ValidateCustomerPaymentProfile.,0.00,CC,auth_only,52353345,John,Smith,,12345 Main Street,Townsville,NJ,12345,,800-555-1234,,user#example.com,,,,,,,,,0.00,0.00,0.00,FALSE,none,32573C7D03376A9052AACA73835EDAEF,P,2,,,,,,,,,,,XXXX4444,MasterCard,,,,,,,,,,,,,,,,,",
"1,1,1,This transaction has been approved.,AO13Y1,Y,40023515436,none,Test transaction for ValidateCustomerPaymentProfile.,0.00,CC,auth_only,52353345,John,Smithberg,,42 Main Street,Townsville,NJ,12345,,800-555-1234,,user#example.com,,,,,,,,,0.00,0.00,0.00,FALSE,none,5B937D29D29F261776859B50DC1C3CF6,P,2,,,,,,,,,,,XXXX5100,MasterCard,,,,,,,,,,,,,,,,,"
],
"messages": {
"resultCode": "Ok",
"message": [
{
"code": "I00001",
"text": "Successful."
}
]
}
}

Get all instances of text in curly braces between brackets

Let's say I have some text like this:
{
"source": "Analytics 13 {Employee_Info.acl} {Employee_Data}",
"lastRecNo": "3",
"columns": {
"ID": "numeric",
"NAME": "character",
"EFFECTIVE_DATE": "date",
"ROLE": "character"
},
"data": [{
"ID": 1,
"NAME": "Bill Smith",
"EFFECTIVE_DATE": "2018-10-01",
"ROLE": "Director {Regional},{Call Center}"
},
{
"ID": 2,
"NAME": "Ellen Jones",
"EFFECTIVE_DATE": "2018-07-01",
"ROLE": "Manager"
},
{
"ID": 3,
"NAME": "Sam Edwards",
"EFFECTIVE_DATE": "2018-09-01",
"ROLE": "Supervisor"
}]
}
And I want to RegEx match every object inside the "data" array (including the curly braces).
So the first match would be:
{
"ID": 1,
"NAME": "Bill Smith",
"EFFECTIVE_DATE": "2018-10-01",
"ROLE": "Director {Regional},{Call Center}"
}
the second would be:
{
"ID": 2,
"NAME": "Ellen Jones",
"EFFECTIVE_DATE": "2018-07-01",
"ROLE": "Manager"
}
and the third would be
{
"ID": 3,
"NAME": "Sam Edwards",
"EFFECTIVE_DATE": "2018-09-01",
"ROLE": "Supervisor"
}
What regex pattern would I use to do that in PowerShell?
Notice the first match actually has some extra curly braces in the text of the "ROLE" field, which shouldn't interfere with the match.
I've tried this so far '(?<={).*?(?=})', but the first match is:
"source": "Analytics 13 {Employee_Info.acl
This result isn't a part of the "data" array and it doesn't include the curly braces in the match. I know I'm missing something that says "make sure we are inside the brackets/"data" array and I'm probably not taking into account the extra curly braces in the "ROLE" field in the first object of the "data" array that I want to ignore.
Your task can be easily done using ConvertFrom-Json and ConvertTo-Json cmdlets.
Here is a brief example:
First you get text file content to variable.
$JSON = #"
[
{
"source": "Analytics 13 {Employee_Info.acl} {Employee_Data}",
"lastRecNo": "3",
"columns": {
"ID": "numeric",
"NAME": "character",
"EFFECTIVE_DATE": "date",
"ROLE": "character"
},
"data": [{
"ID": 1,
"NAME": "Bill Smith",
"EFFECTIVE_DATE": "2018-10-01",
"ROLE": "Director {Regional},{Call Center}"
},
{
"ID": 2,
"NAME": "Ellen Jones",
"EFFECTIVE_DATE": "2018-07-01",
"ROLE": "Manager"
},
{
"ID": 3,
"NAME": "Sam Edwards",
"EFFECTIVE_DATE": "2018-09-01",
"ROLE": "Supervisor"
}]
}
]
"#
Then you just perform converting from JSON using ConvertFrom-Json cmdlet.
ConvertFrom-Json -InputObject $JSON
Output:
source lastRecNo columns data
------ --------- ------- ----
Analytics 13 {Employee_Info.acl} {Employee_Data} 3 #{ID=numeric; NAME=character; EFFECTIVE_DATE=date; ROLE=character} {#{ID=1; NAME=Bill Smith; EFFECTIVE_DATE=2018-10-01; ROLE=Director {Regional},{Call Center}}, #...
You then can return items from DATA to JSON format using ConvertTo-Json cmdlet. All together.
$PSObject = ConvertFrom-Json -InputObject $JSON
foreach ($item in $PSObject.data){
ConvertTo-Json $item
}
Output:
{
"ID": 1,
"NAME": "Bill Smith",
"EFFECTIVE_DATE": "2018-10-01",
"ROLE": "Director {Regional},{Call Center}"
}
{
"ID": 2,
"NAME": "Ellen Jones",
"EFFECTIVE_DATE": "2018-07-01",
"ROLE": "Manager"
}
{
"ID": 3,
"NAME": "Sam Edwards",
"EFFECTIVE_DATE": "2018-09-01",
"ROLE": "Supervisor"
}
You can now add filter conditions for DATA items in foreach loop.