How to send json data using powershell from S3 to Neptune instance? - amazon-web-services

I have a turtle file that I wish to send from S3 server to the Neptune instance on powershell. Below is the command I am using
Invoke-RestMethod -Uri http://edwardspoc.civwhymjvz19.us-east-1.neptune.amazonaws.com:8182 -ContentType application/json -body '
{
"source" : "s3://cedwardsneptune/ATC.ttl",
"format" : "turtle",
"iamRoleArn" : "arn:aws:s3:::cedwardsneptune",
"region" : "us-east-1",
"failOnError" : "FALSE",
"parserConfiguration" : {
"namedGraphUri" : "http://purl.bioontology.org/ontology/ATC"
}
}'
This is giving me the following error
Invoke-RestMethod : Cannot send a content-body with this verb-type.
I made subtle changes but nothing seems to be working. Any help is appreciated !

Error you having says that you are trying to send a BODY when your REQUEST VERB does not allow it.
Basically you are doing GET and trying to send a body. When GET verb is not allowed to send a body part.
Probably you need to use POST method (check specific webservice documentation)
For that I recommend you next format:
$Cred = Get-Credential (If you could have)
$Url = "http://edwardspoc.civwhymjvz19.us-east-1.neptune.amazonaws.com:8182"
$Body = '{
"source" : "s3://cedwardsneptune/ATC.ttl",
"format" : "turtle",
"iamRoleArn" : "arn:aws:s3:::cedwardsneptune",
"region" : "us-east-1",
"failOnError" : "FALSE",
"parserConfiguration" : {
"namedGraphUri" : "http://purl.bioontology.org/ontology/ATC"
}
}'
Invoke-RestMethod -Method 'Post' -Uri $url -Credential $Cred -ContentType $contentType -Body $body -ContentType 'application/json' -OutFile output.csv
(*)Of course -OutFile is optional
And about JSON There is a cmdlet called ConvertFrom-JSON you can use commonly when you need to parse an output(that is a JSON).
p.e:
Invoke-RestMethod -Method 'Post' -Uri $url -Credential $Cred -Body $body |
ConvertFrom-JSON | Select field1, field2, field3
If you expect an output (json) having field1, field2, field3 as fields

Related

How do I obtain temporary AWS credentials for an unauthenticated role in PowerShell using a Cognito IdentityPool?

I was writing a PowerShell script that needed to access an AWS S3 bucket using an unauthenticated role via Cognito and had trouble finding much documentation. All of the documentation I was able to find for the AWS PowerShell SDK discussed storing your AccessKey and SecretKey but never how to get those credentials using Cognito when you aren't using a user pool.
There may be other ways to do this with PowerShell (I haven't been able to find them yet.) but you can obtain temporary credentials through Cognito using AWS's REST API.
The following PowerShell example shows how to:
Set your REST URL
Get an id from the Cognito Identity provider
Use the received id to request temporary credentials (AccessKey will begin with AS instead of AK)
Set the temporary credentials
For more information see:
AWS API Getting Credentials
AWS API GetCredentialsForIdentity
AWS API GetId
function Get-CognitoRestURL {
param(
[parameter(Mandatory)]$Region
)
return "https://cognito-identity.{0}.amazonaws.com/" -f $Region
}
function Get-AWSTempCredentials {
param(
[parameter(Mandatory)]$IdentityPoolId,
[parameter(Mandatory)]$Region
)
try {
$cognitoRestURL = Get-CognitoRestURL -Region $Region
$requestTempId = Invoke-RestMethod -Uri $cognitoRestURL -Method "POST" `
-Headers #{
"authority"=$cognitoRestURL
"x-amz-target"="AWSCognitoIdentityService.GetId"
"x-amz-user-agent"="aws-powershell callback"
} -ContentType "application/x-amz-json-1.1" -Body "{`"IdentityPoolId`":`"$($IdentityPoolId)`"}"
} catch {
Write-Error $_
#Request failed, we don't have the data we need to continue
break
}
try {
$tempCredentials = Invoke-RestMethod -Uri $cognitoRestURL -Method "POST" `
-Headers #{
"x-amz-target"="AWSCognitoIdentityService.GetCredentialsForIdentity"
"x-amz-user-agent"="aws-powershell callback"
} -ContentType "application/x-amz-json-1.1" -Body "{`"IdentityId`":`"$($requestTempId.IdentityId)`"}"
} catch {
Write-Error $_
#Request failed, we don't have the data we need to continue
break
}
return $tempCredentials
}
function Set-AWSTempCredentials {
param(
[parameter(Mandatory)]$AccessKeyId,
[parameter(Mandatory)]$SecretKey,
[parameter(Mandatory)]$SessionToken,
[parameter(Mandatory)]$ProfileName,
[parameter(Mandatory)]$Region
)
Set-AWSCredential -AccessKey $AccessKeyId -SecretKey $SecretKey -SessionToken $SessionToken -StoreAs $ProfileName
return Get-AWSCredential -ProfileName $ProfileName
}
$region = "us-west-1"
$IdentityPoolId = "us-west-1:12a01023-4567-123a-bcd1-12345a0b1abc"
$response = Get-AWSTempCredentials -IdentityPoolId $IdentityPoolId -Region $region
Set-AWSTempCredentials -AccessKeyId $response.Credentials.AccessKeyId `
-SecretKey $response.Credentials.SecretKey `
-SessionToken $response.Credentials.SessionToken `
-ProfileName MyTempCredentials `
-Region $region

how to take latest backupid to JSON for restore data to an instance in a different project GCP

I am working on restore the latest backup restore data to an instance into a different project GCP in Windows power shell.
Using below queries using power shell getting backup id's list
$cred = gcloud auth print-access-token
$headers = #{ "Authorization" = "Bearer $cred" }
Invoke-WebRequest `
-Method GET `
-Headers $headers `
-Uri "https://sqladmin.googleapis.com/v1/projects/source-project/instances/source-instance/backupRuns" | Select-Object -Expand Content
Restore using below command
$cred = gcloud auth print-access-token
$headers = #{ "Authorization" = "Bearer $cred" }
Invoke-WebRequest `
-Method POST `
-Headers $headers `
-ContentType: "application/json; charset=utf-8" `
-InFile request.json `
-Uri "https://sqladmin.googleapis.com/v1/projects/target-project/instances/target-instance/restoreBackup" | Select-Object -Expand Content
In the request.json file updating latest backupid to restore latest backup up.
How to take latest backup id from the list automatically using power shell
Please help any one.

How to creatte an index pattern in Opensearch use the API?

I want to create an index pattern using the Opensearch API. The issue i'm having is with the message body type. I'm getting the following message when running my script
I'm running opensearch 1.3
Invoke-RestMethod: {"statusCode":400,"error":"Bad Request","message":"[request body.attributes]: expected value of type [object] but got [undefined]"}
$baseuri = "https://<clusterendpoint>.<region>.es.amazonaws.com"
#obtain auth cookie
$authbody = #"
{"username":"$apiUserName", "password":"$apiPassword"}
"#
$authUri = "$baseuri/_dashboards/auth/login"
Invoke-RestMethod -Method Post -Uri $authUri -Headers #{"osd-xsrf"="true"} -ContentType "application/json" -Body $authbody -SessionVariable S1
#### create index pattern
$indexid = "indexname"
$body=#"
{
"index_pattern": {
"title": "indexname-*",
"timeFieldName": "#timestamp"
}
}
"#
$uri = "$baseuri/_dashboards/api/saved_objects/index-pattern/$indexid"
Invoke-RestMethod -Method Post -Uri $uri -Headers #{"osd-xsrf"="true"} -ContentType "application/json" -WebSession $S1 -Body $body -Verbose
i've attempted various ways of defining an object with no luck. when I run the invoke with ($body | convertfrom-json) i get a different error message.
Invoke-RestMethod: {"statusCode":400,"error":"Bad Request","message":"[request body]: expected a plain object value, but found [null] instead."}
UPDATE: I did originally attempt to use the create index pattern API for elastic, but this results in a new index being created, not an opensearch index_pattern.
$body=#{
"index_pattern"= #{
"title"= "ai-summary"
}
}
$uri = "$baseuri/api/index_patterns/$indexid"
Invoke-RestMethod -Method Post -Uri $uri -Headers #{"osd-xsrf"="true"} -ContentType "application/json" -WebSession $S1 -Body ($body | ConvertTo-Json) -Verbose -Credential $credObject

How do I get the code reviewer(s) of a build in Azure DevOps pipeline?

Given a build id, how do I get the code reviewer(s) name in azure DevOps pipeline?
Assume, the build was off of a master branch - which devs merge their feature branch after code has been reviewed in a pull request. This is the policy and no one directly commit their change into master. So that means, every build has a code reviewer behind it. How do i get that?
Thanks!
You can use below Rest api to get the PR reviewers.
1, First call below build rest api with the buildId. And in the response you will get the commit id from the build's sourceVersion and the repository id.
GET https://dev.azure.com/{organization}/{project}/_apis/build/builds/{buildId}?api-version=5.1
2, After you get the commit id and repository id. You can call commit rest api to get the associated PR id from the comments in the response.
GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/commits/{commitId}?api-version=5.1
3, Then Call pull request reviewer rest api to get the Reviewers.
GET https://dev.azure.com/{organization}/{project}/_apis/git/repositories/{repositoryId}/pullRequests/{pullRequestId}/reviewers?api-version=5.1
Below is the example scripts in powershell. See this link to get a Personal access token
$buildId= " "
$burl =" https://dev.azure.com/OrgName/ProjName/_apis/build/builds/$($buildId)?api-version=5.1"
$PAT="personel access token"
$base64AuthInfo= [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($PAT)"))
$buildInfo = Invoke-RestMethod -Uri $curl -Headers #{Authorization = ("Basic {0}" -f $base64AuthInfo1)} -Method get -ContentType "application/json"
#get CommitId and repoId
$commitId = $buildInfo.sourceVersion
$repoId=$buildInfo.repository.id
#commit rest api
$curl = "https://dev.azure.com/OrgName/ProjName/_apis/git/repositories/$($repoId)/commits/$($commitId)?api-version=5.1"
$commitInfo = Invoke-RestMethod -Uri $curl -Headers #{Authorization = ("Basic {0}" -f $base64AuthInfo1)} -Method get -ContentType "application/json"
#get PR id
$prId = $commitInfo.comment.split(" ")[2].TrimEnd(":")
$prurl = "https://dev.azure.com/OrgName/ProjName/_apis/git/repositories/$($repoId)/pullRequests/$($prId)/reviewers?api-version=5.1"
Invoke-RestMethod -Uri $prurl -Headers #{Authorization = ("Basic {0}" -f $base64AuthInfo1)} -Method get -ContentType "application/json"
If you can find the build from the pipeline runs history in the UI page with a give buildId. It will be much easier. You can get the PR id from the title directly. See below pic.
You can also click on the commit id shown on above screenshot, to see the details of the commit, where you will get the associated PR.
Here is what I have finally working. Took Levi's code snippet above and just fixed a line to get pull request id working in various scenarios. Kudos to Levi's for helping! Hope it helps someone.
$PAT="personel access token"
$base64EncodedPAT = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes(":$($PAT)"))
$basicAuth = #{Authorization = "Basic $base64EncodedPAT" }
$buildId= "..."
function GetCodeReviewers() {
#Get build info
$buildUrl = "https://dev.azure.com/OrgName/ProjName/_apis/build/builds/$($buildId)?api-version=5.1"
$buildInfo = Invoke-RestMethod -Method Get -Uri $buildUrl -Headers $basicAuth
# Get Commit Info
$commitUrl = "https://dev.azure.com/OrgName/ProjName/_apis/git/repositories/$($buildInfo.repository.id)/commits/$($buildInfo.sourceVersion)?api-version=5.1"
$commitInfo = Invoke-RestMethod -Uri $commitUrl -Method Get -Headers $basicAuth
#Get Code Reviewers
$comment = $commitInfo.comment
#$pullRequestId = $comment.split(" ")[2].TrimEnd(":") # it turns out, the 3rd item may not always be the PullRequestID so the next line may not work for all scenarios
#note that, a comment could come as follows:
# case 1: Merge PR 1234: some other text here including story or bug numbers
# case 2: Merge pull request 1234 some additional text goes here including story or bug numbers
# The following will pick the first number - which I assume will always be the PullRequestID
$pullRequestId = $null
$pullRequestId = $comment.Replace(':', '').Split(" ").Trim() | Where-Object {[int]::TryParse($_, $pullRequestId)} | Select-Object -First 1
$pullRequestUrl = "https://dev.azure.com/OrgName/ProjName/_apis/git/repositories/$($buildInfo.repository.id)/pullRequests/$($pullRequestId)/reviewers?api-version=5.1"
$reviewers = Invoke-RestMethod -Uri $pullRequestUrl -Method Get -Headers $basicAuth
return $reviewers.value
}

List of Google API endpoints call by the SDK

We have previously generated a list of Google's API end-points utilised by the SDK by grepping the source repo. Now that that doesn't seem to be available, has anyone else found a way of obtaining such a list? We need to be able to whitelist these end-points on our corporate firewall/proxy.
Thanks!
PART 1
If your objective is to whitelist URLs for your firewall, the URL *.googleapis.com will cover 99% of everything you need. There are only a few endpoints left:
bookstore.endpoints.endpoints-portal-demo.cloud.goog
cloudvolumesgcp-api.netapp.com
echo-api.endpoints.endpoints-portal-demo.cloud.goog
elasticsearch-service.gcpmarketplace.elastic.co
gcp.redisenterprise.com
payg-prod.gcpmarketplace.confluent.cloud
prod.cloud.datastax.com
PART 2
List the Google API endpoints that are available for your project with this command:
gcloud services list --available --format json | jq -r ".[].config.name"
https://cloud.google.com/sdk/gcloud/reference/services/list
Refer to PART 5 for a PowerShell script that produces a similar list.
PART 3
Process the Discovery Document which provides machine readable information:
Google API Discovery Service
curl https://www.googleapis.com/discovery/v1/apis | jq -r ".items[].discoveryRestUrl"
Once you have a list of discovery documents, process each document and extract the rootUrl key.
curl https://youtubereporting.googleapis.com/$discovery/rest?version=v1 | jq -r ".rootUrl"
PART 4
PowerShell script to process the Discovery Document and generate an API endpoint list:
Copy this code to a file named list_google_apis.ps1. Run the command as follows:
powershell ".\list_google_apis.ps1 | Sort-Object -Unique | Out-File -Encoding ASCII -FilePath apilist.txt"
There will be some errors displayed as some of the discovery document URLs result in 404 (NOT FOUND) errors.
$url_discovery = "https://www.googleapis.com/discovery/v1/apis"
$params = #{
Uri = $url_discovery
ContentType = 'application/json'
}
$r = Invoke-RestMethod #params
foreach($item in $r.items) {
$url = $item.discoveryRestUrl
try {
$p = #{
Uri = $url
ContentType = 'application/json'
}
$doc = Invoke-RestMethod #p
$doc.rootUrl
} catch {
Write-Host "Failed:" $url -ForegroundColor Red
}
}
PART 5
PowerShell script that I wrote a while back that produces similar output to gcloud services list.
Documentation for the API:
https://cloud.google.com/service-usage/docs/reference/rest/v1/services/list
<#
.SYNOPSIS
This program displays a list of Google Cloud services
.DESCRIPTION
Google Service Management allows service producers to publish their services on
Google Cloud Platform so that they can be discovered and used by service consumers.
.NOTES
This program requires the Google Cloud SDK CLI is installed and set up.
https://cloud.google.com/sdk/docs/quickstarts
.LINK
PowerShell Invoke-RestMethod
https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-restmethod?view=powershell-5.1
Google Cloud CLI print-access-token Documentation
https://cloud.google.com/sdk/gcloud/reference/auth/print-access-token
Google Cloud API Documentation
https://cloud.google.com/service-infrastructure/docs/service-management/reference/rest
https://cloud.google.com/service-usage/docs/reference/rest/v1/services
https://cloud.google.com/service-infrastructure/docs/service-management/reference/rest/v1/services/list
#>
function Get-AccessToken {
# Get an OAuth Access Token
$accessToken=gcloud auth print-access-token
return $accessToken
}
function Display-ServiceTable {
Param([array][Parameter(Position = 0, Mandatory = $true)] $serviceList)
if ($serviceList.Count -lt 1) {
Write-Output "No services were found"
return
}
# Display as a table
$serviceList.serviceConfig | Select name, title | Format-Table -Wrap | more
}
function Get-ServiceList {
Param([string][Parameter(Position = 0, Mandatory = $true)] $accessToken)
# Build the url
# https://cloud.google.com/service-infrastructure/docs/service-management/reference/rest/v1/services/list
$url="https://servicemanagement.googleapis.com/v1/services"
# Build the Invoke-RestMethod parameters
# https://learn.microsoft.com/en-us/powershell/module/microsoft.powershell.utility/invoke-restmethod?view=powershell-5.1
$params = #{
Headers = #{
Authorization = "Bearer " + $accessToken
}
Method = 'Get'
ContentType = "application/json"
}
# Create an array to store the API output which is an array of services
$services = #()
# Google APIs page the output
$nextPageToken = $null
do {
if ($nextPageToken -eq $null)
{
$uri = $url
} else {
$uri = $url + "?pageToken=$nextPageToken"
}
try {
# Get the list of services
$output = Invoke-RestMethod #params -Uri $uri
} catch {
Write-Host "Error: REST API failed." -ForegroundColor Red
Write-Host "URL: $url" -ForegroundColor Red
Write-Host $_.Exception.Message -ForegroundColor Red
return $services
}
# Debug: Display as JSON
# $output | ConvertTo-Json
# Append services to list
$services += $output.services
$nextPageToken = $output.nextPageToken
} while ($nextPageToken -ne $null)
return $services
}
############################################################
# Main Program
############################################################
$accessToken = Get-AccessToken
$serviceList = Get-ServiceList $accessToken
Display-ServiceTable $serviceList
Command-line tool JQ