I want to get the results of multiple queries into single call onto freebase,which is there in this chapter http://mql.freebaseapps.com/ch04.html. I am using python for querying. I want to query like this
{ # Start the outer envelope
"q1": { # Query envelope for query named q1
"query":{First MQL query here} # Query property of query envelope
}, # End of first query envelope
"q2": { # Start query envelope for query q2
"query":[{Second MQL query here}] # Query property of q2
} # End of second query envelope
}
and get answers like
{
"q1": {
"result":{First MQL result here},
"code": "/api/status/ok"
},
"q2": {
"result":[{Second MQL result here}],
"code": "/api/status/ok"
},
"status": "200 OK",
"code": "/api/status/ok",
"transaction_id":[opaque string value]
}
As specified on that link. I also came across some of the question on SO, which are -
Freebase python
Multiple Queries in MQL on Freebase
But they seems to be using the old API which is "api.freebase.com". The updated API is "www.googleapis.com/freebase"
I tried the following code, but its not working.
import json
import urllib
api_key = "freebase_api_key"
service_url = 'https://www.googleapis.com/freebase/v1/mqlread'
query1 = [{'id': None, 'name': None, 'type': '/astronomy/planet'}]
query2 = [{'id': None, 'name': None, 'type': '/film/film'}]
envelope = {
'q1':query1,
'q2':query2
}
encoded = json.dumps(envelope)
params = urllib.urlencode({'query':encoded})
url = service_url + '?' + params
print url
response = json.loads(urllib.urlopen(url).read())
print response
I am getting error as
{u'error': {u'code': 400, u'message': u'Type /type/object does not have property q1', u'errors': [{u'domain': u'global', u'message': u'Type /type/object does not have property q1', u'reason': u'invalid'}]}}
How can I embed multiple queries into a single MQL query
I'd suggest looking at the Batch capability of the Python client library for the Google APIs.
Related
I'm trying to use Boto3 to get the number of vulnerabilities from my images in my repositories. I have a list of repository names and image IDs that are getting passed into this function. Based off their documentation
I'm expecting a response like this when I filter for ['imageScanFindings']
'imageScanFindings': {
'imageScanCompletedAt': datetime(2015, 1, 1),
'vulnerabilitySourceUpdatedAt': datetime(2015, 1, 1),
'findingSeverityCounts': {
'string': 123
},
'findings': [
{
'name': 'string',
'description': 'string',
'uri': 'string',
'severity': 'INFORMATIONAL'|'LOW'|'MEDIUM'|'HIGH'|'CRITICAL'|'UNDEFINED',
'attributes': [
{
'key': 'string',
'value': 'string'
},
]
},
],
What I really need is the
'findingSeverityCounts' number, however, it's not showing up in my response. Here's my code and the response I get:
main.py
repo_names = ['cftest/repo1', 'your-repo-name', 'cftest/repo2']
image_ids = ['1.1.1', 'latest', '2.2.2']
def get_vuln_count(repo_names, image_ids):
container_inventory = []
client = boto3.client('ecr')
for n, i in zip(repo_names, image_ids):
response = client.describe_image_scan_findings(
repositoryName=n,
imageId={'imageTag': i}
)
findings = response['imageScanFindings']
print(findings)
Output
{'findings': []}
The only thing that shows up is findings and I was expecting findingSeverityCounts in the response along with the others, but nothing else is showing up.
THEORY
I have 3 repositories and an image in each repository that I uploaded. One of my theories is that I'm not getting the other responses, such as findingSeverityCounts because my images don't have vulnerabilities? I have inspector set-up to scan on push, but they don't have vulnerabilities so nothing shows up in the inspector dashboard. Could that be causing the issue? If so, how would I be able to generate a vulnerability in one of my images to test this out?
My theory was correct and when there are no vulnerabilities, the response completely omits certain values, including the 'findingSeverityCounts' value that I needed.
I created a docker image using python 2.7 to generate vulnerabilities in my scan to test out my script properly. My work around was to implement this if statement- if there's vulnerabilities it will return them, if there aren't any vulnerabilities, that means 'findingSeverityCounts' is omitted from the response, so I'll have it return 0 instead of giving me a key error.
Example Solution:
response = client.describe_image_scan_findings(
repositoryName=n,
imageId={'imageTag': i}
)
if 'findingSeverityCounts' in response['imageScanFindings']:
print(response['imageScanFindings']['findingSeverityCounts'])
else:
print(0)
I have create an AppSynch API on AWS. I can easily query it using the console or some AWS specific package. However I would want to query it using a simple package such as e.g. urllib3. Its surprisingly hard to find anyone doing using a direct api call (everyone uses some kind of aws related packages or solutions that i cant seem to get to work). The query I want to do is:
mutation provision {
provision(
noteId:
{ec2Instance: "t2.micro",
s3Bucket: "dev"})}
I have tried with different variations of:
query = """
mutation provision {
provision(
noteId:
{ec2Instance: "t2.micro",
s3Bucket: "dev"})}
"""
headers = {"x-api-key": api_key}
http = urllib3.PoolManager()
data = json.dumps("query": query, "variables": {}, "operationName": "somename")
r = http.request('POST', url, headers=headers,
data=data.encode('utf8'))
But I somehow cannot get it to work, i keep on getting messages that the API cant understand the POST request
I found the solution:
import requests
import json
APPSYNC_API_KEY = APPSYNC_API_KEY
APPSYNC_API_ENDPOINT_URL = APPSYNC_API_ENDPOINT_URL
headers = {
'Content-Type': "application/graphql",
'x-api-key': APPSYNC_API_KEY,
'cache-control': "no-cache",
}
query = """
mutation provision {
provision(
inputParameters:
{ec2Instance: "t2.micro",
s3Bucket: "dev"})}
"""
payload_obj = {"query": query}
payload = json.dumps(payload_obj)
response = requests.request("POST", APPSYNC_API_ENDPOINT_URL, data=payload, headers=headers)
print(response)
Below is my code for writing into a BigQuery table:
from google.cloud import bigquery
response = bigquery.tabledata.insertAll(projectId=PROJECT_ID,datasetId=DATASET_ID,
tableId=TABLE_ID,
body=data).execute()
However, I'm getting the following error:
no module tabledata in google.cloud.bigquery
Can anyone help me with this?
Which API should I use here?
Please, check the Streaming data into BigQuery documentation. When using python you need to use following function:
insert_rows(table, rows, selected_fields=None, **kwargs)
which insert rows into a table via the streaming API. For more information refer to following BigQuery Python API reference documentation.
You can check Python streaming insert example:
# TODO(developer): Import the client library.
# from google.cloud import bigquery
# TODO(developer): Construct a BigQuery client object.
# client = bigquery.Client()
# TODO(developer): Set table_id to the ID of the model to fetch.
# table_id = "your-project.your_dataset.your_table"
table = client.get_table(table_id) # Make an API request.
rows_to_insert = [(u"Phred Phlyntstone", 32), (u"Wylma Phlyntstone", 29)]
errors = client.insert_rows(table, rows_to_insert) # Make an API request.
if errors == []:
print("New rows have been added.")
You can also use API, calling tabledata.insertAll method to see the request and response of API. You need to specify projectId, datasetId, tableId. You can see JavaScript code snippet that is used to perform request:
function execute() {
return gapi.client.bigquery.tabledata.insertAll({
"projectId": "<your_projectId>",
"datasetId": "<your_datasetId>",
"tableId": "<your_tableId>",
"resource": {}
})
.then(function(response) {
// Handle the results here (response.result has the parsed body).
console.log("Response", response);
},
function(err) { console.error("Execute error", err); });
}
Let me know about the results.
I made a Serverless API backend on AWS console which uses API Gateway, DynamoDB, Lambda functions.
Upon creation I can add the data in dynamoDB online by adding a JSON file, which looks like this:
{
"id": "4",
"k": "key1",
"v": "value1"
}
But when I try to add this using "Postman", by adding the above JSON data in the body of POST message, I get a Positive return (i.e. no errors) but only the "id" field is added in the database and not the "k" or "v".
What is missing?
I think that you need to check on your Lambda function.
As you are using Postman to do the API calls, received event's body will be as follows:
{'resource':
...
}, 'body': '{\n\t"id": 1,\n\t"name": "ben"\n
}', 'isBase64Encoded': False
}
As you can see:
'body': '{\n\t"id": 1,\n\t"name": "ben"\n}'
For example, I will use Python 3 for this case, what I need to do is to load the body into JSON format then we are able to use it.
result = json.loads(event['body'])
id = result['id']
name = result['name']
Then update them into DynamoDB:
item = table.put_item(
Item={
'id': str(id),
'name': str(name)
}
)
I would like to get the usage cost report of each instance in my aws account form a period of time.
I'm able to get linked_account_id and service in the output but I need instance_id as well. Please help
import argparse
import boto3
import datetime
cd = boto3.client('ce', 'ap-south-1')
results = []
token = None
while True:
if token:
kwargs = {'NextPageToken': token}
else:
kwargs = {}
data = cd.get_cost_and_usage(
TimePeriod={'Start': '2019-01-01', 'End': '2019-06-30'},
Granularity='MONTHLY',
Metrics=['BlendedCost','UnblendedCost'],
GroupBy=[
{'Type': 'DIMENSION', 'Key': 'LINKED_ACCOUNT'},
{'Type': 'DIMENSION', 'Key': 'SERVICE'}
], **kwargs)
results += data['ResultsByTime']
token = data.get('NextPageToken')
if not token:
break
print('\t'.join(['Start_date', 'End_date', 'LinkedAccount', 'Service', 'blended_cost','unblended_cost', 'Unit', 'Estimated']))
for result_by_time in results:
for group in result_by_time['Groups']:
blended_cost = group['Metrics']['BlendedCost']['Amount']
unblended_cost = group['Metrics']['UnblendedCost']['Amount']
unit = group['Metrics']['UnblendedCost']['Unit']
print(result_by_time['TimePeriod']['Start'], '\t',
result_by_time['TimePeriod']['End'],'\t',
'\t'.join(group['Keys']), '\t',
blended_cost,'\t',
unblended_cost, '\t',
unit, '\t',
result_by_time['Estimated'])
As far as I know, Cost Explorer can't treat the usage per instance. There is a function Cost and Usage Reports which gives a detailed billing report by dump files. In this file, you can see the instance id.
It can also be connected to the AWS Athena. Once you did this, then directly query to the file on Athena.
Here is my presto example.
select
lineitem_resourceid,
sum(lineitem_unblendedcost) as unblended_cost,
sum(lineitem_blendedcost) as blended_cost
from
<table>
where
lineitem_productcode = 'AmazonEC2' and
product_operation like 'RunInstances%'
group by
lineitem_resourceid
The result is
lineitem_resourceid unblended_cost blended_cost
i-***************** 279.424 279.424
i-***************** 139.948 139.948
i-******** 68.198 68.198
i-***************** 3.848 3.848
i-***************** 0.013 0.013
where the resourceid containes the instance id. The amount of cost is summed for all usage in this month. For other type of product_operation, it will contains different resource ids.
You can add an individual tag to all instances (e.g. Id) and then group by that tag:
GroupBy=[
{
'Type': 'TAG',
'Key': 'Id'
},
],