How can I get a partial response for Method: instances.aggregatedList Compute API in GCP - google-cloud-platform

I am trying to get specific response from Compute API method instances.aggregatedList by setting the fields request param as per https://cloud.google.com/resource-manager/docs/performance#partial-response
But I am getting 400 BAD REQUEST.
Is there a sample which I can refer for getting partial response for aggregated methods?

If you use the following CURL command:
curl -H "Authorization: Bearer "$(gcloud auth application-default print-access-token) "https://compute.googleapis.com/compute/v1/projects/[CHANGE-FOR-YOUR-PROJECT-ID]/aggregated/instances?maxResults=1"
You'll notice that the result will have a similar form to:
{
"id": "projects/[PROJECT-ID]/aggregated/instances",
"items": {
"zones/us-central1-a": {
"instances": [
{
"id": "[INSTANCE-ID]",
"creationTimestamp": "2020-09-21T06:22:21.604-07:00",
"name": "instance-1",
"description": "",
"tags": {
"items": [
"http-server",
"https-server"
],
"fingerprint": "XXXXXX"
},
"machineType": "https://www.googleapis.com/compute/v1/projects/[PROJECT-ID]/zones/us-central1-a/machineTypes/e2-medium",
"status": "RUNNING",
"zone": "https://www.googleapis.com/compute/v1/projects/[PROJECT-ID]/zones/us-central1-a",
"canIpForward": false,
"networkInterfaces": [
{
"network": "https://www.googleapis.com/compute/v1/projects/[PROJECT-ID]/global/networks/default",
"subnetwork": "https://www.googleapis.com/compute/v1/projects/[PROJECT-ID]/regions/us-central1/subnetworks/[SUBNETWORK_NAME]",
"networkIP": "10.8.0.13",
"name": "nic0",
... with a lot more fields
As you can see the result is a little bit different as the response body explained in the documentation:
{
"id": string,
"items": [
{
"scopeName": string,
"instances": [
{
"id": string,
"creationTimestamp": string,
"name": string,
"description": string,
"tags": {
"items": [
string
],
"fingerprint": string
},
"machineType": string,
"status": enum,
"statusMessage": string,
"zone": string,
"canIpForward": boolean,
"networkInterfaces": [
{
"network": string,
"subnetwork": string,
"networkIP": string,
"ipv6Address": string,
"name": string,
.... with a lot more fields
Notice that if you compare both results, the actual response that you receive has an additional "zones/us-central1-a": field before the instances: field that I believe is causing the behavior you experience.
If you are interested in working with partial resources and get only some particular fields on the response you simply need to respect the syntax rules explained on the documentation you've shared and use the escape characters accordingly on your query parameters.
E.g. if you are only interested in getting the id of your project as well as the instances' name, machineType and status I tested the following curl command from the Cloud Shell with my GCP project and it worked without issues:
curl -H "Authorization: Bearer "$(gcloud auth application-default print-access-token) "https://compute.googleapis.com/compute/v1/projects/[PROJECT-ID]/aggregated/instances?fields=id,items/zones%2Finstances(name,machineType,status)"
where I see that something similar to the following is returned:
{
"id": "projects/[PROJECT-ID]/aggregated/instances",
"items": {
"zones/us-central1-a": {
"instances": [
{
"name": "instance-1",
"machineType": "https://www.googleapis.com/compute/v1/projects/[PROJECT-ID]/zones/us-central1-a/machineTypes/e2-medium",
"status": "RUNNING"
},
{
"name": "instance-2",
"machineType": "https://www.googleapis.com/compute/v1/projects/[PROJECT-ID]/zones/us-central1-a/machineTypes/e2-medium",
"status": "TERMINATED"
}
]
}
}
}

Related

Data Fusion could not parse response from JSON

I am using the CDAP reference to start a Data fusion batch pipeline(GCS to GCS).
curl -w "\n" -X POST -H "Authorization: Bearer ${AUTH_TOKEN}" \
"${CDAP_ENDPOINT}/v3/namespaces/default/apps/${PIPELINE_NAME}/workflows/DataPipelineWorkflow/start" \
-d "{ argfile : '${ARGUMENT_FILE}' }"
where argfile is the input URL to the argument setter macro
The argument json file is:
{
"arguments": [
{
"name": "input.path",
"type": "string",
"value": "gs://bucket/employee_sample.csv"
},
{
"name": "directive",
"type": "array",
"value": [
"parse-as-csv :body ',' true",
"drop body"
]
},
{
"name": "output.path",
"type": "string",
"value": "gs://bucket/FusionOutput"
},
{
"name": "PROJECT_ID",
"type": "string",
"value": "project-id"
},
{
"name": "output.schema",
"type": "schema",
"value": [
{
"name": "emp_id",
"type": "string",
"nullable": true
},
{
"name": "name_prefix",
"type": "string",
"nullable": true
},
{
"name": "first_name",
"type": "string",
"nullable": true
}
]
}
]
}
The problem here is that I am receiving this error
Pipeline 'gcs_fusion' failed.
io.cdap.cdap.api.macro.InvalidMacroException: Argument 'input.path' is not defined.
at io.cdap.cdap.etl.common.DefaultMacroEvaluator.lookup(DefaultMacroEvaluator.java:54) ~[na:na]
at io.cdap.cdap.internal.app.runtime.plugin.MacroParser.findRightmostMacro(MacroParser.java:144)
java.lang.RuntimeException: Could not parse response from 'https://storage.cloud.google.com/bucket/argumentj.json': java.lang.IllegalStateException: Expected BEGIN_OBJECT but was STRING at line 2 column 1
at io.cdap.plugin.ArgumentSetter.handleResponse(ArgumentSetter.java:83) ~[na:na]
at io.cdap.plugin.http.HTTPArgumentSetter.run(HTTPArgumentSetter.java:76) ~[na:na]
The same pipeline works with same template in some runs when I try to pass the argument setter URL through the Data Fusion UI Console.
The argument setter JSON follows the syntax prescribed in https://github.com/data-integrations/argument-setter and the argument json file is set to public in storage bucket.
Would be helpful if someone could resolve this.
Note that I've anonymized my project details here
Assuming your bucket is publicly accessible, then the URL you want to provide to the argument setter have the following pattern:
https://storage.googleapis.com/[BUCKET_NAME]/[OBJECT_NAME]
If the bucket is not publicly accessible, you will need to generate a signed URL for the argument file in Google Cloud Storage. You can read more about how to generate signed URL here.
I've had the same problem, and I resolved it by making my bucket public, using the command below:
gsutil iam ch allUsers:objectViewer gs://BUCKET_NAME

List users as non admin with custom fields

As per the documentation, I should be able to get a list of users with a custom schema as long as the field in the schema has a value of ALL_DOMAIN_USERS in the readAccessType property. That is the exact set up I have in the admin console; Moreover, when I perform a get request to the schema get endpoint for the schema in question, I get confirmation that the schema fields are set to ALL_DOMAIN_USERS in the readAccessType property.
The problem is when I perform a users list request, I don't get the custom schema in the response. The request is the following:
GET /admin/directory/v1/users?customer=my_customer&projection=full&query=franc&viewType=domain_public
HTTP/1.1
Host: www.googleapis.com
Content-length: 0
Authorization: Bearer fakeTokena0AfH6SMD6jF2DwJbgiDZ
The response I get back is the following:
{
"nextPageToken": "tokenData",
"kind": "admin#directory#users",
"etag": "etagData",
"users": [
{
"externalIds": [
{
"type": "organization",
"value": "value"
}
],
"organizations": [
{
"department": "department",
"customType": "",
"name": "Name",
"title": "Title"
}
],
"kind": "admin#directory#user",
"name": {
"fullName": "Full Name",
"givenName": "Full",
"familyName": "Name"
},
"phones": [
{
"type": "work",
"value": "(999)999-9999"
}
],
"thumbnailPhotoUrl": "https://photolinkurl",
"primaryEmail": "user#domain.com",
"relations": [
{
"type": "manager",
"value": "user#domain.com"
}
],
"emails": [
{
"primary": true,
"address": "user#domain.com"
}
],
"etag": "etagData",
"thumbnailPhotoEtag": "photoEtagData",
"id": "xxxxxxxxxxxxxxxxxx",
"addresses": [
{
"locality": "Locality",
"region": "XX",
"formatted": "999 Some St Some State 99999",
"primary": true,
"streetAddress": "999 Some St",
"postalCode": "99999",
"type": "work"
}
]
}
]
}
However, if I perform the same request with a super admin user, I get an extra property in the response:
"customSchemas": {
"Dir": {
"fieldOne": false,
"fieldTwo": "value",
"fieldThree": value
}
}
My understanding is that I should get the custom schema with a non admin user as long as the custom schema fields are set to be visible by all domain users. This is not happening. I opened a support ticket with G Suite but the guy that provided "support", send me in this direction. I believe this is a bug or maybe I overlooked something.
I contacted G Suite support and in fact, this issue is a domain specific problem.
It took several weeks for the issue to be addressed by the support engineers at Google but it was finally resolved. The behaviour is the intended one now.

How to use Google Data Prep API using Python

Google Just launched the new API. Link is here.
I want to know what is the host in this case as they are using example.com and using the port 3005.
I am also following this article. But this does not provide example code.
If you open the Dataprep Console and navigate to Settings > Access Tokens you can click on Generate New Token. This will bring up the token, that you can copy to clipboard, but also the instructions on which base endpoint to use:
Then, clicking on a particular Recipe (see image below) will modify the browser URL to be in the form of:
https://clouddataprep.com/flows/<FLOW_ID>?recipe=<RECIPE_ID>&tab=recipe
We'll keep RECIPE_ID> so that our request body (dataprep-request.json) is something like this:
{
"wrangledDataset": {
"id": <RECIPE_ID>
}
}
Then, we can call JobGroups Create:
curl https://api.clouddataprep.com/v4/jobGroups \
-X POST \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d #dataprep-request.json
The response will be similar to this output (I used python3 -m json.tool to pretty print the JSON):
{
"sessionId": "<SESSION_ID>",
"reason": "JobStarted",
"jobGraph": {
"vertices": [
4479390,
4479391
],
"edges": [
{
"source": 4479390,
"target": 4479391
}
]
},
"id": <JOB_GROUP_ID>,
"jobs": {
"data": [
{
"id": 4479390
},
{
"id": 4479391
}
]
}
}
Now, with the retrieved <JOB_GROUP_ID> we can use the JobGroups Get endpoint:
curl https://api.clouddataprep.com/v4/jobGroups/<JOB_GROUP_ID> \
-H "Authorization: Bearer $TOKEN"
Response:
{
"id": <JOB_GROUP_ID>,
"name": null,
"description": null,
"ranfrom": "ui",
"ranfor": "recipe",
"status": "InProgress",
"profilingEnabled": true,
"runParameterReferenceDate": "2019-12-08T21:49:33.000Z",
"createdAt": "2019-12-08T21:49:35.000Z",
"updatedAt": "2019-12-08T21:49:36.000Z",
"workspace": {
"id": REDACTED
},
"creator": {
"id": REDACTED
},
"updater": {
"id": REDACTED
},
"snapshot": {
"id": 4226057
},
"wrangledDataset": {
"id": <RECIPE_ID>
},
"flowRun": null
}

Autodesk BIM360: Folder Creation returns 400 Bad request

I called the POST-Folder Method via Postman with a JSON body according to this example. But I only receive the message "400 Bad Request" without any explanations. This is what my request looks like:
The service adress:
https://developer.api.autodesk.com/data/v1/projects/b.5823d0b2-0000-0000-00/commands
The HTTP-header
Authorization: Bearer 2_legged_token
Content-Type: application/vnd.api+json
The JSON-Body
{
"jsonapi": {
"version": "1.0"
},
"data": {
"type": "commands",
"attributes": {
"extension": {
"type": "commands:autodesk.core:CreateFolder",
"version": "1.0.0",
"data": {
"requiredAction": "create"
}
}
},
"relationships": {
"resources": {
"data": [
{
"type": "folders",
"id": "1"
}
]
}
}
},
"included": [
{
"type": "folders",
"id": "1",
"attributes": {
"name": "test",
"extension": {
"type": "folders:autodesk.bim360:Folder",
"version": "1.0.0"
}
},
"relationships": {
"parent": {
"data": {
"type": "folders",
"id": "urn:adsk.wipprod:fs.folder:co.Ai*****"
}
}
}
}
]
}
The response
{
"jsonapi": {
"version": "1.0"
},
"errors": [
{
"id": "f1266e76-a37e-400b-bff6-de84b11cdb00",
"status": "400",
"detail": "BadRequest"
}
]
}
What I have found out so far:
The project id is right. When I take a wrong project id I receive a different error.
The Json is also valid.
When I take a (surely) wrong parent-folder-urn I'll receive the same error message. So maybe this is a wrong urn format or something?
As of now, you can create a BIM 360 Docs Folder with commands endpoints, as you pointed out. For that you can use:
3-legged token
2-legged token with x-user-id header, this should contain the Autodesk User ID obtained, for instance, from GET users#me endpoint
"pure" 2-legged token will return bad request (as of August/2017)
Sorry about the documentation, the endpoint to create BIM 360 Docs folder via Commands was released a couple weeks back and we're just finishing to write the documentation.

Drupal-8 restful webservices only delivering HTML

I have a freshly configured Drupal 8.0 beta 12 with HAL, HTTP Basic Authentication, RESTful Web Services and Serialization modules enabled. The installation is in a subfolder (D8_beta12). When I access the web services for the first page created (id=1) like
curl -H "Accept: application/json" --request GET http://localhost/d8_beta12/node/1
DRUPAL only delivers HTML format of the node. I have tried to configure via REST UI to JSON only but that didn't help. DRUPAL would answer the requests even if the web services are disabled.
Its probably the user, who's sitting in front of DRUPAL, who is the problem (me) - but I don't get it. Any help deply appreciated - I have been working hours on this, reviewing all of DRUPALs forums and here as well.
Thanks,
Andi
I was getting the same and found a note about it https://drupal.stackexchange.com/questions/161421/d8-services-errors-no-route-found-for-the-specified-formats
So I updated the request to :
node/1?_format=json
Update: Found note about it in a webchick slideshare:
http://www.slideshare.net/webchickenator/plain-english-guide-to-drupal-8-criticals/51
Please try replacing aaplication/json with application/hal+json.
curl -H "Accept: application/hal+json" --request GET http://localhost/d8_beta12/node/1
You may also like to go through this precise tutorial.
from postman if you are trying or from other rest api you are trying make sure you follow below format
http://localhost/da/drupal819/node/253?_format=json
output
{
"nid": [
{
"value": "253"
}
],
"uuid": [
{
"value": "6f255c93-6886-4c38-b4cb-c40469073d1c"
}
],
"vid": [
{
"value": "253"
}
],
"langcode": [
{
"value": "en"
}
],
"type": [
{
"target_id": "article",
"target_type": "node_type",
"target_uuid": "ee2c7b5e-d57e-4714-9193-daff153c63ff"
}
],
"title": [
{
"value": "Commodo Pagus Quia"
}
],
"uid": [
{
"target_id": "14",
"target_type": "user",
"target_uuid": "0b8602ec-8003-4e3d-8406-5bd53e47df71",
"url": "/da/drupal819/user/14"
}
],
"status": [
{
"value": "1"
}
],
"created": [
{
"value": "1474126412"
}
],
"changed": [
{
"value": "1474175616"
}
],
"promote": [
{
"value": "1"
}
],
"sticky": [
{
"value": "0"
}
],
"revision_timestamp": [
{
"value": "1474175616"
}
],
"revision_uid": [
{
"target_id": "14",
"target_type": "user",
"target_uuid": "0b8602ec-8003-4e3d-8406-5bd53e47df71",
"url": "/da/drupal819/user/14"
}
],
"revision_log": [],
"revision_translation_affected": [
{
"value": "1"
}
],
"default_langcode": [
{
"value": "1"
}
],
"path": [],
"body": [
{
"value": "Duis revitas melior paulatim quibus quidne rusticus velit vereor.\n\n",
"format": "plain_text",
"summary": "Duis vitas melior paulatim quibus quidne rusticus velit vereor.\n\n"
}
],
"comment": [
{
"status": "2",
"cid": "165",
"last_comment_timestamp": "1474175616",
"last_comment_name": "",
"last_comment_uid": "13",
"comment_count": "1"
}
],
"field_image": [
{
"target_id": "248",
"alt": "Diam iriure neo quadrum refero valetudo verto ymo.",
"title": "Os patria refoveo si valetudo.",
"width": "225",
"height": "526",
"target_type": "file",
"target_uuid": "02050136-7a51-4183-9c28-7d7cc793183a",
"url": "http://localhost/da/drupal819/sites/default/files/2016-09/gen8C.tmp.jpeg"
}
],
"field_tags": []
}
you can try other format as well
1) http://localhost/da/drupal819/node/253?_format=hal_json
2) http://localhost/da/drupal819/node/253?_format=xml
to support all above format you need to make that format support from rest configuration from admin panel
and even you need to provide get permission for rest call from permission page for anonymous user.