We currently have airflow DAGs running on an Astronomer cloud instance, and we’re trying to develop external functions that would eventually trigger these DAGs.
Referencing the documentation for this → Make Requests to the Airflow REST API | Astronomer Documentation
Making a call to the API would mean sending an HTTP request to:
https://[AIRFLOW-DOMAIN]/api/v1/[method to call]
[AIRFLOW-DOMAIN]: Use https://[your-base-domain]/[deployment-release-name]
i.e. https://gcpXXXY.us-east4.astronomer.io/gravitational-flux-XXXX/api/v1/pools
Not sure if it was necessary, but also set an environment variable AIRFLOW__API__AUTH_BACKEND = airflow.api.auth.backend.default
The issue is, instead of getting 200 or 401 or 403 responses, we’d always get some HTML like below when testing out API calls via the Postman app. When checking the HTML in the browser, it's basically a 404 Page see screenshot
What am I missing? Thanks in advance!
<!DOCTYPE html>
<html>
<head>
<meta name=“robots” content=“noindex”>
<!-- Environment variables injected from NGINX —>
.
.
.
<meta http-equiv=“X-UA-Compatible” content=“IE=edge” />
<meta charset=“utf-8”>
<title>Astronomer
.
.
</head>
<body>
<div id=“root”>
<script src="/assets/runtime.82af4b31.js">
<script src="/assets/2.36213ffe.js">
<script src="/assets/index.b05feb43.js">
</body>
</html>
You need to retrieve an access token first.
curl --location --request POST "https://auth.astronomer.io/oauth/token" \
--header "content-type: application/json" \
--data-raw '{
"client_id": "<api-key-id>",
"client_secret": "<api-key-secret>",
"audience": "astronomer-ee",
"grant_type": "client_credentials"}'
You can then use that token (24 hour expiry) and call the Airflow API on your Astronomer instance.
curl -X GET <your-deployment-url>/api/v1/dags \
-H 'Cache-Control: no-cache' \
-H 'Authorization: Bearer <your-access-token>'
More information can be found here: https://docs.astronomer.io/astro/airflow-api
Related
I am trying to publish a workbook through rest api to the tableau server.
This is the below curl request which I am importing in the postman:
curl "https://MY_SERVER/api/3.4/sites/site-id/workbooks" -X POST -H "X-Tableau-Auth:credentials token" -H "Content-Type: multipart/mixed;" -F "request_payload=#publish-workbook.xml" -F "tableau_workbook=#MY_WORKBOOK.twbx"
But it is throwing this below error:
<?xml version='1.0' encoding='UTF-8'?>
<tsResponse xmlns="http://tableau.com/api" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://tableau.com/api https://help.tableau.com/samples/en-us/rest_api/ts-api_3_18.xsd">
<error code="406000">
<summary>Bad Request</summary>
<detail>Content type 'multipart/form-data;boundary=--------------------------313748027746018569295475' not supported</detail>
</error>
</tsResponse>
Also we are trying to publish datasource using postman. But same type of error is coming. Kindly help regarding this issue.
I am trying to hit this below curl from postman:
curl "https://MY_SERVER/api/3.4/sites/site-id/workbooks" -X POST -H "X-Tableau-Auth:credentials token" -H "Content-Type: multipart/mixed;" -F "request_payload=#publish-workbook.xml" -F "tableau_workbook=#MY_WORKBOOK.twbx"
I am expecting to publish the workbook in tableau server.
References:
https://help.tableau.com/current/api/rest_api/en-us/REST/rest_api_ref_data_sources.htm#publish_data_source
https://help.tableau.com/current/api/rest_api/en-us/REST/rest_api_concepts_publish.htm
I'm using google cloud's "Hello World" demo for cloud functions but the URL it produces gives me an error:
Error: Page not found
The requested URL was not found on this server.
I follow the tutorial, check allow unauthenticated, etc yet the url trigger leads me to the error.
The curl response requested also returns an error:
curl -X POST MY_URL -H "Content-Type:application/json" -d '{"name":"Jane"}'
returns:
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>404 Page not found</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Page not found</h1>
<h2>The requested URL was not found on this server.</h2>
<h2></h2>
</body></html>
Any ideas why?
edit: testing works just fine
Edit2: the url in MY_URL is structured like this:
curl -X POST https://us-west1-PROJECT-NAME.cloudfunctions.net/FUNCTION-NAME -H "Content-Type:application/json" -d '{"message":"Jane"}'
According to this GCP doc:
As of January 15, 2020, all HTTP functions by default require most invokers to be authenticated. To allow unauthenticated invocation you must specify this at or after deployment.
After deployment, manually add the Cloud Functions Invoker permission to allUsers users in the Cloud Functions page in the Google Cloud Console.
However, it's always a best practice to set authorization on your cloud functions.
I was able to curl the endpoint successfully using:
curl -X POST https://us-central1-<project_name>.cloudfunctions.net/<function_name> -H "Authorization:
bearer $(gcloud auth print-identity-token)" -H "Content-Type:application/json" -d '{"name": "Jane"}'
Output:
I followed this page:
https://cloud.google.com/text-to-speech/docs/quickstart-protocol
I generated an access token with:
gcloud auth application-default print-access-token
and upon entering this:
curl -H "Authorization: Bearer "$(my_token_is_in_here) \
-H "Content-Type: application/json; charset=utf-8" \
--data "{
'input':{
'text':'Android is a mobile operating system developed by Google,
based on the Linux kernel and designed primarily for
touchscreen mobile devices such as smartphones and tablets.'
},
'voice':{
'languageCode':'en-gb',
'name':'en-GB-Standard-A',
'ssmlGender':'FEMALE'
},
'audioConfig':{
'audioEncoding':'MP3'
}
}" "https://texttospeech.googleapis.com/v1/text:synthesize" > synthesize-text.txt
I got this:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 605 0 297 100 308 297 308 0:00:01 --:--:-- 0:00:01 3517
Buth then I opened up synthesize-text.txt and found
{
"error": {
"code": 401,
"message": "Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.",
"status": "UNAUTHENTICATED"
}
}
Clearly gcloud is giving me a token, why doesnt text-to-speech recognize it?
[SOLVED]
1) I went to APIs and Services in the 'hamburger' drop down
2) I went to 'credentials' submenu in left column
3) I did 'create credentials' and selected API key
then in my request I formulated as
Curl -H "X-Goog-Api-Key: my-newly-created-API-key" -H "Content-Type: application/json; charset=utf-8" --data "{'input':{'text':'Android is a mobile operating system developed by Google, based on the Linux kernel and designed primarily for touchscreen mobile devices such as smartphones and tablets.'},'voice':{'languageCode':'en-gb','name':'en-GB-Standard-A','ssmlGender':'FEMALE'},'audioConfig':{'audioEncoding':'MP3'}}" "https://texttospeech.googleapis.com/v1beta1/text:synthesize" > synthesize-text.txt
now it works.
So it seems the use of 'Authorization Bearer' with the service account key does not work.
I am using Windows 10.
Like Fredrik has said in his comment, there is an error in Google's Docs and it should be: -H "Authorization: Bearer $(my_token_is_in_here)" (" at the end, not after Bearer)
Trying to fetch EC2 userdata from win EC2 instance
http://169.254.169.254/latest/user-data
I get 404 - Not Found error
I believe you are experiencing that because you have no EC2 user-data attached
I conducted an experiment in my AWS Console. I launched two identical EC2 instances based an Ubuntu 18.04 image. However with one of the instances I attached user-data, the other I didn't
Instance WITH user-data
$ curl http://169.254.169.254/latest
dynamic
meta-data
user-data
$ curl http://169.254.169.254/latest/user-data
(prints my specified user-data)
Instance WITHOUT user-data
$ curl http://169.254.169.254/latest
dynamic
meta-data
(notice the absence of user-data)
$ curl http://169.254.169.254/latest/user-data
.... <title>404 - Not Found</title> ....
This answers the OPs questions, but not what John Bresnahan experienced
You MUST use the "/" at the end for all URIs.
curl http://169.254.169.254/latest/meta-data/
curl http://169.254.169.254/latest/user-data/
Alternatively, you can use the DNS record "instance-data" instead the IP. e.g.:
curl http://instance-data/latest/user-data/
I am seeing this as well:
root#ip-172-31-29-121:/var/lib/cloud# curl http://169.254.169.254/latest/
dynamic
meta-data
user-dataroot#ip-172-31-29-121:/var/lib/cloud#
root#ip-172-31-29-121:/var/lib/cloud# curl http://169.254.169.254/latest/user-data
<?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>404 - Not Found</title>
</head>
<body>
<h1>404 - Not Found</h1>
</body>
</html>
Note that the first curl command indicates that the data should be there.
Curl successfully uploads the file to S3 using a signed url:
curl -v -k -X PUT \
-H "x-amz-server-side-encryption: AES256" \
-H "Content-Type: application/pdf" \
-T "__tests__/resources/test.pdf" \
"http://mybucket.s3.amazonaws.com/test.pdf?AWSAccessKeyId=IDKEY&Expires=1489458783&Signature=SIGNATURE
I've tried replicating this in Grails using the REST client plugin:
String url = "http://mybucket.s3.amazonaws.com/test.pdf?AWSAccessKeyId=IDKEY&Expires=1489458783&Signature=SIGNATURE"
RestResponse resp = rest.put(url){
header "x-amz-server-side-encryption", "AES256"
header "Content-Type", "application/pdf"
body pdf
}
But Amazon rejects the upload, saying the arguments are incorrect...probably due to the pdf being sent as a "body" parameter. Any ideas?
Instead of using a rest client to upload it would be simpler to use the AWS Java SDK in your Grails app.
See an example here of using a pre-signed url to upload http://docs.aws.amazon.com/AmazonS3/latest/dev/PresignedUrlUploadObjectJavaSDK.html