Env : We using GCP cloud composer to run airflow dags.
Aim : To use apache airflow stable apis to trigger dags externally using REST.
We tried to use airflow experimental apis to trigger dags externally by setting airflow override configuration in cloud composer: airflow.api.auth.backend.default and making IAP request. It worked fine. We followed steps described in https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf.
Now starting from Airflow 2.0.0 + , airflow experimental apis are deprecated and new stable apis are introduced. The new stable api doesn't supports airflow.api.auth.backend.default. Hence it is throwing :403 FORBIDDEN.
Also we tried with auth_backend : airflow.api.auth.backend.basic_auth , but this also not working as when we pass user/password in Authorization header, we are not able to pass IAP Bearer token.
Please share the step to access new airflow apis similar to steps shown to access experimental apis in https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf
Do we need to do double Auth i.e first Authorization with IAP and then with Airflow ?
TL'DR version:
In order to make Airflow Stable API work at GCP Composer:
Set "api-auth_backend" to "airflow.composer.api.backend.composer_auth"
Make sure your service account email length is <64 symbols
Make sure your service account has required permissions (Composer User role should be sufficient)
Longread:
We are using Airflow for a while now, and started with version 1.x.x with "experimental" (now deprecated) API's.
To Authorize, we are using "Bearer" token obtained with service account:
# Obtain an OpenID Connect (OIDC) token from metadata server or using service account.
google_open_id_connect_token = id_token.fetch_id_token(Request(), client_id)
# Fetch the Identity-Aware Proxy-protected URL, including an
# Authorization header containing "Bearer " followed by a
# Google-issued OpenID Connect token for the service account.
resp = requests.request(
method, url,
headers={'Authorization': 'Bearer {}'.format(
google_open_id_connect_token)}, **kwargs)
Now we are migrating to Airflow 2.x.x and faced with exact same issue:
403 FORBIDDEN.
Our environment details are:
composer-1.17.3-airflow-2.1.2 (Google Cloud Platform)
"api-auth_backend" is set to "airflow.api.auth.backend.default".
Documentation claims that:
After you set the api-auth_backend configuration option to airflow.api.auth.backend.default, the Airflow web server accepts all API requests without authentication.
However, this does not seem to be true.
In experimental way, we found that if "api-auth_backend" is set to "airflow.composer.api.backend.composer_auth", Stable REST API (Airflow 2.X.X) starting to work.
But there is other caveat to this: for us, some of our service accounts did work, and some did not.
The ones that did not work were throwing "401 Unauthorized" error.
We figured out that accounts having email length > 64 symbols were throwing error. Same was observed at this answer.
So after setting "api-auth_backend" to "airflow.composer.api.backend.composer_auth" and making sure that our service account email length is <64 symbols - our old code for Airflow 1.x.x started to work for Authentication. Then we needed to make changes (API URLs and response handling) and stable Airflow (2.x.x) API started to work for us
in the same way as it was for Airflow 1.x.x.
UPD: this is a defect in Airflow and will be fixed here: https://github.com/apache/airflow/pull/19932
Cloud Composer's support for Airflow2 stable API starts from composer-1.17.0-preview.12 onwards.
The how-to documentation is here:
Access Airflow REST API
Trigger DAG with Cloud Function
Related
I'm trying to setup a VPN connection using a federated login with Google IdP following these instructions.
Previously, I had configured a saml-provider with Google and it worked fine to authenticate users to the AWS console through Google using ARN roles
WHen I setup the VPN connection, it successfully opens the browser and asks me to select my google account, but after selecting the account I'm getting an error message from Google
According to this help section
Verify that the value in the saml:Issuer tag in the SAMLRequest matches the Entity ID value configured in the SAML Service Provider Details section in the Admin console. This value is case-sensitive.
So this is a problem coming from AWS and not from me ? Is Google IdP compatible at all with VPN authentication ? (I found this doc that mentions compatibility with okta)
Edit
Thanks to some of the answers below, I managed to make it work with Google IdP. Here is a screenshot of relevant SAML Google app screens (note that for groups I ended up adding the employees department, but I guess anything else would have worked)
To be able to save an ACS URL starting with http:// in the G Suite interface, use the trick given by teknowlogist: open the inspector > network tab, perform the request to save an URL with https, then right-click copy it as cURL, replace https by http, paste in regular console, and you're good.
I found a workaround to not being able to input http://127.0.0.1:35001 as the ACS URL on the GSuite SAML app page. The Google admin console only does client-side validation for the https requirement, so you can use the Chrome console to monitor the network call made when modifying the ACS URL.
Then, you can copy this as a curl command and change https to http
#Ted Schroeder —
Previous approach (or, plain Google doesn't work)
I just used a reverse proxy:
mitmproxy \
--listen-port 35000 \
--mode 'reverse:http://127.0.0.1:35001' \
--set keep_host_header=true
If you change Google SAML's ACS URL to be https://127.0.0.1:35000 and click "Test SAML Login", Google will take you to https://127.0.0.1:35000, whose traffic will be redirected to http://127.0.0.1:35001. In the browser I get:
Authentication details received, processing details. You may close this window at any time.
However, using the SAML-tracer extension, I found that there was a URL mismatch (https://127.0.0.1:35000 vs. http://127.0.0.1:35001). Seems like the AWS VPN Client is broadcasting its expected URL as being http://127.0.0.1:35001. So this doesn't seem viable.
Current approach (or, Auth0+Google works)
I tried using Auth0 instead, and got it to work! There's a few hoops — for instance, create a new Auth0 application, go to Addons and enable SAML2 Web App, set Application Callback URL to http://127.0.0.1:35001, and then in Settings use the following:
{
"audience": "urn:amazon:webservices:clientvpn",
"mappings": {
"user_id": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier",
"email": "NameID",
"name": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name",
"given_name": "FirstName",
"family_name": "LastName",
"upn": "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/upn",
"groups": "memberOf"
},
"binding": "urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect",
"signResponse": true
}
Then users, if they download the VPN config from AWS and use the AWS VPN Client app, will be taken to an Auth0 login screen where they can login via Google. Voila! (And then for security, you need to add Auth0 Rules to grant only certain users/groups authorization.)
I don't have a full answer yet, but I have the beginnings of one and I actually got past the 403 error above. The key to all this can be found in the AWS Client VPN information here: https://docs.aws.amazon.com/vpn/latest/clientvpn-admin/client-authentication.html
Look for the section entitled "Service provider information for creating an app".
The key is that these are the ACS URL and the Entity ID that need to be used. Unfortunately, G Suite won't let you set the ACS URL to a non-https URL and apparently the AWS Client VPN app won't provide a secure URL for the ACS URL (where the SAML Authenticate response goes).
So, if you set the Entity ID to "urn:amazon:webservices:clientvpn" and have the G Suite SAML app in place according to the instructions, you'll get past the 403. However, since the ACS URL can't be specified you get whatever error message you're likely to get from the ACS URL that the authentication response goes to.
Example scenario
If you set it to https://signon.aws.amazon.com/saml" like you would for AWS Console SSO, you get an error from the AWS sign in that the SAML response was invalid.
And if you set it to https://127.0.0.1:35001 then you get a message from the browser that the "site can't provide a secure connection".
If anybody gets any further with this, I'd love to hear about it. In the meanwhile, I'm going to be looking into non-AWS OpenVPN clients that might actually support G Suite as a SAML IdP.
#alexandergunnarson
Since I don't have the ability to comment (thanks so much for making this easy stackOverflow) I had to edit my answer to get it past the censors.
Unfortunately, we don't have, and probably won't have for some time, G Suite Enterprise because it's too expensive for our startup environment. So OIDP is not a viable option for us now. I figured this would work. Good to know that it does.
I was too having the same issue. In my case, I needed to turn on the two-factor authentication for the account that I was trying to log in with.
Is there any way to restart a VM on GCP from Internet using python, bash or php?
I need to restart when I get some notification.
Thanks!
You need 2 things to do this:
Being authenticated
Use API to communicate with Compute Engine Google APIs (or client libraries that embed the API call)
Authentication
The easiest way is to use gcloud sdk CLI. If you use your personal computer, you can use your personal user account, else, you can use a service account with a service account key file.
If you use a client library, you don't need to have gcloud sdk installed, only the service account key file is enough. One good practice is to define an environment variable GOOGLE_APPLICATION_CREDENTIALS=/path/to/serviceAccountKeyFile.json.
It's a good practice because you don't hardcode the way of authenticating your code. The library search by itself the credential in several location and you can run the same code seamlessly in your local environment and on GCP. No hack, do dirty code, no different entry point.
API Call
If you use gcloud, you can reset your VM by using the REST API
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" -X POST \
https://compute.googleapis.com/compute/v1/projects/<MyProjectId>/zones/<ComputeZone>/instances/<InstanceName>/reset
You can also use the stop (wait 90s) and then restart your VM. Same authentication, juste the URL are different. It's important to wait at least the max stop grace period of 90s before triggering a start
Sadly, there isn't Compute Engine client libraries. You can only use the discovery API. Not as handful as the other existing libraries. You have an example here. To reset, you have to call this method in Python
compute = googleapiclient.discovery.build('compute', 'v1')
result = compute.instances().reset(project=<MyProjectId>, zone=<ComputeZone>, instance=<InstanceName>).execute()
I've built a JHipster application with oauth2 authentication. The latter is implemented by Keycloak through the generated docker-compose file.
Everything works fine in the browser for "human" users. But I need to make some external programs use the API while beng authenticated.
So I started to simulate direct access to API with Postman. I read about XSRF-TOKEN cookie. But to be frank, I don't understand the process of authentication.
Can anyone explain how to perform authenticated requests to the JHipster API regarding keycloak ?
Maybe it isn't the appropriate approach : I also read about implementing a Configuration based on another authentication mecanism, that should be used for controllers exposed on a different endpoint.
Any help figuring out all of that would be really appreciated !
You need machine to machine authentication. Generally Open ID Connect (OIDC) offers client credentials flow for this case. So in theory you just enable it in the Keycloak client configuration and you may use it. But it depends on your API auth implementation. Your API very likely uses different OIDC flow for humans and it may not be ready for client credentials flow.
I got it working: Jan's tip show me the way.
Mainly I followed this tutorial.
The solution would be to add a client configured with Standard flow, Service account and Authorization enabled. Then I added the JHipster client scope provided by the generated docker-compose setup.
Once that was configured, I configured the request to use the Oauth2 authentication. I filled the configuration form for token request according to the tutorial. And I was good to go !
Many thx !
Right now I don't managed to have the Google Cloud Platform Data Loss Prevention (DLP) client library for python working behind a SSL proxy (it works fine with other GCP client lib for example for storage or bigquery):
https://cloud.google.com/dlp/docs/libraries#client-libraries-usage-python
So I tried to use request.post to use the API behind a SSL proxy
url = 'https://dlp.googleapis.com/v2/projects/'+os.environ['PROJECT_ID']+'/content:inspect'
headers = {
'Content-Type': 'application/json',
'Authorization': 'Bearer {}'.format(subprocess.run('gcloud auth print-access-token', shell=True, check=True, stdout=subprocess.PIPE).stdout.decode().replace('\n', '').replace('\r', ''))
}
}
json_response = requests.post(url=url, json=parsed, headers=headers, proxies=proxies, verify=True)
json.loads(json_response.text)
This is working fine on CloudShell but not on my local machine where SDK is installed. The reason is that on CloudShell:
gcloud auth print-access-token
give me the same token for a period of few minutes while on my local machine (Windows or Mac), every time I execute the command, I got a new token. On my local machine if I replace in the header the gcloud command by the token from CloudShell it works fine. I have the latest version of SDK on both my local machine and on CloudShell.
question 1: it is expected that every time we run gcloud auth print-access-token locally (SDK), we get a new token ? (On CloudShell it is the same token for a period of few minutes)
question 2: what is the easiest/best way to generate a token ? since gcloud auth print-access-token doesn't seems the right way to do it when using local machine and SDK. This is not a productive application. This is just to test the DLP API.
question 1: it is expected that every time we run gcloud auth
print-access-token locally (SDK), we get a new token ? (On CloudShell
it is the same token for a period of few minutes)
The answer depends on where you run your code. When running from a Google compute service (Cloud Shell is a VM), the token comes from the metadata server. I am now sure if or how long the token is cached. Tokens have an expiration (default 3600 seconds), so it is easy for the metadata server to cache tokens. If your code is running outside of the Google Cloud, the answer depends on the library used.
question 2: what is the easiest/best way to generate a token ? since
gcloud auth print-access-token doesn't seems the right way to do it
when using local machine and SDK. This is not a productive
application. This is just to test the DLP API.
Obtaining tokens from the CLI is just for testing. The normal method is to use the SDK. However, since you are using the REST API, read this article that I wrote on how to create tokens in your own code and use them in REST APIs. My article includes Python source code and an example calling the Compute API to list instances in a project.
Google Cloud – Creating OAuth Access Tokens for REST API Calls
I have a Bluemix web application (Liberty for Java), which implements some web services. These web services should be called from a mobile application (Android). I have now secured this web application by binding it to the Bluemix Single Sign On service (SSO) with a cloud directory created in the SSO service. Using the web app from a web browser works fine; but, I have problems obtaining an access token from the SSO service, which would allow the mobile application to invoke the services.
From the OAuth2 specification (IETF RFC 6749), I figured that the appropriate way of doing this would be the "native application" profile with a "public client" (as specified in Clause 2.1 of the OAuth2 Spec) using the "password" grant type (OAuth2 spec, Clause 4.3 "Resource Owner Password Credentials Grant").
I used the Spring for Android framework for this purpose, and code for this would look like this:
ResourceOwnerPasswordResourceDetails resourceDetails =
new ResourceOwnerPasswordResourceDetails();
resourceDetails.setId("dtu-se2-e15-cloud-directory");
resourceDetails.setAccessTokenUri(APP_SSO_API_ACCESS_TOKEN_URI);
resourceDetails.setClientId(APP_SSO_API_CLIENT_ID);
resourceDetails.setClientSecret(APP_SSO_API_CLIENT_SECRET);
resourceDetails.setGrantType("password");
resourceDetails.setScope(Arrays.asList(SCOPE));
resourceDetails.setUsername(USERNAME);
resourceDetails.setPassword(PASSWORD);
OAuth2RestTemplate restTemplate = new OAuth2RestTemplate(resourceDetails);
restTemplate.getMessageConverters().add(new MappingJackson2HttpMessageConverter());
UserPosition newPosition = restTemplate.postForObject(
POST_POSITION_SERVICE_URI, position, UserPosition.class);
But, I don't think that this code matters, and ultimately made my experiments with obtaining access tokens directly with curl:
Actually, I tried using two different clients with different OAuth2 token endpoints:
I created a client (and client credentials) in the Cloud directory, that I had created in the Bluemix SSO service. And I tried the OAuth2 Token Endpoint URI and the created client credentials. But it appears that this endpoint does not support the grant type "password" at all. It appears that this client is not considered a public client by the endpoint.
I also tried the credentials and OAuth Token Endpoint URI for the Web application itself (which I looked up in the VCAP_SERVICES environment variable). This end point seems to support the grant type "password"; but all variants of requests I could think of, kept responding: invalid_resource_owner_credential.
As I said, I used curl to try out many different variations of requests to these
token endpoints:
used the token endpoint URI of both the SSO services with the web app credentials
as well as the one created in the cloud directory API Access
tried GET and POST (did not make any difference)
tried Content-Type: application/x-www-form-urlencoded and
Content-Type: application/json (both of them seemed to work with the same effect)
providing the client_id only (which always was unsuccessful)
providing the client credentials in the body or parameters, as well as
authenication information in the header (actually, I do not like the idea
of providing the client secret to the Android app, but I tried that too);
as user name, I tried the name as I had created it in the web browser redirection
when registering a new user; but I also tried the user name which the Principal of
the security context of a request would provide (when successfully invoking
a service from a web browser with the user logged in); I even tried the principal's
accessId (non of these worked, I always got: invalid_resource_owner_credential)
used different scopes, and none at all
None of the above (and different combinations of that) would result in a successful
response and an access token for the user. The "best" I could get was a response
invalid_resource_owner_credential (making me believe that at least the client was
accepted in some situations).
Unfortunately, I did not find many things that could be configured concerning client
access (and public clients, in particular), and I did not find much documentation on
which subset of the OAuth2 protocol (grant types and profiles) is supported by the
Bluemix SSO service and the attached cloud directory.
Can anyone could tell me how to authenticate with a Bluemix web application
(Liberty for Java) from a mobile app (Android) as a public client or how to set
up the Bluemix web app and the SSO service to which it is bound so that this is
possible. In case it would matter, I am working with Bluemix in the "US South"
region and under an IBM Academic Initiative membership account for Bluemix.
I would prefer a solution, where the mobile app would not need to know the client
secret, but if this is the only way to make this work for now, adding the client
credentials to the mobile app would be OK.
I would appreciate any help with this problem, thanks in advance,
Ekkart
Bluemix has a mobile-specific service called Mobile Client Access that could help to facilitate security for your mobile app. To read about it, log into Bluemix and look for it under the Mobile category. To ask questions about it here, use or search using the [bluemix-mobile-services] tag.