Launch Jupyter Notebooks in AWS Sagemaker from a Custom Webapplication - amazon-web-services

We have a requirement where we are building a Webportal/platform that will use services of AWS and Git as both will host certain content to allow users to search for certain artifacts.
We also want to allow a user after they have searched for certain artifacts (lets say certain jupyter notebooks) to be able to launch these notebooks from our web-application. Note the notebooks are in different domain i.e AWS Console application host them.
Now, When user click on the notebook links from the webportal search it should open up the Jupyter notebook in a notebook instance in a new tab.
We understand there is integration of AWS Sagemaker and GIT so some repos that will store notebooks can be configured. When user performs the search in webapp it will pick up the results from github API Call.
The same repos can also be added in the sagemaker-github integration through AWS Console. So when a user launches the notebook he will see the github repos as well.
I understand we call Sagemaker API either through SDK or Rest API(not sure there is a rest api interface exploring on that). See a CLI call example -
aws sagemaker create-presigned-notebook-instance-url --notebook-instance-name notebook-sagemaker-git
this gives me a response url "AuthorizedUrl": "https://notebook-sagemaker-git.notebook.us-east-2.sagemaker.aws?authToken=eyJhbGciOiJIUzI1NiJ9.eyJmYXNDcmVkZW50aWFscyI6IkFZQURlQlR1NHBnZ2dlZGc3VTJNcjZKSmN3UUFYd0FCQUJWaGQzTXRZM0o1Y0hSdkxYQjFZbXhwWXkxclpYa0FSRUZvUVZadGMxSjFSVzV6V1hGVGJFWmphRXhWUTNwcVlucDZaR2x5ZDNGQ1RsZFplV1YyUkRoTGJubHRWRzVQT1dWM1RTdDBTR0p6TjJoYVdXeDJabnBrUVQwOUFBRUFCMkYzY3kxcmJYTUFTMkZ5YmpwaGQzTTZhMjF6T25WekxXVmhjM1F0TWpvMk5qZzJOek15TXpJMk5UUTZhMlY1THpObFlUaGxNMk14TFRSaU56a3RORGd4T0
However, when i open this url it again asks me the aws console username and password. I feel in the webapp when i logged in a user would already authenticate himself through AWS API as well as GIT API.
So there should be no need to re-authenticate themselves when they connect to AWS-Console to access their notebooks.
Is it something that can be circumvent using SIngle sign on etc.
thanks,
Aakash

The URL that you get from a call to CreatePresignedNotebookInstanceUrl is valid only for 5 minutes. If you try to use the URL after the 5-minute limit expires, you are directed to the AWS console sign-in page. See https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_CreatePresignedNotebookInstanceUrl.html
Jun

Related

Automate cloud auth login with no human interaction

I have a virtual machine which runs the scripts daily using cloud services, but for every 150 days I need to manually remote login to VM and do gcloud auth login, and all the scripts fail if I don’t do this step, is there a way to write an shell or Python script and use it with crown job to complete this step with no human interaction. Manually I need to select the link which will open browser and select Allow but can I automate this using no-launch-browser( still no human interaction should be there).
You cannot configure user credentials to automatically re-authenticate. Those credentials require human interaction.
Instead, configure the CLI to use a service account.
gcloud auth activate-service-account
If you are running in Google Cloud, you can use the attached service account.
Compute Engine Service accounts
Using either method means that the CLI will automatically renew credentials.

Unable to connect from Google Cloud to Bitbucket repository using app password

I'm using GCP build triggers connected to Bitbucket repositories. The connection is made using user credentials. Bitbucket has announced they're ending support for account password usage:
Beginning March 1, 2022, Bitbucket users will no longer be able to use
their Atlassian account password when using Basic authentication with
the Bitbucket API or Git over HTTPS. For security reasons, we require
all users to use Bitbucket app passwords.
Problem is, when trying to connect to a repository in GCP, the only option to supply Bitbucket credentials is via a web login (which to the point of app passwords, you cannot login via the bitbucket.org with an app password).
GCP Bitbucket login prompt via bitbucket.org
Expected behavior: GCP provides an option to submit app password credentials when connecting to a Bitbucket repository.
I followed directions for GCP Cloud Build integration with Bitbucket Cloud and successfully built out a functioning trigger for my repository here. I only built the trigger in GCP and used the generated webhook URL when creating the webhook in Bitbucket: I didn't create SSH keys, nor is my cloudbuild.yaml entirely valid - so the builds are failing.
Access to the Bitbucket repository was provided through GCP GUI in Cloud Build.
I have been informed of this change as well. I am trying to understand the scope of the change and its impact. It states that you cannot log in Atlassian account and password. However, besides using app passwords, you can also log in using OAuth2. https://developer.atlassian.com/cloud/bitbucket/oauth-2/
In the case of GCP Build Triggers, when I first set up the Bitbucket repository to connect to, I need to go through the "Authorization Code Grant" flow and acknowledge what access I am granting to Google Cloud Source Repository. If you check the Bitbucket API endpoints being called, they are URLs that are being used for "Authorization Code Grant" flow.
Based on these findings, am I right to say that there is no necessity to change existing triggers or mirrored repositories on GCP since they are using OAuth2 in the first place instead of Atlassian accounts and passwords?
If you can setup the build trigger to be done by a webhook you can configure the build with ssh key. But if you have to configure it as a manual trigger then using the bitbucket login credentials is the only option. Personally, I don't like this config with user login though.
The only good thing is even now(after bitbucket stopped supporting the login credentials for code checkout) the code checkout in GCP is working fine.

In a containerized application that runs in AWS/Azure but needs to access GCLOUD commands, what is the best way to setup gcloud authentication?

I am very new to GCP and I would greatly appreciate some help here ...
I have a docker containerized application that runs in AWS/Azure but needs to access gcloud SDK as well as through "Google cloud client libraries".
what is the best way to setup gcloud authentication from an application that runs outside of GCP?
In my Dockerfile, I have this (cut short for brevity)
ENV CLOUDSDK_INSTALL_DIR /usr/local/gcloud/
RUN curl -sSL https://sdk.cloud.google.com | bash
ENV PATH $PATH:$CLOUDSDK_INSTALL_DIR/google-cloud-sdk/bin
RUN gcloud components install app-engine-java kubectl
This container is currently provisioned from an Azure app service & AWS Fargate. When a new container instance is spawned, we would like it to be gcloud enabled with a service account attached already so our application can deploy stuff on GCP using its deployment manager.
I understand gcloud requires us to run gcloud auth login to authenticate to your account. How we can automate the provisioning of our container if this step has to be manual?
Also, from what I understand, for cloud client libraries, we can store the path to service account key json file in an environment variable (GOOGLE_APPLICATION_CREDENTIALS). So this file either has to be stored inside the docker image itself OR has to be mounted from an external storage at the very least?
How safe is it to store this service account key file in an external storage. What are the best practices around this?
There are two main means of authentication in Google Cloud Platform:
User Accounts: Belong to people, represent people involved in your project and they're associated to a Google Account
Service Accounts: Used by an application or an instance.
Learn more about their differences here.
Therefore, you are not required to use the command gcloud auth login to perform gcloud commands.
You should be using gcloud auth activate-service-account instead, along with the --key-file=<path-to-key-file> flag, which will allow you to authenticate without the need of signing into a Google Account with access to your project every time you need to call an API.
This key should be stored securely, preferably encrypted in the platform of your choice. Learn how to do it in GCP here following these steps as an example.
Take a look at these useful links for storing secrets in Microsoft Azure and AWS.
On the other hand, you can deploy services to GCP programmatically either using Cloud Libraries with your programming language of choice, or using Terraform is very intuitive if you prefer to do so over using the Google Cloud SDK through the CLI.
Hope this helped.

Authenticating to pubsub in a co-lab notebook via the gcloud auth command

I would like to authenticate to pubsub via a co-lab notebook by using the !gcloud auth command. However, when I run that command, I am able to authenticate to cloud sql as well as to the gcp buckets, but I get the following error message when I run the publisher = pubsub.PublisherClient() command:
DefaultCredentialsError: Could not automatically determine credentials. Please set GOOGLE_APPLICATION_CREDENTIALS or explicitly create credentials and re-run the application. For more information, please see https://cloud.google.com/docs/authentication/getting-started
I want to avoid uploading my cloud credentials to a google drive folder to authenticate with pubsub, since i want to be able to share this notebook across my organization so that other users with the correct access rights can also run the notebook directly from their end without needing to upload their own service account credentials. Is there a way that I can do this? Thanks in advance.
Use:
from google.colab import auth
auth.authenticate_user()
As in this example showing Google Cloud Storage access:
https://colab.research.google.com/notebooks/io.ipynb#scrollTo=NQX0hbfYaEKc

How to setup properly Google Cloud Shell?

I know this question is probably a bit vague. I was trying to run one of the examples of Google NLP Library in Google Shell.
I have 0 experience with using API, JSON, Nodejs... I don't understand what they are and how to use them.
Please help
Here is the snapshot of the error:
The error message means that you are using user credentials instead of service account credentials.
When you connect to Google Cloud Shell, you are using your Google Accounts User Credentials. Those credentials are the ones that you used to log in to the Google Cloud Console. When you run an application in Google Cloud Shell, your application is using those credentials unless you explicitly specify different credentials.
The solution is to create a service account in the Google Cloud Console. Then in your program use the service account for credentials for your application.
Google Cloud Service Accounts
When you do not specify the application credentials, the Google Client libraries use a method to locate credentials called ADC (Application Default Credentials). I wrote an article that might help you understand ADC:
Google Cloud Application Default Credentials
The simplest method for you is to create the environment variable GOOGLE_APPLICATION_CREDENTIALS to point to the service account full path location before running your application. Change the path below to point to where the service account is stored on Cloud Shell. You will need to first create the service acount, download it and then upload to Cloud Shell.
export GOOGLE_APPLICATION_CREDENTIALS="$HOME/service-account.json"
Managing files with Cloud Shell
This link will provide more information on how to write applications that use service accounts.
Setting Up Authentication for Server to Server Production Applications