I want to create a publisher for google cloud Pub/Sub that publish to a regional endpoint.
This is the code I am running(pretty much the code from the quickstart guide):
using Google.Cloud.PubSub.V1;
using Grpc.Core;
using System;
using System.Collections.Generic;
using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using static Google.Cloud.PubSub.V1.PublisherClient;
namespace PubSub
{
class PubSubProducer
{
public async Task<int> PublishMessagesAsync(string projectId, string topicId, IEnumerable<string> messageTexts)
{
var creationSettings = new ClientCreationSettings(credentials: await GoogleGrpcCredentials.GetApplicationDefaultAsync().ConfigureAwait(false),
serviceEndpoint: "us-central1-pubsub.googleapis.com");
var customSettings = new PublisherClient.Settings
{
EnableMessageOrdering = true
};
TopicName topicName = TopicName.FromProjectTopic(projectId, topicId);
PublisherClient publisher = await PublisherClient.CreateAsync(topicName, clientCreationSettings: creationSettings, settings: customSettings);
int publishedMessageCount = 0;
var publishTasks = messageTexts.Select(async text =>
{
try
{
string ordering_key;
if (int.Parse(text) > 5)
ordering_key = "a";
else
ordering_key = "b";
string message = await publisher.PublishAsync(ordering_key, text);
Console.WriteLine($"Published message {message}");
Interlocked.Increment(ref publishedMessageCount);
}
catch (Exception exception)
{
Console.WriteLine($"An error ocurred when publishing message {text}: {exception.Message}");
}
});
await Task.WhenAll(publishTasks);
return publishedMessageCount;
}
}
}
I would like to use the ordering feature so I need the messages to be in the same region.
This code returns:
An error ocurred when publishing message 9: Status(StatusCode="Unauthenticated", Detail="Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.", DebugException="Grpc.Core.Internal.CoreErrorDetailException: {"created":"#1608719160.348000000","description":"Error received from peer ipv4:<IP>","file":"T:\src\github\grpc\workspace_csharp_ext_windows_x64\src\core\lib\surface\call.cc","file_line":1062,"grpc_message":"Request had invalid authentication credentials. Expected OAuth 2 access token, login cookie or other valid authentication credential. See https://developers.google.com/identity/sign-in/web/devconsole-project.","grpc_status":16}")
When I am not suppling a regional endpoint it works fine:
var creationSettings = new ClientCreationSettings(credentials: await GoogleGrpcCredentials.GetApplicationDefaultAsync().ConfigureAwait(false));
It seems like my service account missing a permission to use this regional endpoint.
How can I fix this? What permissions am I missing?
The issue was with how I created the ClientCreationSettings.
The correct way for publisher:
var clientCreationSettings = new PublisherClient.ClientCreationSettings(serviceEndpoint: "us-central1-pubsub.googleapis.com:443");
For subscriber:
var clientCreationSettings = new SubscriberClient.ClientCreationSettings(serviceEndpoint: "us-central1-pubsub.googleapis.com:443");
Related
Problem
I am trying to access a Google Cloud service (Cloud Translate API) from my AWS Lambda using Nodejs and serverless framework. The system already works perfectly when I use a Google Service Account Key, so that validates that the two cloud services are operational and functional.
However, I'm trying to follow best practice and use Google's Federated Workforce ID instead of a Service Account Key. (Docs).
However, I'm getting an error:
FetchError: request to http://169.254.169.254/latest/meta-data/iam/security-credentials failed, reason: connect ETIMEDOUT 169.254.169.254:80
I've followed the directions in the docs several times, including creating the workplace pool and downloading the client config file. And I have the environment variable set to the config file:
GOOGLE_APPLICATION_CREDENTIALS: ./clientLibraryConfig-fq-aws-apis.json
The Google Auth picks up the credentials file (I can see by running a console.log on const "client"), and it retrieves my projectId in auth.getProjectId();.
But when it comes to initiating the TranslationServiceClient, I get this:
Error
"errorMessage": "request to http://169.254.169.254/latest/meta-data/iam/security-credentials failed, reason: connect ETIMEDOUT 169.254.169.254:80",
Code
"use strict";
const { GoogleAuth } = require("google-auth-library");
const { TranslationServiceClient } = require("#google-cloud/translate");
//////////
// This function gets translation from Google
//////////
const getTranslations = async (originalClipArray, translateTo) => {
// G Translate params
// const projectId = "rw-frequency";
const location = "global";
const auth = new GoogleAuth({
scopes: 'https://www.googleapis.com/auth/cloud-platform'
});
const client = await auth.getClient();
const projectId = await auth.getProjectId();
const translationClient = new TranslationServiceClient()
console.log("past translationserviceclient constructor");
// Build the params for the translate request
const request = {
parent: `projects/${projectId}/locations/${location}`,
contents: originalClipArray,
mimeType: "text/plain", // mime types: text/plain, text/html
targetLanguageCode: translateTo,
};
// Call Google client
// try {
const response = await translationClient.translateText(request);
console.log(`response`);
console.dir(response);
return response;
// } catch (error) {
// console.log(`Google translate error raised:`);
// console.log(error);
// }
};
module.exports.getTranslations = getTranslations;
The request that gives you a timeout retrieves security credentials for EC2 instances. Apparently, your Lambda is using a GCP library intended for EC2. Hope this helps!
I'm develop app IOS and Android with Xamarin cross-platform.
I'm trying hard to use credentials receiving from Cognito Identity to authorize app invoking API Gateway.
My user flow is:
Authenticate on Cognito User Pool and get tokens
Exchange tokens for credentials on Cognito Identity Pools
Access API Gateway using credentials retrieved in the previous step.
Step 1 and 2 seems to works fine. But when app try to connect to api gateway it's getting error 403 (Forbidden).
My code:
Authenticate on Cognito User Pool and get tokens
**
public async Task<AppUser> Login(string username, string password)
{
CognitoUser cognitoUser = new CognitoUser(username, Aws.COGNITO_CLIENT_ID, CognitoUserPool, CognitoIdentityProviderClient);
AppUser appUser = new AppUser() { Email = username };
// Send a login request and wait for the response from Amazon
try
{
AuthFlowResponse response = await cognitoUser.StartWithSrpAuthAsync(new InitiateSrpAuthRequest()
{
Password = password
});
;
appUser.IsAuthenticated = true;
}
catch (NotAuthorizedException e)
{
appUser.IsAuthenticated = false;
appUser.ErrorMessage = e.Message;
}
await _tokenManagement.SaveTokens(cognitoUser.SessionTokens) ;
return appUser;
}
Exchange tokens for credentials on Cognito Identity Pools
**
public async Task<ImmutableCredentials> GetAppCredentialsAsync()
{
CognitoAWSCredentials cac;
if (_tokenManagement.CheckIsAnonymous())
{
//Anonymous credentials
cac = new CognitoAWSCredentials(Aws.COGINITO_IDENTITY_POLL_ID, RegionEndpoint.USEast1);
}
else
{
//Retrieve saved tokens from previous authentication
var tm = await _tokenManagement.RetrieveTokens();
CognitoUser user = new CognitoUser(null, Aws.COGNITO_CLIENT_ID, CognitoUserPool, CognitoIdentityProviderClient)
{
SessionTokens = new CognitoUserSession(tm.IdToken, tm.AccessToken, tm.RefreshToken, tm.IssuedTime, tm.ExpirationTime)
};
//Retrieve authenticated credentials
cac = user.GetCognitoAWSCredentials(Aws.COGINITO_IDENTITY_POLL_ID, RegionEndpoint.USEast1);
}
}
return await cac.GetCredentialsAsync();
}
Access API Gateway using credentials retrieved in the previous step:
**
public async Task<IList<MediaImage>> GetCoversAWSAsync()
{
// Getting credentials and sign request
var request = await BuildRequestAsync("/listMedia");
var client = new HttpClient();
var response = await client.SendAsync(request);
response.EnsureSuccessStatusCode();
IList<MediaImage> covers = JsonConvert.DeserializeObject<IList<MediaImage>>(await response.Content.ReadAsStringAsync());
return covers;
}
private async Task<HttpRequestMessage> BuildRequestAsync(string service)
{
var request = new HttpRequestMessage()
{
Method = HttpMethod.Get,
RequestUri = new Uri(baseURL + service)
};
ImmutableCredentials awsCredential = await _loginService.GetAppCredentialsAsync( );
var signer = new AWS4RequestSigner(awsCredential.AccessKey, awsCredential.SecretKey);
request = await signer.Sign(request, "execute-api", awsRegion);
return request;
}
This code works fine when I hard code credentials from one IAM user. But credentials retrieved from Cognito Identities its getting Forbidden error.
I did a test using SDK for S3 and I was able to successfully list the buckets with the same credentials received from Cognito Identities, but it is not possible to make requests on the API Gateway.
Can you help me? Where did I get lost?
I figure out what was going on.
After ensuring that the permissions settings were correct on AWS.
I found in the documentation that it is necessary to include in the HTTP header the token returned by the cognito (header name x-amz-security-token). So I changed the code to the following:
private async Task<HttpRequestMessage> BuildRequestAsync(string service)
{
var request = new HttpRequestMessage()
{
Method = HttpMethod.Get,
RequestUri = new Uri(baseURL + service)
};
ImmutableCredentials awsCredential = await _loginService.GetAppCredentialsAsync( );
//This where I add header to the HTTP request
request.Headers.Add("x-amz-security-token", awsCredential.Token);
var signer = new AWS4RequestSigner(awsCredential.AccessKey, awsCredential.SecretKey);
request = await signer.Sign(request, "execute-api", awsRegion);
return request;
}
What i want to do: To call a google function from my server/machine & limit it usage with a (simple) authentication.
What i use: Node.js, google-auth-library library for authentication.
What have I done/tried:
1) Created a project in Google Cloud Functions
2) Created a simple google function
exports.helloWorld = (req, res) => {
let message = req.query.message || req.body.message || 'Hello World!';
res.status(200).send(message);
};
3) Set my custom service account
4) Enabled api:
- Cloud Functions API
- IAM Service Account Credentials API
- Cloud Run API
- Compute Engine API
- IAM Service Account Credentials API
5) Given to my server account necessary authorization (project owner, cloud function admin, IAM project admin... (need more?)
6) Generated key from my service account and saved it in json format
NB: with allUser permission (without authorization required), i can call my endpoint without problem
7) From my project i tried to auth my function in this way
const { JWT } = require('google-auth-library');
const fetch = require('node-fetch');
const keys = require('./service-account-keys.json');
async function callFunction(text) {
const url = `https://europe-west1-myFunction.cloudfunctions.net/test`;
const client = new JWT({
email: keys.client_email,
keyFile: keys,
key: keys.private_key,
scopes: [
'https://www.googleapis.com/auth/cloud-platform',
'https://www.googleapis.com/auth/iam',
],
});
const res = await client.request({ url });
const tokenInfo = await client.getTokenInfo(client.credentials.access_token);
try {
const response = await fetch(url, {
method: 'GET',
headers: {
Authorization: `Bearer ${client.credentials.access_token}`,
},
});
if (response.status !== 200) {
console.log(response);
return {};
}
return response.json();
} catch (e) {
console.error(e);
}
}
ℹ️ if i try to pass at client.request() url without name of function (https://europe-west1-myFunction.cloudfunctions.net), i not received error, but when use the JWT token obtained in fetch call, i received the same error.
RESULT:
Error:
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>401 Unauthorized</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Unauthorized</h1>
<h2>Your client does not have permission to the requested URL <code>/test1</code>.</h2>
<h2></h2>
</body></html>
❓ How do I call a google function with any protection to prevent anyone from using it? (I don't need specific security, just that random users don't use it)
Thanks in advance for any help
When you call a private function (or a private Cloud Run) you have to use a google signed identity token.
In your code, you use an access token
headers: {
Authorization: `Bearer ${client.credentials.access_token}`,
},
Access token work when you have to request Google Cloud API, not your services
And the google signed is important, because you can easily generate a self signed identity token with the google auth lib, but it won't work
You have code sample here and I wrote a tool in Go if you want to have a try on it
** EDIT **
I worked on an example, and, even if I never liked Javascript, I have to admit that I'm jealous!! It's so simple in Node!!
Here my working example
const {GoogleAuth} = require('google-auth-library');
async function main() {
// Define your URL, here with Cloud Run but the security is exactly the same with Cloud Functions (same underlying infrastructure)
const url = "https://go111-vqg64v3fcq-uc.a.run.app"
// Here I use the default credential, not an explicit key like you
const auth = new GoogleAuth();
//Example with the key file, not recommended on GCP environment.
//const auth = new GoogleAuth({keyFilename:"/path/to/key.json"})
//Create your client with an Identity token.
const client = await auth.getIdTokenClient(url);
const res = await client.request({url});
console.log(res.data);
}
main().catch(console.error);
Note: Only Service account can generate and identity token with audience. If you are in your local computer, don't use your user account with the default credential mode.
I've set up an API Gateway using WebSocket protocol. On the '$connect' route request setting, I selected 'AWS_IAM' as the authorization method. The web app needs to make a connection to this WebSocket API after a user logged in via Cognito. How do I then authorize the WebSocket API request from the JavaScript on the web app? With the HTTP API Gateway, I can generate the signature from access key and session token, which got passed in to the request header. But I can't pass headers in a WebSocket request.
This is some example/pseudo code that works for me:
Using AWS Amplify Authenticated user:
import { w3cwebsocket as W3CWebSocket } from "websocket"
import { Auth, Signer } from "aws-amplify"
let wsClient: any = null
export const client = async () => {
if (wsClient) return wsClient
if ((await Auth.currentUserInfo()) === null) return wsClient
const credentials = await Auth.currentCredentials()
const accessInfo = {
access_key: credentials.accessKeyId,
secret_key: credentials.secretAccessKey,
session_token: credentials.sessionToken,
}
const wssUrl = "wss://YOUR-API-ID.execute-api.REGION.amazonaws.com/dev"
const signedUrl = Signer.signUrl(wssUrl, accessInfo)
wsClient = new W3CWebSocket(signedUrl)
wsClient.onerror = function () {
console.log("[client]: Connection Error")
}
wsClient.onopen = function () {
console.log("[client]: WebSocket Client Connected")
}
wsClient.onclose = function () {
console.log("[client]: Client Closed")
}
wsClient.onmessage = function (e: any) {
if (typeof e.data === "string") {
console.log("Received: '" + e.data + "'")
}
}
return wsClient
}
Then also using AWS Cognito needs this permission:
{
"Action": ["execute-api:Invoke"],
"Resource": "arn:aws:execute-api:REGION:ACCOUNT-ID-OR-WILDCARD:*/*/$connect",
"Effect": "Allow"
}
I have got an answer from AWS support. I will need to sign the wss URL. So instead of setting request headers in a HTTP request, the signature information will be passed to the url in the query string parameters. A signed wss URL looks like: wss://API_ID.execute-api.region.amazonaws.com/dev?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=ACCESSKEY/20200131/region/execute-api/aws4_request&X-Amz-Date=20200131T100233Z&X-Amz-Security-Token=SECURITY_TOKEN&X-Amz-SignedHeaders=host&X-Amz-Signature=SIGNATURE.
To generate the signed URL, I can use Signer.signUrl method from #aws-amplify/core library.
I implemented this Dart code which signs the AWS request URLs. This is particularly helpful to connect to a IAM-secured WebSocket API Gateway.
https://github.com/MohammedNoureldin/aws-url-signer
I know that putting links in answers in discouraged, but this will not make sense to copy the whole 100 lines of code here.
The usage of my implementation will look like this:
String getSignedWebSocketUrl(
{String apiId,
String region,
String stage,
String accessKey,
String secretKey,
String sessionToken})
My app first uses the Cognito LOGIN endpoint to obtain an Authorization Code. It then uses the TOKEN endpoint to try and obtain tokens (id_token, access_token, refresh_token) but that fails with unauthorized_client.
I do not understand why, the same client is used to access the LOGIN, and that succeeded in returning an authorization code. I'm following the documentation for the TOKEN endpoint
string clientId = ...
string clientSecret = ...
Uri redirectUri = new Uri("myapp://myhost");
string authorization_code = ... // obtained via HTTP GET on LOGIN endpoint
string accessTokenUrl = "https://<domain>.auth.<region>.amazoncognito.com/oauth2/token";
var queryValues = new Dictionary<string, string>
{
{ "grant_type", "authorization_code" },
{ "code", authorization_code },
{ "redirect_uri", redirectUri.AbsoluteUri },
{ "client_id", clientId},
};
using (HttpClient client = new HttpClient())
{
// Authorization Basic header with Base64Encoded (clientId::clientSecret)
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue(
"Basic",
Convert.ToBase64String(System.Text.ASCIIEncoding.ASCII.GetBytes(
string.Format("{0}:{1}",
clientId,
clientSecret))));
// Url Encoded Content
var content = new FormUrlEncodedContent(queryValues);
// HTTPS POST
HttpResponseMessage response = await client.PostAsync(accessTokenUrl, content).ConfigureAwait(false);
string text = await response.Content.ReadAsStringAsync().ConfigureAwait(false);
// test = {"error" : "unauthorized_client"}
}
The problem is two-fold:
1- System.Uri.AbsoluteUri adds a trailing / in the returned string so that my redirectUri becomes myapp://myhost/ instead of myapp://myhost
2- AWS Cognito TOKEN endpoint does not accept trailing / in a redirectURI.
The solution:
I now call redirectUri.OriginalUri instead of redirectUri.AbsoluteUri where I build the query to preserve the redirectUri as it was when I built it.
(I don't really have control over this since in my case Xamarin.Auth.OAuthAuthenticator2 calls Uri.AbsoluteUri on my behalf and transforms the redirectUri string I gave it, so I'm going to have to fix Xamarin.Auth).