How to pass an aws IAM role to a Java client (through a lambda function) - amazon-web-services

I'm aiming to create a lambda function which it will execute a java client, such (the java client) is supposed to call an aws service endpoint.
Since my java client needs authentication (I am approaching this with the AWS4Signer library). I would like to authenticate my java code with the IMDS of my lambda exception role, as I can't use users due to a sec procedures.
I've been trying to use InstanceProfileCredentialsProvider as my credential provider
https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-roles.html
which in theory it should take the IMDS of the instance.Not sure if it only when using an EC2 instance, or it could also work with any aws compute service, such as lambda is.
with InstanceProfileCredentialsProvider I'm getting the following error:
com.amazonaws.internal.InstanceMetadataServiceResourceFetcher - Token is not supported. Ignoring
Failed to connect to service endpoint: com.amazonaws.SdkClientException: Failed to connect to service endpoint:
at com.amazonaws.internal.EC2ResourceFetcher.doReadResource(EC2ResourceFetcher.java:100)
I came across with the following blogs where a similar issue was resported
https://github.com/aws/aws-sdk-java/issues/2285
https://medium.com/expedia-group-tech/service-slow-to-retrieve-aws-credentials-ebc02a38e95b
and it seems this is happening due to the catched instance credentials is already outdated at the time to authenticate the credentials.
So I have added a logic to refresh the credentialsProvider object (InstanceProfileCredentialsProvider)
public static Optional<AWSCredentials> retrieveCredentials (AWSCredentialsProvider provider){
var attempts = 0;
System.out.println("Retrieving credentials...");
try {
System.out.printf("Retrieving credentials at attempt : %s", attempts);
return Optional.of(provider.getCredentials());
}catch(Exception e){
while(attempts < 15) {
try {
TimeUnit.SECONDS.sleep(30);
} catch (InterruptedException ex) {
ex.printStackTrace();
}
System.out.printf("Retrieving credentials at attempt : %s", attempts);
provider.refresh();
try {
return Optional.of(provider.getCredentials());
}catch (Exception e1){
System.out.printf("Attempt : %s failed due to: %s", attempts, e1.getMessage());
}
attempts ++;
}
e.printStackTrace();
System.exit(1);
}
return Optional.empty();
}
```
But I'm still getting the same error.
Any kind of help will be very appreciated.

Related

AWS Cloudwatch does not log Lambda stacktraces

We have a NodeJS Lambda deployed in AWS. Works fine but whenever errors happen, the entire and details of the error is not shown in AWS Cloudwatch. CW shows all console.info output but does not show the stack trace of the exception. If we run the Lambda in our local machines, the console logs would look like the one below:
****START METHOD EXECUTION****
****END METHOD EXECUTION******
/Users/john.doe/Documents/workspace/myproject/user-service/dbutility.js:45
await connection.query('INSERT INTO users SET ?', record, async function (error, results, fields) {
^
at Handshake.onConnect (/Users/john.doe/Documents/workspace/myproject/user-service/node_modules/mysql/lib/Pool.js:58:9)
at Handshake.<anonymous> (/Users/john.doe/Documents/workspace/myproject/user-service/node_modules/mysql/lib/Connection.js:526:10)
at Handshake._callback (/Users/john.doe/Documents/workspace/myproject/user-service/node_modules/mysql/lib/Connection.js:488:16)
at Sequence.end (/Users/john.doe/Documents/workspace/myproject/user-service/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
at Protocol.handleNetworkError (/Users/john.doe/Documents/workspace/myproject/user-service/node_modules/mysql/lib/protocol/Protocol.js:369:14)
at Connection._handleNetworkError (/Users/john.doe/Documents/workspace/myproject/user-service/node_modules/mysql/lib/Connection.js:418:18)
at Socket.emit (node:events:513:28)
at Socket.emit (node:domain:489:12)
at emitErrorNT (node:internal/streams/destroy:151:8)
But when the same Lambda is deployed in AWS, the Cloudwatch logs we see is only the one below
****START METHOD EXECUTION****
****END METHOD EXECUTION******
In our codes, we catch error using the usual try catch:
try {
} catch (err) {
console.error(err);
}
How can we display error stack trace or details in the CW logs?

Getting access denied error while uploading file to amazon s3 using tranferutility

Getting the following error while running the application in logcat
D/com.amazonaws.request: Received error response: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: null; Status Code: 403; Error Code: AccessDenied; Request ID: ZFW34X6QRZEBNR23), S3 Extended Request ID: cCcs8EOdirUSRRd9MyoUcMWDTjEvFLN/cqSYbAT3NCLR/hDSjxI/ATNX63Y9cbyDNABi8MnTauY=
Without using tranfer utility code works fine. When using transfer utility error occurs. Kindly find the code here
getApplicationContext().startService(new Intent(getApplicationContext(), TransferService.class));
AWSMobileClient.getInstance().initialize(getApplicationContext(), new Callback<UserStateDetails>() {
#Override
public void onResult(UserStateDetails result) {
Log.i("AWSMobile Clinet","initialised awsmobile client");
TransferUtility transferUtility=TransferUtility.builder().context(getApplicationContext())
.awsConfiguration(AWSMobileClient.getInstance().getConfiguration())
.s3Client(new AmazonS3Client(AWSMobileClient.getInstance()))
.build();
try {
File exampleFile = new File("/data/data/com.dieto.app/databases/user.db");
TransferObserver uploadObserver=transferUtility.upload( "dietodata144832-
dev",number+".db",exampleFile);
}
catch (Exception e){
Log.e("File load","File load failed"+e.toString());
}
Check if the user has access to the that specific S3 bucket or not. You can do this by navigating to IAM on AWS console, check the policies attached to your user and check if the user has AWSS3FullAccess policy attached or not.
After you attach this policy, you shouldn't get access denied error.

GCP BigTable Metrics - what do 404 requests mean?

We switched to BigTable some time ago and since then there is a number of "404 requests" and also a high number of errors in the GCP Metrics console.
We see no errors in our logs and even data storage/retrieval seems to work as expected.
What is the cause for these errors and how is it possible to find out what is causing them?
As mentioned previously 404 means resource is not found. The relevant resource here is the Bigtable table (which could mean that either the instance id or table id are misconfigured in your application).
I'm guessing that you are looking at the metrics under APIs & Services > Cloud Bigtable API. These metrics show the response code from the Cloud Bigtable Service. You should be able to see this error rate under Monitoring > Metrics Explorer > metric:bigtable.googleapis.com/server/error_count and grouping by instance, method, error_code and app_profile. This will tell which instance and which RPC is causing the errors. Which let you grep your source code for incorrect usages.
A significantly more complex approach is that you can install an interceptor in Bigtable client that:
dumps the resource name of the RPC
once you identify the problematic table name, logs the stack trace of the caller
Something along these lines:
BigtableDataSettings.Builder builder = BigtableDataSettings.newBuilder()
.setProjectId("...")
.setInstanceId("...");
ConcurrentHashMap<String, Boolean> seenTables = new ConcurrentHashMap<>();
builder.stubSettings().setTransportChannelProvider(
EnhancedBigtableStubSettings.defaultGrpcTransportProviderBuilder()
.setInterceptorProvider(() -> ImmutableList.of(new ClientInterceptor() {
#Override
public <ReqT, RespT> ClientCall<ReqT, RespT> interceptCall(
MethodDescriptor<ReqT, RespT> methodDescriptor, CallOptions callOptions,
Channel channel) {
return new ForwardingClientCall.SimpleForwardingClientCall<ReqT, RespT>(channel.newCall(methodDescriptor, callOptions)) {
#Override
public void sendMessage(ReqT message) {
Message protoMessage = (Message) message;
FieldDescriptor desc = protoMessage.getDescriptorForType()
.findFieldByName("table_name");
if (desc != null) {
String tableName = (String) protoMessage.getField(desc);
if (seenTables.putIfAbsent(tableName, true) == null) {
System.out.println("Found new tableName: " + tableName);
}
if ("projects/my-project/instances/my-instance/tables/my-mispelled-table".equals(
tableName)) {
new RuntimeException(
"Fake error to get caller location of mispelled table id").printStackTrace();
}
}
delegate().sendMessage(message);
}
};
}
}))
.build()
);
Google Cloud Support here,
Without more insight I won’t be able to provide valid information about this 404 issue.
The issue must be either a typo or with the configuration, but cannot confirm with the shared data.
In order to provide more meaningful support, I would suggest you to open a Public Issue Tracker or a Google Cloud Support ticket.

Cannot send request to Twilio from AWS ECS Task (awsvpc network mode)

I am using AWS - ECS service and I have 5 running tasks on the cluster that has initiated as awsvpc network mode.
The problem is that the task is supposed to send request to Twilio for the SMS code but the request to Twilio is being timed out.
const twilioClient = require('twilio')(accountSid, authToken)
try {
await twilioClient.messages.create({
body: `${code}`,
from: phoneNumber,
to: userInput.phone
})
} catch (err) {
console.log('Twilio Error: ', err)
return false
}
The error below shows the error I have logged on CloudWatch.
Twilio Error: {
Error: ETIMEDOUT
at Timeout._onTimeout (/srv/node_modules/request/request.js:849:19)
ontimeout (timers.js:436:11)
at tryOnTimeout (timers.js:300:5)
at listOnTimeout (timers.js:263:5)
at Timer.processTimers (timers.js:223:10) code: 'ETIMEDOUT', connect: true
}
The problem is that the same code works in case of default network mode for Task on AWS ECS.
I am using EC2 mode, not Fargate mode.
Looking forward to the right help on this.
Cheers.

I'm not sure if this is Cloud Pub/Sub internal error

Recently I started to get this error when testing my Google Cloud function:
textPayload: "2019/08/12 11:15:58 error publishing to the topic - rpc error: code = Unauthenticated desc = transport: compute: Received 500 `Could not fetch URI /computeMetadata/v1/instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fpubsub"
It was surprise to me. Recenty I made some changes in the function, but it seems that was not the case. I reverted to the old code which was working before and I got the same error.
I read Cloud Pub/Sub docs on errors:
INTERNAL 500 This error indicates an internal server error; it should not occur. If this error occurs, please report to cloud support. The error should be transient.
I reported it to the cloud support.
Still, I'm not sure is this Google Cloud internal error or the one induced by my code.
EDIT: Here is a pretty minimal code in Go.
package testfcn
import (
"log"
"net/http"
"cloud.google.com/go/pubsub"
)
func TestFcn(w http.ResponseWriter, r *http.Request) {
ctx := r.Context()
client, err := pubsub.NewClient(ctx, "min-fotball-staging")
if err != nil {
log.Print("error creating PubSub client - ", err)
return
}
topic := client.Topic("TestTopic")
result := topic.Publish(ctx, &pubsub.Message{
Data: []byte("TestMessage"),
})
_, err = result.Get(ctx)
if err != nil {
log.Print("error publishing to the topic - ", err)
return
}
}
And I have topic named TestTopic.
This was not a Cloud Pub/Sub client library's internal error.
Project's service-<project_number>#gcf-admin-robot.iam.gserviceaccount.com service account had not assigned Cloud Functions Service Agent role.
It was not possible to add this role through Google Cloud Console, but by using gcloud command:
gcloud projects add-iam-policy-binding <project_name> --role=roles/cloudfunctions.serviceAgent --member=serviceAccount:service-<project_number>#gcf-admin-robot.iam.gserviceaccount.com
Possibly useful links:
https://cloud.google.com/functions/docs/concepts/iam#cloud_functions_service_account
Accessing google cloud storage bucket from cloud functions throws 500 error
https://github.com/googleapis/google-cloud-go/issues/1532
Extra:
In my case, this service account had Owner role. This means Owner role does not have some permission(s) Cloud Functions Service Agent role has.