After setting up AWS Kinesis Firehose along with OpenSearch I've been unable to run Test with Demo data successfully.
Error log suggests Auth permission related issue but the user is an Admin with full permmission access.
{ "deliveryStreamARN": "arn:aws:firehose:us-east-1:...:deliverystream/PUT-...", "destination": "https://....us-east-1.aoss.amazonaws.com", "deliveryStreamVersionId": 1, "message": "{"type":"security_exception","reason":"Authorization failure for the following indices: [analytics-index]"}", "errorCode": "403" }
Related
We have a GCS bucket named 'testfiles' and Pub/Sub topic 'testtopic' with a subscription 'testsubscription'. We have created a notification configuration to receive notifications on the Pub/Sub topic for any event happening on the GCS bucket. When we run the following command to list the notifications on the bucket,
gcloud storage buckets notifications list gs://testfiles
we see the below output,
{
"kind": "storage#notification",
"selfLink": "https://www.googleapis.com/storage/v1/b/testfiles/notificationConfigs/28",
"id": "28",
"topic": "//pubsub.googleapis.com/projects/test-project/topics/testtopic",
"etag": "28",
"payload_format": "JSON_API_V1"
}
Also, we have provided the Cloud Storage Service account the Pub/Sub Publisher role.
Despite these settings, when we upload a file to the bucket 'testfiles', we do not see any JSON messages in the above mentioned topic/subscription (testtopic/testsubscription).
We tried to follow the documentation here
Please advise, if there is something we are missing.
The console (GCP console, console.cloud.google.com) was not displaying the Pub/Sub messages. However, when we connect to Pub/Sub topic's subscription through a listener code, it was able to receive the JSON messages without any issues.
We are working on a POC in AWS Amplify and we have a requirement to store all the deployments logs (individual deploy jobs) for a period of 1 year. At the moment we have a functional application deployed in Amplify and we can see all the access logs in the Amplify console. But the deployment logs are only available through the AWS CLI and I cannot find them anywhere in the console. when I query aws amplify list-jobs --app-id "${AMPLIFY_APP_ID}" --branch-name "${branch}" it will give me the below output,
{
"jobSummaries": [
{
"jobArn": "xxxxxxxxxxxxxx",
"jobId": "xxxxxxxxxxxxxx",
"commitId": "xxxxxxxxxxxxxx",
"commitMessage": "xxxxxxxxxxxxxx",
"commitTime": "xxxxxxxxxxxxxx",
"startTime": "xxxxxxxxxxxxxx",
"status": "SUCCEED",
"endTime": "xxxxxxxxxxxxxx"
}
}
I noticed we can use log groups and the Lambda function to store access logs. I need to find a way to store these deployment logs in an S3 or some kind of Database. Appreciate your suggestions.
I get this error in CloudWatch logs of kinesis firehose
{
"deliveryStreamARN": "arn:aws:firehose:us-west-2:917877325894:deliverystream/test_dynamodb",
"destination": "arn:aws:es:us-west-2:917877325894:domain/test-dynamodb2",
"deliveryStreamVersionId": 1,
"message": "Error received from Elasticsearch cluster. {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"no permissions for [indices:data/write/bulk] and User [name=arn:aws:iam::917877325894:role/firehose_delivery_role2, backend_roles=[arn:aws:iam::917877325894:role/firehose_delivery_role2], requestedTenant=null]\"}],\"type\":\"security_exception\",\"reason\":\"no permissions for [indices:data/write/bulk] and User [name=arn:aws:iam::917877325894:role/firehose_delivery_role2, backend_roles=[arn:aws:iam::917877325894:role/firehose_delivery_role2], requestedTenant=null]\"},\"status\":403}",
"errorCode": "ES.ServiceException"
}
I have added all different policies to the role attached to Firehose but still getting the same error.(btw the role was made by firehose itself but I tried also adding more policies with no different result)
I also have open access policy for the elasticsearch domain
Did anyone face the same thing before?
I had the same problem, instructions for troubleshooting are here
https://aws.amazon.com/premiumsupport/knowledge-center/es-troubleshoot-cloudwatch-logs/
Go to Kibana and add your all_access mapping. Voila.
I created an API by AWS API Gateway and Lambda that is same 'https://github.com/aws-samples/simple-websockets-chat-app'. But the API not working trust. I get an error when i try to connect. Its message is "WebSocket connection to 'wss://b91xftxta9.execute-api.eu-west-1.amazonaws.com/dev' failed: Error during WebSocket handshake: Unexpected response code: 500"
My Connection Code
var ws= new WebSocket("wss://b91xftxta9.execute-api.eu-west-1.amazonaws.com/dev");
ws.onopen=function(d){
console.log(d);
}
Try adding $context.error.validationErrorString and $context.integrationErrorMessage to the logs for the stage.
I added a bunch of stuff to the Log Format section, like this:
{ "requestId":"$context.requestId", "ip": "$context.identity.sourceIp",
"requestTime":"$context.requestTime", "httpMethod":"$context.httpMethod",
"routeKey":"$context.routeKey", "status":"$context.status",
"protocol":"$context.protocol", "errorMessage":"$context.error.message",
"path":"$context.path",
"authorizerPrincipalId":"$context.authorizer.principalId",
"user":"$context.identity.user", "caller":"$context.identity.caller",
"validationErrorString":"$context.error.validationErrorString",
"errorResponseType":"$context.error.responseType",
"integrationErrorMessage":"$context.integrationErrorMessage",
"responseLength":"$context.responseLength" }
In early development this allowed me to see this type of error:
{
"requestId": "QDu0QiP3oANFPZv=",
"ip": "76.54.32.210",
"requestTime": "21/Jul/2020:21:37:31 +0000",
"httpMethod": "POST",
"routeKey": "$default",
"status": "500",
"protocol": "HTTP/1.1",
"integrationErrorMessage": "The IAM role configured on the integration
or API Gateway doesn't have permissions to call the integration.
Check the permissions and try again.",
"responseLength": "35"
}
try using wscat -c wss://b91xftxta9.execute-api.eu-west-1.amazonaws.com/dev in a terminal. This should allow you to connect it. If you don't have wscat installed, just do a npm install -g wscat
To get more details, enable logging for your API: Stages -> Logs/Tracing -> CloudWatch Settings -> Enable CloudWatch Logs. Then, send a connection request again and monitor your API logs in CloudWatch. In my case, I had the next error:
Execution failed due to configuration error: API Gateway does not have permission to assume the provided role {arn_of_my_role}
So, I added API Gateway to my role's Trust Relationships, as it's mentioned here and it fixed the problem.
I am trying to sync my data from MongoDB in to Amazon Elasticsearch Service using mongoosastic For some reason, sync is not happening as expected.
I do not see any error from the mongoosastic plugin. Not sure what is failing in AWS Elasticsearch service
Is there any way to get Elasticsearch logs in Amazon Elasticsearch Service?
elasticsearch = require('elasticsearch');
this.es_connection = new elasticsearch.Client("Amazon Elasticsearch Service address");
return this.es_connection.ping({
requestTimeout: 30000,
hello: 'elasticsearch'
}, function(error) {
if (error) {
console.error('elasticsearch cluster is down!' + JSON.stringify(error));
} else {
logger.info('All is well in elasticsearch');
}
});
In order to troubleshoot AWS elasticsearch service, you'll need to configure the log shipping to CloudWatch:
https://aws.amazon.com/blogs/big-data/viewing-amazon-elasticsearch-service-error-logs/
Then you will be able to use CloudWatch console in order to view your logs and understand if the issue is in Elasticsearch or is it mongoosastic issue / mapping/ index failures issues.