I use amazon services. I have a task to track an IP address and user agent for each who download an image from s3.
I use amazon API gateway and amazon lambda and Amazon S3. Is it possible? I found triggers only on uploading or deleting the file from s3
As at now, S3 doesn't have object read event trigger. What you may do is to use cloudtrail to track the api call to read object of the s3 bucket and create an alarm to trigger a lambda function.
ex: S3 -> CloudTrail -> CloudWatch Event -> Rule -> Lamdba
Another simple solution would be to allow the object download directly via lambda.
ex: API Gateway -> Lambda -> S3
This will return the lambda output which can be the blob (be aware of the size limit) or preferably pre-signed url for the object.
You mention that your goal is "I want to track an IP address and user agent for each request of an image from s3".
To obtain this information, you should activate Amazon S3 server access logging:
Server access logging provides detailed records for the requests that are made to a bucket.
The Amazon S3 Server Access Log Format includes IP address, User Agent and other standard web log information:
79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be awsexamplebucket1 [06/Feb/2019:00:00:38 +0000] 192.0.2.3 79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be 3E57427F3EXAMPLE REST.GET.VERSIONING - "GET /awsexamplebucket1?versioning HTTP/1.1" 200 - 113 - 7 - "-" "S3Console/0.4" - s9lzHYrFp76ZVxRcpX9+5cjAnEH2ROuNkd2BHfIa6UkFVdtjf5mKR3/eTPFvsiP/XV/VLi31234= SigV2 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader awsexamplebucket1.s3.us-west-1.amazonaws.com TLSV1.1
79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be awsexamplebucket1 [06/Feb/2019:00:00:38 +0000] 192.0.2.3 79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be 891CE47D2EXAMPLE REST.GET.LOGGING_STATUS - "GET /awsexamplebucket1?logging HTTP/1.1" 200 - 242 - 11 - "-" "S3Console/0.4" - 9vKBE6vMhrNiWHZmb2L0mXOcqPGzQOI5XLnCtZNPxev+Hf+7tpT6sxDwDty4LHBUOZJG96N1234= SigV2 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader awsexamplebucket1.s3.us-west-1.amazonaws.com TLSV1.1
79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be awsexamplebucket1 [06/Feb/2019:00:00:38 +0000] 192.0.2.3 79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be A1206F460EXAMPLE REST.GET.BUCKETPOLICY - "GET /awsexamplebucket1?policy HTTP/1.1" 404 NoSuchBucketPolicy 297 - 38 - "-" "S3Console/0.4" - BNaBsXZQQDbssi6xMBdBU2sLt+Yf5kZDmeBUP35sFoKa3sLLeMC78iwEIWxs99CRUrbS4n11234= SigV2 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader awsexamplebucket1.s3.us-west-1.amazonaws.com TLSV1.1
79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be awsexamplebucket1 [06/Feb/2019:00:01:00 +0000] 192.0.2.3 79a59df900b949e55d96a1e698fbacedfd6e09d98eacf8f8d5218e7cd47ef2be 7B4A0FABBEXAMPLE REST.GET.VERSIONING - "GET /awsexamplebucket1?versioning HTTP/1.1" 200 - 113 - 33 - "-" "S3Console/0.4" - Ke1bUcazaN1jWuUlPJaxF64cQVpUEhoZKEG/hmy/gijN/I1DeWqDfFvnpybfEseEME/u7ME1234= SigV2 ECDHE-RSA-AES128-GCM-SHA256 AuthHeader awsexamplebucket1.s3.us-west-1.amazonaws.com TLSV1.1
Related
an instance was taken out of service in response to a ELB system
health check failure.
I hit the health check endpoint with my browser and it returns fine, but I'm getting the above message.
How can I debug this?
I've looked at instant settings => Get System Logs and nginx logs,
edit
nginx has
- [27/Mar/2020:05:35:42 +0000] "GET /littlehome/heartbeat/ HTTP/1.1" 200 2 2.920 2.920 "-" "ELB-HealthChecker/2.0"
- [27/Mar/2020:05:35:42 +0000] "GET /littlehome/heartbeat/ HTTP/1.1" 200 2 2.858 2.856 "-" "ELB-HealthChecker/2.0"
it returned 200 for sure..
and still aws think it received 502
{
"Target": {
"Id": "i-085e8dffe8781f876",
"Port": 80
},
"HealthCheckPort": "80",
"TargetHealth": {
"State": "unhealthy",
"Reason": "Target.ResponseCodeMismatch",
"Description": "Health checks failed with these codes: [502]"
}
},
Based on the comments, the issue was that grace period in Auto Scaling Group was too short. The solution was to increase it.
I have EC2 and can access S3 using ec2-user with:
aws s3 ls --profile myprofile s3://
On the same EC2 I created docker container and installed aws cli tool. Created same ~./aws/config file with same content as on host ec2 but this command (created ec2-user to be consistent with host user/setup) from docker container now gives error:
- MainThread - botocore.auth - DEBUG - StringToSign:
AWS4-HMAC-SHA256
- MainThread - botocore.endpoint - DEBUG - Sending http request: <AWSPreparedRequest stream_output=False, method=POST, url=https://sts.amazonaws.com/, headers={'Content-Type': b'application/x-www-form-urlencoded; charset=utf-8', 'User-Agent': b'aws-cli/1.16.190 Python/3.7.3 Linux/3.10.0-957.10.1.el7.x86_64 botocore/1.12.180', 'X-Amz-Date': b'2019Z', 'X-Amz-Security-Token': b'1234=', 'Authorization': b'AWS4-HMAC-SHA256 Credential=abc/2019/us-east-1/sts/aws4_request, SignedHeaders=content-type;host;x-amz-date;x-amz-security-token, Signature=abc', 'Content-Length': '160'}>
- MainThread - urllib3.util.retry - DEBUG - Converted retries value: False -> Retry(total=False, connect=None, read=None, redirect=0, status=None)
- MainThread - urllib3.connectionpool - DEBUG - Starting new HTTPS connection (1): sts.amazonaws.com:443
I have successfully login to facebook:
"GET /accounts/facebook/login/ HTTP/1.0" 302 0
"GET /accounts/facebook/login/callback/?code=AQCMYR8By_NW2ArWZ63Kq00twt4mSUiQ9BBApbvwt7eLWYyiMxYJkOXuRlbXzb9tq4lS-QunyFUlKxgVc0P6D3K-rl6AVkuMUZ2o7XjJi1LNmiaiUdzT6WHzWhyAbRm2SLIkm6mwgOPMI_g47h_yRE4tra1qLMZikfWq9npXC2QWybHQ9XeaFv3zS13EqaG8H9rJ-RMKmZXb9Ti4uSzK3-Vlzk1ORLWEIbIZw3YiEpqg18fSf4hb3PEB-ro7C5FmflEdoxwaig3Vdmddvl9wOyqmx1czE4bIwqtYR3yFilZ2h0o8uEj0g03rbBY5e5GGAcNyjFmgQj1zGsgMJIQDjFXO&state=fBRf1vERs3PD HTTP/1.0" 200 4362
In front I recieves message
Social Network Login Failure
An error occurred while attempting to login via your social network account.
VK auth is working.
What can be wrong and now to debug it?
Any SOCIALACCOUNT_PROVIDERS and any other VIRABLES values.
I have used HTTPS for AWS Application Load Balancer listener by using ACM certificate.
I requested the public certificate for a subdomain from ACM: test.example.com and create a CNAME in route 53 for it:
Name: _xxxxxxxxxxx.test.example.com
Type: CNAME
Value: xxxxxx.xxx.acm-validations.aws.
I can use the DNS of ALB (xxxx.us-east-1.elb.amazonaws.com) to call the API successfully in POSTMAN, however, when I use python requests or cURL to call the same API, it will always tell me there is some issue with the SSL.
cURL:
Code:
curl -X POST \
https://xxxxx.us-east-1.elb.amazonaws.com/prod/testapi \
-H 'Content-Type: application/json' \
-H 'cache-control: no-cache' \
-d '{
"paras1": "xxxxx"
}'
Error:
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: CN=test.example.com
* start date: Nov 11 00:00:00 2018 GMT
* expire date: Dec 11 12:00:00 2019 GMT
* subjectAltName does not match xxxx.us-east-1.elb.amazonaws.com
* SSL: no alternative certificate subject name matches target host name 'xxxx.us-east-1.elb.amazonaws.com'
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (51) SSL: no alternative certificate subject name matches target host name 'xxxx.us-east-1.elb.amazonaws.com'
Python Requests:
Code:
import requests
url = "https://xxxxx.us-east-1.elb.amazonaws.com/prod/testapi"
payload = "{\"paras1\": \"xxxxx\"}"
headers = {
'Content-Type': "application/json",
'cache-control': "no-cache"
}
response = requests.request("POST", url, data=payload, headers=headers)
print(response.text)
Error:
HTTPSConnectionPool(host='xxxx.us-east-1.elb.amazonaws.com', port=443):
Max retries exceeded with url: /prod/testapi
(Caused by SSLError(CertificateError("hostname 'xxxx.us-east-1.elb.amazonaws.com' doesn't match 'test.example.com'",),))
I can use the DNS of ALB (xxxx.us-east-1.elb.amazonaws.com)
That isn't how this is designed to work. You need to point test.example.com to the ELB in DNS, and then:
url = "https://test.example.com/prod/testapi"
Clearly, you call xxxx.us-east-1.elb.amazonaws.com which is set up with a certificate for test.example.com. Despite the certificate may be valid it does not match the URL you are calling which means that the certificate is NOT valid for THIS call. I think you also must set a custom domain for the API gateway. https://docs.aws.amazon.com/apigateway/latest/developerguide/how-to-custom-domains.html
Edit: thanks for the comment. I am not sure how I read "elb" as an api gateway. My fault. Still the DNS for the ELB has to match the one in the certificat. You can create a CNAME from your domain to the ELB domain. This should work (at least this is how we do that).
We are using Chaos monkey for resilience testing on AWS ec2 clients. When chaos monkey is trying to authenticate with the given secret key for frankfurt region, we are getting HTTP/1.1 401 Unauthorized error.We are using AWS SDK for Java.
It is using signature version v2 and eu-central-1(frankfurt) requires v4.
How can we set signature version to v4 or what setting we have to make to AWS SDK?
Here is the snapshot of the error:-
2015-01-19 05:37:05.028 - DEBUG SLF4JLogger - [SLF4JLogger.java:61] Sending request -1804412292: POST https://ec2.eu-central-1.amazonaws.com/ HTTP/1.1
2015-01-19 05:37:05.029 - DEBUG SLF4JLogger - [SLF4JLogger.java:61] >> "Action=DescribeInstances&Signature=Ao5fLfM%2B/rOcbdll0LF0K2F9U8NBlgd%2BAwuFk83GOxo%3D&SignatureMethod=HmacSHA256&SignatureVersion=2&Timestamp=2015-01-19T10%3A37%3A05.024Z&Version=2010-06-15&AWSAccessKeyId=xxx"
2015-01-19 05:37:05.029 - DEBUG SLF4JLogger - [SLF4JLogger.java:61] >> POST https://ec2.eu-central-1.amazonaws.com/ HTTP/1.1
2015-01-19 05:37:05.029 - DEBUG SLF4JLogger - [SLF4JLogger.java:61] >> Host: ec2.eu-central-1.amazonaws.com
2015-01-19 05:37:05.030 - DEBUG SLF4JLogger - [SLF4JLogger.java:61] >> Content-Type: application/x-www-form-urlencoded
2015-01-19 05:37:05.030 - DEBUG SLF4JLogger - [SLF4JLogger.java:61] >> Content-Length: 225
2015-01-19 05:37:05.609 - DEBUG SLF4JLogger - [SLF4JLogger.java:61] Receiving response -1804412292: HTTP/1.1 401 Unauthorized
2015-01-19 05:37:05.610 - DEBUG SLF4JLogger - [SLF4JLogger.java:61] << HTTP/1.1 401 Unauthorized
Setting the v4 signature is explained here. The relevant bits for Java:
Add the following in your code.
System.setProperty(SDKGlobalConfiguration.ENABLE_S3_SIGV4_SYSTEM_PROPERTY, "true");
Or, on the command line, specify the following.
-Dcom.amazonaws.services.s3.enableV4