I'm trying to verify if the public access block of my bucket mypublicbucketname is checked or not through Lambda function. For testing, I create a bucket and I have unchecked the public access block. So, I did this Lambda:
import sys
from pip._internal import main
main(['install', '-I', '-q', 'boto3', '--target', '/tmp/', '--no-cache-dir', '--disable-pip-version-check'])
sys.path.insert(0,'/tmp/')
import json
import boto3
import botocore
def lambda_handler(event, context):
# TODO implement
print(boto3.__version__)
print(botocore.__version__)
client = boto3.client('s3')
response = client.get_public_access_block(Bucket='mypublicbucketname')
print("response:>>",response)
I updated the latest version of boto3 and botocore.
1.16.40 #for boto3
1.19.40 #for botocore
Even if I uploaded them and the function seems correct I got this exception:
[ERROR] ClientError: An error occurred (NoSuchPublicAccessBlockConfiguration) when calling the GetPublicAccessBlock operation: The public access block configuration was not found
Someone can explain me why I have this error ?
For futur users. If you got the same problem with get_public_access_block(). Use this solution:
try:
response = client.get_public_access_block(Bucket='mypublicbucketname')
except botocore.exceptions.ClientError as e:
if e.response['Error']['Code'] == 'NoSuchPublicAccessBlockConfiguration':
print('No Public Access')
else:
print("unexpected error: %s" % (e.response))
for put_public_access_block, it works fine.
Related
My lambda_function.py file looks like this:
from urllib.request import urlopen
from google.cloud import bigquery
import json
url = "https://data2.unhcr.org/population/get/timeseries?widget_id=286725&sv_id=54&population_group=5460&frequency=day&fromDate=1900-01-01"
bq_client = bigquery.Client()
def lambda_helper(event, context):
response = urlopen(url)
data_json = json.loads(response.read())
bq_client.load_table_from_json(data_json['data']['timeseries'], "xxx.xxxx.tablename")
But every time I zip it up and upload it to my Lambda I get this error:
{
"errorMessage": "Handler 'lambda_handler' missing on module 'lambda_function'",
"errorType": "Runtime.HandlerNotFound",
"stackTrace": []
}
Is there a reason this error would be thrown even though I clearly have that function written in this module? This is driving me nuts. Thanks for any help!
It should be:
def lambda_handler(event, context):
not
def lambda_helper(event, context):
this is basic thing and it seems obvious but am stuck in this one
am using boto3, I have this access key and secret key,
upload file route:
#app.route('/up')
def up():
main = request.files['mainimg']
bucket=<bucketname>
if main:
upload_to_aws(main)
the upload_to_aws function (from github):
import os
import boto3
from werkzeug.utils import secure_filename
def upload_to_aws(file, acl="public-read"):
filename = secure_filename(file.filename)
s3 = boto3.client(
's3',
aws_access_key_id=os.environ.get('FASO_S3_ACCESS_KEY'),
aws_secret_access_key=os.environ.get('FASO_S3_SECRET_KEY')
)
try:
s3.upload_fileobj(
file,
"fasofashion",
file.filename,
ExtraArgs={
"ACL": acl,
"ContentType": file.content_type
}
)
print('uploadeed')
except Exception as e:
# This is a catch all exception, edit this part to fit your needs.
print("Something Happened: ", e)
return e
I keep getting these errors:
Access denied file must be a string
File must be a string
I am trying to run a python script that is present in AWS Lambda /tmp directory. The scripts require some extra dependencies like boto3 etc to run the file. When AWS Lambda runs the file it gives out the following error:
ModuleNotFoundError: No module named 'boto3'
However when i run this file directly as a lambda function then it runs easily whithout any import errors.
The Lambda Code that is trying to execute the code present in /tmp directory :
import json
import os
import urllib.parse
import boto3
s3 = boto3.client('s3')
def lambda_handler(event, context):
records = [x for x in event.get('Records', []) if x.get('eventName') == 'ObjectCreated:Put']
sorted_events = sorted(records, key=lambda e: e.get('eventTime'))
latest_event = sorted_events[-1] if sorted_events else {}
info = latest_event.get('s3', {})
file_key = info.get('object', {}).get('key')
bucket_name = info.get('bucket', {}).get('name')
s3 = boto3.resource('s3')
BUCKET_NAME = bucket_name
keys = [file_key]
for KEY in keys:
local_file_name = '/tmp/'+KEY
s3.Bucket(BUCKET_NAME).download_file(KEY, local_file_name)
print("Running Incoming File !! ")
os.system('python ' + local_file_name)
The /tmp code that is trying to get some data from S3 using boto3 :
import sys
import boto3
import json
def main():
session = boto3.Session(
aws_access_key_id='##',
aws_secret_access_key='##',
region_name='##')
s3 = session.resource('s3')
# get a handle on the bucket that holds your file
bucket = s3.Bucket('##')
# get a handle on the object you want (i.e. your file)
obj = bucket.Object(key='8.json')
# get the object
response = obj.get()
# read the contents of the file
lines = response['Body'].read().decode()
data = json.loads(lines)
transactions = data['dataset']['fields']
print(str(len(transactions)))
return str(len(transactions))
main()
So boto3 is imported in both the codes . But its only successful when the lambda code is executing it . However /tmp code cant import boto3 .
What can be the reason and how can i resolve it ?
Executing another python process does not copy Lambda's PYTHONPATH by default:
os.system('python ' + local_file_name)
Rewrite like this:
os.system('PYTHONPATH=/var/runtime python ' + local_file_name)
In order to find out complete PYTHONPATH the current Lambda version is using, add the following to the first script (one executed by Lambda):
import sys
print(sys.path)
Using Python, I followed and when it came to Test it, the following error popped up:
{
"errorMessage": "module initialization error"
}
What could I have done wrong?
from __future__ import print_function
import os
from datetime import datetime
from urllib2 import urlopen
SITE = os.environ['site'] # URL of the site to check, stored in the site environment variable, e.g. https://aws.amazon.com
EXPECTED = os.environ['expected'] # String expected to be on the page, stored in the expected environment variable, e.g. Amazon
def validate(res):
'''Return False to trigger the canary
Currently this simply checks whether the EXPECTED string is present.
However, you could modify this to perform any number of arbitrary
checks on the contents of SITE.
'''
return EXPECTED in res
def lambda_handler(event, context):
print('Checking {} at {}...'.format(SITE, event['time']))
try:
if not validate(urlopen(SITE).read()):
raise Exception('Validation failed')
except:
print('Check failed!')
raise
else:
print('Check passed!')
return event['time']
finally:
print('Check complete at {}'.format(str(datetime.now())))
You don't need any environment variables. Just keep it simple
from __future__ import print_function
import os
from datetime import datetime
from urllib2 import urlopen
def lambda_handler(event, context):
url = 'https://www.google.com' # change it with your own
print('Checking {} at {}...'.format(url, datetime.utcnow()))
html = urlopen(url).read()
# do some processing
return html
Here is another simple example.
from __future__ import print_function
def lambda_handler(event, context):
first = event.get('first', 0)
second = event.get('second', 0)
sum = first + second
return sum
Here is a sample event which will be used to invoke this lambda. you can configure event from Lambda web interface. (or google it)
{
"first": 10,
"second": 23
}
In my case, I missed adding the logging_config.ini to the lambda function.
I guess you would face similar error when the lambda function doesn't find the referenced file or package.
Thanks to the new cloud9 IDE integration, I was able to create one on the fly.
I am using boto3 in aws lambda to fecth object in S3 located in Frankfurt Region.
v4 is necessary. otherwise following error will return
"errorMessage": "An error occurred (InvalidRequest) when calling
the GetObject operation: The authorization mechanism you have
provided is not supported. Please use AWS4-HMAC-SHA256."
Realized ways to configure signature_version http://boto3.readthedocs.org/en/latest/guide/configuration.html
But since I am using AWS lambda, I do not have access to underlying configuration profiles
The code of my AWS lambda function
from __future__ import print_function
import boto3
def lambda_handler (event, context):
input_file_bucket = event["Records"][0]["s3"]["bucket"]["name"]
input_file_key = event["Records"][0]["s3"]["object"]["key"]
input_file_name = input_file_bucket+"/"+input_file_key
s3=boto3.resource("s3")
obj = s3.Object(bucket_name=input_file_bucket, key=input_file_key)
response = obj.get()
return event #echo first key valuesdf
Is that possible to configure signature_version within this code ? use Session for example. Or is there any workaround on this?
Instead of using the default session, try using custom session and Config from boto3.session
import boto3
import boto3.session
session = boto3.session.Session(region_name='eu-central-1')
s3client = session.client('s3', config= boto3.session.Config(signature_version='s3v4'))
s3client.get_object(Bucket='<Bkt-Name>', Key='S3-Object-Key')
I tried the session approach, but I had issues. This method worked better for me, your mileage may vary:
s3 = boto3.resource('s3', config=Config(signature_version='s3v4'))
You will need to import Config from botocore.client in order to make this work. See below for a functional method to test a bucket (list objects). This assumes you are running it from an environment where your authentication is managed, such as Amazon EC2 or Lambda with a IAM Role:
import boto3
from botocore.client import Config
from botocore.exceptions import ClientError
def test_bucket(bucket):
print 'testing bucket: ' + bucket
try:
s3 = boto3.resource('s3', config=Config(signature_version='s3v4'))
b = s3.Bucket(bucket)
objects = b.objects.all()
for obj in objects:
print obj.key
print 'bucket test SUCCESS'
except ClientError as e:
print 'Client Error'
print e
print 'bucket test FAIL'
To test it, simply call the method with a bucket name. Your role will have to grant proper permissions.
Using a resource worked for me.
from botocore.client import Config
import boto3
s3 = boto3.resource("s3", config=Config(signature_version="s3v4"))
return s3.meta.client.generate_presigned_url(
"get_object", Params={"Bucket": AIRFLOW_BUCKET, "Key": key}, ExpiresIn=expTime
)