I am trying to deploy a Lambda function to AWS from S3.
My organization currently does not provide the ability for me to upload files to the root of an S3 bucket, but only to a folder (ie: s3://application-code-bucket/Application1/).
Is there any way to deploy the Lambda function code through S3, from a directory other than the bucket root? I checked the documentation for Lambda's CreateFunction AWS command and could not find anything obvious.
You need to zip your lambda package and upload to S3 in any folder.
You can then provide an https S3 url of the file to upload to lambda
function.
The S3 bucket needs to be in the same region as that of the lambda
function.
Make sure you zip from the folder, i.e when the package is unzipped,
the files should be extracted in the same directory as the unzip
command, and should not create a new directory for the contents.
I have this old script of mine that I used to automate lambda deployments.
It needs to be refactored a bit, but still usable.
It gets as input the lambda name and the zip file path located locally on your PC.
It uploads it to S3 and publishes to the AWS Lambda.
You need to set AWS credentials with IAM roles that allows:
S3 upload permission
AWS Lambda update permission
You need to modify the bucket name and the path you want your zip to be uploaded to. (lines 36-37).
That's it.
Related
My ClaudiaJS API is working fine but I cannot store images files on an S3 bucket.
The problem is because the Lambda Function has no permission to Write on S3.
Question: How can I tell ClaudiaJS to include an S3 Write permission into the Lambda Function Permissions?
I was able to follow this example1 and let my ec2 instance read from S3.
In order to write to the same bucket I thought changing line 572 from grant_read() to grant_read_write()
should work.
...
# Userdata executes script from S3
instance.user_data.add_execute_file_command(
file_path=local_path
)
# asset.grant_read(instance.role)
asset.grant_read_write(instance.role)
...
Yet the documented3 function cannot be accessed according to the error message.
>> 57: Pyright: Cannot access member "grant_read_write" for type "Asset"
What am I missing?
1 https://github.com/aws-samples/aws-cdk-examples/tree/master/python/ec2/instance
2 https://github.com/aws-samples/aws-cdk-examples/blob/master/python/ec2/instance/app.py#L57
3 https://docs.aws.amazon.com/cdk/latest/guide/permissions.html#permissions_grants
This is the documentation for Asset:
An asset represents a local file or directory, which is automatically
uploaded to S3 and then can be referenced within a CDK application.
The method grant_read_write isn't provided, as it is pointless. The documentation you've linked doesn't apply here.
an asset is just a Zip file that will be uploded to the bootstraped CDK s3 bucket, then referenced by Cloudformation when deploying.
if you have an script you want ot put into an s3 bucket, you dont want to use any form of asset cause that is a zip file. You would be better suited using a boto3 command to upload it once the bucket already exists, or making it part of a codePipeline to create the bucket with CDK then the next step in the pipeline uploads it.
the grant_read_write is for aws_cdk.aws_s3.Bucket constructs in this case.
I'm trying to use an S3 bucket to upload files to as part of a build, it is configured to provide files as a static site and the content is protected using a Lambda and CloudFront. When I manually create files in the bucket they are all visible and everything is happy, but when the files are uploaded what is created are not available, resulting in an access denied response.
The user that's pushing to the bucket does not belong in the same AWS environment, but it has been set up with an ACL that allows it to push to the bucket, and the bucket with a policy that allows it to be pushed to by that user.
The command that I'm using is:
aws s3 sync --no-progress --delete docs/_build/html "s3://my-bucket" --acl bucket-owner-full-control
Is there something else that I can try that basically uses the bucket permissions for anything that's created?
According to OP's feedback in the comment section, setting Object Ownership to Bucket owner preferred fixed the issue.
I subscribed to the free synthetic dataset.
Now I have "Revision ID", "Revision ARN", "Data set ID" and 28 CSV files which I can not download in a pack. I must manually download them one after another or I can export them all to the AWS e3 (I do not want to do that).
Is there a way to download it all in a single archive or somehow automate the process via AWS S3 CLI?
I've tried
./venv/bin/awscliv2 s3 cp s3://arn:aws:dataexchange:us-east-1::data-sets/b0b14e86c092855166507c15e045b844/revisions/6011536d595840f7bd4412fca59e0f6b/assets/7cd4a5cbedb0c5c83e37c20f668b3708 ./
fatal error: Parameter validation failed:
Invalid bucket name "arn:aws:dataexchange:us-east-1::data-sets": Bucket
name must match the regex "^[a-zA-Z0-9.\-_]{1,255}$" or be an ARN
matching the regex "^arn:(aws).*:s3:[a-z\-0-9]+:[0-9]{12}:accesspoint[/:]
[a-zA-Z0-9\-]{1,63}$|^arn:(aws).*:s3-outposts:[a-z\-0-9]+:
[0-9]{12}:outpost[/:][a-zA-Z0-9\-]{1,63}[/:]accesspoint[/:][a-zA-Z0-9\-]
{1,63}$"
UPD: I've found a python snippet, which uses boto3 and creates a temporary bucket in the process.
UPD 2:: From https://docs.aws.amazon.com/data-exchange/latest/userguide/jobs.html#exporting-assets
There are two ways you can export assets from a published revision of a product:
To an Amazon S3 bucket that you have permissions to access.
By using a signed URL.
Therefore, I can't do that stuff without bucket through AWS CLI, but I can use EXPORT_ASSET_TO_SIGNED_URL
UPD 3: I've created a gist for downloading a dataset from AWS DataExchange via signed urls
I dont know why u using such complicated aws s3 cp. 2 line script can be something like
#syncing all files in folder to local directory ~/csvs/
aws s3 sync s3://bucketname/<folder>/ ~/csvs/
#you can zip or tar full foder whatever u want
zip ~/csvs csvs.zip
I want to get started with zappa framework, but when I'm running init zappa I got this notification:
How to disable autocreate s3 bucket?
You can't. Zappa first uploads your .zip file to a bucket and from there does the deployment.
From the official repo:
Zappa will automatically package up your application, [...] upload the archive to S3, create and manage the necessary Amazon IAM policies and roles, register it as a new Lambda function, create a new API Gateway resource, create WSGI-compatible routes for it, link it to the new Lambda function, and finally delete the archive from your S3 bucket. Handy!
So your option is to dig into Zappa and circumvent this on your own or perhaps try Chalice that does the upload directly.
#mislav is correct that Zappa does need an S3 bucket. But one only gets auto-created if you don't specify one. Simply provide a valid bucket name at the prompt, and Zappa will use that bucket instead of creating one for you.