I was wondering what's the best way (if it is possible) to access local files from a lambda function.
Basically I want to get a .txt file that I have at C://Users/User/Desktop or any directory and put that file inside an s3 bucket.
I have been able to put information into a bucket from a lambda function (Hardcoded info) but I'm struggling in getting info from host to AWS. From what I have seen, the way to go is to use AWS IoT Greengrass but after hours of trying to make it work, things are not looking good.
Is AWS IoT Greengrass the only option or is it there a simpler way of accessing local files?
It is not at all possible to have a lambda function access local files directly - something running on your local machine is going to need to serve those files TO lambda, lambda can't retrieve them.
Without knowing anything about the problem you are trying to solve, I would certainly start with the thought of pushing those files to S3, and then let lambda do its thing.
Related
As i'm New on aws and a little confused by all the similar services, I would like to have some leads and know if I am in the right direction.
I have tar.gz archives stored on aws s3 glacier deep archives. I would like that when requesting a restore, the archive is automatically extracted and the folders and files it contains put in s3 (with an expiration date).
these archives are too big to be extracted via lambda (300GB or more).
My idea would be to trigger a lambda function when the restore is complete and use that lambda function to start another aws service that does the extraction. I was thinking either aws batch or fargate. Which service do you think is the most suitable? For this kind of simple task it is preferable to use an arm architecture?
If someone has already done this before and has codes to share I'm interested (if not I'll try to put my final solution here for others).
I'm writing to you because I'm quite a novice with AWS... I only worked before with EC2 instances for simple tasks...
I am currently looking for an AWS service for reciving data using REST API calls (to external AWS services).
So far I have used EC2 where I deployed my library (python) that made calls and stored data in S3.
What more efficient ways does AWS offer for this? some SaaS?
I know that they are still more details to know in order to choose a good services but I would like to know from where I can start looking.
Many thanks in advance :)
I make API requests using AWS Lambda. Specifically, I leave code that makes requests, writes the response to a file and pushes the response object (file) to AWS S3.
You'll need a relative/absolute path to push the files to wherever you want to ingest. By default lambda servers current working directory is: /var/task but you may want to write your files to /tmp/ instead.
You can automate the ingestion process by setting a CloudWatch rule to trigger your function. Sometimes I chain lambda functions when I need to loop requests with changing parameters instead of packing all requests within a single function,
i.e.
I leave the base request (parameterized) in one function and expose the function through an API Gateway endpoint.
I create a second function to call the base function once for each value I need by using the Event object (which is the JSON body of a regular request). This data will replace parameters within the base function.
I automate the second function.
Tip:
Lambda sometimes will run your requests inside the same server. So if you're continuously running these for testing the server may have files from past calls that you don't want, so I usually have a clean-up step at the beginning of my functions that iterates through my filesystem to make sure there are no files before making the requests.
Using python 3.8 as a runtime I use the requests module to send the request, I write the file and use boto3 to push the response object to an aws S3 bucket.
To invoke an external service you need some "compute resources" to run your client. Under compute resources in aws we understand ec2, ecs (docker container) or lambda (serverless - my favorite)
You had your code already running on EC2 so you should already know you need VPC with a public subnet and ip address to make an outbound call regardless the compute resource you choose
I have seen a lot of examples of using AWS Transfer to move files into S3. I have the opposite need.
In a separate project, someone drops a file in S3. I have code that pgp encrypts that file with a third party public key. The code then moves the encrypted file to another S3 location. This part is done and is working.
Problem: I now need to SFTP this S3 encrypted file out of AWS to the third party.
It seems like AWS Transfer is the way to go. The examples I see are all for incoming files.
I would prefer to do this in Lambda. I have a trigger set up in S3 to run a Lambda function every time an encrypted file is dropped. I have that set up and working but nothing else.
Has anyone else done this?
It appears that your requirement is:
When a file is placed into a particular Amazon S3 bucket / path
Trigger an AWS Lambda function
The Lambda function should:
Encrypt the file
Send it via SFTP to a remote location
If you have everything working except the last step, then you simply need to write code that will send the file via SFTP. You would place this code in your Lambda function.
For example, in Node there is a library called ssh2-sftp-client.
See:
ssh2-sftp-client - npm
What is the best way to connect to SFTP in NodeJS? - Quora
Since the connection is outbound to a remote location, such a connection would not involve any AWS services (apart from the fact that the code is running within AWS Lambda).
I have a key that is being shared among different services and it is currently stored in an s3 bucket inside a text file.
My goal is to read that variable and pass it to my lambda service through cloudformation.
for an ec2 instance it was easy because I could download the file and read it, and that was easily achievable by putting the scripts inside my cloudformation json file. But I don't have any idea how to do it for my lambdas....!
I tried to put my credentials in gitlab pipeline but because of the access permissions it doesn't let gitlab pass it on, so my best and least expensive option right now is to do it in cloud formation.
The easiest method would be to have the Lambda function read the information from Amazon S3.
The only way to get CloudFormation to "read" some information from Amazon S3 would be to create a Custom Resource, which involves writing an AWS Lambda function. However, since you already have a Lambda function, it would be easier to simply have that function read the object.
It's worth mentioning that, rather than storing such information in Amazon S3, you could use the AWS Systems Manager Parameter Store, which is a great place to store configuration information. Your various applications can then use Parameter Store to store and retrieve the configuration. CloudFormation can also access the Parameter Store.
I'm a newbie in AWS Lambda functions. I used a script in AWS CLI in order to create an aws function in Node.js. This script has a config file called config.json. After function creation, I'm able to see the code on Lambda AWS Console and here comes my doubt. The code has this line:
var config = require('./config.json');
So, where this "./config.json" file is actually stored. Could I be able to edit the contents of config.json after deployment of lambda function?
Thanks in advance.
So, where this ./config.json file is actually stored?
It should be stored in the same directory as your Lambda handler function. They should be bundled in a zip file and deployed to AWS. If you didn't deploy it that way then that file doesn't currently exist.
If your Lambda function consists of multiple files you will have to bundle your files and deploy it to AWS as a zip file.
You cannot edit the source of external libraries/files via the AWS Lambda web console. You can only edit the source of the Lambda function handler via the web console.
Your files are placed into the directory specified in the environment variable LAMBDA_TASK_ROOT. You can read this via nodejs as process.env.LAMBDA_TASK_ROOT.
The code you deploy, including the config.json file are read-only, but if you do wish to modify files on the server, you may do so underneath /tmp. Mind, those changes will only be valid for that single container, for its lifecycle (4m30s - 4hrs). Lambda will auto-scale up and down between 0 and 100 containers, by default.
Global variables are also retained across invocations, so if you read config.json into a global variable, then modify that variable, those changes will persist throughout the lifecycle of the underlying container(s). This can be useful for caching information across invocations, for instance.