I have a bunch of settings and configs that I need AWS Lambda to be able to access. This is not a .py file, so I cannot just import it. In order to load the YAML, I need to specify a path to the file, so how do I do that in Lambda?
My understanding is that Lambda can only read/write to the /tmp folder. But I would like to include this config file with either the lambda layer or as a file in the lambda code package. Either of those two options are not in the /tmp folder and therefore does not provide direct read/write to files there.
Where can I put this YAML file and how can I reach it to read it during runtime?
Lambda has read access to everything. It only has write access to the /tmp folder. If you are thinking about including the YAML file in a Lambda layer, or in the code package, your code will be able to read the file just fine.
Related
I have a lambda function written in .NET 6 to which I have attached a lambda layer which is a zip file with many JSON files and folders in it.
I want to read each JSON file in my lambda function.
When I do this I get an error saying Could not find a part of the path '/var/task/geojson-files/UK.json':
var filePath = Path.Combine(AppDomain.CurrentDomain.BaseDirectory, "geojson-files", "UK.json");
var geojson = await File.ReadAllTextAsync(filePath);
I also tried /opt/geojson-files/UK.json. But still unable to read the file.
Is there any way to read the file in a lambda function from a lambda layer?
Thanks to #jarmod for his suggestion in the comments to enumerate the contents of /opt
I iterated over the /opt folder and could get the root cause of my problem i.e. unable to read files from the lambda layer.
I zipped the contents of geojson-files from inside that folder. So when I enumerated /opt, I could directly see /opt/UK.json file. So then I zipped the geojson-files folder and now I can see /opt/geojson-files/UK.json file.
The example code in the MWAA docs for connecting MWAA to EKS has the following:
#use a kube_config stored in s3 dags folder for now
kube_config_path = '/usr/local/airflow/dags/kube_config.yaml'
This doesn't make me think that putting the kube_config.yaml file in the dags/ directory is a sensible long-term solution.
But I can't find any mention in the docs about where would be a sensible place to store this file.
Can anyone link me to a reliable source on this? Or make a sensible suggestion?
From KubernetesPodOperator Airflow documentation:
Users can specify a kubeconfig file using the config_file parameter, otherwise the operator will default to ~/.kube/config.
In a local environment, the kube_config.yaml file can be stored in specific directory reserved for Kubernetes (e.g. .kube, kubeconfig). Reference: KubernetesPodOperator (Airflow).
In the MWAA environment, where DAG files are stored in S3, the kube_config.yaml file can be stored anywhere in the root DAG folder (including any subdirectory in the root DAG folder, e.g. /dags/kube). The location of the file is less important than explicitly excluding it from DAG parsing via the .airflowignore file. Reference: .airflowignore (Airflow).
Example S3 directory layout:
s3://<bucket>/dags/dag_1.py
s3://<bucket>/dags/dag_2.py
s3://<bucket>/dags/kube/kube_config.yaml
s3://<bucket>/dags/operators/operator_1.py
s3://<bucket>/dags/operators/operator_2.py
s3://<bucket>/dags/.airflowignore
Example .airflowignore file:
kube/
operators/
I am using cloudformation to create my lambda function. I have opted to pull the code from S3.
However, it appears to create a nested structure when the lambda function gets created, and I am unable to import my packages unless I move the lambda and associated library packages up to the root level of the lambda function.
Cloudformation value for code section:
Code:
S3Bucket: youll_never_guess-bucket-12345
S3Key: python_data_collector.zip
How it appears in lambda, aws console:
Full path for handler in console:
I've tried: python_data_collector/lambda.lambda_handler and python_data_collector.lambda.lambda_handler
Error message:
Unable to import module 'python_data_collector/lambda': No module named 'requests'"
The Python dependencies should reside at the root level of you lambda deployment package. You can indeed point to a nested file as the entry point of your function, but this does not change the dependency behaviour of your function.
However, the structure of your lambda code has nothing to do with where your zip file is located in the S3 bucket. Presumably, when you are creating the zip file, you are adding a folder at the root level which contains the code & dependencies. You should not have that extra folder in the zip file and simply put the code (nested or not) and the dependencies (not nested) at the root of you zip package. Lambda will simply unzip the file and place the contents as is, in your lambda function.
Firstly, I am a newbie to AWS. I was able to edit my Lambda code in line, but I recently uploaded a zip file to it(30MB) to S3 bucket and added this zip to my Lambda from S3, and now my Lambda inline editor doesn't open anymore saying the following error
"The deployment package of your Lambda function "LF2" is too large to
enable inline code editing. However, you can still invoke your function."
I tried deleting my zip file from S3 bucket hoping that the URL of zip would not be reachable and the lambda would lose the zip file and let me edit the function again. But, my lambda size still consists of the 30MB zip file size. I am unable to delete this zip and can't figure out a way to get rid of this it and edit my lambda code again.
Note: My Lambda code was written in-line and different from the zip file(which only contains elastic search setup files which I uploaded for using in my code since import elastic search wasn't working). I know there would have been a better way to do this without uploading it's zip.
Yes, you can download the Lambda function. Go to the AWS console for the Lambda function, make sure you are in the Configuration view, then click Actions | Export function. This will allow you to download a ZIP file containing the Lambda function.
Note that once you upload a Lambda function via S3, it's copied by the Lambda service. There's no connection at that point back to the S3 object that you uploaded. One reason for this is that your Lambda function would break if you, accidentally or otherwise, deleted the file from S3.
I had this problem yesterday then I somehow managed to find my code but not that full code that was vanished from AWS lambda. I wrote that code again, tested it, then tried to upload it with the same name of the lambda function and at the same lambda function by compressing it in my own system.
While uploading it, lambda gave me the option to choose between the remote file I uploaded and the local file it had saved previously. I opted for the local file and boom! I got my code back as it was last saved.
So, I suggest you to just try to upload a random blank compressed zip file containing one file name same as the lambda function. It would give you the option to choose from both files, then choose for "local" file. It would take you to the in-line editor where your code was.
I just ran into same soup .. seem like the upload replaced the previous index.js with export handler.
How and what’s to be configured in API gateway to accept file as an input in AJAX call. Also How to read the contents of the file in Python Lambda?
There ain't much help with examples available in the official aws docs
We need to zip the dependencies along with python file and upload it.
For example if you wish to read a file called input.conf in python file reader.py,
you need to create a zip file which includes the above files within it.
/uploader.zip
|- input.conf
| - reader.py
The name of the python file should be the first part of lambda handler, like:
reader.lambda_handler (in case of python)