i need to process csv file using numpy and pandas , i could do it in local machine its running as a expected manner then i am trying to do the same thing in aws lambda function with the same code i was used in my local machine.i am trying to import numpy inside my aws lambda function but error was thrown as Unable to import module 'aws_lambda_demo/lambda_function': No module named 'numpy'. then i create a zip file with aws lambda function along with numpy package after that also facing same issue Unable to import module 'aws_lambda_demo/lambda_function': No module named 'numpy'. Notes: i kept my csv files in s3 storage .any help will be appreciated . thanks in advance.
The problem is that your local numpy and pandas are compiled for the local machine's architecture. Since AWS Lambda uses custom Linux, they are probably not compatible.
So if you want to use them, you have two choices:
Compile dependencies on EC2 instance which uses the same Amazon Linux version as AWS Lambda and create a deployment package.
Use one of the precompiled packages from here
Related
I am trying to write code using the AWS Lambda function to extract Pdf Invoice data from the AWS Textract Service and save the data into Excel. To do this, I installed the openpyxl library, created a zip file for it, and created a layer in lambda function that uses the openpyxl library. I am getting the following error ( No module named 'openpyxl'",). I would appreciate your assistance in resolving it.
Have you tried Textractor pip install amazon-textract-textractor? It comes built-in with the export-to-excel features and comes with the lamdba layers pre-built on the official GitHub repository: https://aws-samples.github.io/amazon-textract-textractor/using_in_lambda.html
Note that the available lamdba layer uses XlsxWriter instead of openpyxl.
Disclaimer: I am a maintainer of Textractor.
I've had a lambda function in Python 3.9 which is pretty simple. Just sends a post request. It has worked fine for the past few weeks.
I've deleted and remade the stack several times in CloudFormation, and have had no issues.
Randomly, today it started throwing the error
"errorMessage": "Unable to import module 'index': No module named 'requests'",
"errorType": "Runtime.ImportModuleError"
index.py is the name of the lambda function.
So I looked it up, and everyone says that 'requests' is not prepackaged into Lambda, and you need to create a layer where you manually install it into a zip etc.
How is this possible? It worked for weeks without having to create a layer, and the code was not touched.
But now 'import requests' just stops working?
There are of course other alternatives to the requests module. 'urllib' comes to mind. But this is bugging me. How is it possible that it worked for weeks, but randomly just stops being able to import this module?
Its like some kind of Mandela effect. Every single person says you need to create a venv, and package it with dependencies manually. I've never had to do that and it worked for AGES. I feel like im going crazy...
Did you possibly change the version of Python used by the function?
From Upcoming changes to the Python SDK in AWS Lambda | AWS Compute Blog:
In response to customer feedback, we have decided to cancel the change described in this blog post. The version of the AWS SDK included in the AWS Lambda runtimes for Python 2.7, Python 3.6 and Python 3.7 will continue to include the ‘requests’ module in Botocore. No action is required for customers using these runtimes. The Lambda runtimes for Python 3.8 and later do not include the ‘requests’ module. Customers using ‘requests’ with these runtimes should package the ‘requests’ module with their function code or as a Lambda layer.
I am using AWS glue to create ETL workflow, where I am fetching the data from the API and loading it into RDS. In AWS Glue, I used pyspark script. In the same script, I have used the 'aiohttp' and 'asyncio' modules of python to call my API asynchronously. But in AWS glue it is throwing me an error that Module Not found for the only aiohttp.
I have already tried with different versions of aiohttp module and tested in the glue job but still throwing me the same error. Can someone please help me with this topic?
Glue 2.0
AWS Glue version 2.0 lets you provide additional Python modules or different versions at the job level. You can use the --additional-python-modules job parameter with a list of comma-separated Python modules to add a new module or change the version of an existing module.
Also, within the --additional-python-modules option you can specify an Amazon S3 path to a Python wheel module.
This link to official documentation lists all modules already available. If you need a different version or need one to be installed, it can be specified in the parameter mentioned above.
Glue 1.0 & 2.0
You can zip the python library, upload it so s3 and specify the path as --extra-py-files job parameter.
See link to official documentation for more information.
I have need to run a custom file parser to parse incoming files live for further upstream use. The parser is an executable and takes some arguments. I have written the following lambda test function:
import subprocess, sys
import os
def lambda_handler(event, context):
path = os.environ['LAMBDA_TASK_ROOT']
subprocess.call([path+'/*exefile*',*args*])
However I get an error, and this is even without any input, meaning it's not related to the writing of new files in the lambda environment:
PermissionError errno13 Permission denied
It seems to me that it's simply not allowed to run the executable. For the IAM role I have given it the full lambda access. The custom exe file is not compiled for linux, but wouldn't I get another error if that was the issue?
Help appreciated!
Per this AWS blog post it needs to compiled for Amazon's Linux:
If you compile your own binaries, ensure that they’re either
statically linked or built for the matching version of Amazon Linux.
The current version of Amazon Linux in use within AWS Lambda can
always be found on the Supported Versions page of the Lambda docs.
when using cost explorer client in lambda function , 'ce' is not supported. but it may available in new version of botocore. so is it possible to import the new version of botocore in lambda python function ? if yes, how we can implement it. please provide some example
AWS Lambda provides a way to to use a version of Boto3 other than the one included by default, you can include it in your deployment package.
From Creating a Deployment Package (Python)
AWS Lambda includes the AWS SDK for Python (Boto 3), so you don't need
to include it in your deployment package. However, if you want to use
a version of Boto3 other than the one included by default, you can
include it in your deployment package.