Running Taurus BlazeMeter on AWS Lambda - amazon-web-services

I am trying to run a BlazeMeter Taurus script with a JMeter script inside via AWS Lambda. I'm hoping that there is a way to run bzt via a local installation in /tmp/bzt instead of looking for a bzt installation on the system which doesn't really exist since its lambda.
This is my lambda_handler.py:
import subprocess
import json
def run_taurus_test(event, context):
subprocess.call(['mkdir', '/tmp/bzt/'])
subprocess.call(['pip', 'install', '--target', '/tmp/bzt/', 'bzt'])
# subprocess.call('ls /tmp/bzt/bin'.split())
subprocess.call(['/tmp/bzt/bin/bzt', 'tests/taurus_test.yaml'])
return {
'statusCode': 200,
'body': json.dumps('Executing Taurus Test hopefully!')
}
The taurus_test.yaml runs as expected when testing on my computer with bzt installed via pip normally, so I know the issue isn't with the test script. The same traceback as below appears if I uninstall bzt from my system and try use a local installation targeted in a certain directory.
This is the traceback in the execution results:
Traceback (most recent call last):
File "/tmp/bzt/bin/bzt", line 5, in <module>
from bzt.cli import main
ModuleNotFoundError: No module named 'bzt'
It's technically failing in /tmp/bzt/bin/bzt which is the executable that's failing, and I think it is because it's not using the local/targeted installation.
So, I'm hoping there is a way to tell bzt to use keep using the targeted installation in /tmp/bzt instead of calling the executable there and then trying to pass it on to an installation that doesn't exist elsewhere. Feedback if AWS Fargate or EC2 would be better suited for this is also appreciated.

Depending on the size of the bzt package, the solutions are:
Use Lambda Docker recent feature, and this way, what you run locally is what you get on Lambda.
Use Lambda layers (similar to Docker), this layer as the btz module in the python directory as described there
When you package your Lambda, instead of uploading a simple Python file, create a ZIP file containing both: /path/to/zip_root/lambda_handler.py and pip install --target /path/to/zip_root

Related

How to install python packages within Amazon Sagemaker Processing Job?

I am trying to create a Sklearn processing job in Amazon Sagemekar to perform some data transformation of my input data before I do model training.
I wrote a custom python script preprocessing.py which does the needful. I use some python package in this script. Here is the Sagemaker example I followed.
When I try to submit the Processing Job I get an error -
............................Traceback (most recent call last):
File "/opt/ml/processing/input/code/preprocessing.py", line 6, in <module>
import snowflake.connector
ModuleNotFoundError: No module named 'snowflake.connector'
I understand that my processing job is unable to find this package and I need to install it. My question is how can I accomplish this using Sagemaker Processing Job API? Ideally there should be a way to define a requirements.txt in the API call, but I don't see such functionality in the docs.
I know I can create a custom Image with relevant packages and later use this image in the Processing Job, but this seems too much work for something that should be built-in?
Is there an easier/elegant way to install packages needed in Sagemaker Processing Job ?
One way would be to call pip from Python:
subprocess.check_call([sys.executable, "-m", "pip", "install", package])
Another way would be to use an SKLearn Estimator (training job) instead, to do the same thing. You can provide the source_dir, which can include a requirements.txt file, and these requirements will be installed for you
estimator = SKLearn(
entry_point="foo.py",
source_dir="./foo", # no trailing slash! put requirements.txt here
framework_version="0.23-1",
role = ...,
instance_count = 1,
instance_type = "ml.m5.large"
)

Deploy Pytidylib module in aws lambda by using Lamda Layers

I am trying to deploy pytidylib python module into AWS lambda function by using layers .
I have created the path as described in aws docs and created new layer.
Now the code of pytidylib needs some libraries from /usr/lib but i have installed libraries in /python/lib/python3.7/site-packages/ , so to resolve this i added the path in environ PATH of aws linux server platform , but still the issue is not resolved.
Below is my code :-
def lambda_handler(event, context):
"""Read file from s3 on trigger."""
s3 = boto3.client("s3")
print(sys.platform)
ld_library_path = os.environ["LD_LIBRARY_PATH"]
print("old ld_library_path is ",ld_library_path)
ld_library_path = ld_library_path +
":/opt/python/lib/python3.7/site-packages/"
os.environ["LD_LIBRARY_PATH"] = ld_library_path
print("ld_library_path after set is
",os.environ["LD_LIBRARY_PATH"])
ld_library_path after set is /var/lang/lib:/lib64:/usr/lib64:/var/runtime:/var/runtime/lib:/var/task:/var/task/lib:/opt/lib:/opt/python/lib/python3.7/site-packages/
I want to understand is there any way i can make this work through some changes in code and make the pytidylib module run through layers .
Below is the error:-
[ERROR] OSError: Could not load libtidy using any of these names:
libtidy,libtidy.so,libtidy-0.99.so.0,cygtidy-0-99-0,tidylib,libtidy.dylib,tidy
Traceback (most recent call last):
File "/var/task/lambda_function.py", line 68, in lambda_handler
document, errors = tidy_document(doc)
File "/opt/python/lib/python3.7/site-packages/tidylib/tidy.py", line 222, in tidy_document
return get_module_tidy().tidy_document(text, options)
File "/opt/python/lib/python3.7/site-packages/tidylib/tidy.py", line 234, in get_module_tidy
_tidy = Tidy()
File "/opt/python/lib/python3.7/site-packages/tidylib/tidy.py", line 99, in __init__
+ ",".join(lib_names))
I tried to replicate your issue and for me Pytidylib layer works as expected.
This is the way I used to construct the layer, in case you want to give it a try. It involves a docker tool described in the recent AWS blog:
How do I create a Lambda layer using a simulated Lambda environment with Docker?
I created Pytidylib layer as follows:
Create empty folder, e.g. mylayer.
Go to the folder and create requirements.txt file with the content of
Pytidylib
Run the following docker command (may adjust python version to your needs):
docker run -v "$PWD":/var/task "lambci/lambda:build-python3.8" /bin/sh -c "pip install -r requirements.txt -t python/lib/python3.8/site-packages/; exit"
Create layer as zip:
zip -r pytidylayer.zip python > /dev/null
Create lambda layer based on pytidylayer.zip in the AWS Console. Don't forget to specify Compatible runtimes to python3.8.
Add the layer to the lambda and test using the following lambda function:
import json
import tidylib
def lambda_handler(event, context):
print(dir(tidylib))
return {
'statusCode': 200,
'body': json.dumps('Hello from Lambda!')
}
The function executed correctly:
['PersistentTidy', 'Tidy', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'release_tidy_doc', 'sink', 'tidy', 'tidy_document', 'tidy_fragment']
I solved this by adding path of the tidy library (libtidy.so.5.2.0) into the environment variable of LD_LIBRARY_PATH of linux server
For me library was pre installed in ubuntu 18.04 server in /usr/lib .Copy the library from this path ,place it inside the tidylib folder create a zip, and follow the steps of lambda-layers creation .

How can I run a HIT Task with ParlAi and MTurk

I tried to follow this tutorial on publishing a HIT Task on MTurk using ParlAI:
how to setup a task with parlAI
The tutorial says that I should run the following code in order to run the task on sandbox:
$ python run.py -nh 2 -r 0.05 --sandbox --verbose
Normally, it should then download the dataset that has been specified in the agents.py file. But when I run this in powershell, the following error message occurs:
PS C:\py3\ParlAI\parlai\mturk\tasks\model_evaluator> python run.py -nh 2 -r
0.05 --sandbox --verbose
Traceback (most recent call last):
File "run.py", line 6, in <module>
from parlai.core.params import ParlaiParser
ModuleNotFoundError: No module named 'parlai'
PS C:\py3\ParlAI\parlai\mturk\tasks\model_evaluator>
It says that module parlai is missing. How can I make it work?
Thanks for your answer!
Best regards
Rainer
This error means that ParlAI just hasn't been installed to your python path yet (python setup.py develop in the ParlAI home directory).
However, note that ParlAI doesn't officially support windows, so you're likely to run into further errors after that.
(The ParlAI team usually responds pretty quickly if you open issues on the Github page if you have questions directly for them.)

AWS Lambda, Python: Unable to import module that is definitely within Zip package (xmlsec)

I am using the Python module xmlsec in my lambda function. The import looks like import dm.xmlsec.binding as xmlsec. The proper directory structure exists. at the root of the archive there is dm/xmlsec/binding/__init__.py and the rest of the module is there. However, when executing the function on lambda, I get the error "No module named dm.xmlsec.binding"
I have built many Python27 lambda functions in the same way as this one with no issues. I install all of the needed python modules to my build directory, with the lambda function at the root. I then zip the package recursively and update the existing function with the resulting archive using the AWS CLI. I've also tried manually uploading the archive in the console as well, with the same result.
I was honestly expecting some trouble with this module, but I did expect lambda to at least see it. What is going on?

Deployment of Django app to AWS Lambda using zappa fails even though Zappa says your app is live at the following link

I came across the amazing serverless AWS Lambda recently and thought it would be great to have my app up there and not have to worry about auto scaling, load balancing and all for apparently a fraction of the cost.
So then I found out about Zappa which takes care of deploying your python app to AWS Lambda for you. Amazing is what I thought.
Its actually on paper very easy to do. just follow the instructions here..
https://github.com/Miserlou/Zappa
Anyway I followed the instructions with just a very basic django app using virtualenv that just contained the django rest framework tutorial in it..
Tested it locally and works fine.
Then I set up my s3 bucket and authenticated my credentials with the awscli.
Then I ran the 2 thing you need to deply.
Zappa init,
Zappa deploy dev.
Then it went through all its processes, packaging into zip, deploying etc...
Then at the end it said your app is live and here is the url
It gave me a url to try.
I pasted the url into the browser and this is what the browser displayed for me.
Oh yeah and my s3 bucket is still empty and so is my aws lambda service.
{
"message": "An uncaught exception happened while servicing this request.",
"traceback": [
"Traceback (most recent call last):\n",
" File \"/var/task/handler.py\", line 395, in handler\n response = Response.from_app(self.wsgi_app, environ)\n",
" File \"/home/donagh/projects/vizzydev/vizzy/visualid/vizzy_django/env/build/Werkzeug/werkzeug/wrappers.py\", line 865, in from_app\n",
" File \"/home/donagh/projects/vizzydev/vizzy/visualid/vizzy_django/env/build/Werkzeug/werkzeug/wrappers.py\", line 57, in _run_wsgi_app\n",
" File \"/home/donagh/projects/vizzydev/vizzy/visualid/vizzy_django/env/build/Werkzeug/werkzeug/test.py\", line 871, in run_wsgi_app\n",
"TypeError: 'NoneType' object is not callable\n"
]
}
If anyone as any ideas I would be very grateful. I would love to get this working. It would be an incredibly powerful resource.
When I get errors related to werkzeug wrapper it is usually because my packages were not installed in my virtual environment.
virtualenv venv
source venv/bin/activate
pip install Django
pip install zappa
# pip install any other packages
# or with a requirements.txt file
pip install -r requirements.txt
Then run the zappa deploy commands.