Unable to run chromedriver using selenium in Python in aws cloud9 - amazon-web-services

After I install selenium, chrome, and chromedriver to aws cloud9, I tried to execute chromedriver. But PATH error occurred.
I checked chromedriver path is correct. And I don't know why this error happened.
aws:~/environment/seleniumTest $ ll /home/ec2-user/environment/seleniumTest/chromedriver
-rwxr-xr-x 1 ec2-user ec2-user 8496784 Sep 13 2018 /home/ec2-user/environment/seleniumTest/chromedriver
Are there any solutions to solve this problem?
My steps is as below,
1.install selenium
pip install selenium -t ./
2.install chrome
curl https://intoli.com/install-google-chrome.sh | bash
3.download chromedriver
curl -SL https://chromedriver.storage.googleapis.com/2.42/chromedriver_linux64.zip > chromedriver.zip
unzip chromedriver.zip
4.Execute Python code as below
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
def lambda_handler(event, context):
options = Options()
options.add_argument('--headless')
driver = webdriver.Chrome('/home/ec2-use/environment/seleniumTest/chromedriver', chrome_options=options)
return(0)
I got "'chromedriver' executable needs to be in PATH" error.
Response
{
"errorMessage": "Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home\n",
"errorType": "WebDriverException",
"stackTrace": [
[
"/var/task/seleniumTest/lambda_function.py",
9,
"lambda_handler",
"driver = webdriver.Chrome('/home/ec2-use/environment/seleniumTest/chromedriver', chrome_options=options)"
],
[
"/var/task/selenium/webdriver/chrome/webdriver.py",
73,
"__init__",
"self.service.start()"
],
[
"/var/task/selenium/webdriver/common/service.py",
83,
"start",
"os.path.basename(self.path), self.start_error_message)"
]
]
}
Function Logs
Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home
: WebDriverException
Traceback (most recent call last):
File "/var/task/seleniumTest/lambda_function.py", line 9, in lambda_handler
driver = webdriver.Chrome('/home/ec2-use/environment/seleniumTest/chromedriver', chrome_options=options)
File "/var/task/selenium/webdriver/chrome/webdriver.py", line 73, in __init__
self.service.start()
File "/var/task/selenium/webdriver/common/service.py", line 83, in start
os.path.basename(self.path), self.start_error_message)
selenium.common.exceptions.WebDriverException: Message: 'chromedriver' executable needs to be in PATH. Please see https://sites.google.com/a/chromium.org/chromedriver/home

I'm not sure you're specifying the executable_path for Chromedriver properly, I think the below should work:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
def lambda_handler(event, context):
options = Options()
options.add_argument('--headless')
driver = webdriver.Chrome(executable_path='/home/ec2-user/environment/seleniumTest/chromedriver', chrome_options=options)
return(0)

Related

Aws Transribe unable to start_transcription_job without LanguageCode in boto3

I have an audio file in S3.
I don't know the language of the audio file. So I need to use IdentifyLanguage for start_transcription_job().
LanguageCode will be blank since I don't know the language of the audio file.
Envirionment
Using
Python 3.8 runtime,
boto3 version 1.16.5 ,
botocore version: 1.19.5,
no Lambda Layer.
Here is my code for the Transcribe job:
mediaFileUri = 's3://'+ bucket_name+'/'+prefixKey
transcribe_client = boto3.client('transcribe')
response = transcribe_client.start_transcription_job(
TranscriptionJobName="abc",
IdentifyLanguage=True,
Media={
'MediaFileUri':mediaFileUri
},
)
Then I get this error:
{
"errorMessage": "Parameter validation failed:\nMissing required parameter in input: \"LanguageCode\"\nUnknown parameter in input: \"IdentifyLanguage\", must be one of: TranscriptionJobName, LanguageCode, MediaSampleRateHertz, MediaFormat, Media, OutputBucketName, OutputEncryptionKMSKeyId, Settings, ModelSettings, JobExecutionSettings, ContentRedaction",
"errorType": "ParamValidationError",
"stackTrace": [
" File \"/var/task/app.py\", line 27, in TranscribeSoundToWordHandler\n response = response = transcribe_client.start_transcription_job(\n",
" File \"/var/runtime/botocore/client.py\", line 316, in _api_call\n return self._make_api_call(operation_name, kwargs)\n",
" File \"/var/runtime/botocore/client.py\", line 607, in _make_api_call\n request_dict = self._convert_to_request_dict(\n",
" File \"/var/runtime/botocore/client.py\", line 655, in _convert_to_request_dict\n request_dict = self._serializer.serialize_to_request(\n",
" File \"/var/runtime/botocore/validate.py\", line 297, in serialize_to_request\n raise ParamValidationError(report=report.generate_report())\n"
]
}
With this error, means that I must specify the LanguageCode and IdentifyLanguage is an invalid parameter.
100% sure the audio file exist in S3. But without LanguageCode it don't work, and IdentifyLanguage parameter is unknown parameter
I using SAM application to test locally using this command:
sam local invoke MyHandler -e lambda\TheDirectory\event.json
And I cdk deploy, and check in Aws Lambda Console as well, tested it the same events.json, but still getting the same error
This I think is Lambda Execution environment, I didn't use any Lambda Layer.
I look at this docs from Aws Transcribe:
https://docs.aws.amazon.com/transcribe/latest/dg/API_StartTranscriptionJob.html
and this docs of boto3:
https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/transcribe.html#TranscribeService.Client.start_transcription_job
Clearly state that LanguageCode is not required and IdentifyLanguage is a valid parameter.
So what I missing out? Any idea on this? What should I do?
Update:
I keep searching and asked couple person online, I think I should build the function container 1st to let SAM package the boto3 into the container.
So what I do is, cdk synth a template file:
cdk synth --no-staging > template.yaml
Then:
sam build --use-container
sam local invoke MyHandler78A95900 -e lambda\TheDirectory\event.json
But still, I get the same error, but post the stack trace as well
[ERROR] ParamValidationError: Parameter validation failed:
Missing required parameter in input: "LanguageCode"
Unknown parameter in input: "IdentifyLanguage", must be one of: TranscriptionJobName, LanguageCode, MediaSampleRateHertz, MediaFormat, Media, OutputBucketName, OutputEncryptionKMSKeyId, Settings, JobExecutionSettings, ContentRedaction
Traceback (most recent call last):
File "/var/task/app.py", line 27, in TranscribeSoundToWordHandler
response = response = transcribe_client.start_transcription_job(
File "/var/runtime/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/var/runtime/botocore/client.py", line 607, in _make_api_call
request_dict = self._convert_to_request_dict(
File "/var/runtime/botocore/client.py", line 655, in _convert_to_request_dict
request_dict = self._serializer.serialize_to_request(
File "/var/runtime/botocore/validate.py", line 297, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
Really no clue what I doing wrong here. I also report a github issue here, but seem like cant reproduce the issue.
Main Question/Problem:
Unable to start_transription_job
without LanguageCode
with IdentifyLanguage=True
What possible reason cause this, and how can I solve this problem(Dont know the languange of the audio file, I want to identify language of audio file without given the LanguageCode) ?
Check whether you are using the latest boto3 version.
boto3.__version__
'1.16.5'
I tried it and it works.
import boto3
transcribe = boto3.client('transcribe')
response = transcribe.start_transcription_job(TranscriptionJobName='Test-20201-27',IdentifyLanguage=True,Media={'MediaFileUri':'s3://BucketName/DemoData/Object.mp4'})
print(response)
{
"TranscriptionJob": {
"TranscriptionJobName": "Test-20201-27",
"TranscriptionJobStatus": "IN_PROGRESS",
"Media": {
"MediaFileUri": "s3://BucketName/DemoData/Object.mp4"
},
"StartTime": "datetime.datetime(2020, 10, 27, 15, 41, 2, 599000, tzinfo=tzlocal())",
"CreationTime": "datetime.datetime(2020, 10, 27, 15, 41, 2, 565000, tzinfo=tzlocal())",
"IdentifyLanguage": "True"
},
"ResponseMetadata": {
"RequestId": "9e4f94a4-20e4-4ca0-9c6e-e21a8934084b",
"HTTPStatusCode": 200,
"HTTPHeaders": {
"content-type": "application/x-amz-json-1.1",
"date": "Tue, 27 Oct 2020 14:41:02 GMT",
"x-amzn-requestid": "9e4f94a4-20e4-4ca0-9c6e-e21a8934084b",
"content-length": "268",
"connection": "keep-alive"
},
"RetryAttempts": 0
}
}
End up I notice this is because my packaged lambda function isn’t being uploaded for some reason. Here is how I solved it after getting help from couple of people.
First modify CDK stack which define my lambda function like this:
from aws_cdk import (
aws_lambda as lambda_,
core
)
from aws_cdk.aws_lambda_python import PythonFunction
class MyCdkStack(core.Stack):
def __init__(self, scope: core.Construct, id: str, **kwargs) -> None:
super().__init__(scope, id, **kwargs)
# define lambda
my_lambda = PythonFunction(
self, 'MyHandler',
entry='lambda/MyHandler',
index='app.py',
runtime=lambda_.Runtime.PYTHON_3_8,
handler='MyHandler',
timeout=core.Duration.seconds(10)
)
This will use aws-lambda-python module ,it will handle installing all required modules into the docker.
Next, cdk synth a template file
cdk synth --no-staging > template.yaml
At this point, it will bundling all the stuff inside entry path which define in PythonFunction and install all the necessary dependencies defined in requirements.txt inside that entry path.
Next, build the docker container
$ sam build --use-container
Make sure template.yaml file in root directory. This will build a docker container, and the artifact will build inside .aws-sam/build directory in my root directory.
Last step, invoke the function using sam:
sam local invoke MyHandler78A95900 -e path\to\event.json
Now finally successfully call start_transcription_job as stated in my question above without any error.
In Conclusion:
At the very beginning I only pip install boto3, this only will
install the boto3 in my local system.
Then, I sam local invoke without build the container 1st by sam build --use-container
Lastly, I have sam build at last, but in that point, I didn't
bundle what defined inside requirements.txt into the
.aws-sam/build, therefore need to use aws-lambda-python
module as stated above.

How to run moderngl in Colab?

I'm trying to run moderngl in Colab. I installed it and ran a virtual display:
!sudo apt-get update --fix-missing && apt-get -qqq install x11-utils > /dev/null
!sudo apt-get update --fix-missing && apt-get -qqq install xvfb > /dev/null
!python3 -m pip install -U -qqq moderngl
!python3 -m pip install -U -qqq moderngl-window
!python3 -m pip install -U -qqq pyvirtualdisplay
from pyvirtualdisplay import Display
display = Display(visible=0, size=(960, 540)).start()
import moderngl
ctx = moderngl.create_standalone_context()
buf = ctx.buffer(b'Hello World!') # allocated on the GPU
buf.read()
b'Hello World!'
It printed as expected, but when I run an example I see the error:
!python3 /content/moderngl/examples/basic_alpha_blending.py --window pyglet
2020-03-28 10:25:48,312 - moderngl_window - INFO - Attempting to load window class: moderngl_window.context.pyglet.Window
Traceback (most recent call last):
File "/content/moderngl/examples/basic_alpha_blending.py", line 74, in <module>
AlphaBlending.run()
File "/content/moderngl/examples/ported/_example.py", line 21, in run
mglw.run_window_config(cls)
File "/usr/local/lib/python3.6/dist-packages/moderngl_window/__init__.py", line 185, in run_window_config
cursor=show_cursor if show_cursor is not None else True,
File "/usr/local/lib/python3.6/dist-packages/moderngl_window/context/pyglet/window.py", line 54, in __init__
config=config,
File "/usr/local/lib/python3.6/dist-packages/pyglet/window/xlib/__init__.py", line 165, in __init__
super(XlibWindow, self).__init__(*args, **kwargs)
File "/usr/local/lib/python3.6/dist-packages/pyglet/window/__init__.py", line 588, in __init__
config = screen.get_best_config(config)
File "/usr/local/lib/python3.6/dist-packages/pyglet/canvas/base.py", line 194, in get_best_config
raise window.NoSuchConfigException()
pyglet.window.NoSuchConfigException
I also tried with another virtual display, but the result is the same:
!python3 -m pip install -U -qqq xvfbwrapper
from xvfbwrapper import Xvfb
display = Xvfb(width=960, height=540).start()
pyglet.window.NoSuchConfigException
In Google Colab you can use the EGL backend with moderngl 5.6.
ctx = moderngl.create_context(standalone=True, backend='egl')
print(ctx.info)
Output (partial):
{
'GL_VENDOR': 'NVIDIA Corporation',
'GL_RENDERER': 'Tesla P100-PCIE-16GB/PCIe/SSE2',
'GL_VERSION': '3.3.0 NVIDIA 418.67',
....
}
moderngl creates an OpenGL 3.3 core context by default. If you need a higher context version you can pass in require=430 for OpenGL 4.3 for example. I don't know what these Tesla cards support.
There is a standard example in moderngl for this. It would be able to create the standard RGB triangle: https://github.com/moderngl/moderngl/blob/master/examples/headless_egl.py
The underlying library creating the contexts is glcontext (https://github.com/moderngl/glcontext).
if you are using the moderngl-window package you have to use the headless.Window because pyglet currently is not able to work in headless mode (It might in the future: https://github.com/pyglet/pyglet/issues/51)
If you run into issue make an issue in the moderngl project: https://github.com/moderngl/moderngl or invade their discord server.

Install adwords Api for python - 'Given yaml file, %s, could not be opened.' % path)'

I try install the adwords api for python following the steps of this reference guide : https://github.com/googleads/googleads-python-lib/wiki/API-access-using-own-credentials-(installed-application-flow)#step-2---setting-up-the-client-library
Everything is okay, but in the last step(6), I have one problem.
I try run the code:
from googleads import adwords
# Initialize the AdWords client.
adwords_client = adwords.AdWordsClient.LoadFromStorage()
And Error is:
> >pythonw -u "teste_adwords_api.py" Traceback (most recent call last): File "teste_adwords_api.py", line 3, in <module>
> adwords_client = adwords.AdWordsClient.LoadFromStorage() File "C:\Users\Flávio\Google Drive\BI Caiçara\Python\googleads\adwords.py",
> line 243, in LoadFromStorage
> cls._OPTIONAL_INIT_VALUES)) File "C:\Users\Flávio\Google Drive\BI Caiçara\Python\googleads\common.py", line 128, in
> LoadFromStorage
> 'Given yaml file, %s, could not be opened.' % path) googleads.errors.GoogleAdsValueError: Given yaml file,
> C:\Users\Flávio\googleads.yaml, could not be opened.
My googleads.yaml is:
adwords:
client_id: xxxxxxx
client_secret: xxxxxx
refresh_token: xxxxxx
Where xxxx is my passwords keys
I can't understand what problem in my install process.
Had faced similar problem.
By default it searchs for the googleads.yaml file in the home directory, you can point it to your location while creating your AdWords client, e.g.,
adwords_client = AdWordsClient.LoadFromStorage("full_path_to_your_googleads.yaml")
eg:
adwords_client=AdWordsClient.LoadFromStorage(""C:\\MacUSer\\Documents\\googleads.yaml")
Hope this solves it.
does the yaml file exist in C:\Users\Flávio\ or another path? I was getting the same error before copying the yaml file to the Users\.<

difficulties with cx_freeze setup for kivy with Python 2.7

I am trying to build a package for just a simple Python 2.7 + kivy 1.9.1 script using cx_freeze. Unfortunately I get the following error when executing the DoScroll.exe file:
File "C:\Utils\Pyhton27\lib\site-packages\kivy\lang.py", line 1829, in load_file
with open(filename, 'r') as fd:
IOError: [Errno 2] No such file or directory: 'C:\Projects\Testing\build\exe.win32-2.7\library.zip\kivy\data\style.kv'
The zip file does not contain the kivy\data folder. So my idea was to add the kivy package explicitly to the setup.py file. But this does not help.
How could I resolve this issue?
The setup.py file I am using:
import sys
from cx_Freeze import setup, Executable
base = None
if sys.platform == "win32":
base = "Win32GUI"
options = {
'build_exe': {
'packages': ['kivy']
}
}
executables = [
Executable('DoScroll.py', base=base)
]
setup(name='scroller',
version='0.1',
description='demo scroller',
executables = executables,
options=options
)

Running stem with tor gives "Process terminated: Timed out"

While trying to run Stem's To Russia With Love example, I am getting the following error:
~$ python practice.py
Starting Tor:
Traceback (most recent call last):
File "practice.py", line 49, in <module>
init_msg_handler = print_bootstrap_lines,
File "/usr/local/lib/python2.7/dist-packages/stem/process.py", line 266, in launch_tor_with_config
return launch_tor(tor_cmd, args, torrc_path, completion_percent, init_msg_handler, timeout, take_ownership)
File "/usr/local/lib/python2.7/dist-packages/stem/process.py", line 143, in launch_tor
raise OSError('Process terminated: %s' % last_problem)
OSError: Process terminated: Timed out
I was initially getting the path error that was solved over here. I tried restarting the Ubuntu instance (I am running Ubuntu 14.04 in VirtualBox) just in case if any other running tor was conflicting but its giving the same error. Could anyone please help?
EDIT: My torrc file also seems to be empty right now if this is in any way connected.
It might be failing because you are missing the GeoIP database which is required to use an exit node from a specific country.
Try removing the 'ExitNodes': '{ru}', line from the python script, or since you're on Ubuntu, try sudo apt-get install tor-geoipdb and see if that helps get the connection up and running.
Since it takes time to build the circuits, you can try increasing the timeout a bit as well (though this probably isn't why its failing).
tor_process = stem.process.launch_tor_with_config(
#tor_cmd = '/usr/bin/tor',
timeout = 300,
config = {
'SocksPort': str(SOCKS_PORT),
# 'ExitNodes': '{ru}',
'DataDir': '/tmp/tor',
'Log': [
'NOTICE file /tmp/tor.notice.log',
'ERR file /tmp/tor.log',
],
},
init_msg_handler = print_bootstrap_lines,
)