Selenium Firefox webdriver for python KeyError 'value' - python-2.7

I started with the following basic python script for using selenium:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import Select, WebDriverWait
from selenium.common.exceptions import NoSuchElementException
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
path_bin='/usr/bin/firefox'
path_dr='/usr/local/bin/geckodriver'
profile = webdriver.FirefoxProfile()
binary=FirefoxBinary(path_bin)
self.driver = webdriver.Firefox(executable_path=path_dr,firefox_profile=profile,firefox_binary=binary)
self.driver.implicitly_wait(30)
Then, I tried executing it twice (first time without sudo and second time with sudo) as follows:
user4#pc-4:~/Scripts$ python test.py
Traceback (most recent call last):
File "test.py", line 2049, in <module>
self.driver = webdriver.Firefox(executable_path=path_dr,firefox_profile=profile,firefox_binary=binary)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 144, in __init__
self.service = Service(executable_path, log_path=log_path)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/service.py", line 44, in __init__
log_file = open(log_path, "a+") if log_path is not None and log_path != "" else None
IOError: [Errno 13] Permission denied: 'geckodriver.log'
user4#pc-4:~/Scripts$ sudo python test.py
Traceback (most recent call last):
File "test.py", line 2049, in <module>
self.driver = webdriver.Firefox(executable_path=path_dr,firefox_profile=profile,firefox_binary=binary)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 155, in __init__
keep_alive=True)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 92, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 183, in start_session
self.capabilities = response['value']
KeyError: 'value'
So, I have to understand the following:
As observed IOError: [Errno 13] Permission denied: 'geckodriver.log' is being observed when I run the program without a sudo. Is that really required?
Even though the browser window is opened, in spite of having resolved all the dependencies (I'm using Ubuntu 16.04) like geckodriver for Firefox, I'm getting the above error KeyError: 'value'. How can I resolve this? Do I have to make any changes in my code to avoid this?
I could use some helpful advice/tips/tweaks to get my program running.
P.S.: Digging into the web to learn on selenium, points to some architecture-specific issues as mentioned in here and here and none of them seems to have been solved and that concerns me. Would chrome be a better option?
Update: The following is the output of ls -alh:
-rw-r--r-- 1 root root 238K Aug 22 17:12 geckodriver.log
-rwxrwxrwx 1 user4 user 82K Aug 22 17:08 test.py

As you were trying to pass a couple of arguments when invoking webdriver.Firefox(), we need to consider that executable_path and firefox_binary both these arguments takes string type of arguments. So you need to pass the absolute path of geckodriver binary to executable_path and the absolute path of firefox binary to firefox_binary. Further you don't need to use binary=FirefoxBinary(path_bin). The following code block works fine on my Windows 8 machine:
Code is based on Python 3.6.1
from selenium import webdriver
path_bin=r'C:\Program Files\Mozilla Firefox\firefox.exe'
path_dr=r'C:\Utility\BrowserDrivers\geckodriver.exe'
profile = webdriver.FirefoxProfile()
driver = webdriver.Firefox(executable_path=path_dr,firefox_profile=profile,firefox_binary=path_bin)
driver.get('https://www.google.co.in')
As you are on ubuntu the following code block must work for you:
from selenium import webdriver
path_bin='/usr/bin/firefox'
path_dr='/usr/local/bin/geckodriver'
profile = webdriver.FirefoxProfile()
driver = webdriver.Firefox(executable_path=path_dr,firefox_profile=profile,firefox_binary=path_bin)
driver.get('https://www.google.co.in')

I had this same issue with
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 183, in start_session
self.capabilities = response['value']
KeyError: 'value'
So I have updated selenium and geckodriver to the latest version. This was the solution in my case.

Related

Django + heroku - import error on line that doesn't exist

I'm new to django (using python 2.7) and I was just trying to use heroku for the first time but I always get the following error:
remote: -----> Preparing static assets
remote: Collectstatic configuration error. To debug, run:
remote: $ heroku run python manage.py collectstatic --noinput
When I run that command, an import error regarding the django registration redux library shows up. I've had this problem before in Django and I fixed it by placing RequestSite under 'requests' and Site under 'models'. That solved the problem but the error still shows up in Heroku.
(venv) C:\Users\Carolina\Desktop\Coding\venv\project1>heroku run python manage.p
y collectstatic --noinput
Running python manage.py collectstatic --noinput on acla-acla... up, run.3645
Traceback (most recent call last):
File "manage.py", line 10, in <module>
execute_from_command_line(sys.argv)
File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/_
_init__.py", line 350, in execute_from_command_line
utility.execute()
File "/app/.heroku/python/lib/python2.7/site-packages/django/core/management/_
_init__.py", line 324, in execute
django.setup()
File "/app/.heroku/python/lib/python2.7/site-packages/django/__init__.py", lin
e 18, in setup
apps.populate(settings.INSTALLED_APPS)
File "/app/.heroku/python/lib/python2.7/site-packages/django/apps/registry.py"
, line 115, in populate
app_config.ready()
File "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/admin/app
s.py", line 22, in ready
self.module.autodiscover()
File "/app/.heroku/python/lib/python2.7/site-packages/django/contrib/admin/__i
nit__.py", line 26, in autodiscover
autodiscover_modules('admin', register_to=site)
File "/app/.heroku/python/lib/python2.7/site-packages/django/utils/module_load
ing.py", line 50, in autodiscover_modules
import_module('%s.%s' % (app_config.name, module_to_search))
File "/app/.heroku/python/lib/python2.7/importlib/__init__.py", line 37, in im
port_module
__import__(name)
File "/app/.heroku/python/lib/python2.7/site-packages/registration/admin.py",
line 2, in <module>
from django.contrib.sites.models import RequestSite
ImportError: cannot import name RequestSite
The thing is - that line doesn't exist. I went to venv/lib/site-packages/registration/admin.py and line 2 is this one:
from django.contrib import admin
from django.contrib.sites.requests import RequestSite # HERE
from django.contrib.sites.models import Site
from django.utils.translation import ugettext_lazy as _
from registration.models import RegistrationProfile
from registration.users import UsernameField
class RegistrationAdmin(admin.ModelAdmin):
actions = ['activate_users', 'resend_activation_email']
list_display = ('user', 'activation_key_expired')
raw_id_fields = ['user']
search_fields = ('user__{0}'.format(UsernameField()), 'user__first_name', 'user__last_name')
def activate_users(self, request, queryset):
"""
Activates the selected users, if they are not already
activated.
"""
for profile in queryset:
RegistrationProfile.objects.activate_user(profile.activation_key)
activate_users.short_description = _("Activate users")
def resend_activation_email(self, request, queryset):
"""
Re-sends activation emails for the selected users.
Note that this will *only* send activation emails for users
who are eligible to activate; emails will not be sent to users
whose activation keys have expired or who have already
activated.
"""
if Site._meta.installed:
site = Site.objects.get_current()
else:
site = RequestSite(request)
for profile in queryset:
if not profile.activation_key_expired():
profile.send_activation_email(site)
resend_activation_email.short_description = _("Re-send activation emails")
admin.site.register(RegistrationProfile, RegistrationAdmin)
This is what I get with pip freeze, just in case:
Django==1.9
django-crispy-forms==1.5.2
django-registration==2.0.3
django-registration-redux==1.2
django-tinymce==2.2.0
Pillow==3.0.0
requests==2.9.0
South==1.0.2
stripe==1.27.1
wheel==0.24.0
Anyone knows why this is happening? Thanks in advance!
EDIT ----
Ok, so the problem was the one mentioned by Daniel Roseman. The library is broken in pypi and I had to tell heroku to install it from github (where the package is fixed).
So, I went to my requirements.txt file and replaced this line:
django-registration-redux==1.2
with this one:
-e git://github.com/macropin/django-registration.git#egg=django-registration==1.2
(I also removed 'django-registration==2.0.3' because it is an old version of django-registration-redux and was creating problems).
Hope this helps people with the same issue!
It sounds like you edited the code for your locally installed copy of django-registration-redux. But that won't have any effect on Heroku, since the library will be installed directly from PyPI, according to the version in your requirements.txt.
If the library is really broken, you will need to fork it and point your requirements.txt to your fixed version. However, looking at the code on GitHub, it doesn't actually seem to be broken; you just need to update the version you are pointing to.

Django on Pycharm: ImproperlyConfigured with DJANGO_SETTINGS_MODULE

I am trying to use Pycharm Community Edition to improve on my code in my Django application, but I cannot run all of my Django code that I'd like. I keep getting this traceback...
Traceback (most recent call last):
File "C:/Users/Jaysp_000/firstSite/PROJECTone/blog_static/views.py", line 1, in <module>
from django.views.decorators.csrf import csrf_exempt
File "C:\Python34\lib\site-packages\django\views\decorators\csrf.py", line 3, in <module>
from django.middleware.csrf import CsrfViewMiddleware, get_token
File "C:\Python34\lib\site-packages\django\middleware\csrf.py", line 14, in <module>
from django.utils.cache import patch_vary_headers
File "C:\Python34\lib\site-packages\django\utils\cache.py", line 26, in <module>
from django.core.cache import caches
File "C:\Python34\lib\site-packages\django\core\cache\__init__.py", line 34, in <module>
if DEFAULT_CACHE_ALIAS not in settings.CACHES:
File "C:\Python34\lib\site-packages\django\conf\__init__.py", line 48, in __getattr__
self._setup(name)
File "C:\Python34\lib\site-packages\django\conf\__init__.py", line 42, in _setup
% (desc, ENVIRONMENT_VARIABLE))
django.core.exceptions.ImproperlyConfigured: Requested setting CACHES, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
This error seems to involve the django.views.decortors.csrf.csrf_exempt that I imported to my views.py file. I've tried other files, and they have given me no issues. There is something in particular about this import, but I don't know what.
from django.views.decortors.csrf import csrf_exempt
#csrf_exempt
def handle_hook(request):
from django.http import HttpResponse
from django.core.management import call_command
result = call_command('update_blog', verbosity = 0)
return HttpResponse(result)
The same kind of issue shows up when I am trying to run the code on the python shell (I use 3.4) and when I import django.http.request as request. I type in handle_hook(request), and I get the same kind of error.
Im being told that I must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings, but I haven't a clue on how to do that. I've looked around and I am not certain if those methods specifically speak to my issue. Any clues?
Go to the Run Menu, select Edit Configurations..., then select the run configuration for you tests.
Select the environment variables button. You'll see one existing variable, which is PYTHONUNBUFFERED
Under that add (for example) DJANGO_SETTINGS_MODULE=mysitename.settings

How do you confirm django is using memcached?

I have a python django webserver that I am trying to use memcached to make it faster.
I have downloaded and installed memcached and started it a user called virtual as follows:
/usr/local/bin/memcached -u virtual &
on the django setting.py, I have put the memcached server as this:
MEMCACHE_HOSTS = ['192.168.101.1:11211']
I can do telnet 192.168.101.1 11211 and stats, I do see some statistics there etc.
How do I really know if my django server utilizing the memcached? Is there directory that I can look at or some files to confirm?
Thank you for any insight.
content of manage.py:
#!/usr/bin/env python
import os
import sys
from django.core.management import execute_from_command_line
if __name__ == "__main__":
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "settings")
from django.core.management import execute_from_command_line
execute_from_command_line(sys.argv)
from the command line:
phthon
from django.core.management import execute_from_command_line
and when I do this:
from django.core.cache import cache
I get these errors:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/local/lib64/python2.6/site-packages/django/core/cache/__init__.py", line 69, in <module>
if DEFAULT_CACHE_ALIAS not in settings.CACHES:
File "/usr/local/lib64/python2.6/site-packages/django/conf/__init__.py", line 54, in __getattr__
self._setup(name)
File "/usr/local/lib64/python2.6/site-packages/django/conf/__init__.py", line 47, in _setup
% (desc, ENVIRONMENT_VARIABLE))
django.core.exceptions.ImproperlyConfigured: Requested setting CACHES, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.
You can test cache in Django's shell (python manage.py shell):
>>> from django.core.cache import cache
>>> cache.set('foo', 'bar', 600)
>>> cache.get('foo')
'bar'
If cache.get() returns the set value it means that cache is working as it should. Otherwise it will return None.
An other option is to start memcached with $ memcached -vv, since it will log all the cache accesses to the terminal. Or alternatively you can install monitoring tool for memcached (i.e. memcache-top) and check that something is happening in memcached while using your app.

error with Firefox Binary 'The browser appears to have exited'

I have problem with selenium and Firefox when i run my code below:
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from bs4 import BeautifulSoup
import urllib2,cookielib
import re
from pyvirtualdisplay import Display
import urllib
import urllib2
import httplib2
from os import system
display = Display(visible=0, size=(800, 600))
display.start()
name = 'test'
driver = webdriver.Firefox()
driver.get("http://example.com")
driver.switch_to_default_content()
inputElement = driver.find_element_by_name("name")
inputElement.send_keys(name)
inputElement.submit()
print "Please wait for 5s to get results.."
content = driver.page_source.encode("utf-8")
html = str(content)
soup = BeautifulSoup(html)
print soup
display.stop()
driver.close()
And i get this error:
Exception in thread command9:
Traceback (most recent call last):
File "/usr/local/lib/python2.7/threading.py", line 551, in __bootstrap_inner
self.run()
File "/usr/local/lib/python2.7/threading.py", line 504, in run
self.__target(*self.__args, **self.__kwargs)
File "plugins/help_plugin.py", line 165, in handler_search
search_number_ar(type, source, parameters)
File "plugins/test.py", line 230, in search_test
driver = webdriver.Firefox()
File "/usr/local/lib/python2.7/site-packages/selenium-2.45.0-py2.7.egg/selenium/webdriver/firefox/webdriver.py", line 59, in __init__
self.binary, timeout),
File "/usr/local/lib/python2.7/site-packages/selenium-2.45.0-py2.7.egg/selenium/webdriver/firefox/extension_connection.py", line 47, in __init__
self.binary.launch_browser(self.profile)
File "/usr/local/lib/python2.7/site-packages/selenium-2.45.0-py2.7.egg/selenium/webdriver/firefox/firefox_binary.py", line 66, in launch_browser
self._wait_until_connectable()
File "/usr/local/lib/python2.7/site-packages/selenium-2.45.0-py2.7.egg/selenium/webdriver/firefox/firefox_binary.py", line 100, in _wait_until_connectable
raise WebDriverException("The browser appears to have exited "
WebDriverException: Message: The browser appears to have exited before we could connect. If you specified a log_file in the FirefoxBinary constructor, check it for details.
my code running in vps and i use selenium-2.45.0 and firefox.x86_64-31.6.0.
I try alot of ways to solve my problem but nothing working with me.
I try to update selenium and i try to downgrade firefox and alot of solution.
Just to know when i run my code for the first time it work but when i run it again it show me the above error.
I hope someone help.

Django / mod_wsgi / postgresql_psycopg2: can't connect authentication fails (but settings work fine under django runserver or dbshell)

I'm deploying my django site on a Ubuntu 8.10 server that I've set up from scratch. I'm totally new to this so there may be plenty of stuff that I've missed, but happily have found my way through to the point of deploying a website.
My last hurdle:
I'm getting an authentication error with postgresql
[Sun Apr 19 18:44:05 2009] [error] [client 124.254.102.127]
mod_wsgi (pid=30304): Exception occurred processing WSGI script '/home/acacian/webapps/acacian/deploy/acacian.wsgi'.
Traceback (most recent call last):
File "/usr/lib/python2.5/site-packages/Django-1.0.2_final-py2.5.egg/django/core/handlers/wsgi.py", line 241, in __call__
response = self.get_response(request)
File "/usr/lib/python2.5/site-packages/Django-1.0.2_final-py2.5.egg/django/core/handlers/base.py", line 73, in get_response
response = middleware_method(request)
File "/home/acacian/webapps/pinax-env/src/django-openid/django_openid/consumer.py", line 383, in process_request
if self.session_key in request.session:
File "/usr/lib/python2.5/site-packages/Django-1.0.2_final-py2.5.egg/django/contrib/sessions/backends/base.py", line 43, in __contains__
return key in self._session
File "/usr/lib/python2.5/site-packages/Django-1.0.2_final-py2.5.egg/django/contrib/sessions/backends/base.py", line 172, in _get_session
self._session_cache = self.load()
File "/usr/lib/python2.5/site-packages/Django-1.0.2_final-py2.5.egg/django/contrib/sessions/backends/db.py", line 16, in load
expire_date__gt=datetime.datetime.now()
File "/usr/lib/python2.5/site-packages/Django-1.0.2_final-py2.5.egg/django/db/models/manager.py", line 120, in get
return self.get_query_set().get(*args, **kwargs)
File "/usr/lib/python2.5/site-packages/Django-1.0.2_final-py2.5.egg/django/db/models/query.py", line 269, in get
num = len(clone)
File "/usr/lib/python2.5/site-packages/Django-1.0.2_final-py2.5.egg/django/db/models/query.py", line 68, in __len__
self._result_cache = list(self.iterator())
File "/usr/lib/python2.5/site-packages/Django-1.0.2_final-py2.5.egg/django/db/models/query.py", line 207, in iterator
for row in self.query.results_iter():
File "/usr/lib/python2.5/site-packages/Django-1.0.2_final-py2.5.egg/django/db/models/sql/query.py", line 262, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/lib/python2.5/site-packages/Django-1.0.2_final-py2.5.egg/django/db/models/sql/query.py", line 2288, in execute_sql
cursor = self.connection.cursor()
File "/usr/lib/python2.5/site-packages/Django-1.0.2_final-py2.5.egg/django/db/backends/__init__.py", line 81, in cursor
cursor = self._cursor()
File "/usr/lib/python2.5/site-packages/Django-1.0.2_final-py2.5.egg/django/db/backends/postgresql_psycopg2/base.py", line 98, in _cursor
self.connection = Database.connect(**conn_params)
OperationalError: FATAL: Ident authentication failed for user "acacian"
Note:
the same settings work fine under runserver and dbshell
mod_wsgi has been working with sqlite3 (i'm at the point of switching over)
acacian.wsgi (a little messy)
# basic_project.wsgi is configured to live in projects/basic_project/deploy.
import os
import sys
import site
sys.stdout = sys.stderr
from os.path import abspath, dirname, join
from site import addsitedir
# the project path & pinax src paths
sys.path.insert(0, abspath(join(dirname(__file__), "../../")))
sys.path.insert(0, abspath(join(dirname(__file__), "../../pinax-env/src/pinax")))
sys.path.insert(0, abspath(join(dirname(__file__), "../../pinax-env/src")))
sys.path = ['/home/acacian/webapps/acacian/apps', '/home/acacian/webapps/pinax-env/src/pinax/pinax/apps', '/home/acacian/webapps/acacian', '/home/acacian/webapps/pinax-env/lib/python2.5/site-packages/setuptools-0.6c9-py2.5.egg', '/home/acacian/webapps/pinax-env/lib/python2.5/site-packages/pip-0.3.1-py2.5.egg', '/home/acacian/webapps/pinax-env/src/pinax', '/home/acacian/webapps/pinax-env/src/django-openid', '/home/acacian/webapps/pinax-env/src/atom-format', '/home/acacian/webapps/pinax-env/src/django-app-plugins', '/home/acacian/webapps/pinax-env/src/diff-match-patch', '/usr/lib/python2.5/site-packages/Django-1.0.2_final-py2.5.egg', '/usr/lib/python2.5/site-packages/pip-0.3.1-py2.5.egg', '/home/acacian/webapps/pinax-env/lib/python2.5', '/home/acacian/webapps/pinax-env/lib/python2.5/plat-linux2', '/home/acacian/webapps/pinax-env/lib/python2.5/lib-tk', '/home/acacian/webapps/pinax-env/lib/python2.5/lib-dynload', '/usr/lib/python2.5', '/usr/lib/python2.5/plat-linux2', '/usr/lib/python2.5/lib-tk', '/home/acacian/webapps/pinax-env/lib/python2.5/site-packages', '/usr/local/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages', '/usr/lib/python2.5/site-packages/PIL', '/var/lib/python-support/python2.5'] + sys.path
# A version of activate_this from the mod_wsgi documentation site.
ALLDIRS = ['/home/acacian/webapps/acacian/pinax-env/lib/python2.5/site-packages']
# Remember original sys.path.
prev_sys_path = list(sys.path)
# Add each new site-packages directory.
for directory in ALLDIRS:
site.addsitedir(directory)
# Reorder sys.path so new directories at the front.
new_sys_path = []
for item in list(sys.path):
if item not in prev_sys_path:
new_sys_path.append(item)
sys.path.remove(item)
sys.path[:0] = new_sys_path
#print sys.path
from django.conf import settings
os.environ["DJANGO_SETTINGS_MODULE"] = "acacian.settings"
sys.path.insert(0, join(settings.PINAX_ROOT, "apps"))
sys.path.insert(0, join(settings.PROJECT_ROOT, "apps"))
from django.core.handlers.wsgi import WSGIHandler
application = WSGIHandler()
Thank you!
The error:
Ident authentication failed for user "acacian"
means just that.
When you are using runserver or dbshell the Django process is running with your userid as process owner.
When you run Django under mod_wsgi, in daemon mode, it is running as the user specified in the WSGIDaemonProcess directive. If you are running mod_wsgi in embedded (non-daemon) mode then the userid will match the Apache process User directive.
So, what you need to do is add your user acacian as a user to postgresql using createuser just like you did with your own user name.
It could be your postgres connection settings. I think postgres authenticates by default using the unix userid, but in this case you might want to use standard username/password authentication (since that's what you've hardcoded into the django settings). So you might try:
sudo vi /etc/postgresql/8.4/main/pg_hba.conf
(or emacs or nano etc). Then change the following line:
local all all ident
to:
local all all password
You might find this blogpost helpful http://www.depesz.com/index.php/2007/10/04/ident/ - it tries to explain the "Ident authentication failed" error, it's source, and what can you do about it.
My guess is that your issue is in the database connection settings (settings.py). I am assuming you have already created the database and syncdb'd etc?
This article is awesome for setting up Django/Postgres/etc on Ubuntu 8.10 (I have followed it to set up a couple of VMs already):
http://lethain.com/entry/2009/feb/13/the-django-and-ubuntu-intrepid-almanac/