My question is about setting proxy in selenium (3.4.3.) coding in python (2.7) for Firefox (Geckodriver v0.18.0-win64).
The spec at
http://www.seleniumhq.org/docs/04_webdriver_advanced.jsp
provides only a java example.
from selenium import webdriver
PROXY = "94.56.171.137:8080"
class Proxy(object):
def __call__(self):
self.base_url = "https://whatismyip.com"
print self.base_url
# proxy json object
desired_capability = webdriver.DesiredCapabilities.FIREFOX['proxy']={
"httpProxy":PROXY,
"ftpProxy":PROXY,
"sslProxy":PROXY,
#"noProxy":None,
"proxyType":"manual"
}
firefox_profile = webdriver.FirefoxProfile()
firefox_profile.set_preference("browser.privatebrowsing.autostart", True)
self.driver = webdriver.Firefox(executable_path='D:\Code\Drivers\geckodriver',firefox_profile=firefox_profile, capabilities=desired_capability)
self.driver.get(self.base_url)
if __name__ == "__main__":
proxy_test = Proxy()
proxy_test()
I am getting the following Error Message:
selenium.common.exceptions.WebDriverException: Message: Can't load the
profile. Possible firefox version mismatch. You must use GeckoDriver
instead for Firefox 48+.
If I comment the code regarding the proxy, I am able to get the page, in private mode as the profile specified. I think it is the proxy that is messing things up.
Yaso's answer didn't work for me, instead i used this
proxyString = "Ip:port"
desired_capability = webdriver.DesiredCapabilities.FIREFOX
desired_capability['proxy'] = {
"proxyType": "manual",
"httpProxy": proxyString,
"ftpProxy": proxyString,
"sslProxy": proxyString
}
I spent hours finding an answer and I want to share it.
The simple problem was in the proxy specification.
Initially the proxy and port were one string
PROXY = "94.56.171.137:8080"
the answer should make the port as a number
PROXY = "94.56.171.137"
PORT = 8080
Here is the rest of the code
from selenium import webdriver
PROXY = "94.56.171.137"
PORT = 8080
class Proxy(object):
def __call__(self):
self.base_url = "https://whatismyip.com"
print self.base_url
# https://github.com/mozilla/geckodriver
# proxy json object
desired_capability = webdriver.DesiredCapabilities.FIREFOX
desired_capability['proxy']={
"proxyType":"manual",
"httpProxy":PROXY,
"httpProxyPort": PORT,
"ftpProxy":PROXY,
"ftpProxyPort": PORT,
"sslProxy":PROXY,
"sslProxyPort" : PORT
}
firefox_profile = webdriver.FirefoxProfile()
firefox_profile.set_preference("browser.privatebrowsing.autostart", True)
self.driver = webdriver.Firefox(executable_path='D:\Drivers\geckodriver',firefox_profile=firefox_profile, capabilities=desired_capability)
self.driver.get(self.base_url)
if __name__ == "__main__":
proxy_test = Proxy()
proxy_test() code here
Related
I'd like to host multiple dashapp into a flak server. Each dashapp shall be accessible with a login and password.
Some users can access different dashapps.
I tried the dash_auth.BasicAuth. It works perfectly but only for one dashapp.
So I tried to authenticate with flask_httpauth. Here again, it works well for one dashboard, but not for 2 and more because of blueprints.
My flask_app.py:
import dash
from flask import Flask, render_template, redirect, Blueprint
import dash_bootstrap_components as dbc
from flask_httpauth import HTTPDigestAuth
from apps.dashboard import Dashboard
app = Flask(__name__)
#app.route('/')
def hello_world():
return 'Hello from Flask!'
#others routes
auth = HTTPDigestAuth()
users = {
"john": "hello",
"susan": "bye"
}
#auth.get_password
def get_pw(username):
if username in users:
return users.get(username)
return None
url1 = '/dahsboard1'
dash_app1 = dash.Dash(__name__, server = app, external_stylesheets=[dbc.themes.BOOTSTRAP])
dash_app1.config.suppress_callback_exceptions = True
dash_app1.layout = Dashboard(dash_app1, 'data1', 'Title1', url1).layout
#app.route(url1)
#app.route(url1 + '/')
#app.route('/dash1')
#auth.login_required
def render_dashboard1():
return dash_app1.index()
url2 = '/dashboard2'
dash_app2 = dash.Dash(name='app2', server = app, external_stylesheets=[dbc.themes.BOOTSTRAP])
dash_app2.config.suppress_callback_exceptions = True
dash_app2.layout = Dashboard(dash_app2, 'data2', 'Title2', url2).layout
#app.route(url2)
#app.route(url2 + '/')
#app.route('/dash2')
#auth.login_required
def render_dashboard2():
return dash_app2.index()
if __name__ == '__main__':
app.run(debug=True)
The error:
ValueError: The name '_dash_assets' is already registered for a different blueprint. Use 'name=' to provide a unique name.
I undestand that a blueprint is created at each dashapp creation. After the first call :
print(app.blueprints)
returns
{'_dash_assets': <Blueprint '_dash_assets'>}
How can I add different blueprint names for each dashapp created ? Or more generally, how can I manage authentification for several dashapps running on one flask server ?
EDTIT:
I can solve this problem using this argument at dashboard creation
url_base_pathname = '/fake-url/'
But it leads to another problem: I can't protect this route with
#app.route('/fake-url/')
#auth.login_required(role=['admin'])
def render_dashboard():
return dash_app.app.index()
So the question is: how can I protect the route used in the dash creation with the argument url_base_pathname ?
You may have already solved this by now but will leave the solution here for the community. First you will need to set url_base_pathname for example:
dash_app2 = dash.Dash(
name='app2',
server = app,
url_base_pathname='/your_url_of_choice/'
external_stylesheets=[dbc.themes.BOOTSTRAP])
This will resolve that error.
(My english is not perfect, sorry)
I'm making an app using flask and flask-restx.
for changing between production and development envs,
I designed to access env variables to change environment(like server IP and Port)
finally, I found there is 'current_app', and wrote to access variables from DB model file. but an error message comes up. and then I searched this message in official document and wrote again following solution, but error message is still there.
could you help me?
RuntimeError: Working outside of application context.
here is my code
db.py
app = current_app
host = app.config['config_data']['host']
port = app.config['config_data']['port']
username = app.config['config_data']['username']
password = app.config['config_data']['password']
client = MongoClient(host=host, port=int(port), username=username, password=password)
return client
app.py
def create_app(test_config = None):
api = Api()
app = Flask(__name__)
CORS(app)
api.init_app(app)
app.config['config_data'] = app.config.get('DB_CONFIG')
with app.app_context():
from lib.db import getDb, DBHandler
getDb()
api.add_namespace(AuthControl, '/auth')
api.add_namespace(Board, '/board')
api.add_namespace(Manager, '/manager')
api.add_namespace(Admin, '/admin')
api.add_namespace(Loopback, '/loopback')
if test_config is None:
app.config.from_envvar('APP_CONFIG_FILE')
print("PRODUCTION RELEASE")
else:
app.config.from_envvar('APP_CONFIG_FILE')
print("DEVLEOPMENT")
return app
and here is execution code
#!/bin/sh
export APP_CONFIG_FILE="./config/prod_config.py"
export FLASK_ENV=production
gunicorn -b 0.0.0.0:5050 "app:create_app()"
So I am trying to work with python 2.7 to do various things that require pulling data from the internet. I have not been very successful, and I am looking for help to diagnose what I am doing wrong.
Firstly I managed to get pip to work by by defining the proxy like so, pip install --proxy=http://username:password#someproxy.com:8080 numpy. Hence python must be capable of getting through it!
However when it came to actually writing a .py script that could do the same I have had no success. I tried using the following code with urllib2 first:
import urllib2
uri = "http://www.python.org"
http_proxy_server = "someproxyserver.com"
http_proxy_port = "8080"
http_proxy_realm = http_proxy_server
http_proxy_user = "username"
http_proxy_passwd = "password"
# Next line = "http://username:password#someproxyserver.com:8080"
http_proxy_full_auth_string = "http://%s:%s#%s:%s" % (http_proxy_user,
http_proxy_passwd,
http_proxy_server,
http_proxy_port)
def open_url_no_proxy():
urllib2.urlopen(uri)
print "Apparent success without proxy server!"
def open_url_installed_opener():
proxy_handler = urllib2.ProxyHandler({"http": http_proxy_full_auth_string})
opener = urllib2.build_opener(proxy_handler)
urllib2.install_opener(opener)
urllib2.urlopen(uri)
print "Apparent success through proxy server!"
if __name__ == "__main__":
open_url_no_proxy()
open_url_installed_opener()
However I just get this error:
URLError: <urlopen error [Errno 10060] A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond>
Then I tried urllib3 as this is the module used by pip to handle proxies:
from urllib3 import ProxyManager, make_headers
# Establish the Authentication Settings
default_headers = make_headers(basic_auth='username:password')
http = ProxyManager("https://www.proxy.com:8080/", headers=default_headers)
# Now you can use `http` as you would a normal PoolManager
r = http.request('GET', 'https://www.python.org/')
# Check data is from destination
print(r.data)
I got this error:
raise MaxRetryError(_pool, url, error or ResponseError(cause)) MaxRetryError: HTTPSConnectionPool(host='www.python.org', port=443): Max retries exceeded with url: / (Caused by ProxyError('Cannot connect to proxy.', error('Tunnel connection failed: 407 Proxy Authorization Required',)))
I would really appreciate any help diagnosing this issue.
The solution to my problem was to use the requests module, see the below thread:
Proxies with Python 'Requests' module
mtt2p list this code which worked for me.
import requests
import time
class BaseCheck():
def __init__(self, url):
self.http_proxy = "http://user:pw#proxy:8080"
self.https_proxy = "http://user:pw#proxy:8080"
self.ftp_proxy = "http://user:pw#proxy:8080"
self.proxyDict = {
"http" : self.http_proxy,
"https" : self.https_proxy,
"ftp" : self.ftp_proxy
}
self.url = url
def makearr(tsteps):
global stemps
global steps
stemps = {}
for step in tsteps:
stemps[step] = { 'start': 0, 'end': 0 }
steps = tsteps
makearr(['init','check'])
def starttime(typ = ""):
for stemp in stemps:
if typ == "":
stemps[stemp]['start'] = time.time()
else:
stemps[stemp][typ] = time.time()
starttime()
def __str__(self):
return str(self.url)
def getrequests(self):
g=requests.get(self.url,proxies=self.proxyDict)
print g.status_code
print g.content
print self.url
stemps['init']['end'] = time.time()
#print stemps['init']['end'] - stemps['init']['start']
x= stemps['init']['end'] - stemps['init']['start']
print x
test=BaseCheck(url='http://google.com')
test.getrequests()
I have a basic TCP server implemented in twisted, to which a client is connected. The client connects and sends data necessary to start a websocket resource. Using these details sent by the TCP client, I want to add an autobahn websocket resource as a child under a twisted web resource. and when the client disconnects, I want to remove this child from the twisted web resource. please suggest, what should be the best way to implement this? Can I use resource.delEntity(child)?
so far, the code looks like this:
from twisted.internet.protocol import Protocol, Factory
from autobahn.twisted.websocket import WebSocketServerFactory, WebSocketServerProtocol
from autobahn.twisted.resource import WebSocketResource
class ProtoPS(WebSocketServerProtocol):
def onMessage(self,payload,isBinary):
if not isBinary:
print("Text message received: {}".format(payload.decode('utf8')))
self.sendMessage(payload,isBinary)
class BaseResource:
def __init__(self,proto,vid):
self.vid = vid
self.factory = WebSocketServerFactory()
self.factory.protocol = proto
self.resource = WebSocketResource(self.factory)
wsroot.putChild(self.vid,self.resource)
class BackendProto(Protocol):
def __init__(self):
self.SERVICEMAP = {}
def dataReceived(self,data):
msg = json.loads(data)
if ('cmd' in msg) and (msg['cmd'] == "create"):
self.vid = msg['client']['id']
self.SERVICEMAP[self.vid] = BaseResource(ProtoPS,self.vid)
def connectionLost(self,reason):
wsroot.delEntity(self.vid)
del self.SERVICEMAP[self.vid]
class BackendFactory(Factory):
protocol = BackendProto
if __name__ == '__main__':
reactor.listenTCP(8081,BackendFactory())
wsroot = Data("","text/plain")
wssite = Site(wsroot)
reactor.listenTCP(9000,wssite)
I am running django on twisted. I have a special variable which is my engine being passed to each request. Take a loook at the following code:
# Django setup
sys.path.append("shoout_web")
os.environ['DJANGO_SETTINGS_MODULE'] = 'shoout_web.settings'
from django.core.handlers.wsgi import WSGIHandler
def wsgi_resource():
pool = threadpool.ThreadPool()
pool.start()
# Allow Ctrl-C to get you out cleanly:
reactor.addSystemEventTrigger('after', 'shutdown', pool.stop)
generic = WSGIHandler()
def wrapper(environ, start_response):
environ['engine'] = engine
return generic(environ, start_response)
wsgi_resource = wsgi.WSGIResource(reactor, pool, wrapper)
return wsgi_resource
wsgi_root = wsgi_resource()
reactor.listenTCP(DJANGO_PORT, server.Site(wsgi_root, logPath=os.path.join(log_dir, '.django.log')))
Note the line " environ['engine'] = engine "
Right now I am interested in writing test all my django views. How should I go about doing this?
Sample view function:
def push_message(request):
engine = request.META['engine']
if request.method == "POST":
user_hexid = request.session['user_hexid']
room_hexid = request.POST['room_hexid']
message_body = request.POST['message_body']
ret = blockingCallFromThread( reactor, engine.push_public_message, user_hexid, room_hexid, message_body)
return HttpResponse(cjson.encode( {'thread_hexid':ret} ))
EDIT:
Just to clear up some doubts:
I don't think I am able to put that engine within settings because the engine is actually a twisted server which is listening on a specific port
Apparently it's not documented, but from looking at the test client code, you can pass extra environ keys using the defaults keyword argument:
client = Client(defaults={'engine': engine})