My Django application uses elasticsearch to index several ressources.
Now I wanted to protect my elasticsearch instance with as password which is working fine if I use "curl -u" or so. Anyways from the elasticsearch_dsl documentation, found here: https://elasticsearch-dsl.readthedocs.io/en/latest/configuration.html, I do not understand what I have to do in order to setup elasticsearch that way that it uses a password for authentication and where do I have to place this code?! is smb. maybe able to pass show me some snippets of his configuration?
My current state looks like this:
settingy.py
ELASTICSEARCH_DSL = {
'default': {
'hosts': env.str('ELASTICSEARCH_HOST') + str(':') + env.str('ELASTICSEARCH_PORT'),
},
}
ELASTICSEARCH_DSL_SIGNAL_PROCESSOR = 'django_elasticsearch_dsl.signals.RealTimeSignalProcessor'
documents.py
from django_elasticsearch_dsl import Document, Index, fields
from elasticsearch_dsl import analyzer
from App.models import Post
# elasticsearch index
posts = Index('posts')
html_strip = analyzer(
'html_strip',
tokenizer="standard",
filter=["lowercase", "stop", "snowball"],
char_filter=["html_strip"]
)
#posts.document
class PostDocument(Document):
... more index stuff
According to the docs, I have to manually setup a default client connection where I can also pass the password and username for authentication which to me seems not to be possible at settings.py at moment.
Kind regards
You can pass the elasticsearch URL as
from urllib.parse import quote_plus as urlquote
elk_base_url = 'elasticsearch://{user_name}:{password}#{host_ip}:{host_port}'
elastic_search_url = elk_base_url.format(user_name='my_username',
password=urlquote('mysecret_password'),
# password may contain special characters
host_ip='my-elastic-host-ip',
host_port=9200)
ELASTICSEARCH_DSL = {
'default': {
'hosts': [elastic_search_url]
},
}
This solution has been tested under the following circumstances
Django==3.0.4
django-elasticsearch-dsl==7.1.1
logstash == kibana == elasticsearch == 7.6.0
If you are experiencing AuthenticationException(401, '') exception, it means you were provide the wrong credentials. Please do check the value of elastic_search_url and make sure the values are correct.
Related
I would like to redirect my users to specific location areas in my website, by detecting their location from their IP address.
What would be the best way to achieve this under Django 1.1.1 ?
Thanks
Edit: I want city based locationing on europe.
GeoDjango looks like it will suit your needs. I'm not sure exactly how you would want to direct users, but using the GeoIP API, you can do something like:
from django.contrib.gis.utils import GeoIP
g = GeoIP()
ip = request.META.get('REMOTE_ADDR', None)
if ip:
city = g.city(ip)['city']
else:
city = 'Rome' # default city
# proceed with city
The Docs explain things in great detail; I would take a moment to read through them thoroughly.
GeoIP is already mentioned, but I find pygeoip less troublesome to install and no-brainer if you want to embed it in you application instead of installing in Python's site-packages. Still, it works great with free MaxMind databases, e.g GeoLite City one.
Example of use (almost the same as for GeoIP):
>>> import pygeoip
>>> gi = pygeoip.GeoIP(GEOIP_DATABASE, pygeoip.GEOIP_STANDARD)
>>> gi.record_by_addr(ip)
{'country': '...', 'country_code': '...', ...}
This is one solution, from DjangoSnippets; btw, not sure why the code below doesn't use urlparse; but that could be fixed :-)
(Looking at the other answers, it seems you have plenty of options to choose from. This option may not be preferred because it relies on a free 3rd party service.)
from urllib2 import urlopen, Request
import re, socket
from django.conf import settings
domain_re = re.compile('^(http|https):\/\/?([^\/]+)')
domain = domain_re.match(settings.SITE_URL).group(2)
def getUserCountry(ip):
url = "http://api.wipmania.com/" + ip + "?" + domain
socket.setdefaulttimeout(5)
headers = {'Typ':'django','Ver':'1.1.1','Connection':'Close'}
try:
req = Request(url, None, headers)
urlfile = urlopen(req)
land = urlfile.read()
urlfile.close()
return land[:2]
except Exception:
return "XX"
Note from WIPmania: "Using API is free for any purpose, personal or business, if you are making fewer than 10.000 requests per calendar day. A simple yet powerful API allowing you to query the WorldIP database with a single link."
You could create a view which gets the user's IP and then issues an HTTP redirect which will cause their browser to load the page you want:
def redirect_based_on_ip(request):
ip = request.meta['REMOTE_ADDR']
if ip == SOMETHING:
return HttpResponseRedirect('/something')
elif ip == SOMETHING_ELSE:
return HttpResponseRedirect('/something_else')
# ...
You might find the SubnetTree library for Python helpful if you want to test to see if an IP is in a particular block.
Based on some free services.
it's not fast, but you can add more free services:
settings:
IPCOUNTRY_APYKEY = [
{# free tier 2 querys per second
"url": "http://api.ipinfodb.com/v3/ip-country/?ip={ip}&key={key}&format=json",
"params": {
"key": "*****************************",
},
"fieldname": "countryCode",
},
{# free tier 150 querys per minute and https is not suported in free tier
"url": "http://ip-api.com/json/{ip}?fields=2",
"params": {},
"fieldname": "countryCode",
},
{# free tier 1.500 queries per day
"url": "https://api.ipdata.co/{ip}?api-key={key}",
"params": {
"key": "*****************************",
},
"fieldname": "country_code",
},
{# free tier 10.000 queries per month and https is not suported in free tier
"url": "http://api.ipstack.com/{ip}?access_key={key}",
"params": {
"key": "*****************************",
},
"fieldname": "country_code",
},
{# free tier 10.000 queries per month and https is not suported in free tier
"url": "http://api.ipapi.com/{ip}?access_key={key}",
"params": {
"key": "*****************************",
},
"fieldname": "country_code",
},
]
CODE:
import json
import urllib3
from django.conf import settings
for service in settings.IPCOUNTRY_APYKEY:
url = service["url"].format(ip=ip,**service["params"])
headers = {'Type': 'django', 'Ver': '1.1.1', 'Connection': 'Close'}
urllib3.disable_warnings()
http_call = urllib3.PoolManager()
try:
r = http_call.request('GET', url, headers=headers, timeout=1.0)
if r.status == 200:
json_response = json.loads(r.data.decode("utf-8"))
print(json_response[service["fieldname"]])
except Exception as e:
pass
return None
I have made SO answer where I am using Cloudflare CDN, they provide extra header with GEO location of each visitor. Advantage is that we don't have to install any external library or make any API call. Redirection can be achieved by Django middleware.
You can achieve this by making use of IP2Location Python library and IP2Location BIN database. First, purchase a commercial IP2Location database from https://www.ip2location.com/database/ip2location or download a free IP2Location LITE database from https://lite.ip2location.com/. And also install the IP2Location Python library by using this command: pip install IP2Location
After that, open your views.py, and add the following code into the file:
from django.http import HttpResponse
from django.shortcuts import redirect
import IP2Location
database = IP2Location.IP2Location(YOUR_DATABASE_PATH)
def get_client_ip(request):
x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR')
if x_forwarded_for:
ip = x_forwarded_for.split(',')[0]
else:
ip = request.META.get('REMOTE_ADDR')
if ip == '127.0.0.1': # Only define the IP if you are testing on localhost.
ip = '8.8.8.8'
return ip
def redirect_view(request):
ip = get_client_ip(request)
rec = database.get_all(ip)
if rec.city == 'London': # Or any other city name
response = redirect('PATH_REDIRECT_TO')
return response
def redirectsucess_view(request):
return HttpResponse('Welcome!')
After that, open the urls.py, and add the following codes into the urlpatterns list:
path('redirect/', views.redirect_view),
path('PATH_REDIRECT_TO/', views.redirectsucess_view)
I am trying to use a local (mac) python program to append a row to a google sheet. I naively thought the snippet below would be sufficient:
import requests
url = "https://sheets.googleapis.com/v4/spreadsheets/SHEETID/values/Expenses!A1:D1:append?valueInputOption=USER_ENTERED"
data = {
"range": "Expenses!A1:D1",
"majorDimension": "ROWS",
"values": [
[NEW ROW DATA]]
],
}
resp = requests.post(url, data)
I am getting the error:
401: "Request is missing required authentication credential.
Expected OAuth 2 access token, login cookie or other valid
authentication credential.
I am not to sure how to set-up the authentication for the google sheets rest api.
Can anyone provide an example of how to go about this.
You can try the sample python code in the documentation.
"""
BEFORE RUNNING:
---------------
1. If not already done, enable the Google Sheets API
and check the quota for your project at
https://console.developers.google.com/apis/api/sheets
2. Install the Python client library for Google APIs by running
`pip install --upgrade google-api-python-client`
"""
from pprint import pprint
from googleapiclient import discovery
# TODO: Change placeholder below to generate authentication credentials. See
# https://developers.google.com/sheets/quickstart/python#step_3_set_up_the_sample
#
# Authorize using one of the following scopes:
# 'https://www.googleapis.com/auth/drive'
# 'https://www.googleapis.com/auth/drive.file'
# 'https://www.googleapis.com/auth/spreadsheets'
credentials = None
service = discovery.build('sheets', 'v4', credentials=credentials)
# The ID of the spreadsheet to update.
spreadsheet_id = 'my-spreadsheet-id' # TODO: Update placeholder value.
# The A1 notation of a range to search for a logical table of data.
# Values will be appended after the last row of the table.
range_ = 'my-range' # TODO: Update placeholder value.
# How the input data should be interpreted.
value_input_option = '' # TODO: Update placeholder value.
# How the input data should be inserted.
insert_data_option = '' # TODO: Update placeholder value.
value_range_body = {
# TODO: Add desired entries to the request body.
}
request = service.spreadsheets().values().append(spreadsheetId=spreadsheet_id, range=range_, valueInputOption=value_input_option, insertDataOption=insert_data_option, body=value_range_body)
response = request.execute()
# TODO: Change code below to process the `response` dict:
pprint(response)
and also
Since you are working with the app that access data from the other user open this guide for Authorize Requests to know more about authentication credentials.
I'm trying to make a simple Python Lambda that makes snapshots of our Elasticsearch database. This is done through Elasticsearch's REST API using simple HTTP requests.
However, for AWS, I have to sign these requests. I have a feeling it can be achieved through boto3's low-level clients probably with generate_presigned_url, but I cannot for the life of me figure out how to invoke this function correctly. For example, what are the valid ClientMethods? I've tried ESHttpGet but to no avail.
Can anyone point me in the right direction?
Edit: Apparently this workaround has been broken by Elastic.
I struggled for a while to do a similar thing. Currently the boto3 library doesn't support making signed es requests, though since I raised an issue with them it's become a feature request.
Here's what I've done in the meantime using DavidMuller's library mentioned above and boto3 to get my STS session credentials:
import boto3
from aws_requests_auth.aws_auth import AWSRequestsAuth
from elasticsearch import Elasticsearch, RequestsHttpConnection
session = boto3.session.Session()
credentials = session.get_credentials().get_frozen_credentials()
es_host = 'search-my-es-domain.eu-west-1.es.amazonaws.com'
awsauth = AWSRequestsAuth(
aws_access_key=credentials.access_key,
aws_secret_access_key=credentials.secret_key,
aws_token=credentials.token,
aws_host=es_host,
aws_region=session.region_name,
aws_service='es'
)
# use the requests connection_class and pass in our custom auth class
es = Elasticsearch(
hosts=[{'host': es_host, 'port': 443}],
http_auth=awsauth,
use_ssl=True,
verify_certs=True,
connection_class=RequestsHttpConnection
)
print(es.info())
Hope this saves somebody some time.
There are several Python extensions to the requests library that will perform the SigV4 signing for you. I have used this one and it works well.
While other answers are perfectly fine, I wanted to eliminate the use of external packages. Obviously, botocore itself has all the required functionality to sign requests it was just a matter of looking at the source code. This is what I ended up with for sending AWS API requests directly (things are hardcoded for the demonstration purposes):
import boto3
import botocore.credentials
from botocore.awsrequest import AWSRequest
from botocore.endpoint import URLLib3Session
from botocore.auth import SigV4Auth
params = '{"name": "hello"}'
headers = {
'Host': 'ram.ap-southeast-2.amazonaws.com',
}
request = AWSRequest(method="POST", url="https://ram.ap-southeast-2.amazonaws.com/createresourceshare", data=params, headers=headers)
SigV4Auth(boto3.Session().get_credentials(), "ram", "ap-southeast-2").add_auth(request)
session = URLLib3Session()
r = session.send(request.prepare())
I recently published requests-aws-sign, which provides AWS V4 request signing for the Python requests library.
If you look at this code you will see how you can use Botocore to generate the V4 request signing.
why not just use requests?
import requests
headers = {'Content-Type': 'application/json',}
data = '{"director": "Burton, Tim", "genre": ["Comedy","Sci-Fi","R-rated"],"profit" : 98 , "year": 1996, "actor": ["Jack Nicholson","PierceBrosnan","Sarah Jessica Parker"], "title": "Mars Attacks!"}'
response = requests.post('https://search-your-awsendpoint.us-west-2.es.amazonaws.com/yourindex/_yourdoc/', headers=headers, data=data)
this worked for me
I am trying to scrape using proxy from proxymesh.com
I am currently using the following code,
It opens Chrome and creates a javascript alert to input username and password.
I am currently doing is manually everytime I run a new instance of the script.
If someone could please help in automating it.
There could be 2 ways to do it,
Either somehow pass the username and password through Chrome Options
OR
Somehow make webdriver switch to javascript alert and enter the username and password there.
Here is my code so far,
from selenium import webdriver
chrome_option = webdriver.ChromeOptions()
chrome_option.add_argument("--proxy-server=http://us.proxymesh.com:31280")
b = webdriver.Chrome('chromedriver.exe',
chrome_options=chrome_option)
"Do Something"
Thanks in advance
This is what has worked for me using Chrome and proxy authentication. I don't have a proxymesh account, so you'll have to verify if it works or not yourself:
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
proxy = {'address': 'us.proxymesh.com:31280',
'usernmae': 'johnsmith123',
'password': 'iliketurtles'}
capabilities = dict(DesiredCapabilities.CHROME)
capabilities['proxy'] = {'proxyType': 'MANUAL',
'httpProxy': proxy['address'],
'ftpProxy': proxy['address'],
'sslProxy': proxy['address'],
'noProxy': '',
'class': "org.openqa.selenium.Proxy",
'autodetect': False}
capabilities['proxy']['socksUsername'] = proxy['username']
capabilities['proxy']['socksPassword'] = proxy['password']
driver = webdriver.Chrome(executable_path=[path to your chromedriver], desired_capabilities=capabilities)
I would like to redirect my users to specific location areas in my website, by detecting their location from their IP address.
What would be the best way to achieve this under Django 1.1.1 ?
Thanks
Edit: I want city based locationing on europe.
GeoDjango looks like it will suit your needs. I'm not sure exactly how you would want to direct users, but using the GeoIP API, you can do something like:
from django.contrib.gis.utils import GeoIP
g = GeoIP()
ip = request.META.get('REMOTE_ADDR', None)
if ip:
city = g.city(ip)['city']
else:
city = 'Rome' # default city
# proceed with city
The Docs explain things in great detail; I would take a moment to read through them thoroughly.
GeoIP is already mentioned, but I find pygeoip less troublesome to install and no-brainer if you want to embed it in you application instead of installing in Python's site-packages. Still, it works great with free MaxMind databases, e.g GeoLite City one.
Example of use (almost the same as for GeoIP):
>>> import pygeoip
>>> gi = pygeoip.GeoIP(GEOIP_DATABASE, pygeoip.GEOIP_STANDARD)
>>> gi.record_by_addr(ip)
{'country': '...', 'country_code': '...', ...}
This is one solution, from DjangoSnippets; btw, not sure why the code below doesn't use urlparse; but that could be fixed :-)
(Looking at the other answers, it seems you have plenty of options to choose from. This option may not be preferred because it relies on a free 3rd party service.)
from urllib2 import urlopen, Request
import re, socket
from django.conf import settings
domain_re = re.compile('^(http|https):\/\/?([^\/]+)')
domain = domain_re.match(settings.SITE_URL).group(2)
def getUserCountry(ip):
url = "http://api.wipmania.com/" + ip + "?" + domain
socket.setdefaulttimeout(5)
headers = {'Typ':'django','Ver':'1.1.1','Connection':'Close'}
try:
req = Request(url, None, headers)
urlfile = urlopen(req)
land = urlfile.read()
urlfile.close()
return land[:2]
except Exception:
return "XX"
Note from WIPmania: "Using API is free for any purpose, personal or business, if you are making fewer than 10.000 requests per calendar day. A simple yet powerful API allowing you to query the WorldIP database with a single link."
You could create a view which gets the user's IP and then issues an HTTP redirect which will cause their browser to load the page you want:
def redirect_based_on_ip(request):
ip = request.meta['REMOTE_ADDR']
if ip == SOMETHING:
return HttpResponseRedirect('/something')
elif ip == SOMETHING_ELSE:
return HttpResponseRedirect('/something_else')
# ...
You might find the SubnetTree library for Python helpful if you want to test to see if an IP is in a particular block.
Based on some free services.
it's not fast, but you can add more free services:
settings:
IPCOUNTRY_APYKEY = [
{# free tier 2 querys per second
"url": "http://api.ipinfodb.com/v3/ip-country/?ip={ip}&key={key}&format=json",
"params": {
"key": "*****************************",
},
"fieldname": "countryCode",
},
{# free tier 150 querys per minute and https is not suported in free tier
"url": "http://ip-api.com/json/{ip}?fields=2",
"params": {},
"fieldname": "countryCode",
},
{# free tier 1.500 queries per day
"url": "https://api.ipdata.co/{ip}?api-key={key}",
"params": {
"key": "*****************************",
},
"fieldname": "country_code",
},
{# free tier 10.000 queries per month and https is not suported in free tier
"url": "http://api.ipstack.com/{ip}?access_key={key}",
"params": {
"key": "*****************************",
},
"fieldname": "country_code",
},
{# free tier 10.000 queries per month and https is not suported in free tier
"url": "http://api.ipapi.com/{ip}?access_key={key}",
"params": {
"key": "*****************************",
},
"fieldname": "country_code",
},
]
CODE:
import json
import urllib3
from django.conf import settings
for service in settings.IPCOUNTRY_APYKEY:
url = service["url"].format(ip=ip,**service["params"])
headers = {'Type': 'django', 'Ver': '1.1.1', 'Connection': 'Close'}
urllib3.disable_warnings()
http_call = urllib3.PoolManager()
try:
r = http_call.request('GET', url, headers=headers, timeout=1.0)
if r.status == 200:
json_response = json.loads(r.data.decode("utf-8"))
print(json_response[service["fieldname"]])
except Exception as e:
pass
return None
I have made SO answer where I am using Cloudflare CDN, they provide extra header with GEO location of each visitor. Advantage is that we don't have to install any external library or make any API call. Redirection can be achieved by Django middleware.
You can achieve this by making use of IP2Location Python library and IP2Location BIN database. First, purchase a commercial IP2Location database from https://www.ip2location.com/database/ip2location or download a free IP2Location LITE database from https://lite.ip2location.com/. And also install the IP2Location Python library by using this command: pip install IP2Location
After that, open your views.py, and add the following code into the file:
from django.http import HttpResponse
from django.shortcuts import redirect
import IP2Location
database = IP2Location.IP2Location(YOUR_DATABASE_PATH)
def get_client_ip(request):
x_forwarded_for = request.META.get('HTTP_X_FORWARDED_FOR')
if x_forwarded_for:
ip = x_forwarded_for.split(',')[0]
else:
ip = request.META.get('REMOTE_ADDR')
if ip == '127.0.0.1': # Only define the IP if you are testing on localhost.
ip = '8.8.8.8'
return ip
def redirect_view(request):
ip = get_client_ip(request)
rec = database.get_all(ip)
if rec.city == 'London': # Or any other city name
response = redirect('PATH_REDIRECT_TO')
return response
def redirectsucess_view(request):
return HttpResponse('Welcome!')
After that, open the urls.py, and add the following codes into the urlpatterns list:
path('redirect/', views.redirect_view),
path('PATH_REDIRECT_TO/', views.redirectsucess_view)