mail address validation in python/django - django

We are developing our website in django framework and with python. currently we are looking for an api/a tool to validate the physical addres the user enters. It doesn't need to be vey sophisticated. We just need to avoid spam registrations. For this step, we need to verify if country, state, city, and zipcode are belong to each other. For instance, Berkeley as a city with Zipcode 94704 is not in NY state, etc.
Any thoughts?

You can give a try to pygeocoder. It's a Python wrapper for Google's Geocoding V3 API. It can fulfill some of your validation needs.
from pygeocoder import Geocoder
g = Geocoder()
g.geocode("1 wellington, ottawa").valid_address
>>> True
g.geocode("1 wellington, chicago").valid_address
>>> False
It can take some minor misspelling too
result = g.geocode("1600 amphiteather, mountain view")
result.valid_address
>>> True
print result
>>> 1600 Amphitheatre Pkwy, Mountain View, CA 94043, USA
g.geocode("16000000 amphitheater, mountain view").valid_address
>>> False
But Google doesn't always have the right postal/zip code. USPS has a public API for the US I believe.
disclamer: I made pygeocoder

You could probably use a GeoCoding service (like Google's) to verify that an address exists.

Related

Raw_input & If Statements

I am trying to make a personal assistant for python named "Sansa".
I am a noob and I want my code to execute my initial_request variable only if users raw_input is yes. If it is no I want the code to execute the last engine.say command. I tried to achieve this using a dictionary. Thanks and sorry for the messy code.
import webbrowser
import pyttsx
#adding voice (i will change)
engine = pyttsx.init()
engine = pyttsx.init()
voices = engine.getProperty('voices')
for voice in voices:
engine.setProperty('voice', voice.id)
#Introducing Sansa and trying to navigate through the codes.
#serviceP =input
first_answer = {'pos' : 'Yes', 'neg' : 'No'}
response = engine.say("Hello my Lord. My name is Sansa and I am your personal assistant. I am here to serve you. Can I be of service?")
engine.runAndWait()
st = raw_input("Hello my Lord. My name is Sansa and I am your personal assistant. I am here to serve you. Can I be of service?")
initial_Request =('What would you like to do? I can visit various websites, perform a google or youtube search, and so much more.')
#evaluate yes or no to execute initial_Request
if st == first_answer['pos']:
engine.say(initial_Request)
engine.runAndWait()
elif st == first_answer['neg']:
engine.say("Ok. I'm here if you need me.")
engine.runAndWait()
Okay I have found the problem to make the program run through the code such as adding the variable unknownResponse, adding print statements under the engine.say() functions,etc. However I am still working through a few bugs such as the voice not properly continuing throughout the if and elif statements. The program is coming together though. Here is the updated code.
import webbrowser
import pyttsx
#adding voice (i will change)
engine = pyttsx.init()
engine = pyttsx.init()
voices = engine.getProperty('voices')
for voice in voices:
engine.setProperty('voice', voice.id)
#Introducing Sansa and trying to navigate through the codes. Might need some type of loop to redirect the user
first_answer = {'pos' : 'Yes', 'neg' : 'No'}
engine.say("Hello my Lord. My name is Sansa and I am your personal assistant. I am here to serve you. Can I be of service?")
engine.runAndWait()
st = raw_input("Hello my Lord. My name is Sansa and I am your personal assistant. I am here to serve you. Can I be of service?")
initial_Request =("What would you like to do? I can visit various websites, perform a google or youtube search, and so much more.")
negative_Response =("Okay Master. I'm here if you change your mind.")
#evaluate yes or no to execute initial_Request
if st == first_answer['pos']:
engine.say("What would you like to do? I can visit various websites, perform a google or youtube search, and so much more.")
engine.runAndWait()
print initial_Request
elif st == first_answer['neg']:
engine.say("Okay Master. I'm here if you change your mind.")
engine.runAndWait()
print negative_Response
#add loop to make questions repeat
unknownResponse = True
while unknownResponse:
againTry =raw_input("Sorry I don't understand. Please try again.")
if st != first_answer['pos', 'neg']:
engine.say("Sorry I don't understand. Please try again.")
engine.runAndWait()
unknownResponse = False

What does it mean to normalize an email address?

In this example, Django talks about normalizing an email address with self.normalize_email(email) where self is BaseUserManager. When I search for "normalizing emails" it seems to be a practice across all platforms. I see tutorials of how to do it, but nothing really explaining what it is and what it's used for.
For email addresses, foo#bar.com and foo#BAR.com are equivalent; the domain part is case-insensitive according to the RFC specs. Normalizing means providing a canonical representation, so that any two equivalent email strings normalize to the same thing.
The comments on the Django method explain:
Normalize the email address by lowercasing the domain part of it.
One application of normalizing emails is to prevent multiple signups. If your application lets the public to sign up, your application might attract the "unkind" types, and they could attempt to sign up multiple times with the same email address by mixing symbols, upper and lower cases to make variants of the same email address.
From Django's repository, the docstring of normalize_email is the following:
Normalize the email address by lowercasing the domain part of it.
What this method does is to lowercase the domain part of an email, so this part is case insensitive, so consider the following examples:
>>> from django.contrib.auth.models import BaseUserManager
>>> BaseUserManager.normalize_email("user#example.com")
user#example.com
>>> BaseUserManager.normalize_email("user#EXAMPLE.COM")
user#example.com
>>> BaseUserManager.normalize_email("user#example.COM")
user#example.com
>>> BaseUserManager.normalize_email("user#EXAMPLE.com")
user#example.com
>>> BaseUserManager.normalize_email("user#ExAmPlE.CoM")
user#example.com
As you can see all emails are equivalent because the case after # is irrelevant.

GQL Queries - Retrieving specific data from query object

I'm building a database using Google Datastore. Here is my model...
class UserInfo(db.Model):
name = db.StringProperty(required = True)
password = db.StringProperty(required = True)
email = db.StringProperty(required = False)
...and below is my GQL Query. How would I go about retrieving the user's password and ID from the user_data object? I've gone through all the google documentation, find it hard to follow, and have spent ages trying various things I've read online but nothing has helped! I'm on Python 2.7.
user_data = db.GqlQuery('SELECT * FROM UserInfo WHERE name=:1', name_in)
user_info = user_data.get()
This is basic Python.
From the query, you get a UserInfo instance, which you have stored in the user_info variable. You can access the data of an instance via dot notation: user_info.password and user_info.email.
If this isn't clear, you really should do a basic Python tutorial before going any further.
You are almost there. Treat the query object like a class.
name = user_info.name
Documentation on queries here gives some examples
There are some python tips that might help you
dir(user_info)
help(user_info)
you can also print almost anything, like
print user_info[0]
print user_info[0].name
Setup logging for your app
Logging and python etc

Amazon Product Advertising API: Get Average Customer Rating

When using Amazon's web service to get any product's information, is there a direct way to get the Average Customer Rating (1-5 stars)? Here are the parameters I'm using:
Service=AWSECommerceService
Version=2011-08-01
Operation=ItemSearch
SearchIndex=Books
Title=A Game of Thrones
ResponseGroup=Large
I would expect it to have a customer rating of 4.5 and total reviews of 2177. But instead I get the following in the response.
<CustomerReviews><IFrameURL>http://www.amazon.com/reviews/iframe?...</IFrameURL></CustomerReviews>
Is there a way to get the overall customer rating, besides for reading the <IFrameURL/> value, making another HTTP request for that page of reviews, and then screen scraping the HTML? That approach is fragile since Amazon could easily change the reviews page structure which would bust my application.
You can scrape from here. Just replace the asin with what you need.
http://www.amazon.com/gp/customer-reviews/widgets/average-customer-review/popover/ref=dpx_acr_pop_?contextId=dpx&asin=B000P0ZSHK
As far as i know, Amazon changed it's API so its not possible anymore to get the reviewrank information. If you check this Link the note sais:
As of November 8, 2010, only the iframe URL is returned in the request
content.
However, testing with the params you used to get the Iframe it seems that now even the Iframe dosn't work anymore. Thus, even in the latest API Reference in the chapter "Motivating Customers to Buy" the part "reviews" is compleatly missing.
However: Since i'm also very interested if its still possible somehow to get the reviewrank information - maybe even not using amazon API but a competitors API to get review rank informations - i'll set up a bounty if anybody can provide something helpful on that. Bounty will be set in this topic in two days.
You can grab the iframe review url and then use css to position it so only the star rating shows. It's not ideal since you're not getting raw data, but it's an easy way to add the rating to your page.
Sample of this in action - http://spamtech.co.uk/positioning-content-inside-an-iframe/
Here is a VBS script that would scrape the rating. Paste the code below to a text file, rename it to Test.vbs and double click to run on Windows.
sAsin = InputBox("What is your ASIN?", "Amazon Standard Identification Number (ASIN)", "B000P0ZSHK")
if sAsin <> "" Then
sHtml = SendData("http://www.amazon.com/gp/customer-reviews/widgets/average-customer-review/popover/ref=dpx_acr_pop_?contextId=dpx&asin=" & sAsin)
sRating = ExtractHtml(sHtml, "<span class=""a-size-base a-color-secondary"">(.*?)<\/span>")
sReviews = ExtractHtml(sHtml, "<a class=""a-size-small a-link-emphasis"".*?>.*?See all(.*?)<\/a>")
MsgBox sRating & vbCrLf & sReviews
End If
Function ExtractHtml(sHtml,sPattern)
Set oRegExp = New RegExp
oRegExp.Pattern = sPattern
oRegExp.IgnoreCase = True
Set oMatch = oRegExp.Execute(sHtml)
If oMatch.Count = 1 Then
ExtractHtml = Trim(oMatch.Item(0).SubMatches(0))
End If
End Function
Function SendData(sUrl)
Dim oHttp 'As XMLHTTP30
Set oHttp = CreateObject("Msxml2.XMLHTTP")
oHttp.open "GET", sUrl, False
oHttp.send
SendData = Replace(oHttp.responseText,vbLf,"")
End Function
Amazon has completely removed support for accessing rating/review information from their API. The docs mention a Response Element in the form of customer rating, but that doesn't work either.
Google shopping using Viewpoints for some reviews and other sources
This is not possible from PAPI. You either need to scrape it by yourself, or you can use other free/cheaper third-party alternatives for that.
We use the amazon-price API from RapidAPI for this, it supports price/rating/review count fetching for up to 1000 products in a single request.

How to obtain longitude and latitude for a street address programmatically (and legally)

Supposedly, it is possible to get this from Google Maps or some such service. (US addresses only is not good enough.)
The term you're looking for is geocoding and yes Google does provide this service.
New V3 API: http://code.google.com/apis/maps/documentation/geocoding/
Old V2 API: http://code.google.com/apis/maps/documentation/services.html#Geocoding
In addition to the aforementioned Google geocoding web service, there is also a competing service provided by Yahoo. In a recent project where geocoding is done with user interaction, I included support for both. The reason is I have found that, especially outside the U.S., their handling of more obscure locations varies widely. Sometimes Google will have the best answer, sometimes Yahoo.
One gotcha to be aware of: if Google really thinks they don't know where your place is, they will return a 602 error indicating failure. Yahoo, on the other hand, if it can peel out a city/province/state/etc out of your bad address, will return the location of the center of that town. So you do have to pay attention to the results you get to see if they are really what you want. There are ancillary fields in some results that tell you about this: Yahoo calls this field "precision" and Google calls it "accuracy".
If you want to do this without relying on a service, then you download the TIGER Shapefiles from the US Census.
You look up the street you're interested in, which will have several segments. Each segment will have a start address and end address, and you interpolate along the segment to find where on the segment your house number lies.
This will provide you with a lon/lat pair.
Keep in mind, however, that online services employ a great deal of address checking and correction, which you'd have to duplicate as well to get good results.
Also note that as nice as free data is, it's not perfect - the latest streets aren't in there (they might be in the data Google uses), and the streets may be off their real location by some amount due to survey inaccuracies. But for 98% of geocoding needs it works perfectly, is free, and you control everything so you're reducing dependencies in your app.
Openstreetmaps has the aim of mapping everything in the world, though they aren't quite there it's worth keeping tabs on as they provide their data under a CC license
However, many (most?) other countries are only mapped by gov't or services for which you need to pay a fee. If you don't need to geocode very much data, then using Google, Yahoo, or some of the other free worldwide mapping services may be enough.
If you have to geocode a lot of data, then you will be best served by leasing map data from a major provider, such as teleatlas.
-Adam
You could also try the OpenStreetMap NameFinder, which contains open source, wiki-like street data for (potentially) the entire world. NameFinder will cease to exist at the end of august, but Nominatim is its replacement.
Google requires you to show a Google map with their data, with a max of 2.5k (was 25k) HTTP requests per day. Useful but you have to watch usage.
They do have
http://groups.google.com/group/Google-Maps-API/web/resources-non-google-geocoders
(Google has since removed this. If you see a duplicate or cache, I'll link to that.)
...in which I found GeoNames which has both a downloadable db, free web service and a commercial web service.
Google's terms of service will let you use their geocoding API for free if your website is in turn free for consumers to use. If not you will have to get a license for the Enterprise Maps.
For use with Drupal and PHP (and easily modified):
function get_lat_long($address) {
$res = drupal_http_request('http://maps.googleapis.com/maps/api/geocode/json?address=' . $address .'&sensor=false');
return json_decode($res->data)->results[0]->geometry->location;
}
You can have a look at the Google Maps API docs here to get a start on this:
http://code.google.com/apis/maps/documentation/services.html#Geocoding
It also seems to be something that you can do for international addresses using Live Maps also:
http://virtualearth.spaces.live.com/blog/cns!2BBC66E99FDCDB98!1588.entry
You can also do this with Microsoft's MapPoint Web Services.
Here's a blog post that explains how:
http://www.codestrider.com/BlogRead.aspx?b=b5e8e275-cd18-4c24-b321-0da26e01bec5
R Code to get the latitude and longitude of a street address
# CODE TO GET THE LATITUDE AND LONGITUDE OF A STREET ADDRESS WITH GOOGLE API
addr <- '6th Main Rd, New Thippasandra, Bengaluru, Karnataka' # set your address here
url = paste('http://maps.google.com/maps/api/geocode/xml?address=', addr,'&sensor=false',sep='') # construct the URL
doc = xmlTreeParse(url)
root = xmlRoot(doc)
lat = xmlValue(root[['result']][['geometry']][['location']][['lat']])
long = xmlValue(root[['result']][['geometry']][['location']][['lng']])
lat
[1] "12.9725020"
long
[1] "77.6510688"
If you want to do this in Python:
import json, urllib, urllib2
address = "Your address, New York, NY"
encodedAddress = urllib.quote_plus(address)
data = urllib2.urlopen("http://maps.googleapis.com/maps/api/geocode/json?address=" + encodedAddress + '&sensor=false').read()
location = json.loads(data)['results'][0]['geometry']['location']
lat = location['lat']
lng = location['lng']
print lat, lng
Note that Google does seem to throttle requests if it sees more than a certain amount, so you do want to use an API key in your HTTP request.
I had a batch of 100,000 records to be geocode and ran into Google API's limit (and since it was for an internal enterprise app, we had to upgrade to their premium service which is $10K plus)
So, I used this instead: http://geoservices.tamu.edu/Services/Geocode/BatchProcess/ -- they also have an API. (the total cost was around ~200$)
You can try this in JavaScript for city like kohat
var geocoder = new google.maps.Geocoder();
var address = "kohat";
geocoder.geocode( { 'address': address}, function(results, status) {
var latitude = results[0].geometry.location.lat();
var longitude = results[0].geometry.location.lng();
alert(latitude+" and "+longitude);
}
});
In python using geopy PyPI used to get the lattitude,langitude,zipcode etc..
Here is the working sample code..
from geopy.geocoders import Nominatim
geolocator = Nominatim(user_agent="your-app-id")
location = geolocator.geocode("Your required address ")
if location:
print('\n Nominatim ADDRESS :',location.address)
print('\n Nominatim LATLANG :',(location.latitude, location.longitude))
print('\n Nominatim FULL RESPONSE :',location.raw)
else:
print('Cannot Find')
In Nominatim - Some addresses can't working, so i just tried MapQuest.
It returns correctly.
Mapquest provides free-plan 15000 transactions/month. It is enough for me.
Sample code:
import geocoder
g = geocoder.mapquest("Your required address ",key='your-api-key')
for result in g:
# print(result.address, result.latlng)
print('\n mapquest ADDRESS :',result.address,result.city,result.state,result.country)
print('\n mapquest LATLANG :', result.latlng)
print('\n mapquest FULL RESPONSE :',result.raw)
Hope it helps.
I know this is old question but google changing way to get latitude and longitude on regular based.
HTML code
<form>
<input type="text" name="address" id="address" style="width:100%;">
<input type="button" onclick="return getLatLong()" value="Get Lat Long" />
</form>
<div id="latlong">
<p>Latitude: <input size="20" type="text" id="latbox" name="lat" ></p>
<p>Longitude: <input size="20" type="text" id="lngbox" name="lng" ></p>
</div>
JavaScript Code
<script src="https://maps.googleapis.com/maps/api/js?key=YOUR_API_KEY&callback=initMap" async defer></script>
<script>
function getLatLong()
{
var address = document.getElementById("address").value;
var geocoder = new google.maps.Geocoder();
geocoder.geocode( { 'address': address}, function(results, status) {
if (status == google.maps.GeocoderStatus.OK) {
var latitude = results[0].geometry.location.lat();
document.getElementById("latbox").value=latitude;
var longitude = results[0].geometry.location.lng();
document.getElementById("lngbox").value=longitude;
}
});
}
</script>