I have a django parse function as:
def parse_org(request):
try:
org = simplejson.loads(request.POST['org'])
except Exception:
traceback.print_exc()
print org
I got the decode error.
On the client side, the script version(code are pasted in later part) works fine, but recently I want to write a python version to do load test, so I write the following code in a python client script to send request:
data_dict = {}
org = ["UCSD", "MIT"]
data_dict["org"] = org
req = urllib2.Request(request_url, urllib.urlencode(data_dict), headers)
response = urllib2.urlopen(req, timeout = 5)
Then the parse code at the django site gets parsing error. I compare the correct javascript version and the wrong python clients, the only difference is the single and double quote,
the wrong parsed input at django side is:
POST:<QueryDict: {u'org': [u"['UCSD', 'MIT']"], ....
the correct input is:
POST:<QueryDict: {u'org': [u'["UCSD","MIT"]'], ....
For your reference, the javascript side looks like(the django can correctly parse the org as array):
var org = [];
org.push("UCSD")
org.push("MIT")
var data = {"org": JSON.stringify(org), ...
}
$.post(url, data, function(data){
callback(data);
});
I searched a lot, but still can't find why the python client can't work but the javascript client can work. Is it related with the urllib's urlencode? and why there is single and double quote difference there?
Thanks a lot!
Related
Here is my Django view:
def sendMail(request):
url = 'https://example.com/example'
dictwithlist = request.FILES['dictwithlist']
parsed = json.load(dictwithlist)
// the log file shows that the lists are intact after json.loads
logip(request, json.loads(parsed))
x = requests.post(url, json.loads(clockoutJSON))
return HttpResponse(status=204)
If I just send parsed data my express server receives an empty dict, {}. When I log the json.loads(parsed) I find good data, with the lists intact. When the data gets to the other side though, the dictionaries inside the nested list are all removed, replaced by only strings of their keys.
I tried using headers as described here: Sending list of dicts as value of dict with requests.post going wrong but I just get 500 errors. I don't know if I'm formatting the headers wrong or not. (because the code has line spacing and I'm copying it)
Can anyone help me understand why this is failing? I need that list to get through with its dictionaries intact.
I believe you may need to use dumps rather than loads when sending the request:
x = requests.post(url, json.dumps(parsed))
Alternatively if you're using the python requests library you can send json as a dict as below:
response = requests.post(url=url, json=parsed)
I'm trying to send multiple cookies to a url until I get the right one and I don't know why my current code isn't working
I've looked at the existing answers for sending cookies to a url but none of them seem to work in my case
The comments in the code are instructions for the task
# Write a script that can guess cookie values
# and send them to the url http://127.0.0.1:8082/cookiestore
# Read the response from the right cookie value to get the flag.
# The cookie id the aliens are using is alien_id
# the id is a number between 1 and 75
import urllib2
req = urllib2.Request('http://127.0.0.1:8082/cookiestore')
for i in range(75):
req.add_header('alien_id', i)
response = urllib2.urlopen(req)
html = response.read()
print(html)
I expected that one of the iterations would print something different, but they are all the same
This code is Python 3.8 as Python 2.7 is no longer supported, and your req.add_header line did need modifying. Here is your solution:
import urllib.request
url = "http://127.0.0.1:8082/cookiestore"
y = 5
for i in range(75):
request = urllib.request.Request(url)
request.add_header("Cookie", "alien_id = "+str(i))
response = urllib.request.urlopen(request)
responseStr = str(response.read().decode("utf-8"))
print(responseStr)
If you are still using Python 2.7, you can just copy the request.add_header line.
Cheers!
Ahaha cyber discovery i see ;)
You were very close, infact i used your code as my base except for the add header line.
Not sure if you even still need to know but ill leave this here for others
When you send a cookie as a header you send it like so:
req.add_header('Cookie', 'cookiename=cookievalue')
Hope this helps :D
I'm using the following code (from django management commands) to listen to the Twitter stream - I've used the same code on a seperate command to track keywords successfully - I've branched this out to use location, and (apparently rightly) wanted to test this out without disrupting my existing analysis that's running.
I've followed the docs and have made sure the box is in Long/Lat format (in fact, I'm using the example long/lat from the Twitter docs now). It looks broadly the same as the question here, and I tried using their version of the code from the answer - same error. If I switch back to using 'track=...', the same code works, so it's a problem with the location filter.
Adding a print debug inside streaming.py in tweepy so I can see what's happening, I print out the self.parameters self.url and self.headers from _run, and get:
{'track': 't,w,i,t,t,e,r', 'delimited': 'length', 'locations': '-121.7500,36.8000,-122.7500,37.8000'}
/1.1/statuses/filter.json?delimited=length and
{'Content-type': 'application/x-www-form-urlencoded'}
respectively - seems to me to be missing the search for location in some way shape or form. I don't believe I'm/I'm obviously not the only one using tweepy location search, so think it's more likely a problem in my use of it than a bug in tweepy (I'm on 2.3.0), but my implementation looks right afaict.
My stream handling code is here:
consumer_key = 'stuff'
consumer_secret = 'stuff'
access_token='stuff'
access_token_secret_var='stuff'
import tweepy
import json
# This is the listener, resposible for receiving data
class StdOutListener(tweepy.StreamListener):
def on_data(self, data):
# Twitter returns data in JSON format - we need to decode it first
decoded = json.loads(data)
#print type(decoded), decoded
# Also, we convert UTF-8 to ASCII ignoring all bad characters sent by users
try:
user, created = read_user(decoded)
print "DEBUG USER", user, created
if decoded['lang'] == 'en':
tweet, created = read_tweet(decoded, user)
print "DEBUG TWEET", tweet, created
else:
pass
except KeyError,e:
print "Error on Key", e
pass
except DataError, e:
print "DataError", e
pass
#print user, created
print ''
return True
def on_error(self, status):
print status
l = StdOutListener()
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_token, access_token_secret_var)
stream = tweepy.Stream(auth, l)
#locations must be long, lat
stream.filter(locations=[-121.75,36.8,-122.75,37.8], track='twitter')
The issue here was the order of the coordinates.
Correct format is:
SouthWest Corner(Long, Lat), NorthEast Corner(Long, Lat). I had them transposed. :(
The streaming API doesn't allow to filter by location AND keyword simultaneously.
you must refer to this answer i had the same problem earlier
https://stackoverflow.com/a/22889470/4432830
I am using Beautiful Soup to try and scrape a page.
I am trying to follow this tutorial.
I am trying to get the contents of the following page after submitting a Stock Ticker Symbol:
http://www.cboe.com/delayedquote/quotetable.aspx
The tutorial is for a page with a "GET" method, my page is a "POST". I wonder if that is part of the problem?
I want use the first text box – under where it says:
“Enter a Stock or Index symbol below for delayed quotes.”
Relevant code:
user_agent = 'Mozilla/5 (Solaris 10) Gecko'
headers = { 'User-Agent' : user_agent }
values = {'ctl00$ctl00$AllContent$ContentMain$ucQuoteTableCtl$txtSymbol' : 'IBM' }
data = urllib.urlencode(values)
request = urllib2.Request("http://www.cboe.com/delayedquote/quotetable.aspx", data, headers)
response = urllib2.urlopen(request)
The call does not fail, I do not get a set of options and prices returned to me like when I run the page interactively. I a bunch of garbled HTML.
Thanks in advance!
Ok - I think I figured out the problem (and found another). I decided to switch to 'mechanize' from 'urllib2'. Unfortunately, I kept having problems getting the data. Finally, I realized that there are two 'submit' buttons, so I tried passing the name parameter when submitting the form. That did the trick as far as getting the correct response.
However, the next problem was that I could not get BeautifulSoup to parse the HTML and find the necessary tags. A brief Google search revealed others having similar problems. So, I gave up on BeautifulSoup and just did a basic regex on the HTML. Not as elegant as BeautifulSoup, but effective.
Ok - enough speechifying. Here's what I came up with:
import mechanize
import re
br = mechanize.Browser()
url = 'http://www.cboe.com/delayedquote/quotetable.aspx'
br.open(url)
br.select_form(name='aspnetForm')
br['ctl00$ctl00$AllContent$ContentMain$ucQuoteTableCtl$txtSymbol'] = 'IBM'
# here's the key step that was causing the trouble - pass the name parameter
# for the button when calling submit
response = br.submit(name="ctl00$ctl00$AllContent$ContentMain$ucQuoteTableCtl$btnSubmit")
data = response.read()
match = re.search( r'Bid</font><span> \s*([0-9]{1,4}\.[0-9]{2})', data, re.MULTILINE|re.M|re.I)
if match:
print match.group(1)
else:
print "There was a problem retrieving the quote"
Is there a way to programmatically log into Yahoo!, providing email id and password as inputs, and fetch the user's contacts?
I've achieved the same thing with Gmail, using BeautifulSoup.py
Yahoo Address book API provides BBAuth, which requires the user to be redirected to Yahoo login page. But I'm looking for a way to authenticate the user with Yahoo without the redirection.
I have tried this :
http://pypi.python.org/pypi/ContactGrabber/0.1
but I am getting this Error:
Warning (from warnings module): File
"C:\Python26\lib\site-packages\contactgrabber-0.1-py2.6.egg\contactgrabber\base.py",
line 31
RuntimeWarning: tempnam is a potential security risk to your program
Invalid UserID/Password
Exception WindowsError: (2, 'The system cannot find the file
specified', 'C:\DOCUME~1\sjain\LOCALS~1\Temp\2') in > ignored
I solved this problem by using Urllib here is the code :
LoginUrl = "https://login.yahoo.com/config/login?"
ExportUrl = "http://address.yahoo.com/"
def import_yahoo_contacts(login,passwd):
try :
form_data = {'login' : login, 'passwd' : passwd}
jar = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(jar))
form_data = urllib.urlencode(form_data)
resp = opener.open(LoginUrl, form_data)
resp = opener.open(ExportUrl)
page = resp.read()
index = page.find('InitialContacts')
startjson = page.index('[',index)
endjson = page.index(']',index)
Jsondata = page[startjson:endjson+1]
user_contacts = []
data =json.JSONDecoder().decode(Jsondata)
for r in data:
userfriends = []
userfriends.append(r.get('contactName'))
userfriends.append(r.get('email'))
user_contacts.append(userfriends)
except:
return []
return user_contacts
This really work for me :)
You could write one or just use Pinax. Pinax is a collection of tools built on top of Django. They have a application which imports contact imfo (from vCard, Google or Yahoo).
I suggest you use this as you don't have to maintain it plus to avoid reinventing the cycle.