I am trying to convert working Python 2.7 code into Python 3 code and I am receiving a type error from the urllib request module.
I used the inbuilt 2to3 Python tool to convert the below working urllib and urllib2 Python 2.7 code:
import urllib2
import urllib
url = "https://www.customdomain.com"
d = dict(parameter1="value1", parameter2="value2")
req = urllib2.Request(url, data=urllib.urlencode(d))
f = urllib2.urlopen(req)
resp = f.read()
The output from the 2to3 module was the below Python 3 code:
import urllib.request, urllib.error, urllib.parse
url = "https://www.customdomain.com"
d = dict(parameter1="value1", parameter2="value2")
req = urllib.request.Request(url, data=urllib.parse.urlencode(d))
f = urllib.request.urlopen(req)
resp = f.read()
When the Python 3 code is run the following error is produced:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-56-206954140899> in <module>()
5
6 req = urllib.request.Request(url, data=urllib.parse.urlencode(d))
----> 7 f = urllib.request.urlopen(req)
8 resp = f.read()
C:\Users\Admin\Anaconda3\lib\urllib\request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
159 else:
160 opener = _opener
--> 161 return opener.open(url, data, timeout)
162
163 def install_opener(opener):
C:\Users\Admin\Anaconda3\lib\urllib\request.py in open(self, fullurl, data, timeout)
459 for processor in self.process_request.get(protocol, []):
460 meth = getattr(processor, meth_name)
--> 461 req = meth(req)
462
463 response = self._open(req, data)
C:\Users\Admin\Anaconda3\lib\urllib\request.py in do_request_(self, request)
1110 msg = "POST data should be bytes or an iterable of bytes. " \
1111 "It cannot be of type str."
-> 1112 raise TypeError(msg)
1113 if not request.has_header('Content-type'):
1114 request.add_unredirected_header(
TypeError: POST data should be bytes or an iterable of bytes. It cannot be of type str.
I have also read two other tickets (ticket1 and ticket2) which mentioned encoding the date.
When I changed the line f = urllib.request.urlopen(req) to f = urllib.request.urlopen(req.encode('utf-8')) I received the following error: AttributeError: 'Request' object has no attribute 'encode'
I am stuck as to how to make the Python 3 code work. Could you please help me?
From the docs Note that params output from urlencode is encoded to bytes before it is sent to urlopen as data:
data = urllib.parse.urlencode(d).encode("utf-8")
req = urllib.request.Request(url)
with urllib.request.urlopen(req,data=data) as f:
resp = f.read()
print(resp)
Try this:
url = 'https://www.customdomain.com'
d = dict(parameter1="value1", parameter2="value2")
f = urllib.parse.urlencode(d)
f = f.encode('utf-8')
req = urllib.request.Request(url, f)
Your problem lies in the way you were handling the dictionary.
I used python requests module with ZOHO CRM API V2. It worked without any issues.
Here is a sample working code with GET request:
import json
import requests
# API methods - https://www.zoho.com/crm/developer/docs/api/api-methods.html
# getrecords API Call
module_name = 'Deals'
authtoken = '*****'
api_url = "https://crm.zoho.com/crm/private/json/"+module_name+"/getRecords?authtoken="+authtoken+"&scope=crmapi&fromIndex=1&toIndex=2"
# GET Request
request_response = requests.get(
url=api_url
)
print(json.dumps(json.loads(request_response.text), sort_keys=True, indent=4, separators=(",", ": ")))
json_response = json.loads(request_response.text)
Related
Fairly New to Django Here. I tried sending a CSV successfully using this code but I'm getting the following error sending a pdf file I generated using PDFpages from matplotlib
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xac in position
10: invalid start byte
Here's the code
def download_file(request):
# fill these variables with real values
fl_path = 'file.pdf'
filename = 'report.pdf'
fl = open(fl_path, 'r', encoding='utf-8')
mime_type, _ = mimetypes.guess_type(fl_path)
print(mime_type)
response = HttpResponse(fl, content_type=mime_type)
response['Content-Disposition'] = "attachment; filename=%s" % filename
return response
Is there a way to know which is the correct encoding I need to send my file through?
Use django's FileResponse instead of HttpResponse,
FileResponse is more ideal for send file data.
return FileResponse(fl, filename=''.format(filename ), as_attachment=True,
content_type='application/pdf')
Trying to run the following program to post the faces stored in a folder using ImageUrl but there exists an error.
The API I used is MS's Cognitive Face API v1.0
Please help
import sys
import os, time
import cognitive_face as CF
from global_variables import personGroupId
import urllib
import sqlite3
Key = '---some key----' #I removed the key her for some security Purpose
CF.Key.set(Key)
def get_person_id():
person_id = ''
extractId = str(sys.argv[1])[-2:]
connect = sqlite3.connect("Face-DataBase")
c = connect.cursor()
cmd = "SELECT * FROM Students WHERE ID = " + extractId
c.execute(cmd)
row = c.fetchone()
person_id = row[3]
connect.close()
return person_id
if len(sys.argv) is not 1:
currentDir = os.path.dirname(os.path.abspath(os.path.realpath(__file__)))
imageFolder = os.path.join(currentDir, "dataset/" + str(sys.argv[1]))
person_id = get_person_id()
for filename in os.listdir(imageFolder):
if filename.endswith(".jpg"):
print(filename)
imgurl = urllib.request.pathname2url(os.path.join(imageFolder, filename))
res = CF.face.detect(imgurl)
if len(res) != 1:
print ("No face detected in image")
else:
res = CF.person.add_face(imgurl, personGroupId, person_id)
print(res)
time.sleep(6)
The error I got is a Invalid Image Url with Status Code '400'
User.22.1.jpg
Traceback (most recent call last):
File "add_person_faces.py", line 31, in <module>
res = CF.face.detect(imgurl)
File "C:\Users\Avina\Anaconda3\envs\virtual_platform\lib\site-packages\cognitive_face\face.py", line 41, in detect
data=data)
File "C:\Users\Avina\Anaconda3\envs\virtual_platform\lib\site-packages\cognitive_face\util.py", line 102, in request
error_msg.get('message'))
cognitive_face.util.CognitiveFaceException: Error when calling Cognitive Face API:
status_code: 400
code: InvalidURL
message: Invalid image URL.
You'll have to send a public url or send the image in binary format for the API to be able to access it.
Here is how you read the image:
try:
with open('mypic.jpg', 'rb') as image:
img = image.read()
face_to_detect = bytearray(img)
print("image reading done!")
except Exception as ex:
print("exception in reading image bytes {0}".format(ex.args))
has_exception = True
And then post it:
result = cf.face.detect(face_to_detect)
I am a beginner in scrapy, python. I tried to deploy the spider code in scrapinghub and I encountered the following error. Below is the code.
import scrapy
from bs4 import BeautifulSoup,SoupStrainer
import urllib2
from scrapy.selector import Selector
from scrapy.http import HtmlResponse
import re
import pkgutil
from pkg_resources import resource_string
from tues1402.items import Tues1402Item
data = pkgutil.get_data("tues1402","resources/urllist.txt")
class SpiderTuesday (scrapy.Spider):
name = 'tuesday'
self.start_urls = [url.strip() for url in data]
def parse(self, response):
story = Tues1402Item()
story['url'] = response.url
story['title'] = response.xpath("//title/text()").extract()
return story
is my spider.py code
import scrapy
class Tues1402Item(scrapy.Item):
# define the fields for your item here like:
# name = scrapy.Field()
title = scrapy.Field()
url = scrapy.Field()
is the items.py code and
from setuptools import setup, find_packages
setup(
name = 'tues1402',
version = '1.0',
packages = find_packages(),
entry_points = {'scrapy': ['settings = tues1402.settings']},
package_data = {'tues1402':['resources/urllist.txt']},
zip_safe = False,
)
is the setup.py code.
The following is the error.
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/scrapy/core/engine.py", line 126, in _next_request
request = next(slot.start_requests)
File "/usr/local/lib/python2.7/site-packages/scrapy/spiders/init.py", line 70, in start_requests
yield self.make_requests_from_url(url)
File "/usr/local/lib/python2.7/site-packages/scrapy/spiders/init.py", line 73, in make_requests_from_url
return Request(url, dont_filter=True)
File "/usr/local/lib/python2.7/site-packages/scrapy/http/request/init.py", line 25, in init
self._set_url(url)
File "/usr/local/lib/python2.7/site-packages/scrapy/http/request/init.py", line 57, in _set_url
raise ValueError('Missing scheme in request url: %s' % self._url)
ValueError: Missing scheme in request url: h
Thank you in Advance
Your error means that url h is not a valid url. You should print out your self.start_urls and see what urls you have there, you most likely have a string h as your first url.
Seems like your spider iterates through text instead of a list of urls here:
data = pkgutil.get_data("tues1402","resources/urllist.txt")
class SpiderTuesday (scrapy.Spider):
name = 'tuesday'
self.start_urls = [url.strip() for url in data]
Assuming that you store your urls with some separator in urllist.txt file you should split that:
# assuming file has url in every line
self.start_urls = [url.strip() for url in data.splitlines()]
I am working on a project on django 1.9 and using django-rest-framework.
Views.py:
#parser_classes((FileUploadParser,))
class upload_image(APIView):
def post(self, request, format=None):
file_obj = request.data['images']
upload_s3 = FileUpload()
folder_name = 'images'
file_url = upload_s3.put(file_obj, folder_name, file_obj.name)
class File_Upload:
def __init__(self):
self.env = {
"aws_access_key_id": "*************",
"aws_secret_access_key": "***********",
"region_name": "Asia Pacific (Singapore)"
}
self.bucketname = "my_bucket"
self.session = Session(**self.env)
self.s3_client = self.session.client('s3')
def put(self, bytes, folder, file_name):
self.s3_client.put_object(Body=bytes, Bucket=self.bucketname, Key=folder + "/" + file_name)
return "https://s3.%s.amazonaws.com/%s/%s/%s" % (self.env['region_name'], self.bucketname, folder, file_name)
On python shell I am able to upload as well as download images to and from amazon s3. I am using boto for this.
Below is the code that I am using :
>>> import boto
>>> import sys
>>> from boto.s3.connection import S3Connection
>>> from boto.s3.key import Key
>>> conn = boto.connect_s3("ACESS_KEY_ID","SECRET_ACCESS_KEY",host="s3-ap-southeast-1.amazonaws.com",is_secure = False)
>>> k = bucket.get_key("KEY_NAME")
>>> k.get_contents_to_filename("test.file")
>>> k.set_canned_acl('public-read')
>>> url = k.generate_url(0,query_auth=False,force_http=True)
I get a working url. But the problem comes when I want to do the same in the views.py(above code), I get this error:
S3ResponseError: 400 Bad Request
Request Method: POST Request
URL: http://127.0.0.1:8000/admin/imagedata/image/add/
Django Version: 1.9
Exception Type: S3ResponseError
Exception Value: S3ResponseError: 400
Bad Request
Exception Location:/Library/Python/2.7/site-packages/boto/s3/bucket.py in_get_key_internal, line 235
Python Executable: /usr/bin/python Python Version: 2.7.10
kindly tell me where the problem is?
ok here is my code , what I am trying to do is post to a page that is password protected can you have a look at the code below at see where I am going wrong getting
!/usr/bin/python
import requests, sys, socket, json
from requests.auth import HTTPDigestAuth ,HTTPBasicAuth
172.168.101.214
params = {'#Generate': 'New'}
response = requests.post('https://TerraceQ.internal.ca/views/Debug_Dump/1', auth=HTTPDigestAuth('user', 'fakepassword'), data=params)
print response.status_code
there this worked
ip="172.168.99.99"
try:
response = requests.get('https://' + ip + '/views', auth=HTTPDigestAuth('username', 'password'), verify=False)
except urllib3.exceptions.SSLError as e:
sys.exit('test')