I am facing a bit of a situation,
Scenario: I got a django rest api running on my localhost:8000 and I want to access the api using my command line. I have tried urllib2 and python requests libs to talk to the api but failed(i'm getting a 503 error). But when I pass google.com as the url, I am getting the expected response. So I believe my approach is correct but I'm doing something wrong. please see the code below :
import urllib, urllib2, httplib
url = 'http://localhost:8000'
httplib.HTTPConnection.debuglevel = 1
print "urllib"
data = urllib.urlopen(url);
print "urllib2"
request = urllib2.Request(url)
opener = urllib2.build_opener()
feeddata = opener.open(request).read()
print "End\n"
Envioroments:
OS Win7
python v2.7.5
Django==1.6
Markdown==2.3.1
colorconsole==0.6
django-filter==0.7
django-ping==0.2.0
djangorestframework==2.3.10
httplib2==0.8
ipython==1.0.0
jenkinsapi==0.2.14
names==0.3.0
phonenumbers==5.8b1
requests==2.1.0
simplejson==3.3.1
termcolor==1.1.0
virtualenv==1.10.1
Thanks
I had a similar problem, but found that it was the company's proxy that was preventing from pinging myself.
503 Reponse when trying to use python request on local website
Try:
>>> import requests
>>> session = requests.Session()
>>> session.trust_env = False
>>> r = session.get("http://localhost:5000/")
>>> r
<Response [200]>
>>> r.content
'Hello World!'
If you are registering your serializers with DefaultRouter then your api will appear at
http://localhost:8000/api/ for an html view of the index
http://localhost:8000/api/.json for a JSON view of the index
http://localhost:8000/api/appname for an html view of the individual resource
http://localhost:8000/api/appname/.json for a JSON view of the individual resource
you can check the response in your browser to make sure your URL is working as you expect.
Related
I have created a simple Flask RESTful API with a single function used to service GET requests and which expects several URL parameters:
import logging
from flask import Flask
from flask_restful import Api, Resource
from webargs import fields
from webargs.flaskparser import abort, parser, use_args
# initialize the Flask application and API
app = Flask(__name__)
api = Api(app)
# ------------------------------------------------------------------------------
# set up a basic, global logger object which will write to the console
logging.basicConfig(level=logging.INFO,
format="%(asctime)s %(levelname)s %(message)s",
datefmt="%Y-%m-%d %H:%M:%S")
_logger = logging.getLogger(__name__)
# ------------------------------------------------------------------------------
class Abc(Resource):
abc_args = {"rtsp": fields.Url(required=True),
"start": fields.Integer(required=True),
"duration": fields.Integer(required=True),
"bucket": fields.String(required=True),
"prefix": fields.String(missing="")}
#use_args(abc_args)
def get(self, args) -> (str, int):
_logger.info("Recording video clip with the following parameters:\n"
f"\tRTSP URL: {args['rtsp']}"
f"\tStart seconds: {args['start']}"
f"\tDuration seconds: {args['duration']}"
f"\tS3 bucket: {args['bucket']}"
f"\tS3 key prefix: {args['prefix']}")
return "OK", 200
# ------------------------------------------------------------------------------
# This error handler is necessary for usage with Flask-RESTful
#parser.error_handler
def handle_request_parsing_error(err, req, schema, error_status_code, error_headers):
"""
webargs error handler that uses Flask-RESTful's abort function to return
a JSON error response to the client.
"""
abort(error_status_code, errors=err.messages)
# ------------------------------------------------------------------------------
if __name__ == '__main__':
api.add_resource(Abc, "/abc", endpoint="abc")
app.run(debug=True)
When I send a GET request to the endpoint with or without parameters I get none of the expected behavior -- if good parameters are included in the GET request then I expect to see a log message in the console, and if none of the required parameters are present then I expect to get some sort of error as a result. Instead, I get what appear to be 200 response codes in the console and simply the phrase "Unknown Error" in the main browser window.
For example, if I enter the following URL without the expected parameters into my Chrome browser's address bar: http://127.0.0.1:5000/abc
I then see this in the console:
2019-05-28 17:42:14 INFO 127.0.0.1 - - [28/May/2019 17:42:14] "GET /abc HTTP/1.1" 200 -
My assumption is that the above should throw an error of some sort indicating the missing URL parameters.
If I enter the following URL with the expected parameters into my Chrome browser's address bar: http://127.0.0.1:5000/abc?rtsp=rtsp://user:passwd#171.25.14.15:554&start=1559076593&duration=10&bucket=testbucket&prefix=test.
I then see this in the console:
2019-05-28 17:45:31 INFO 127.0.0.1 - - [28/May/2019 17:45:31] "GET /abc?rtsp=rtsp://user:passwd#171.25.14.15:554&start=1559076593&duration=10&bucket=testbucket&prefix=test. HTTP/1.1" 200 -
My assumption is that the above should cause the logger to print the information message to the console as is defined in the Abc.get function.
If I use curl at the command line then I get the following result:
$ curl "http://127.0.0.1:5000/abc?rtsp=rtsp://user:passwd#171.25.14.15:554&start=1559076593&duration=10&bucket=testbucket&prefix=test."
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>None Unknown Error</title>
<h1>Unknown Error</h1>
<p></p>
Whi is this going amiss, and how I can achieve the expected behavior? (My intention is to use this approach to pass arguments to a more realistic GET handler that will launch a video recording function when the request is received, the above has been simplified as much as possible for clarity.)
I am using Flask 1.0.3, Flask-Restful 0.3.6, and Webargs 5.3.1 in an Anaconda environment (Python 3.7) on Ubuntu 18.04.
I have managed to get the expected behavior using a simpler code that does not rely upon flask_restful. See below:
import logging
from flask import Flask, request
# initialize the Flask application
app = Flask(__name__)
# ------------------------------------------------------------------------------
# set up a basic, global logger object which will write to the console
logging.basicConfig(level=logging.INFO,
format="%(asctime)s %(levelname)s %(message)s",
datefmt="%Y-%m-%d %H:%M:%S")
_logger = logging.getLogger(__name__)
# ------------------------------------------------------------------------------
#app.route('/clip', methods=['GET'])
def record_and_store_clip():
message = "Recording video clip with the following parameters:\n" + \
f"\tRTSP URL: {request.args.get('rtsp')}\n" + \
f"\tStart seconds: {request.args.get('start')}\n" + \
f"\tDuration seconds: {request.args.get('duration')}\n" + \
f"\tS3 bucket: {request.args.get('bucket')}\n" + \
f"\tS3 key prefix: {request.args.get('prefix')}\n"
_logger.info(message)
return message
# ------------------------------------------------------------------------------
if __name__ == '__main__':
app.run(debug=True)
I am trying to execute below code (reading content from a html) using FancyURLopener. The code was working fine for last 2 months or so , but now it has started to throw the error : IOError: [Errno socket error] [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:590)
When I try to run it locally , it works like a charm.
from urllib import urlopen
from urllib import FancyURLopener
from bs4 import BeautifulSoup
import requests
doc_name = "XYZ"
class MyOpener(FancyURLopener):
version = 'Mozilla/5.0 (Windows; U; Windows NT 5.1; it; rv:1.8.1.11) Gecko/20071127 Firefox/2.0.0.11'
mopen = MyOpener()
def extract_count_from_url(url, tag_name, tag_type, html_tag) :
html = mopen.open(url).read()
soup = BeautifulSoup(html, "html.parser")
I have searched it over on stackoverflow and google. The answers I am getting is mostly to use urllib2 / urllib libraries and use user agents + set the context to ssl.CERT_NONE (How do I disable the ssl check in python 3.x?)
But I guess same is not applicable when I use FancyURLopener , as when I set context in the open() method along with url , it throws invalid arguments error.
python version = Python 2.7.12
Any leads would be helpful.
Thanks in advance.
I was able to figure out a workaround.
Have added the below part in the code and it bypassed the security.
import ssl
ssl._create_default_https_context = ssl._create_unverified_context
I appreciate the help in advance. I am trying to write a python script that posts an IP address to a site referenced below, and get the results printed out in the terminal or file, and then read the file immediately after.
Here is my script:
#!/usr/bin/env python
import requests
IP = raw_input("Enter IP address here: ")
Alert_URL = 'http://www.blacklistalert.org'
def submit_form():
"""Submit a form"""
payload = IP
# make a get request
resp = requests.get(Alert_URL)
print "Response to GET request: %s" % resp.content
# send POST request
resp = requests.post(Alert_URL, payload)
print "Headers from a POST request response: %s" % resp.headers
# print "HTML Response: %s" %resp.read()
if __name__ == '__main__':
submit_form()
The site has section to input IP addresses on the web page, and inspecting the site I found lines to input as follows:
<form method=POST onsubmit="document.forms[0].submit.disabled='true';">
IP or Domain <input onclick="this.value='';" name=q value=11.11.154.23>
I would like to post an IP address that I want to check to the site using the input section above somehow. For instance using raw_input to post into the 'value=' section, and get the result.
Thanks for the help.
You need to parse the PHPSESSID and post:
import requests
from bs4 import BeautifulSoup
ip = raw_input("Enter IP address here: ")
data = {"q": ip} # ip goes here
url = "http://www.blacklistalert.org/"
with requests.Session() as s:
# get the page first to parse
soup = BeautifulSoup(s.get(url).content)
# extract and add the PHPSESSID
PHPSESSID = soup.select_one("input[name=PHPSESSID]")["value"]
data["PHPSESSID"] = PHPSESSID
# finally post
res = s.post(url, data=data)
print(res)
print(res.content)
I want to build a python client to talk to my python Google Cloud Endpoints API. My simple HelloWorld example is suffering from an HTTPException in the python client and I can't figure out why.
I've setup simple examples as suggested in this extremely helpful thread. The GAE Endpoints API is running on localhost:8080 with no problems - I can successfully access it in the API Explorer. Before I added the offending service = build() line, my simple client ran fine on localhost:8080.
When trying to get the client to talk to the endpoints API, I get the following error:
File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/dist27/gae_override/httplib.py", line 526, in getresponse
raise HTTPException(str(e))
HTTPException: Deadline exceeded while waiting for HTTP response from URL: http://localhost:8080/_ah/api/discovery/v1/apis/helloworldendpoints/v1/rest?userIp=%3A%3A1
I've tried extending the http deadline. Not only did that not help, but such a simple first call on localhost should not be exceeding a default 5s deadline. I've also tried accessing the discovery URL directly within a browser and that works fine, too.
Here is my simple code. First the client, main.py:
import webapp2
import os
import httplib2
from apiclient.discovery import build
http = httplib2.Http()
# HTTPException happens on the following line:
# Note that I am using http, not https
service = build("helloworldendpoints", "v1", http=http,
discoveryServiceUrl=("http://localhost:8080/_ah/api/discovery/v1/apis/{api}/{apiVersion}/rest"))
# result = service.resource().method([parameters]).execute()
class MainPage(webapp2.RequestHandler):
def get(self):
self.response.headers['Content-type'] = 'text/plain'
self.response.out.write("Hey, this is working!")
app = webapp2.WSGIApplication(
[('/', MainPage)],
debug=True)
Here's the Hello World endpoint, helloworld.py:
"""Hello World API implemented using Google Cloud Endpoints.
Contains declarations of endpoint, endpoint methods,
as well as the ProtoRPC message class and container required
for endpoint method definition.
"""
import endpoints
from protorpc import messages
from protorpc import message_types
from protorpc import remote
# If the request contains path or querystring arguments,
# you cannot use a simple Message class.
# Instead, you must use a ResourceContainer class
REQUEST_CONTAINER = endpoints.ResourceContainer(
message_types.VoidMessage,
name=messages.StringField(1),
)
package = 'Hello'
class Hello(messages.Message):
"""String that stores a message."""
greeting = messages.StringField(1)
#endpoints.api(name='helloworldendpoints', version='v1')
class HelloWorldApi(remote.Service):
"""Helloworld API v1."""
#endpoints.method(message_types.VoidMessage, Hello,
path = "sayHello", http_method='GET', name = "sayHello")
def say_hello(self, request):
return Hello(greeting="Hello World")
#endpoints.method(REQUEST_CONTAINER, Hello,
path = "sayHelloByName", http_method='GET', name = "sayHelloByName")
def say_hello_by_name(self, request):
greet = "Hello {}".format(request.name)
return Hello(greeting=greet)
api = endpoints.api_server([HelloWorldApi])
Finally, here is my app.yaml file:
application: <<my web client id removed for stack overflow>>
version: 1
runtime: python27
api_version: 1
threadsafe: yes
handlers:
- url: /_ah/spi/.*
script: helloworld.api
secure: always
# catchall - must come last!
- url: /.*
script: main.app
secure: always
libraries:
- name: endpoints
version: latest
- name: webapp2
version: latest
Why am I getting an HTTP Deadline Exceeded and how to I fix it?
On your main.py you forgot to add some variables to your discovery service url string, or you just copied the code here without it. By the looks of it you were probably suppose to use the format string method.
"http://localhost:8080/_ah/api/discovery/v1/apis/{api}/{apiVersion}/rest".format(api='helloworldendpoints', apiVersion="v1")
By looking at the logs you'll probably see something like this:
INFO 2015-11-19 18:44:51,562 module.py:794] default: "GET /HTTP/1.1" 500 -
INFO 2015-11-19 18:44:51,595 module.py:794] default: "POST /_ah/spi/BackendService.getApiConfigs HTTP/1.1" 200 3109
INFO 2015-11-19 18:44:52,110 module.py:794] default: "GET /_ah/api/discovery/v1/apis/helloworldendpoints/v1/rest?userIp=127.0.0.1 HTTP/1.1" 200 3719
It's timing out first and then "working".
Move the service discovery request inside the request handler:
class MainPage(webapp2.RequestHandler):
def get(self):
service = build("helloworldendpoints", "v1",
http=http,
discoveryServiceUrl=("http://localhost:8080/_ah/api/discovery/v1/apis/{api}/{apiVersion}/rest")
.format(api='helloworldendpoints', apiVersion='v1'))
I'm learning web scraping and building a simple web app at the moment, and I decided to practice scraping a schedule of classes. Here's a code snippet I'm having trouble with in my application, using Python 2.7.4, Flask, Heroku, BeautifulSoup4, and Requests.
import requests
from bs4 import BeautifulSoup as Soup
url = "https://telebears.berkeley.edu/enrollment-osoc/osc"
code = "26187"
values = dict(_InField1 = "RESTRIC", _InField2 = code, _InField3 = "13D2")
html = requests.post(url, params=values)
soup = Soup(html.content, from_encoding="utf-8")
sp = soup.find_all("div", {"class" : "layout-div"})[2]
print sp.text
This works great locally. It gives me back the string "Computer Science 61A P 001 LEC:" as expected. However, when I tried to run it on Heroku (using heroku run bash and then run python), I got back an error,403 Forbidden.
Am I missing some settings on Heroku? At first I thought it's the school settings, but then I was wondering why it works locally without any trouble... Any explanation/suggestion would be really appreciated! Thank you in advance.
I was having a similar issue, request was working locally but getting blocked on Heroku. It looks like the issue is that some websites block requests coming from Heroku (which on on AWS Servers). To get around this you can send your requests via a proxy server.
There are a bunch of different add-ons in heroku to achieve this, I went with fixie which has a reasonably sized free tier.
To install:
heroku addons:create fixie:tricycle
Then import into your local environment so you can try locally:
heroku config -s | grep FIXIE_URL >> .env
then in your python file you just add a couple of lines:
import os
import requests
from bs4 import BeautifulSoup as Soup
proxyDict = {
"http" : os.environ.get('FIXIE_URL', ''),
"https" : os.environ.get('FIXIE_URL', '')
}
url = "https://telebears.berkeley.edu/enrollment-osoc/osc"
code = "26187"
values = dict(_InField1 = "RESTRIC", _InField2 = code, _InField3 = "13D2")
html = requests.post(url, params=values, proxies=proxyDict)
soup = Soup(html.content, from_encoding="utf-8")
sp = soup.find_all("div", {"class" : "layout-div"})[2]
print sp.text
Docs for Fixie are here:
https://devcenter.heroku.com/articles/fixie