i'm trying to fetch a whole website include the JavaScript and css file while using python.
The script get a "GET" request and send back the website (local proxy).
here is my code :
class myHandler(BaseHTTPRequestHandler):
# Handler for the GET requests
def do_GET(self):
opener = urllib.FancyURLopener({})
f = opener.open("http://www.ynet.co.il")
self.wfile.write(f.read())
return
try:
# Create a web server and define the handler to manage the
# incoming request
server = HTTPServer(('', PORT_NUMBER), myHandler)
print 'Started httpserver on port ', PORT_NUMBER
# Wait forever for incoming htto requests
server.serve_forever()
except KeyboardInterrupt:
print '^C received, shutting down the web server'
server.socket.close()
The result for this code is only the html is present to the client.
Thanks a lot for the help, i'm Trying to solve that for few days with no result any .
Related
I am trying to understand more precisely how Http connections work with Flask, so I tried writing a very simple app and another simple connection with requests and requests-toolbelt :
app = Flask('file-streamer')
#app.route("/uploadDumb", methods=["POST"])
def upload_dumb():
print("Hello")
return Response(status=200)
So basically this server should just receive a request and return a response.
Then I implemented a simple piece of code that sends requests with toolbelt :
import requests
from requests_toolbelt.multipart import encoder
values = {"file": ("test.zip", open("test.zip", "rb"), "application/zip"), "test": "hello"}
m = encoder.MultipartEncoder(fields=values)
r = requests.post(url="http://localhost:5000/uploadDumb", data=m, headers={"Content-Type": m.content_type})
The file I'm sending is a pretty large file that I want to upload with streaming.
The thing is, I expected the Flask server to wait for the whole file to be sent (even if the file is useless), then return a response, but that's not what's happening.
Actually, Flask responds at the very beginning of the sending process, returns a 200 response, which causes the 'requests' side to end with a "BrokenPipeError".
Could someone explain to me what is happening there ?
I have a siddhi cep application running on Android. Now I want to send events from event simulator from stream processing editor to android app via a socket connection. Till now I have been successful in making android server socket which listens to python client simulator made by me. But to ease the process, is it possible that I can use event simulator to send events to android siddhi app?
I was wondering if I can change some configurations such that event simulator sends events to android socket, so I looked at setting in deployment.yaml file
but the sending configurations are defined for HTTP
senderConfigurations:
-
id: "http-sender"
# Configuration used for the databridge communication
databridge.config:
# No of worker threads to consume events
# THIS IS A MANDATORY FIELD
workerThreads: 10
# Maximum amount of messages that can be queued internally in MB
# THIS IS A MANDATORY FIELD
maxEventBufferCapacity: 10000000
# Queue size; the maximum number of events that can be stored in the queue
# THIS IS A MANDATORY FIELD
eventBufferSize: 2000
# Keystore file path
# THIS IS A MANDATORY FIELD
keyStoreLocation : ${sys:carbon.home}/resources/security/wso2carbon.jks
# Keystore password
# THIS IS A MANDATORY FIELD
keyStorePassword : wso2carbon
# Session Timeout value in mins
# THIS IS A MANDATORY FIELD
clientTimeoutMin: 30
# Data receiver configurations
# THIS IS A MANDATO
Thanks in advance. If you need some more details please let me know
Edit 1
I actually found a way around to do it but it's having some issues. So basically I redirected output sink of event generator to port such that sink has all the data streams. The code for Stream Processor Studio editor is
#App:name("PatternMatching")
#App:description('Identify event patterns based on the order of event arrival')
define stream RoomTemperatureStream(roomNo string, temp double);
#sink(type="tcp", url='tcp://localhost:5001/abc', sync='false', tcp.no.delay='true', keep.alive='true', worker.threads="10", #map(type='text'))
define stream RoomTemperatureAlertStream(roomNo string, initialTemp double, finalTemp double);
--Capture a pattern where the temperature of a room increases by 5 degrees within 2 minutes
#info(name='query1')
from RoomTemperatureStream
select e1.roomNo, e1.temp as initialTemp, e2.temp as finalTemp
insert into RoomTemperatureAlertStream;
it sends the streams as text to python server, which needs to be started first, code of which is
#!/usr/bin/env python
# Author : Amarjit Singh
import pickle
import socket
import pandas
from pandas import json
if __name__ == "__main__":
# ------------------ create a socket object-----------------------#
try:
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error as err:
serversocket.close()
print("socket creation failed with error %s" % (err))
except KeyboardInterrupt:
serversocket.close()
print("KeyboardInterrupt - but server socket was closed ")
host = "127.0.0.1"
Server_port = 5001
# ------------------ binding to the port -----------------------#
try:
serversocket.bind((host, Server_port))
serversocket.listen(50) # queue up to 5 requests
print("\n Server has started and waiting for connection request ..... ")
# bind to the port
while True: # extra while is created so that server runs even if there is no data
running = True
clientsocket, addr = serversocket.accept() # accept a connection from client
print("Got a connection from Server%s" % str(addr)) # show connection success message
while running:
receivedData = clientsocket.recv(2048)
# json = receivedData
if receivedData:
print(receivedData)
print(receivedData[0])
print(receivedData[1])
print(receivedData[2])
# roomNo = str(receivedData[0])
# temp = int(client_tuple[1]) # from unicode to int
#
# print(" roomNo = %d: UUID = %s temp = %d" % (roomNo, temp))
except socket.error as err:
serversocket.close()
print("socket creation failed with error %s" % (err))
except KeyboardInterrupt:
serversocket.close()
print("KeyboardInterrupt - but server socket was closed ")
Initially, I was sending json data from simulator, but pickle.loads and json.loads did not work. but the problem with text is that data is displayed as
b'\x02\x00\x00\x00t\x00\x00\x003bed14d31-6697-4a74-8a3f-30dc012914ad-localhost:5001\x00\x00\x00\x03abc\x00\x00\x002roomNo:"X0ZYp",\ninitialTemp:15.97,\nfinalTemp:17.22'
b'\x02\x00\x00\x00t\x00\x00\x003bed14d31-6697-4a74-8a3f-30dc012914ad-localhost:5001\x00\x00\x00\x03abc\x00\x00\x002roomNo:"2X951",\ninitialTemp:13.42,\nfinalTemp:10.76'
b'\x02\x00\x00\x00t\x00\x00\x003bed14d31-6697-4a74-8a3f-30dc012914ad-localhost:5001\x00\x00\x00\x03abc\x00\x00\x002roomNo:"PUaJA",\ninitialTemp:15.46,\nfinalTemp:16.26'
b'\x02\x00\x00\x00t\x00\x00\x003bed14d31-6697-4a74-8a3f-30dc012914ad-localhost:5001\x00\x00\x00\x03abc\x00\x00\x002roomNo:"pnz0i",\ninitialTemp:10.42,\nfinalTemp:15.82'
how to remove this extra data?
Siddhi has a WebSocket connector[1] and it's still WIP. Using this dependency you will be able to add a WebSocket sink to your app and send events from that.
Unfortunately, you cannot directly send events from Stream Processor[2] Studio/Editor but if you have an app running in Stream Processor Editor and if it has a WebSocket sink then you can send events to App's sink stream from the Simulator which will intern send that message via WebSocket to the Siddhi app in android.
You can only simulate apps running inside the editor via the Event Simulator or simulate apps deployed in Stream Processor worker nodes via Event Simulator API.
[1]https://github.com/wso2-extensions/siddhi-io-websocket
[2]https://wso2.com/analytics
I appreciate the help in advance. I am trying to write a python script that posts an IP address to a site referenced below, and get the results printed out in the terminal or file, and then read the file immediately after.
Here is my script:
#!/usr/bin/env python
import requests
IP = raw_input("Enter IP address here: ")
Alert_URL = 'http://www.blacklistalert.org'
def submit_form():
"""Submit a form"""
payload = IP
# make a get request
resp = requests.get(Alert_URL)
print "Response to GET request: %s" % resp.content
# send POST request
resp = requests.post(Alert_URL, payload)
print "Headers from a POST request response: %s" % resp.headers
# print "HTML Response: %s" %resp.read()
if __name__ == '__main__':
submit_form()
The site has section to input IP addresses on the web page, and inspecting the site I found lines to input as follows:
<form method=POST onsubmit="document.forms[0].submit.disabled='true';">
IP or Domain <input onclick="this.value='';" name=q value=11.11.154.23>
I would like to post an IP address that I want to check to the site using the input section above somehow. For instance using raw_input to post into the 'value=' section, and get the result.
Thanks for the help.
You need to parse the PHPSESSID and post:
import requests
from bs4 import BeautifulSoup
ip = raw_input("Enter IP address here: ")
data = {"q": ip} # ip goes here
url = "http://www.blacklistalert.org/"
with requests.Session() as s:
# get the page first to parse
soup = BeautifulSoup(s.get(url).content)
# extract and add the PHPSESSID
PHPSESSID = soup.select_one("input[name=PHPSESSID]")["value"]
data["PHPSESSID"] = PHPSESSID
# finally post
res = s.post(url, data=data)
print(res)
print(res.content)
The situation
I'm trying to connect to a server on https protocol with python script. Could someone give me the working example that sends a GET request to https server, or web resource to how to create https connection with python?
An attempt so far
I have learned that the module httplib on python supports creation of http connection, but not https connection?
import httplib
conn = httplib.HTTPConnection('https://adsche.skplanet.com/api/startNewTurn')
header = {"Content-type" : "application/json"}
conn.request('GET', '/announce?info_hash=%da', '', header)
r1 = conn.getresponse()
print r1.status, r1.reason
data1 = r1.read()
print data1
Hi am absolutely new to django,Now I am tring for builtin reset password view..
I follow the link link
But I got the error when I click on reset password button at /password_reset/ :
error at /accounts/password_reset/
[Errno 10061] No connection could be made because the target machine actively refused it.
Exception Location: C:\Python27\lib\socket.py in create_connection, line 571
'urls.py'
(r'^accounts/password_reset$','django.contrib.auth.views.password_reset','template_name':'user/password_reset_form.html','email_template_name':'user/password_reset_email.html'}),
(r'^accounts/password/reset/confirm/(?P[0-9A-Za-z]{1,13})-(?P[0-9Aa-z]{1,13}-[0-9A-Za-z]{1,20})/$', 'django.contrib.auth.views.password_reset_confirm', {'template_name' : 'user/password_reset.html', 'post_reset_redirect': '/logout/' }),
(r'^accounts/password_reset/done/$',<b>'django.contrib.auth.views.password_reset_done'</b>,{'template_name':'user/password_reset_done.html'}),
(r'^accounts/change_password/$',<b> 'password_change'</b>, {'post_change_redirect' : '/accounts/change_password/done/'}),
(r'^accounts/change_password/done/$',<b> 'password_change_done'</b>,{'template_name':'user/password_change_done.html'}),
<b>password_reset_email.html</b>
{% extends 'base.html'%}
{% block content%}
{% url django.contrib.auth.views.password_reset_confirm uidb36=uid, token=token %}
{% endblock %}
I add necessary templates in the folder 'user'.Please help me,Thanks in advance.
As rdegges said, it's a connection error. Check which port the request is trying to access, and make sure windows firewall is set up to receive on that port. You can check the port by looking through django's traceback page, and looking at the "local vars".
From the looks of it, it's an email port. Take another look at the traceback and look out for django trying to send a request to port 25. If it does, make sure your port 25 is configured to receive.
Also, you'll need a makeshift SMTP server for testing purposes because you probably wouldn't want to be using a real one. Just have this running in a separate command prompt window while you're running django, and any emails that django tries to send through your port 25 will be saved in an "emails" folder in the working directory.
#/usr/bin/env python
from datetime import datetime
import asyncore
from smtpd import SMTPServer
class EmlServer(SMTPServer):
no = 0
def process_message(self, peer, mailfrom, rcpttos, data):
filename = '%s-%d.eml' % (datetime.now().strftime('%Y%m%d%H%M%S'), self.no)
f = open(filename, 'w')
f.write(data)
f.close()
print '%s saved.' % filename
self.no += 1
def run():
foo = EmlServer(('localhost', 25), None)
try:
asyncore.loop()
except KeyboardInterrupt:
pass
if __name__ == '__main__':
from os.path import exists, join
from os import mkdir, chdir, getcwd
target_directory = join( getcwd(), "emails" )
if exists(target_directory):
chdir(target_directory)
else:
try:
mkdir(target_directory)
except OSError:
from sys import exit
exit("The containing folder couldn't be created, check your permissions.")
chdir(target_directory)
print "Using directory %s" % target_directory
run()
The error you're getting is a connection error--that means that the server you're using to run your Django site is likely not working correctly. Here are some things you can try on your Django server:
If you're running your Django site via python manage.py runserver, you can try to simply re-run that command.
If you're running your site via a webserver like apache, try restarting apache.
If neither of those work, post a comment and let me know what operating system you're running your site on, and how you're running it, and we can do further debugging.