I am trying to understand more precisely how Http connections work with Flask, so I tried writing a very simple app and another simple connection with requests and requests-toolbelt :
app = Flask('file-streamer')
#app.route("/uploadDumb", methods=["POST"])
def upload_dumb():
print("Hello")
return Response(status=200)
So basically this server should just receive a request and return a response.
Then I implemented a simple piece of code that sends requests with toolbelt :
import requests
from requests_toolbelt.multipart import encoder
values = {"file": ("test.zip", open("test.zip", "rb"), "application/zip"), "test": "hello"}
m = encoder.MultipartEncoder(fields=values)
r = requests.post(url="http://localhost:5000/uploadDumb", data=m, headers={"Content-Type": m.content_type})
The file I'm sending is a pretty large file that I want to upload with streaming.
The thing is, I expected the Flask server to wait for the whole file to be sent (even if the file is useless), then return a response, but that's not what's happening.
Actually, Flask responds at the very beginning of the sending process, returns a 200 response, which causes the 'requests' side to end with a "BrokenPipeError".
Could someone explain to me what is happening there ?
Related
New to using Pytest on APIs. From my understanding, testing creates another instance of Flask. Additionally, from the tutorials I have seen, they also suggest to create a separate DB table instance to add, fetch and remove data for test purposes. However, I simply plan to use the remote api URL as host to simply make the call.
Now, I set my conftest like this, where the flag --testenv would indicate to make the get/post call on the host listed below:
import pytest
import subprocess
def pytest_addoption(parser):
"""Add option to pass --testenv=api_server to pytest cli command"""
parser.addoption(
"--testenv", action="store", default="exodemo", help="my option: type1 or type2"
)
#pytest.fixture(scope="module")
def testenv(request):
return request.config.getoption("--testenv")
#pytest.fixture(scope="module")
def testurl(testenv):
if testenv == 'api_server':
return 'http://api_url:5000/'
else:
return 'http://locahost:5000'
And my test file is written like this:
import pytest
from app import app
from flask import request
def test_nodes(app):
t_client = app.test_client()
truth = [
{
*body*
}
]
res = t_client.get('/topology/nodes')
print (res)
assert res.status_code == 200
assert truth == json.loads(res.get_data)
I run the code using this:
python3 -m pytest --testenv api_server
The thing I expect is that the test file would simply make a call to the remote api with the creds, fetch the data regardless of how it gets pulled in the remote code, and bring it here for assertion. However, I am getting the 400 BAD REQUEST error, with the error being like this:
assert 400 == 200
E + where 400 = <WrapperTestResponse streamed [400 BAD REQUEST]>.status_code
single_test.py:97: AssertionError
--------------------- Captured stdout call ----------------------
{"timestamp": "2022-07-28 22:11:14,032", "level": "ERROR", "func": "connect_to_mysql_db", "line": 23, "message": "Error connecting to the mysql database (2003, \"Can't connect to MySQL server on 'mysql' ([Errno -3] Temporary failure in name resolution)\")"}
<WrapperTestResponse streamed [400 BAD REQUEST]>
Does this mean that the test file is still trying to lookup the database locally for fetching? I am unable to figure out on which host are they sending the test url as well, so I am kind of stuck here. Looking to get some help around here.
Thanks.
I have a setup that lets users download files that are stored in the DB as BYTEA data. Everything works OK, except the download speed is very slow...it seems to download in 33KB chunks, one chunk per second.
Is there a setting I can specify to speed this up?
views.py
from django.http import FileResponse
def getFileResponse(filedata, filename, filesize, contenttype):
response = FileResponse(filedata, content_type=contenttype)
response['Content-Disposition'] = 'attachment; filename=%s' % filename
response['Content-Length'] = filesize
return response
return getFileResponse(
filedata = myfile.filedata, # Binary data from DB
filename = myfile.filename + myfile.fileextension,
filesize = myfile.filesize,
contenttype = myfile.filetype
)
Previously, I had the binary data returned as an HttpResponse and it downloaded like a normal file, with normal speeds. This worked fine locally, but when I pushed to Heroku, it wouldn't download the file -- instead displaying <Memory at XXX> in the download file.
And another side issue...when I include a text file with non-ASCII data (i.e. รก), I get an error as well:
UnicodeEncodeError: 'ascii' codec can't encode characters...: ordinal not in range(128)
How can I handle files with Unicode data?
Update
Anyone know why the download speed gets so slow when changing from HTTPResponse to FileResponse? Or alternatively, why the HTTPResponse to return a file doesn't work on Heroku?
Update - Google Drive
I re-worked my application and hooked it up with a Google Drive back-end for serving files. It employs BytesIO() suggested by Eric below:
def download_file(self, fileid, mimetype=None):
# Get binary file data
request = self.get_file(fileid=fileid, mediaflag=True)
stream = io.BytesIO()
downloader = MediaIoBaseDownload(stream, request)
done = False
# Retry if we received HTTPError
for retry in range(0, 5):
try:
while done is False:
status, done = downloader.next_chunk()
print("Download %d%%." % int(status.progress() * 100))
return stream.getvalue()
except (HTTPError) as error:
return ('API error: {}. Try # {} failed.'.format(error.response, retry))
I think the difference you observe between HttpResponse vs. FileResponse is caused by the spec: https://www.python.org/dev/peps/pep-3333/#buffering-and-streaming
In your previous code, an HttpResponse was created with one huge byte string containing your whole file, and the first iteration pass returned the complete response body. With a a FileResponse, the file is iterated in chunks (of 4kb, 8kb or other depending on your WSGI app server), which (I think) are streamed immediately upstream (to the reverse proxy then client), which may add overhead (more communication over process boundaries?).
It would help to know the app server used (uwsgi, gunicorn, waitress, other) and its relevant config. Also more details about the heroku error in case that can be solved!
why you store whole file in database.
best case is to store file on hard and store only path on database
then according to your web server you can let web server to serve file.
web services serve file better than Django.
if files have no access check store them on media
if your files have access control you according to your web server you can use some response headers
if you use Nginx must use X-Accel-Redirect and use any alternative on other web services tutorial on https://wellfire.co/learn/nginx-django-x-accel-redirects/
Language: Python 2.7
Hello all. I found a really helpful script here: Python to Slack Web Hook
that shows how to send messages to a Slack web hook.
import json
import requests
# Set the webhook_url to the one provided by Slack when you create the webhook at https://my.slack.com/services/new/incoming-webhook/
webhook_url = 'https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX'
slack_data = {"text": "<https://alert-system.com/alerts/1234|Click here> for details!"}
response = requests.post(
webhook_url, data=json.dumps(slack_data),
headers={'Content-Type': 'application/json'}
)
if response.status_code != 200:
raise ValueError(
'Request to slack returned an error %s, the response is:\n%s'
% (response.status_code, response.text)
)
It works flawlessly when I run .py file.
Now, I have a file that has many lines of messages that I want to send to Slack. I have it formatted correctly already in the file, no spaces etc.. It's just a matter of grabbing it and passing it so slack_data = line1 etc..
So, I modify the file with something like this:
with open('export.txt', 'r') as e:
for line in e:
slack_data = line
Now if I do a print slack_data right after that, the information returns on the screen exactly as it should be, so I'm thinking it's good. I haven't began to get it working for each line yet, because it's not even working on the first line.
I get an invalid payload 400 when I run it.
EDIT: Slack support said the what they were receiving has escape characters inserted into for some reason.
"{\"text\": \"<https://alert-system.com/alerts/1234|Click here> for details!"}\n"
Any direction or assistance is appreciated.
Thanks!!
Just posting as it might help somebody. For me the below snippet worked:
data = json.dumps(slack_data)
response = requests.post(
URL, json={"text": data},
headers={'Content-Type': 'application/json'}
)
As #Geo pointed out the final payload that we are going to send should have keyword "text", else it will fail.
Moreover, in post method I have to replace data= with json= else it kept throwing error for invalid payload with 400
Since I already had the data preformatted in the file as JSON already, it was just a matter of removing json.dumps out of the code.
OLD:
#response = requests.post(webhook_url, data=json.dumps(slack_data), headers={'Content-Type': 'application/json'})
NEW:
response = requests.post(webhook_url, data=slack_data, headers={'Content-Type': 'application/json'})
Once I did that, everything worked like a charm.
If you change the code to this:
with open('export.txt', 'r') as e:
slack_data = e.read()
do you still get the 400?
i'm trying to fetch a whole website include the JavaScript and css file while using python.
The script get a "GET" request and send back the website (local proxy).
here is my code :
class myHandler(BaseHTTPRequestHandler):
# Handler for the GET requests
def do_GET(self):
opener = urllib.FancyURLopener({})
f = opener.open("http://www.ynet.co.il")
self.wfile.write(f.read())
return
try:
# Create a web server and define the handler to manage the
# incoming request
server = HTTPServer(('', PORT_NUMBER), myHandler)
print 'Started httpserver on port ', PORT_NUMBER
# Wait forever for incoming htto requests
server.serve_forever()
except KeyboardInterrupt:
print '^C received, shutting down the web server'
server.socket.close()
The result for this code is only the html is present to the client.
Thanks a lot for the help, i'm Trying to solve that for few days with no result any .
I am trying to send mail using python via my own setup mail server
import smtplib
SERVER='myserverdomain.com'
server = smtplib.SMTP(SERVER,587)
server.starttls()
server.login(USER,PASS)
server.sendmail(FROM, TO, message)
server.quit()
But when I run the above code after line 3 it is not executing.Kind of like goes into an infinite loop.Why could this be
I am able to ping my server successfully.
import smtplib
SERVER='myserverdomain.com'
server = smtplib.SMTP_SSL(SERVER,587)
server.login(USER,PASS)
server.sendmail(FROM, TO, message)
server.quit()
This answer worked